Skip to content

Ch 4: GenAI, Red Teaming and Test Levels ​

Official Reference

ISTQB® CT-AI Syllabus v2.0, Chapter 4 (195 minutes), sections 4.2–4.3, Learning Objectives (LOs) AI-4.2.1 (K2), AI-4.2.2 (K3), AI-4.3.1 (K2), AI-4.3.2 (K2), and Hands-on Objective HO-4.2.3 (H2). Official syllabus and documents

Exam weight: four questions from this page. AI-4.2.2 is a 2-point K3 scenario question: you will be asked for the MOST effective red teaming strategy in a situation. The answer almost always combines before deployment, end-to-end, diverse team and several harm types together.

This page covers how to test generative AI (GenAI) and large language models (LLMs), including red teaming, and how testing an ML system is organized: test levels and risk-based testing. For using GenAI to perform testing, the syllabus refers to the separate ISTQB® CT-GenAI syllabus.

Large language model: A generative AI system trained on very large collections of language data. (ISTQB® Glossary; syllabus Chapter 9 says "text-to-text generative AI system")
Attack: A test approach in which a set of deliberate actions is used to reveal a particular defect type.
Exploratory testing: A test approach in which tests are dynamically designed and executed based on tester's knowledge, exploration of a test item, and previous test results.
Risk-based testing: A test approach in which the management, selection, prioritization, and use of test activities and resources are based on corresponding risk types and risk levels.
Source: ISTQB® Glossary (all Chapter 4 keywords)

4.2.1 Testing Generative AI (K2) ​

Testing GenAI evaluates the correctness, coherence and creativity of outputs, including originality and novelty, and verifies that functional and non-functional requirements are met. GenAI produces text, images, video and audio, so test strategies must be adapted to the quality characteristics of each kind of content.

AspectWhat the syllabus says
Common approach: black-box testingFeed various inputs (prompts, images, partial data) and assess the results for clarity, originality, adherence to domain-specific rules. Manual or automated. Especially useful for end-user applications (chatbots, design tools), where user satisfaction depends on the usefulness and plausibility of the content. Red teaming (4.2.2) is a form of this
The input explosion problemInputs are highly diverse and hard to control: optional system prompts plus a user prompt (possibly with huge amounts of disparate data), API access, parameters such as temperature and maximum tokens, and the context window (earlier parts of the conversation), all of which affect the output
Deciding pass or failOften manual review against qualitative evaluation criteria defined in the requirements. Alternatively, a second GenAI system can determine the result automatically (e.g., image recognition checks a generated image), but carefully, since it may reproduce similar biases or errors
Non-functional testingCan be as necessary as functional testing: resource utilization during inference and training (efficiency, cost-effectiveness), measured by CPU and GPU usage, memory consumption, network bandwidth, response times
Benchmark suitesCurated datasets and tasks that give standardized, consistent comparison across models, from language understanding to reasoning and coding, and identify areas for improvement systematically and reproducibly
Temperature: A parameter that controls the randomness of GenAI outputs, with lower values generating more predictable results and higher values generating more creative results.
System prompt: A predefined instruction set, typically hidden from the chatbot's users, that consistently establishes the context, tone, and boundaries for an LLM's responses and guides its behavior throughout interactions.
Source: ISTQB® Glossary

Testing analogy

The input explosion problem is the combinatorial-testing problem on steroids: the "parameters" include free-text prompts, conversation history and sampling settings. Instead of trying to cover combinations, you control what you can (fix temperature, reset the context window between tests), assess properties (rules, plausibility), and use statistical judgement over many runs.

Exam Distinction

The common GenAI test approach is to vary inputs and parameters and assess whether the output follows rules and is plausible. It is not exact matching against predefined expected outputs, and it is not "manual only". Testing is not unnecessary just because outputs vary.

4.2.2 Red Teaming (K3) ​

Red teaming: A test approach that involves attacking a system to deliberately cause it to produce harmful or undesirable results. (ISTQB® Glossary)

Red teaming (RT) is a systematic, often black-box form of fault attack that probes an AI-based system to identify harmful capabilities, particularly in its outputs. It is inspired by military wargaming and NASA's tiger teams. The aim is to deliberately cause harmful or undesirable results, such as privacy violations, racist views, or guidance for chemical, biological, radiological or nuclear (CBRN) attacks. The harmful capabilities found are then used to update and strengthen the system.

PointWhat the syllabus says
Applies toAny AI-based system, but especially critical for GenAI: the vast range of inputs and outputs creates an extensive "attack space"
ScopeTypically the complete end-to-end AI-based system, but can be the AI model only
Harm typesOften security and safety, but also reliability, privacy, fairness, bias, misinformation
RegulationIncreasingly a regulatory expectation for some AI-based systems (e.g., EU AI Act)
vs. benchmarksRT is adaptive and dynamic (prompts change in response to outputs). It complements static benchmarking by testing under extreme and unexpected conditions
Security RTTests vulnerabilities to external attacks, both non-AI and AI-specific, e.g., indirect prompt injection and malicious content hidden in documents used by RAG. It focuses on malicious inputs
Safety (and other) RTFinds harmful outputs under ordinary use, without adversarial intent, e.g., unsafe medical advice
TimingMost effective before deployment, but after initial internal quality assessments
Core activityInteractive prompting: multi-turn dialogues (e.g., 15–20 turns) to elicit defective or policy-violating behavior
Scaling coverageBeyond small expert teams: crowd-sourced prompt generation (manual); automated: an LLM generates attack prompts, another LLM checks the outputs; hybrid: human creativity plus automation
Blue teamingRT is proactive and pre-deployment. Blue teaming is ongoing, real-time defensive monitoring and filtering of inputs to an operational system. RT insights improve blue-team monitoring and filters

The five-step approach (syllabus):

mermaid
flowchart TB
  S1["1. Assemble a diverse team<br/>wide range of perspectives<br/>and attack vectors"] --> S2["2. Provide access<br/>in a safe test environment"]
  S2 --> S3["3. Prompt to find vulnerabilities<br/>open-ended exploration<br/>or checklists"]
  S3 --> S4["4. Analyze the failures<br/>to understand threats"]
  S4 --> S5["5. Create datasets from the threats<br/>to support mitigations<br/>and improvements"]
  S5 -.->|"insights"| BT["Blue teaming<br/>monitoring and input filters<br/>in operation"]

Red teaming ends with reusable datasets, not just a bug list; those datasets drive mitigations before release and feed the blue team's filters after release.

Worked example: implementing red teaming (K3) ​

Scenario (illustrative): a health insurer will launch "AskCover", an LLM chatbot for members. It answers questions using RAG over policy documents and can read documents that members upload (e.g., invoices). Go-live is in eight weeks.

StepDecisions for AskCoverSyllabus basis
Timing and scopeStart after the internal QA cycle, before go-live. Attack the end-to-end system (UI → RAG → LLM → output filters), not just the modelBefore deployment, after internal quality assessments; end-to-end typically
1. TeamSecurity tester, claims specialist, a clinician, privacy officer, two staff from different cultural backgrounds, plus crowd-sourced prompts from customer-service agentsDiverse team; crowd-sourcing
2. EnvironmentIsolated copy with synthetic member data, production-equivalent configuration and RAG indexSafe test environment
3. PromptingChecklist and open-ended sessions per harm type: security: indirect prompt injection via an uploaded "invoice" containing hidden instructions, poisoned text in RAG sources; privacy: trying to get another member's data; safety: questions that tempt unsafe medical advice under ordinary use; fairness/bias: the same claim question phrased by different personas; misinformation: invented policy terms. 15–20-turn dialogues to wear down refusals. An LLM generates attack variants and another LLM pre-screens outputs; humans review the flagged onesInteractive multi-turn prompting; open-ended or checklists; harm types; security vs. safety RT; automated and hybrid approaches
4. AnalysisCluster failures by harm type and root cause (e.g., "RAG content treated as instructions") and rate severityAnalyze failures to understand threats
5. DatasetsTurn every successful attack into a regression dataset of prompts plus expected safe behavior; use it to fix the system prompt and filters, and hand it to the blue team for live monitoringCreate datasets for mitigations; insights for blue teaming

Reusable red teaming plan template:

text
System under test ......: (end-to-end | model only)  version/config: ......
Timing .................: after internal QA on ..../..../....  — before deployment on ..../..../....
Harm types in scope ....: [ ] security (incl. prompt injection, RAG content) [ ] safety (ordinary use)
                          [ ] privacy [ ] fairness/bias [ ] reliability [ ] misinformation [ ] other: ....
Team (diverse) .........: roles/perspectives .......... crowd-sourcing? ...... automation (LLM attacker + LLM judge)? ......
Safe test environment ..: data (synthetic?) ...... isolation ...... configuration parity ......
Techniques .............: open-ended exploration | checklists | multi-turn dialogues (15–20 turns)
Failure analysis .......: categorize by harm type / root cause / severity
Outputs ................: datasets of successful attacks -> mitigations, regression tests, blue-team filters

Exam Distinction

The best red teaming strategy in a scenario is almost always the one that is proactive (before deployment) and covers several vulnerability types together. Watch for these traps: "run it after go-live to save time" or "react to incidents" (that is blue teaming), "focus only on one team's concern", and "rely on benchmarks instead".

Hands-on exercise HO-4.2.3 (H2): Exploratory testing of an LLM performing boundary value analysis

Goal (syllabus): perform exploratory testing of an LLM using a session sheet focused on the LLM's ability to generate test cases with 2-value and 3-value boundary value analysis (BVA), then verify the correctness and completeness of the output in a debrief.

Session sheet (template, supplementary)

FieldContent
CharterExplore how well the LLM applies 2-value and 3-value BVA (as defined in CTFL v4.0) to simple numeric specifications
Time box45 minutes
Test objectAny chat LLM; note model name, version, temperature (if visible)
Ideas to exploreinteger vs. decimal ranges; open-ended ranges ("at least 18"); several partitions; ambiguous wording; asking for 2-value then 3-value in the same conversation (context window!); repeating the same prompt in a fresh chat (non-determinism)
Notes / defects / questions…

Start with this specification and oracle: "An order qualifies for a bulk discount if it contains 100 to 500 items (whole numbers)." Partitions: ≤ 99 (invalid), 100–500 (valid), ≥ 501 (invalid).

BVA variantCorrect test values
2-value BVA (boundary + nearest value in the neighboring partition)99, 100, 500, 501
3-value BVA (boundary + both neighbors)99, 100, 101, 499, 500, 501

Debrief questions: Did the LLM confuse 2-value and 3-value BVA? Did it add extra or missing values? Did it treat "100 to 500" as inclusive? Did the answer change when you repeated the prompt? Which defects would matter if a team relied on this output?

4.3.1 Test Levels for Machine Learning Systems (K2) ​

The non-AI components of an ML system (MLS) are tested with conventional test levels. ML adds two specialized test levels:

Input data testing: A test level that focuses on the quality of data used for training an ML model.
ML model testing: A test level that focuses on the ability of an ML model to meet required ML functional performance criteria and non-functional criteria.
Source: ISTQB® Glossary. Note: section 4.3.1 extends input data testing to the production data used for predictions as well.
mermaid
flowchart LR
  S["Testing an<br/>ML system"] --> MLS["ML-specific<br/>test levels"]
  S --> CV["Conventional test levels<br/>(as risks require)"]
  MLS --> IDT["Input data testing<br/>training + production data (Ch. 5)"]
  MLS --> MT["ML model testing<br/>the ML model (Ch. 6)"]
  CV --> CT["Component testing<br/>non-AI components, data pipeline"]
  CV --> CIT["Component integration testing<br/>pipeline ↔ model ↔ UI, AIaaS API"]
  CV --> ST["System testing<br/>performance once embedded, non-functional"]
  CV --> SIT["System integration testing<br/>external systems and services"]
  CV --> AT["Acceptance testing<br/>suitability of an AI service"]

The two ML-specific levels do not replace the conventional ones: an ML system still needs component, integration, system and acceptance testing for everything around the model.

Test levelFocus for an MLS (syllabus)
Input data testing (Ch. 5)The training data used to train the model and the production data used to generate predictions in operation
ML model testing (Ch. 6)The ML models, which are the ultimate output of the ML workflow
Component testingAny non-AI components: user interface, data pipeline, communication components
Component integration testingInputs from the data pipeline are received as expected by the model; predictions are exchanged with the relevant components (e.g., UI) and used correctly. Where AI is provided as a service, API testing of the service is part of this level
System testingConfirmation testing that the ML functional performance from model testing is not adversely affected when the model is embedded in the complete system, especially if the model was deliberately changed (e.g., compressing a DNN). Also non-functional testing, e.g., the time to deliver a prediction
System integration testingInterfaces and data exchanges between the AI-based system and external systems or services, in an environment representative of operation
Acceptance testingWhere AI is used as a service: the suitability of the service for the intended system, and whether, e.g., ML functional performance criteria have been achieved

Exam Distinction

If a scenario says a model performs badly for one region, group or condition that is rare in the data (e.g., high-altitude weather stations), the test level that should have caught it is input data testing (representativeness). Later levels might detect it, but input data testing would catch it earliest.

4.3.2 Risk-Based Testing of Machine Learning Systems (K2) ​

Risk-based testing should be applied to all systems, with or without AI. Most regulatory frameworks require a risk-based approach for AI-based systems. AI poses unique risks, so its testing differs. There is no standard way to categorize ML risks. They concern both the development of the MLS (project risks) and the MLS itself (product risks).

Project risk: A risk that impacts project success. Product risk: A risk that impacts the quality of a product. (ISTQB® Glossary)

One categorization, by the ML workflow (three areas):

AreaConcerned withExample risks (syllabus)TypeCovered in
DevelopmentThe ML algorithm, model development, the ML development frameworkSub-optimal algorithm selection, poor selection of evaluation approach, framework security vulnerabilitiesProject risksChapter 7
Input dataTraining data and production dataBiased training data, data-pipeline defects, unrepresentative training dataProduct risksChapter 5
ModelThe generated ML modelFailure to achieve required ML functional performance, overfitted model, susceptibility to adversarial examplesProduct risksChapter 6

Another categorization: by the ISO/IEC 25059 quality characteristics (Chapter 2).

Testing designed specifically for these risks includes data pipeline testing, adversarial testing, and review of algorithm/model suitability. Chapters 5–7 cover these in detail.

Exam Distinction

Generic risks such as security and usability apply to any system. Risks such as data bias and ML model performance are specific to ML systems. The official sample exam's rationale also notes that risk management should be dynamic for both conventional and AI systems; it is not "static for conventional, dynamic for AI".

Unit Quiz ​

Unit Quiz: Chapter 4, page 2 (4.2–4.3)

Q1. What is the "input explosion problem" when testing a GenAI model?

AInputs can be highly diverse and hard to control: optional system prompts, user prompts with large amounts of disparate data, API access, parameters such as temperature and maximum tokens, and the context window
BThe model crashes when the input is too long
CTraining data grows faster than storage capacity
DThe number of test cases doubles with every release

Q2. A team uses an image recognition model to automatically check whether images generated by a GenAI tool show the requested objects. What does the syllabus warn about this approach?

AIt is forbidden for GenAI systems
BIt always needs a human to approve each result
CIt should be used carefully, because the second system may reproduce similar biases or errors
DIt only works for text outputs

Q3. A bank will launch a GenAI customer assistant that uses RAG over uploaded customer documents. The security team wants red teaming only for prompt injection; the compliance team wants only bias checks; the project manager wants to red team after go-live. What is the MOST appropriate red teaming approach?

ARed team after go-live, based on customer complaints
BRed team only for prompt injection now, and for bias in a later release
CSkip red teaming and rely on benchmark scores
DBefore deployment (after internal quality assessments), red team the end-to-end system with a diverse team covering security (including indirect prompt injection via the RAG documents) and other harms such as bias

Q4. Which sequence matches the typical red teaming approach in the syllabus?

ADeploy → monitor → filter inputs → retrain
BAssemble a diverse team → provide access in a safe test environment → prompt to identify vulnerabilities (open-ended or with checklists) → analyze the failures → create datasets from the threats to support mitigations
CWrite test cases from the specification → execute → report pass/fail
DRun a benchmark suite → publish the score → compare with competitors

Q5. Which TWO test levels does the syllabus describe as specialized test levels for ML systems?

Select 2 options.

AInput data testing
BComponent integration testing
CML model testing
DSystem integration testing
EAcceptance testing

Q6. A model is compressed to a smaller size so it can run inside a mobile app. At which test level does the syllabus emphasize confirming that its ML functional performance has not been adversely affected?

AComponent testing
BInput data testing
CSystem testing
DSystem integration testing

Q7. Using the ML workflow to categorize risks, "sub-optimal algorithm selection" belongs to which area, and what type of risk is it?

ADevelopment area: a project risk
BInput data area: a product risk
CModel area: a product risk
DModel area: a project risk

0 / 7 answered

Topic Drills: 4.2.1 · 4.2.2 · 4.3.1 · 4.3.2 · Next: Chapter 5 →

Unofficial study guide for the ISTQB® Certified Tester AI Testing (CT-AI) exam. Not affiliated with International Software Testing Qualifications Board (ISTQB®).