Appearance
Ch 4: GenAI, Red Teaming and Test Levels
Official Reference
ISTQB® CT-AI Syllabus v2.0, Chapter 4 (195 minutes), sections 4.2–4.3, Learning Objectives (LOs) AI-4.2.1 (K2), AI-4.2.2 (K3), AI-4.3.1 (K2), AI-4.3.2 (K2), and Hands-on Objective HO-4.2.3 (H2). Official syllabus and documents
Exam weight: four questions from this page. AI-4.2.2 is a 2-point K3 scenario question: you will be asked for the MOST effective red teaming strategy in a situation. The answer almost always combines before deployment, end-to-end, diverse team and several harm types together.
This page covers how to test generative AI (GenAI) and large language models (LLMs), including red teaming, and how testing an ML system is organized: test levels and risk-based testing. For using GenAI to perform testing, the syllabus refers to the separate ISTQB® CT-GenAI syllabus.
Large language model: A generative AI system trained on very large collections of language data. (ISTQB® Glossary; syllabus Chapter 9 says "text-to-text generative AI system")
Attack: A test approach in which a set of deliberate actions is used to reveal a particular defect type.
Exploratory testing: A test approach in which tests are dynamically designed and executed based on tester's knowledge, exploration of a test item, and previous test results.
Risk-based testing: A test approach in which the management, selection, prioritization, and use of test activities and resources are based on corresponding risk types and risk levels.
Source: ISTQB® Glossary (all Chapter 4 keywords)
Attack: A test approach in which a set of deliberate actions is used to reveal a particular defect type.
Exploratory testing: A test approach in which tests are dynamically designed and executed based on tester's knowledge, exploration of a test item, and previous test results.
Risk-based testing: A test approach in which the management, selection, prioritization, and use of test activities and resources are based on corresponding risk types and risk levels.
Source: ISTQB® Glossary (all Chapter 4 keywords)
4.2.1 Testing Generative AI (K2)
Testing GenAI evaluates the correctness, coherence and creativity of outputs, including originality and novelty, and verifies that functional and non-functional requirements are met. GenAI produces text, images, video and audio, so test strategies must be adapted to the quality characteristics of each kind of content.
| Aspect | What the syllabus says |
|---|---|
| Common approach: black-box testing | Feed various inputs (prompts, images, partial data) and assess the results for clarity, originality, adherence to domain-specific rules. Manual or automated. Especially useful for end-user applications (chatbots, design tools), where user satisfaction depends on the usefulness and plausibility of the content. Red teaming (4.2.2) is a form of this |
| The input explosion problem | Inputs are highly diverse and hard to control: optional system prompts plus a user prompt (possibly with huge amounts of disparate data), API access, parameters such as temperature and maximum tokens, and the context window (earlier parts of the conversation), all of which affect the output |
| Deciding pass or fail | Often manual review against qualitative evaluation criteria defined in the requirements. Alternatively, a second GenAI system can determine the result automatically (e.g., image recognition checks a generated image), but carefully, since it may reproduce similar biases or errors |
| Non-functional testing | Can be as necessary as functional testing: resource utilization during inference and training (efficiency, cost-effectiveness), measured by CPU and GPU usage, memory consumption, network bandwidth, response times |
| Benchmark suites | Curated datasets and tasks that give standardized, consistent comparison across models, from language understanding to reasoning and coding, and identify areas for improvement systematically and reproducibly |
Temperature: A parameter that controls the randomness of GenAI outputs, with lower values generating more predictable results and higher values generating more creative results.
System prompt: A predefined instruction set, typically hidden from the chatbot's users, that consistently establishes the context, tone, and boundaries for an LLM's responses and guides its behavior throughout interactions.
Source: ISTQB® Glossary
System prompt: A predefined instruction set, typically hidden from the chatbot's users, that consistently establishes the context, tone, and boundaries for an LLM's responses and guides its behavior throughout interactions.
Source: ISTQB® Glossary
Testing analogy
The input explosion problem is the combinatorial-testing problem on steroids: the "parameters" include free-text prompts, conversation history and sampling settings. Instead of trying to cover combinations, you control what you can (fix temperature, reset the context window between tests), assess properties (rules, plausibility), and use statistical judgement over many runs.
Exam Distinction
The common GenAI test approach is to vary inputs and parameters and assess whether the output follows rules and is plausible. It is not exact matching against predefined expected outputs, and it is not "manual only". Testing is not unnecessary just because outputs vary.
4.2.2 Red Teaming (K3)
Red teaming: A test approach that involves attacking a system to deliberately cause it to produce harmful or undesirable results. (ISTQB® Glossary)
Red teaming (RT) is a systematic, often black-box form of fault attack that probes an AI-based system to identify harmful capabilities, particularly in its outputs. It is inspired by military wargaming and NASA's tiger teams. The aim is to deliberately cause harmful or undesirable results, such as privacy violations, racist views, or guidance for chemical, biological, radiological or nuclear (CBRN) attacks. The harmful capabilities found are then used to update and strengthen the system.
| Point | What the syllabus says |
|---|---|
| Applies to | Any AI-based system, but especially critical for GenAI: the vast range of inputs and outputs creates an extensive "attack space" |
| Scope | Typically the complete end-to-end AI-based system, but can be the AI model only |
| Harm types | Often security and safety, but also reliability, privacy, fairness, bias, misinformation |
| Regulation | Increasingly a regulatory expectation for some AI-based systems (e.g., EU AI Act) |
| vs. benchmarks | RT is adaptive and dynamic (prompts change in response to outputs). It complements static benchmarking by testing under extreme and unexpected conditions |
| Security RT | Tests vulnerabilities to external attacks, both non-AI and AI-specific, e.g., indirect prompt injection and malicious content hidden in documents used by RAG. It focuses on malicious inputs |
| Safety (and other) RT | Finds harmful outputs under ordinary use, without adversarial intent, e.g., unsafe medical advice |
| Timing | Most effective before deployment, but after initial internal quality assessments |
| Core activity | Interactive prompting: multi-turn dialogues (e.g., 15–20 turns) to elicit defective or policy-violating behavior |
| Scaling coverage | Beyond small expert teams: crowd-sourced prompt generation (manual); automated: an LLM generates attack prompts, another LLM checks the outputs; hybrid: human creativity plus automation |
| Blue teaming | RT is proactive and pre-deployment. Blue teaming is ongoing, real-time defensive monitoring and filtering of inputs to an operational system. RT insights improve blue-team monitoring and filters |
The five-step approach (syllabus):
mermaid
flowchart TB
S1["1. Assemble a diverse team<br/>wide range of perspectives<br/>and attack vectors"] --> S2["2. Provide access<br/>in a safe test environment"]
S2 --> S3["3. Prompt to find vulnerabilities<br/>open-ended exploration<br/>or checklists"]
S3 --> S4["4. Analyze the failures<br/>to understand threats"]
S4 --> S5["5. Create datasets from the threats<br/>to support mitigations<br/>and improvements"]
S5 -.->|"insights"| BT["Blue teaming<br/>monitoring and input filters<br/>in operation"]Red teaming ends with reusable datasets, not just a bug list; those datasets drive mitigations before release and feed the blue team's filters after release.
Worked example: implementing red teaming (K3)
Scenario (illustrative): a health insurer will launch "AskCover", an LLM chatbot for members. It answers questions using RAG over policy documents and can read documents that members upload (e.g., invoices). Go-live is in eight weeks.
| Step | Decisions for AskCover | Syllabus basis |
|---|---|---|
| Timing and scope | Start after the internal QA cycle, before go-live. Attack the end-to-end system (UI → RAG → LLM → output filters), not just the model | Before deployment, after internal quality assessments; end-to-end typically |
| 1. Team | Security tester, claims specialist, a clinician, privacy officer, two staff from different cultural backgrounds, plus crowd-sourced prompts from customer-service agents | Diverse team; crowd-sourcing |
| 2. Environment | Isolated copy with synthetic member data, production-equivalent configuration and RAG index | Safe test environment |
| 3. Prompting | Checklist and open-ended sessions per harm type: security: indirect prompt injection via an uploaded "invoice" containing hidden instructions, poisoned text in RAG sources; privacy: trying to get another member's data; safety: questions that tempt unsafe medical advice under ordinary use; fairness/bias: the same claim question phrased by different personas; misinformation: invented policy terms. 15–20-turn dialogues to wear down refusals. An LLM generates attack variants and another LLM pre-screens outputs; humans review the flagged ones | Interactive multi-turn prompting; open-ended or checklists; harm types; security vs. safety RT; automated and hybrid approaches |
| 4. Analysis | Cluster failures by harm type and root cause (e.g., "RAG content treated as instructions") and rate severity | Analyze failures to understand threats |
| 5. Datasets | Turn every successful attack into a regression dataset of prompts plus expected safe behavior; use it to fix the system prompt and filters, and hand it to the blue team for live monitoring | Create datasets for mitigations; insights for blue teaming |
Reusable red teaming plan template:
text
System under test ......: (end-to-end | model only) version/config: ......
Timing .................: after internal QA on ..../..../.... — before deployment on ..../..../....
Harm types in scope ....: [ ] security (incl. prompt injection, RAG content) [ ] safety (ordinary use)
[ ] privacy [ ] fairness/bias [ ] reliability [ ] misinformation [ ] other: ....
Team (diverse) .........: roles/perspectives .......... crowd-sourcing? ...... automation (LLM attacker + LLM judge)? ......
Safe test environment ..: data (synthetic?) ...... isolation ...... configuration parity ......
Techniques .............: open-ended exploration | checklists | multi-turn dialogues (15–20 turns)
Failure analysis .......: categorize by harm type / root cause / severity
Outputs ................: datasets of successful attacks -> mitigations, regression tests, blue-team filtersExam Distinction
The best red teaming strategy in a scenario is almost always the one that is proactive (before deployment) and covers several vulnerability types together. Watch for these traps: "run it after go-live to save time" or "react to incidents" (that is blue teaming), "focus only on one team's concern", and "rely on benchmarks instead".
Hands-on exercise HO-4.2.3 (H2): Exploratory testing of an LLM performing boundary value analysis
Goal (syllabus): perform exploratory testing of an LLM using a session sheet focused on the LLM's ability to generate test cases with 2-value and 3-value boundary value analysis (BVA), then verify the correctness and completeness of the output in a debrief.
Session sheet (template, supplementary)
| Field | Content |
|---|---|
| Charter | Explore how well the LLM applies 2-value and 3-value BVA (as defined in CTFL v4.0) to simple numeric specifications |
| Time box | 45 minutes |
| Test object | Any chat LLM; note model name, version, temperature (if visible) |
| Ideas to explore | integer vs. decimal ranges; open-ended ranges ("at least 18"); several partitions; ambiguous wording; asking for 2-value then 3-value in the same conversation (context window!); repeating the same prompt in a fresh chat (non-determinism) |
| Notes / defects / questions | … |
Start with this specification and oracle: "An order qualifies for a bulk discount if it contains 100 to 500 items (whole numbers)." Partitions: ≤ 99 (invalid), 100–500 (valid), ≥ 501 (invalid).
| BVA variant | Correct test values |
|---|---|
| 2-value BVA (boundary + nearest value in the neighboring partition) | 99, 100, 500, 501 |
| 3-value BVA (boundary + both neighbors) | 99, 100, 101, 499, 500, 501 |
Debrief questions: Did the LLM confuse 2-value and 3-value BVA? Did it add extra or missing values? Did it treat "100 to 500" as inclusive? Did the answer change when you repeated the prompt? Which defects would matter if a team relied on this output?
4.3.1 Test Levels for Machine Learning Systems (K2)
The non-AI components of an ML system (MLS) are tested with conventional test levels. ML adds two specialized test levels:
Input data testing: A test level that focuses on the quality of data used for training an ML model.
ML model testing: A test level that focuses on the ability of an ML model to meet required ML functional performance criteria and non-functional criteria.
Source: ISTQB® Glossary. Note: section 4.3.1 extends input data testing to the production data used for predictions as well.
ML model testing: A test level that focuses on the ability of an ML model to meet required ML functional performance criteria and non-functional criteria.
Source: ISTQB® Glossary. Note: section 4.3.1 extends input data testing to the production data used for predictions as well.
mermaid
flowchart LR
S["Testing an<br/>ML system"] --> MLS["ML-specific<br/>test levels"]
S --> CV["Conventional test levels<br/>(as risks require)"]
MLS --> IDT["Input data testing<br/>training + production data (Ch. 5)"]
MLS --> MT["ML model testing<br/>the ML model (Ch. 6)"]
CV --> CT["Component testing<br/>non-AI components, data pipeline"]
CV --> CIT["Component integration testing<br/>pipeline ↔ model ↔ UI, AIaaS API"]
CV --> ST["System testing<br/>performance once embedded, non-functional"]
CV --> SIT["System integration testing<br/>external systems and services"]
CV --> AT["Acceptance testing<br/>suitability of an AI service"]The two ML-specific levels do not replace the conventional ones: an ML system still needs component, integration, system and acceptance testing for everything around the model.
| Test level | Focus for an MLS (syllabus) |
|---|---|
| Input data testing (Ch. 5) | The training data used to train the model and the production data used to generate predictions in operation |
| ML model testing (Ch. 6) | The ML models, which are the ultimate output of the ML workflow |
| Component testing | Any non-AI components: user interface, data pipeline, communication components |
| Component integration testing | Inputs from the data pipeline are received as expected by the model; predictions are exchanged with the relevant components (e.g., UI) and used correctly. Where AI is provided as a service, API testing of the service is part of this level |
| System testing | Confirmation testing that the ML functional performance from model testing is not adversely affected when the model is embedded in the complete system, especially if the model was deliberately changed (e.g., compressing a DNN). Also non-functional testing, e.g., the time to deliver a prediction |
| System integration testing | Interfaces and data exchanges between the AI-based system and external systems or services, in an environment representative of operation |
| Acceptance testing | Where AI is used as a service: the suitability of the service for the intended system, and whether, e.g., ML functional performance criteria have been achieved |
Exam Distinction
If a scenario says a model performs badly for one region, group or condition that is rare in the data (e.g., high-altitude weather stations), the test level that should have caught it is input data testing (representativeness). Later levels might detect it, but input data testing would catch it earliest.
4.3.2 Risk-Based Testing of Machine Learning Systems (K2)
Risk-based testing should be applied to all systems, with or without AI. Most regulatory frameworks require a risk-based approach for AI-based systems. AI poses unique risks, so its testing differs. There is no standard way to categorize ML risks. They concern both the development of the MLS (project risks) and the MLS itself (product risks).
Project risk: A risk that impacts project success. Product risk: A risk that impacts the quality of a product. (ISTQB® Glossary)
One categorization, by the ML workflow (three areas):
| Area | Concerned with | Example risks (syllabus) | Type | Covered in |
|---|---|---|---|---|
| Development | The ML algorithm, model development, the ML development framework | Sub-optimal algorithm selection, poor selection of evaluation approach, framework security vulnerabilities | Project risks | Chapter 7 |
| Input data | Training data and production data | Biased training data, data-pipeline defects, unrepresentative training data | Product risks | Chapter 5 |
| Model | The generated ML model | Failure to achieve required ML functional performance, overfitted model, susceptibility to adversarial examples | Product risks | Chapter 6 |
Another categorization: by the ISO/IEC 25059 quality characteristics (Chapter 2).
Testing designed specifically for these risks includes data pipeline testing, adversarial testing, and review of algorithm/model suitability. Chapters 5–7 cover these in detail.
Exam Distinction
Generic risks such as security and usability apply to any system. Risks such as data bias and ML model performance are specific to ML systems. The official sample exam's rationale also notes that risk management should be dynamic for both conventional and AI systems; it is not "static for conventional, dynamic for AI".
Unit Quiz
Unit Quiz: Chapter 4, page 2 (4.2–4.3)
Q1. What is the "input explosion problem" when testing a GenAI model?
AInputs can be highly diverse and hard to control: optional system prompts, user prompts with large amounts of disparate data, API access, parameters such as temperature and maximum tokens, and the context window
BThe model crashes when the input is too long
CTraining data grows faster than storage capacity
DThe number of test cases doubles with every release
Q2. A team uses an image recognition model to automatically check whether images generated by a GenAI tool show the requested objects. What does the syllabus warn about this approach?
AIt is forbidden for GenAI systems
BIt always needs a human to approve each result
CIt should be used carefully, because the second system may reproduce similar biases or errors
DIt only works for text outputs
Q3. A bank will launch a GenAI customer assistant that uses RAG over uploaded customer documents. The security team wants red teaming only for prompt injection; the compliance team wants only bias checks; the project manager wants to red team after go-live. What is the MOST appropriate red teaming approach?
ARed team after go-live, based on customer complaints
BRed team only for prompt injection now, and for bias in a later release
CSkip red teaming and rely on benchmark scores
DBefore deployment (after internal quality assessments), red team the end-to-end system with a diverse team covering security (including indirect prompt injection via the RAG documents) and other harms such as bias
Q4. Which sequence matches the typical red teaming approach in the syllabus?
ADeploy → monitor → filter inputs → retrain
BAssemble a diverse team → provide access in a safe test environment → prompt to identify vulnerabilities (open-ended or with checklists) → analyze the failures → create datasets from the threats to support mitigations
CWrite test cases from the specification → execute → report pass/fail
DRun a benchmark suite → publish the score → compare with competitors
Q5. Which TWO test levels does the syllabus describe as specialized test levels for ML systems?
Select 2 options.
AInput data testing
BComponent integration testing
CML model testing
DSystem integration testing
EAcceptance testing
Q6. A model is compressed to a smaller size so it can run inside a mobile app. At which test level does the syllabus emphasize confirming that its ML functional performance has not been adversely affected?
AComponent testing
BInput data testing
CSystem testing
DSystem integration testing
Q7. Using the ML workflow to categorize risks, "sub-optimal algorithm selection" belongs to which area, and what type of risk is it?
ADevelopment area: a project risk
BInput data area: a product risk
CModel area: a product risk
DModel area: a project risk
0 / 7 answered
Topic Drills: 4.2.1 · 4.2.2 · 4.3.1 · 4.3.2 · Next: Chapter 5 →