Appearance
Ch 4: Introduction to Testing AI-Based Systems
Official Reference
ISTQB® CT-AI Syllabus v2.0, Chapter 4 (195 minutes), section 4.1, Learning Objectives (LOs) AI-4.1.1, AI-4.1.2, AI-4.1.3 (all K2). Official syllabus and documents
Exam weight (whole Chapter 4): ~18% of points (7 questions, 8 points). Every Chapter 4 LO gets its own question, and the K3 red teaming question (AI-4.2.2) is worth 2 points. This page covers the three 4.1 LOs, one question each.
This is where the syllabus moves from "what AI is" to "why testing it is different". Testing AI-based systems has three challenges that conventional testing rarely faces together:
- AI-based systems can be locked (fixed after deployment, mostly deterministic, easier to test) or adaptive (they evolve and learn, so their behavior can change unpredictably).
- The probabilistic nature of many AI models often needs statistical testing: assessing performance distributions and confidence levels across diverse scenarios.
- The test oracle problem: deciding whether an output is correct is hard or impossible for complex or subjective tasks, tasks that exceed human capabilities, fuzzy tasks, or tasks with no objective ground truth. Vague or incomplete requirements make it worse. Solutions include statistical evaluations, consulting domain experts, and defining a ground truth.
Chapter 4 keywords (know name and definition, K1): attack, exploratory testing, risk-based testing, test oracle, adaptive AI-based system, AI-based system, generative AI, large language model, locked AI-based system.
Test oracle: A source to determine an expected result.
Test oracle problem: The challenge of determining whether a test has passed or failed for a given set of test inputs and state.
Ground truth: The information provided by direct observation and measurement that is known to be real or true.
Source: ISTQB® Glossary
Test oracle: A source to determine an expected result.
Test oracle problem: The challenge of determining whether a test has passed or failed for a given set of test inputs and state.
Ground truth: The information provided by direct observation and measurement that is known to be real or true.
Source: ISTQB® Glossary
4.1.1 Locked and Adaptive AI-Based Systems (K2)
Locked AI-based system: A deterministic AI-based system with a fixed unchanging model which does not change its behavior once it is deployed.
Adaptive AI-based system: An AI-based system that adjusts its behavior in response to changes in its operational environment.
Source: ISTQB® Glossary
Adaptive AI-based system: An AI-based system that adjusts its behavior in response to changes in its operational environment.
Source: ISTQB® Glossary
mermaid
flowchart TB
L["Fully locked<br/>deterministic, same output<br/>for the same input<br/>e.g. DNN with fixed weights:<br/>lane detection, traffic signs"] --- M["In between<br/>locked at runtime but<br/>updated periodically<br/>e.g. LLM-based chatbots"] --- A["Fully adaptive<br/>non-deterministic, self-learning<br/>e.g. reinforcement learning,<br/>adaptive product recommendations"]Real systems sit on a continuum: the further down a system sits, the harder it is to test, because its expected behavior keeps moving.
| Locked AI-based system | Adaptive AI-based system | |
|---|---|---|
| Behavior after deployment | Does not change; a DNN's weights and biases are fixed and only change by retraining | Adapts, e.g., based on a reward function or to a new operational environment; the details cannot be predicted in advance |
| Syllabus examples | Many AI-based systems in use today; safety-related self-driving functions such as lane detection or traffic sign recognition | A reinforcement learning system; an e-commerce platform recommending products from evolving preferences |
| Testability | Far easier: largely deterministic, so expected results do not change. An updated locked system is a new system, so a fresh round of testing is required | More complex: the system may change its behavior during testing or as a result of testing |
| Caveat | Large neural networks can still behave non-deterministically: floating-point precision limits, hardware execution variations, especially with parallel computation or GPUs | New behaviors can't be predicted, so they can't be tested in advance and no test cases can be prepared for them |
GenAI in between: many GenAI systems, such as LLM-based chatbots, are deployed as locked models at runtime but updated periodically, which places them between fully locked and fully adaptive.
How to test an adaptive system (syllabus):
- Before deployment: test rigorously, e.g., by simulating changes in environmental conditions, testing the learning mechanism itself, and testing its ability to adapt appropriately in controlled scenarios.
- After deployment: build an automated test suite focused on core functionality and run it whenever the system undergoes significant changes, to verify that its adaptations are safe.
- Ongoing: check that performance has not degraded beyond a threshold, in response to significant changes or as part of ongoing monitoring.
Testing analogy
For a locked system, your regression suite stays valid until the next release. For an adaptive system, every "release" happens silently in production, so the core-functionality suite becomes a safety net that runs on every significant adaptation, not just on every deployment.
Exam Distinction
"AI" does not automatically mean "adaptive". A rule-based expert system (e.g., a spam filter with predefined rules) or a DNN with fixed weights is locked: its behavior only changes when it is rebuilt or retrained. Look for words like learns from real-time data, continuously, based on changing habits. Those signal an adaptive system.
4.1.2 Rationale for a Statistical Approach to Testing AI-Based Systems (K2)
Testing AI-based systems is challenging because of their data-driven and probabilistic nature. The syllabus gives four reasons for a statistical approach:
| Reason | What it means (syllabus) |
|---|---|
| Non-determinism | AI-based systems are fundamentally probabilistic; the same inputs may not always yield the same output (stochastic architecture elements, or probabilistic mappings learned from data). A single test, e.g., a cat misclassified as a dog, cannot reflect overall AI functional correctness. The test suite must be large enough for statistically significant results |
| Distributional performance evaluation | Models are trained on data distributions that do not exactly match the operational environment. Test a statistically significant sample of scenarios from the relevant operational data distributions to capture operational variability |
| Handling uncertainty and bias | AI is susceptible to data biases and can produce confident but incorrect predictions. Statistical testing quantifies and analyzes accuracy, fairness and AI robustness using confidence intervals, hypothesis testing and error analysis |
| Regulatory and safety context | Regulated industries (e.g., healthcare, transport) often require showing that safety or fairness thresholds are met with high confidence. Statistical methods support claims across a broad range of scenarios, not just specific examples |
How to do it in practice is covered in 6.1.3 (margin of error, confidence level, sample size).
Non-determinism: A property of a system or process in which an outcome is not uniquely determined by its initial conditions.
Probabilistic: Behavior described in terms of probabilities, where outcomes are uncertain and described by likelihoods rather than certainty.
Source: ISTQB® Glossary
Probabilistic: Behavior described in terms of probabilities, where outcomes are uncertain and described by likelihoods rather than certainty.
Source: ISTQB® Glossary
Exam Distinction
The main driver of statistical testing is non-determinism / probabilistic behavior, not the size or complexity of the system and not "automation is impractical". And statistical testing never replaces a separate, representative test dataset; it depends on one.
4.1.3 Test Oracles for AI-Based Systems (K2)
Why AI makes the oracle hard (five causes):
| Cause | Syllabus explanation |
|---|---|
| Probabilistic and non-deterministic nature | Outputs can vary for identical inputs. Even with a single correct target value, a strict pass/fail oracle often needs thresholds or tolerance ranges |
| Exploratory development and incomplete specifications | Requirements may evolve, be incomplete or be missing, so there is no detailed specification to derive precise expected results from |
| Complexity of tasks | Tasks may be too complex for straightforward human verification, so manual checking is impractical |
| Subjectivity of behavior | Correctness can be subjective, e.g., virtual assistants, where user expectations vary widely |
| Self-learning systems | They update their internal models after deployment, so "correct" behavior changes over time and an initial set of expected results can quickly become invalid |
How testers address it (five solutions):
mermaid
flowchart LR
P["Test oracle problem"] --> O["Output boundaries<br/>acceptable ranges, distributions,<br/>limits, tolerances"]
P --> E["Environmental boundaries<br/>fix lighting, temperature,<br/>network latency"]
P --> X["Expert consultation<br/>domain experts<br/>(may differ / be fallible)"]
P --> S["Specialized testing<br/>A/B, back-to-back,<br/>metamorphic"]
P --> Y["Proxy oracles<br/>secondary systems/models,<br/>incl. other AI"]None of the five gives a perfect expected result; each replaces "the exact correct output" with something checkable: a range, a controlled environment, an expert judgement, a relation, or a second opinion.
| Solution | Syllabus example / note |
|---|---|
| Defining output boundaries | Agreed acceptable ranges, distributions, specified limits, tolerances, e.g., an autonomous car stopping within a maximum distance |
| Defining environmental boundaries | Specify test environment conditions (lighting levels, temperature, network latency) so outputs are predictable and repeatable |
| Expert consultation | Domain experts help define expected results, but their opinions may differ or be fallible |
| Specialized testing | A/B testing, back-to-back testing, metamorphic testing compare behaviors or verify properties, often without explicit expected outputs (Chapter 6) |
| Proxy oracles | Use secondary systems or models (including other AI) to assess outputs, e.g., training a proxy model on labeled data to predict for unlabeled tests |
Pseudo-oracle (used by back-to-back testing, 6.1.10): An independently derived variant of the test item used to generate results, which are compared with the results of the original test item based on the same test inputs. (ISTQB® Glossary)
Exam Distinction: self-learning and the oracle problem
For self-learning systems the core issue is that the expected behavior changes after release while the specification does not. It is not that the requirements changed during development (that is the exploratory development cause) and not that the tasks are complex.
Unit Quiz
Unit Quiz: Chapter 4, page 1 (4.1)
Q1. Which system is MOST likely a LOCKED AI-based system?
AA smart thermostat that keeps learning the household's habits after installation
BA traffic-sign recognition model in a car whose DNN weights were fixed at release and only change when the model is retrained
CA recommendation engine that updates its suggestions as each user's preferences change
DA reinforcement learning trading agent that keeps adjusting its strategy in production
Q2. Why are locked AI-based systems generally easier to test than adaptive ones?
AThey are largely deterministic, so expected results do not change
BThey never contain neural networks
CThey never need regression testing
DThey cannot exhibit any non-determinism
Q3. During testing, an image classifier labels one photo of a cat as a dog. Why should the tester NOT conclude from this one test that the model fails its accuracy requirement?
ABecause image classifiers are never wrong about cats
BBecause the test oracle was missing
CBecause the model is probabilistic, so a single test cannot reflect its overall AI functional correctness; a sufficiently large test suite is needed for a statistically significant result
DBecause only adaptive systems need statistical testing
Q4. A medical AI vendor must show regulators that its model meets a fairness threshold "with high confidence". Which reason for statistical testing does this illustrate?
ADistributional performance evaluation
BNon-determinism
CHandling uncertainty and bias
DRegulatory and safety context
Q5. A virtual assistant's answers are judged "good" by some users and "unhelpful" by others for the same question. Which cause of the test oracle problem does this BEST illustrate?
ASubjectivity of behavior
BSelf-learning systems
CComplexity of tasks
DProbabilistic and non-deterministic nature
Q6. There is no way to know the exact correct output of a route-optimization AI, but testers can check that adding a closed road never produces a SHORTER route. Which test oracle solution is this?
AProxy oracle
BExpert consultation
CSpecialized testing, such as metamorphic testing, which verifies properties without explicit expected outputs
DDefining environmental boundaries
0 / 6 answered
Topic Drills: 4.1.1 · 4.1.2 · 4.1.3 · Next page: GenAI, Red Teaming and Test Levels →