Skip to content

Ch 4: Introduction to Testing AI-Based Systems ​

Official Reference

ISTQB® CT-AI Syllabus v2.0, Chapter 4 (195 minutes), section 4.1, Learning Objectives (LOs) AI-4.1.1, AI-4.1.2, AI-4.1.3 (all K2). Official syllabus and documents

Exam weight (whole Chapter 4): ~18% of points (7 questions, 8 points). Every Chapter 4 LO gets its own question, and the K3 red teaming question (AI-4.2.2) is worth 2 points. This page covers the three 4.1 LOs, one question each.

This is where the syllabus moves from "what AI is" to "why testing it is different". Testing AI-based systems has three challenges that conventional testing rarely faces together:

  1. AI-based systems can be locked (fixed after deployment, mostly deterministic, easier to test) or adaptive (they evolve and learn, so their behavior can change unpredictably).
  2. The probabilistic nature of many AI models often needs statistical testing: assessing performance distributions and confidence levels across diverse scenarios.
  3. The test oracle problem: deciding whether an output is correct is hard or impossible for complex or subjective tasks, tasks that exceed human capabilities, fuzzy tasks, or tasks with no objective ground truth. Vague or incomplete requirements make it worse. Solutions include statistical evaluations, consulting domain experts, and defining a ground truth.
Chapter 4 keywords (know name and definition, K1): attack, exploratory testing, risk-based testing, test oracle, adaptive AI-based system, AI-based system, generative AI, large language model, locked AI-based system.
Test oracle: A source to determine an expected result.
Test oracle problem: The challenge of determining whether a test has passed or failed for a given set of test inputs and state.
Ground truth: The information provided by direct observation and measurement that is known to be real or true.
Source: ISTQB® Glossary

4.1.1 Locked and Adaptive AI-Based Systems (K2) ​

Locked AI-based system: A deterministic AI-based system with a fixed unchanging model which does not change its behavior once it is deployed.
Adaptive AI-based system: An AI-based system that adjusts its behavior in response to changes in its operational environment.
Source: ISTQB® Glossary
mermaid
flowchart TB
  L["Fully locked<br/>deterministic, same output<br/>for the same input<br/>e.g. DNN with fixed weights:<br/>lane detection, traffic signs"] --- M["In between<br/>locked at runtime but<br/>updated periodically<br/>e.g. LLM-based chatbots"] --- A["Fully adaptive<br/>non-deterministic, self-learning<br/>e.g. reinforcement learning,<br/>adaptive product recommendations"]

Real systems sit on a continuum: the further down a system sits, the harder it is to test, because its expected behavior keeps moving.

Locked AI-based systemAdaptive AI-based system
Behavior after deploymentDoes not change; a DNN's weights and biases are fixed and only change by retrainingAdapts, e.g., based on a reward function or to a new operational environment; the details cannot be predicted in advance
Syllabus examplesMany AI-based systems in use today; safety-related self-driving functions such as lane detection or traffic sign recognitionA reinforcement learning system; an e-commerce platform recommending products from evolving preferences
TestabilityFar easier: largely deterministic, so expected results do not change. An updated locked system is a new system, so a fresh round of testing is requiredMore complex: the system may change its behavior during testing or as a result of testing
CaveatLarge neural networks can still behave non-deterministically: floating-point precision limits, hardware execution variations, especially with parallel computation or GPUsNew behaviors can't be predicted, so they can't be tested in advance and no test cases can be prepared for them

GenAI in between: many GenAI systems, such as LLM-based chatbots, are deployed as locked models at runtime but updated periodically, which places them between fully locked and fully adaptive.

How to test an adaptive system (syllabus):

  • Before deployment: test rigorously, e.g., by simulating changes in environmental conditions, testing the learning mechanism itself, and testing its ability to adapt appropriately in controlled scenarios.
  • After deployment: build an automated test suite focused on core functionality and run it whenever the system undergoes significant changes, to verify that its adaptations are safe.
  • Ongoing: check that performance has not degraded beyond a threshold, in response to significant changes or as part of ongoing monitoring.

Testing analogy

For a locked system, your regression suite stays valid until the next release. For an adaptive system, every "release" happens silently in production, so the core-functionality suite becomes a safety net that runs on every significant adaptation, not just on every deployment.

Exam Distinction

"AI" does not automatically mean "adaptive". A rule-based expert system (e.g., a spam filter with predefined rules) or a DNN with fixed weights is locked: its behavior only changes when it is rebuilt or retrained. Look for words like learns from real-time data, continuously, based on changing habits. Those signal an adaptive system.

4.1.2 Rationale for a Statistical Approach to Testing AI-Based Systems (K2) ​

Testing AI-based systems is challenging because of their data-driven and probabilistic nature. The syllabus gives four reasons for a statistical approach:

ReasonWhat it means (syllabus)
Non-determinismAI-based systems are fundamentally probabilistic; the same inputs may not always yield the same output (stochastic architecture elements, or probabilistic mappings learned from data). A single test, e.g., a cat misclassified as a dog, cannot reflect overall AI functional correctness. The test suite must be large enough for statistically significant results
Distributional performance evaluationModels are trained on data distributions that do not exactly match the operational environment. Test a statistically significant sample of scenarios from the relevant operational data distributions to capture operational variability
Handling uncertainty and biasAI is susceptible to data biases and can produce confident but incorrect predictions. Statistical testing quantifies and analyzes accuracy, fairness and AI robustness using confidence intervals, hypothesis testing and error analysis
Regulatory and safety contextRegulated industries (e.g., healthcare, transport) often require showing that safety or fairness thresholds are met with high confidence. Statistical methods support claims across a broad range of scenarios, not just specific examples

How to do it in practice is covered in 6.1.3 (margin of error, confidence level, sample size).

Non-determinism: A property of a system or process in which an outcome is not uniquely determined by its initial conditions.
Probabilistic: Behavior described in terms of probabilities, where outcomes are uncertain and described by likelihoods rather than certainty.
Source: ISTQB® Glossary

Exam Distinction

The main driver of statistical testing is non-determinism / probabilistic behavior, not the size or complexity of the system and not "automation is impractical". And statistical testing never replaces a separate, representative test dataset; it depends on one.

4.1.3 Test Oracles for AI-Based Systems (K2) ​

Why AI makes the oracle hard (five causes):

CauseSyllabus explanation
Probabilistic and non-deterministic natureOutputs can vary for identical inputs. Even with a single correct target value, a strict pass/fail oracle often needs thresholds or tolerance ranges
Exploratory development and incomplete specificationsRequirements may evolve, be incomplete or be missing, so there is no detailed specification to derive precise expected results from
Complexity of tasksTasks may be too complex for straightforward human verification, so manual checking is impractical
Subjectivity of behaviorCorrectness can be subjective, e.g., virtual assistants, where user expectations vary widely
Self-learning systemsThey update their internal models after deployment, so "correct" behavior changes over time and an initial set of expected results can quickly become invalid

How testers address it (five solutions):

mermaid
flowchart LR
  P["Test oracle problem"] --> O["Output boundaries<br/>acceptable ranges, distributions,<br/>limits, tolerances"]
  P --> E["Environmental boundaries<br/>fix lighting, temperature,<br/>network latency"]
  P --> X["Expert consultation<br/>domain experts<br/>(may differ / be fallible)"]
  P --> S["Specialized testing<br/>A/B, back-to-back,<br/>metamorphic"]
  P --> Y["Proxy oracles<br/>secondary systems/models,<br/>incl. other AI"]

None of the five gives a perfect expected result; each replaces "the exact correct output" with something checkable: a range, a controlled environment, an expert judgement, a relation, or a second opinion.

SolutionSyllabus example / note
Defining output boundariesAgreed acceptable ranges, distributions, specified limits, tolerances, e.g., an autonomous car stopping within a maximum distance
Defining environmental boundariesSpecify test environment conditions (lighting levels, temperature, network latency) so outputs are predictable and repeatable
Expert consultationDomain experts help define expected results, but their opinions may differ or be fallible
Specialized testingA/B testing, back-to-back testing, metamorphic testing compare behaviors or verify properties, often without explicit expected outputs (Chapter 6)
Proxy oraclesUse secondary systems or models (including other AI) to assess outputs, e.g., training a proxy model on labeled data to predict for unlabeled tests
Pseudo-oracle (used by back-to-back testing, 6.1.10): An independently derived variant of the test item used to generate results, which are compared with the results of the original test item based on the same test inputs. (ISTQB® Glossary)

Exam Distinction: self-learning and the oracle problem

For self-learning systems the core issue is that the expected behavior changes after release while the specification does not. It is not that the requirements changed during development (that is the exploratory development cause) and not that the tasks are complex.

Unit Quiz ​

Unit Quiz: Chapter 4, page 1 (4.1)

Q1. Which system is MOST likely a LOCKED AI-based system?

AA smart thermostat that keeps learning the household's habits after installation
BA traffic-sign recognition model in a car whose DNN weights were fixed at release and only change when the model is retrained
CA recommendation engine that updates its suggestions as each user's preferences change
DA reinforcement learning trading agent that keeps adjusting its strategy in production

Q2. Why are locked AI-based systems generally easier to test than adaptive ones?

AThey are largely deterministic, so expected results do not change
BThey never contain neural networks
CThey never need regression testing
DThey cannot exhibit any non-determinism

Q3. During testing, an image classifier labels one photo of a cat as a dog. Why should the tester NOT conclude from this one test that the model fails its accuracy requirement?

ABecause image classifiers are never wrong about cats
BBecause the test oracle was missing
CBecause the model is probabilistic, so a single test cannot reflect its overall AI functional correctness; a sufficiently large test suite is needed for a statistically significant result
DBecause only adaptive systems need statistical testing

Q4. A medical AI vendor must show regulators that its model meets a fairness threshold "with high confidence". Which reason for statistical testing does this illustrate?

ADistributional performance evaluation
BNon-determinism
CHandling uncertainty and bias
DRegulatory and safety context

Q5. A virtual assistant's answers are judged "good" by some users and "unhelpful" by others for the same question. Which cause of the test oracle problem does this BEST illustrate?

ASubjectivity of behavior
BSelf-learning systems
CComplexity of tasks
DProbabilistic and non-deterministic nature

Q6. There is no way to know the exact correct output of a route-optimization AI, but testers can check that adding a closed road never produces a SHORTER route. Which test oracle solution is this?

AProxy oracle
BExpert consultation
CSpecialized testing, such as metamorphic testing, which verifies properties without explicit expected outputs
DDefining environmental boundaries

0 / 6 answered

Topic Drills: 4.1.1 · 4.1.2 · 4.1.3 · Next page: GenAI, Red Teaming and Test Levels →

Unofficial study guide for the ISTQB® Certified Tester AI Testing (CT-AI) exam. Not affiliated with International Software Testing Qualifications Board (ISTQB®).