Skip to content

Ch 6: Model Risks, Documentation, Performance and Adversarial Testing ​

Official Reference

ISTQB® CT-AI Syllabus v2.0, Chapter 6 (225 minutes), sections 6.1.1–6.1.4, Learning Objectives (LOs) AI-6.1.1, AI-6.1.2, AI-6.1.3, AI-6.1.4 (all K2). Official syllabus and documents

Exam weight (whole Chapter 6): ~23% of points (9 questions, 10 points), the heaviest chapter. Each of its nine LOs gets one question, and metamorphic testing (AI-6.1.5, page 2) is the 2-point K3 question. On this page, note that the 6.1.3 formulas are not calculated in the exam; you need to reason about margin of error and confidence level.

ML model testing is the second ML-specific test level (4.3.1). It addresses functional risks (bias, overfitting, adversarial vulnerabilities), non-functional risks (lack of AI robustness, performance efficiency) and deployment risks.

Chapter 6 keywords (know name and definition, K1): A/B testing, adversarial testing, back-to-back testing, concept drift, data drift, drift testing, metamorphic testing, ML functional performance, ML model testing, review, overfitting, underfitting.
ML model testing: A test level that focuses on the ability of an ML model to meet required ML functional performance criteria and non-functional criteria.
ML functional performance: The degree to which an ML model meets ML functional performance criteria.
Source: ISTQB® Glossary

6.1.1 Machine Learning Model Risks and Mitigations (K2) ​

The syllabus gives this table of example ML model risks and mitigations:

Potential riskPossible risk mitigation
Biased or unfair ML modelTesting for bias (5.1.2)
Unethical modelEthical system testing
Adversarial examplesAdversarial testing (6.1.4)
Overfitted modelTesting for overfitting (6.1.8)
Underfitted modelTesting for underfitting (6.1.8)
Unacceptable data drift · Unacceptable concept driftDrift testing (6.1.7)
Model causes side-effectsSide-effects testing
Model exhibits reward hackingReward hacking testing
Model API defectAPI testing (7.1.2)
Failure to achieve required ML model performance measures (e.g., lack of accuracy, recall)ML functional performance testing (6.1.3)
Functional incorrectness · Non-functional defectsMetamorphic testing (6.1.5)
Test oracle problemMetamorphic testing (6.1.5) · Back-to-back testing (6.1.10) · A/B testing (6.1.9)
Poor system requirementsRequirements review · Red teaming (4.2.2) · Exploratory testing
Lack of model AI robustness due to unexpected inputsAdversarial testing (6.1.4) · Fuzz testing
Inadequate ML model performance efficiencyPerformance testing
Poor model documentation (e.g., function, accuracy, interface)Model documentation review (6.1.2)
Model updates introduce defectsBack-to-back testing (6.1.10)
Model updates decrease ML model functional performanceA/B testing (6.1.9)
Deployment of updated model causes immediate failureSmoke testing
Deployment of updated model causes regressionRegression testing
Security vulnerabilities · Safety vulnerabilities · Privacy violations · Harmful or undesirable outputs (e.g., racist views, dangerous guidance)Red teaming (4.2.2)
Reward hacking: The activity performed by an intelligent agent to maximize its reward function to the detriment of meeting the original objective.
Fuzz testing: A test technique in which high volumes of random data are used to generate test inputs.
Smoke testing: A test type to gain sufficient confidence that a test object is ready for planned testing.
Source: ISTQB® Glossary

Exam Distinction: four "update" rows

After a model update: new defects → back-to-back testing. Lower functional performance → A/B testing. Immediate failure on deployment → smoke testing. Regression → regression testing. The test oracle problem has three mitigations (metamorphic, back-to-back, A/B), and so do poor system requirements (requirements review, red teaming, exploratory testing).

6.1.2 Machine Learning Model Documentation and Review (K2) ​

Why documentation is critical for ML models: unlike code you can inspect, MLS have limited understandability of machine-generated code, a black-box nature, and dependency on data. Models are also frequently updated. So documentation becomes the main tool for developers, testers and regulators to understand, evaluate and trust AI-based systems. Transparency lets stakeholders trace model behavior, decision logic and data lineage throughout the lifecycle.

  • Standardized documentation improves communication, supports informed decisions, and verifies the quality and maintainability of ML models.
  • Regulatory compliance: the EU AI Act emphasizes transparency obligations that require documenting model decisions, limitations and interpretability measures. For high-risk systems, passing a documentation audit is often a prerequisite for deployment. For all systems, a thorough review helps verify quality.
  • Frameworks: Model Cards (concise overview of intended uses, evaluation results, ethical considerations) and Datasheets for Datasets (standardized description of a dataset's motivation, composition, collection process and uses).

Checklist of typical contents (eight categories):

CategoryTypical contents (syllabus)
GeneralIdentifiers, description, developer, version, date, contact, license, hardware needs
DesignAssumptions, technical decisions, ML algorithm
UsageIntended purpose, primary/secondary uses, users, self-learning approach, bias, ethics, safety, transparency, thresholds, platform, data drift, concept drift
DatasetsFeatures, source, collection, availability, preprocessing, use, content, labels, size, privacy, security, bias/fairness, restrictions
TestingTest dataset details, independence of testing, test results, test activities (e.g., functional, adversarial)
FunctionalMeasures, validation dataset, thresholds, actual performance
Non-functionalScalability, reliability, availability, performance efficiency (e.g., latency, resource utilization), maintainability, AI robustness
OperationalDeployment plan, deployment environment, computational resources, monitoring metrics/alerts, retraining strategy, model update/rollback plan, deprecation plan, security (adversarial risks), explainability methods

Reviewing documentation against such a checklist is a core test activity. It aims to:

  1. identify missing information, inaccuracies and inconsistencies;
  2. improve clarity and readability;
  3. enhance maintainability by finding where documentation needs improvement;
  4. provide sufficient information for test and deployment activities;
  5. verify that all relevant regulatory requirements have been met.

Testing analogy

Treat a model card like a requirements specification in a review: check it for completeness, consistency and testability. If "intended use", "thresholds" or "test dataset details" are missing, you can't design meaningful tests. That is a review finding, just as a missing acceptance criterion would be.

Exam Distinction

Prediction speed (latency) belongs in the model's documentation (non-functional). The source of the training data belongs there too (datasets). How AI and non-AI components interact is a system concern, not model documentation (this last point comes from the official sample exam's rationale, not from the syllabus text).

6.1.3 ML Functional Performance Testing of Probabilistic ML Systems (K2) ​

ML functional performance testing evaluates how well a model performs its intended functions, using accuracy, recall, precision and F1-score (3.3), against acceptance criteria. For probabilistic MLS it moves beyond a single pass/fail to statistically measure performance, because of non-determinism (4.1.2). That means evaluating behavior across a large, representative dataset.

Precondition: acceptance criteria stated in statistical terms: a performance metric, a margin of error (MoE) and a confidence level (CL). Such a requirement determines the minimum number of tests. As the number of tests increases:

Keep fixed……and this changesMeaning
Confidence levelMoE decreasesThe result becomes more precise
Margin of errorCL increasesThe result becomes more certain
mermaid
flowchart TB
  C["Acceptance criterion<br/>metric + MoE + CL<br/>e.g. 98% accuracy ±4% at 95% CL"] --> N["Minimum number of tests<br/>e.g. 601 (conservative)"]
  N --> T["Run on a large, independent,<br/>representative test dataset<br/>(sequential testing may stop early)"]
  T --> R["Report with statistical confidence<br/>e.g. 94% ±4% at 95% CL"]

The statistical criterion drives how many tests you need, and the report states a range with a confidence level, never a bare pass/fail ratio.

The syllabus example (understand it; you will not calculate it in the exam):

  • Criterion: 98% accuracy, MoE ±4%, 95% CL → 601 test cases, using the conservative assumption that the true accuracy could be anywhere from 0 to 100%. To meet 98%, at least 589 of the 601 must pass.
  • If 98% is observed after all 601, the measured MoE at 95% CL is narrower (about ±1.1%), because variance is lower at high accuracy.
  • So you don't always need all 601. Sequential testing analyzes results as they accumulate and stops early when there is enough evidence for (or against) the target. With consistently high accuracy, ±4% at 95% CL is reached after about 170 test cases.
  • Safety-critical example: "99% reliability with 95% confidence" needs 299 test cases, all of which must pass.

How the testing is done: a large test dataset, completely independent of the training and validation datasets, that is a representative sample of the operational input domain (5.1.4). The tests run in an ML development framework capable of statistical analysis. Results are reported with statistical confidence, e.g., "The model achieved an accuracy of 94% ±4% at the 95% CL." This lets stakeholders see the range of expected operational performance and decide whether it is acceptable for deployment.

Exam Distinction: reason, don't calculate

Typical question: "more tests were run and the observed accuracy stayed the same; what is MOST likely now?" With the same accuracy and the same CL, more tests give a smaller MoE. More tests never make the CL go down or the MoE go up, all else being equal. The syllabus states the formulas are for illustration only and are not examined.

6.1.4 Adversarial Testing of Machine Learning Systems (K2) ​

Adversarial testing: A test technique in which tests are designed based on adversarial examples.
Adversarial example: An input to an ML model created by applying small perturbations to a working example that results in the model outputting an incorrect result with high confidence.
Adversarial attack: The deliberate use of adversarial examples to cause an ML model to fail.
Source: ISTQB® Glossary

Adversarial testing deliberately feeds the model perturbations of input data, often imperceptible to humans, designed to cause incorrect predictions. Successful ones are adversarial examples. Test inputs are usually slightly modified versions of legitimate inputs that cause misclassification.

  • Why: found vulnerabilities let developers add safeguards and make the model more robust. They protect against accidental adversarial examples the model might meet, and against adversarial attacks (malicious use).
  • Difficulty: generating effective inputs is technically complex, and keeping up with evolving attack techniques is an ongoing challenge.
ApproachHow (syllabus)
Black-boxFocus on input/output behavior, no knowledge of internals. Build an equivalent model whose internals are known, craft adversarial inputs from it, and apply them to the original, assuming equivalent models share classification boundaries (transferability). Or use brute force: a vast number of tests, hoping some random tests hit an adversarial example
White-boxKnowing the model's internals (architecture, parameters, training process) typically makes it easier to craft adversarial examples
Manual or automatedCrafting specific examples by hand, or automated algorithms that generate large numbers of variations
mermaid
flowchart TB
  L["Legitimate input<br/>correctly classified<br/>'stop sign'"] --> P["Small, often<br/>imperceptible perturbation"]
  P --> X["Adversarial example<br/>misclassified with high confidence<br/>'speed limit 80'"]
  X --> S["Safeguards / robustness<br/>e.g. augment training data<br/>with adversarial examples (3.2.1)"]

An adversarial example is a working input plus a tiny change; the defect it reveals is the model's fragile decision boundary, which robustness measures then harden.

Exam Distinction

Adversarial testing is not "black-box only" (white-box knowledge helps), not "manual only" (automated generation is common), and it is not re-running tests that already passed. Its defining idea is minimally perturbed inputs that cause misclassification.

Unit Quiz ​

Unit Quiz: Chapter 6, page 1 (6.1.1–6.1.4)

Q1. A newly retrained model is released, and a team wants to know whether the update made its ML functional performance worse than the previous version. Which mitigation does the 6.1.1 table map to "model updates decrease ML model functional performance"?

ASmoke testing
BFuzz testing
CA/B testing
DModel documentation review

Q2. Which risk does the syllabus mitigate with RED TEAMING in the ML model risk table?

AOverfitted model
BSecurity vulnerabilities, safety vulnerabilities, privacy violations and harmful or undesirable outputs
CUnacceptable data drift
DModel API defect

Q3. Why is documentation especially important for ML models, compared with conventional systems?

ABecause ML models have no source code at all
BBecause regulators forbid code reviews of ML models
CBecause documentation replaces testing for ML models
DBecause machine-generated models are hard to understand, are black boxes, depend on data and are frequently updated, so documentation becomes the main tool for understanding, evaluating and trusting them

Q4. In the documentation checklist, which category contains "latency, resource utilization, scalability, reliability, availability, maintainability and AI robustness"?

ANon-functional
BFunctional
COperational
DUsage

Q5. An acceptance criterion is stated as "98% accuracy with a margin of error of ±4% at a 95% confidence level". The confidence level is kept fixed and MORE tests are run. What happens?

AThe margin of error increases
BThe confidence level decreases
CThe margin of error decreases, so the result becomes more precise
DNothing changes

Q6. How should the result of ML functional performance testing of a probabilistic model be reported, according to the syllabus?

AAs a simple pass/fail ratio
BWith statistical confidence, e.g., "The model achieved an accuracy of 94% ±4% at the 95% CL"
COnly as the number of failed test cases
DAs the accuracy on the training data

Q7. How can adversarial testing be performed as BLACK-BOX testing, according to the syllabus?

ABy reading the model's weights and gradients
BBy retraining the model with adversarial examples
CBy reviewing the model documentation
DBy building an equivalent model whose internals are known, crafting adversarial inputs from it, and applying them to the original (relying on transferability), or by brute force with a vast number of tests

Q8. What is an adversarial example?

AAny test case that fails
BAn input created by applying small perturbations to a working example that results in the model outputting an incorrect result with high confidence
CA training example with a missing label
DA prompt that asks an LLM for harmful content

0 / 8 answered

Topic Drills: 6.1.1 · 6.1.2 · 6.1.3 · 6.1.4 · Next page: Metamorphic, Drift, Fitting, A/B and Back-to-Back →

Unofficial study guide for the ISTQB® Certified Tester AI Testing (CT-AI) exam. Not affiliated with International Software Testing Qualifications Board (ISTQB®).