Appearance
Ch 7: ML Development Testing
Official Reference
ISTQB® CT-AI Syllabus v2.0, Chapter 7 (K2, 30 minutes), section 7.1, Learning Objectives (LOs) AI-7.1.1 and AI-7.1.2. Official syllabus and documents
Exam weight: ~5% of points (2 questions, 2 points, K2): one question per LO. Typical forms: "which TWO risks can ML functional performance testing mitigate?" and "how does shadow testing differ from canary testing?"
The shortest chapter, and the most familiar territory for a tester. It covers the risks introduced by ML development tools, configuration choices and deployment mechanisms, rather than by the ML model itself. Test approaches and test types include API testing, ML functional performance testing, A/B testing, back-to-back testing and reviews, used to verify AI robustness and efficiency.
Chapter 7 keywords (know name and definition, K1): ML development testing, ML functional performance, shadow testing.
ML development testing (Unofficial: compiled from syllabus section 7.1; no ISTQB® Glossary entry): testing that addresses the risks introduced by ML development tools, configuration choices and deployment mechanisms, rather than by the ML model itself.
ML functional performance: The degree to which an ML model meets ML functional performance criteria. (ISTQB® Glossary)
Shadow testing: A test type used to compare a new version of a component or system with the current version by running the former in the background. (ISTQB® Glossary)
ML development testing (Unofficial: compiled from syllabus section 7.1; no ISTQB® Glossary entry): testing that addresses the risks introduced by ML development tools, configuration choices and deployment mechanisms, rather than by the ML model itself.
ML functional performance: The degree to which an ML model meets ML functional performance criteria. (ISTQB® Glossary)
Shadow testing: A test type used to compare a new version of a component or system with the current version by running the former in the background. (ISTQB® Glossary)
7.1.1 Machine Learning Development Risks and Mitigations (K2)
The syllabus gives this table of example ML development risks and mitigations:
| Potential risk | Possible risk mitigation |
|---|---|
| Incorrect or unintended use of library or framework APIs (e.g., TensorFlow, PyTorch) | API testing (7.1.2) |
| Sub-optimal framework selection | Framework suitability review |
| Problems within the algorithm, model or development framework that introduce systemic unfairness | Testing for bias (5.1.2) |
| Defective framework installation or build | Smoke testing |
| Defective implementation of the evaluation by the framework | Reviews of framework evaluation code · Cross-check framework evaluation results (e.g., against manual benchmarks) |
| Poor performance efficiency (e.g., framework is slow to respond) | Performance testing |
| Poor usability of the framework | Usability testing |
| Defect in a library used by the framework (e.g., a defect in PyTorch) · Defective algorithm implementation | ML functional performance testing (6.1.3) · Back-to-back testing (6.1.10) |
| Security vulnerabilities in the framework | Security testing |
| Poor user documentation for the framework | Framework documentation review |
| Sub-optimal algorithm selection | Algorithm suitability review · A/B testing (6.1.9) |
| Sub-optimal hyperparameter selection (e.g., network structure, learning rate) | ML functional performance testing (6.1.3) · A/B testing (6.1.9) |
| Defective allocation of data to training, validation and testing datasets | Data allocation review |
| Poor selection of evaluation approach (e.g., k-fold cross-validation) | ML functional performance testing (6.1.3) |
| Incorrect interpretation of test results due to the stochastic nature of the learning process | ML functional performance testing (6.1.3) |
| Deployment defect (e.g., from generating a modified version for a target platform) | Smoke testing · ML functional performance testing (6.1.3) · A/B testing (6.1.9) |
| Deployed model is incompatible with the operational environment | Smoke testing · MLS deployment testing (7.1.2) |
| Deployed model is no improvement over the current model | Shadow testing (7.1.2) |
How to read the table quickly
- ML functional performance testing (6.1.3) appears in five rows. It is the answer whenever a risk shows up as wrong or unreliable model results: a library defect or defective algorithm implementation, poor hyperparameters, a poor evaluation approach, stochastic results, or a deployment defect.
- Smoke testing is for "does it even start / run?": installation or build, deployment defects, incompatible environment.
- Reviews (framework suitability, evaluation code, framework documentation, algorithm suitability, data allocation) are for choices and artefacts made before the model runs.
Exam Distinction
Performance testing (speed, resources: "the framework is slow") ≠ ML functional performance testing (correctness of results: accuracy, recall…). And choosing between two algorithms is an algorithm suitability review or A/B testing, not ML functional performance testing of a single algorithm.
7.1.2 Machine Learning System Deployment Testing (K2)
Deploying an MLS involves test activities that verify the AI-based system functions correctly and reliably in its target environment (e.g., cloud, edge device, mobile). Each test type addresses specific deployment risks:
| Test type | What it verifies (syllabus) |
|---|---|
| Installability testing | The MLS can be successfully installed, configured and subsequently uninstalled across all supported environments, including system dependencies (e.g., GPU drivers), compatibility with frameworks, and successful execution of installation scripts |
| Rollback testing | The system can revert to a previously stable, operational state after a degraded or failed deployment. It may cover only the model or the complete system (e.g., including the data pipeline). It must be performed before deployment to confirm rollback readiness |
| Canary testing | Validates new deployments by releasing an updated model to a small subset of production traffic (e.g., 5% of users). Real-time metrics (latency, accuracy, error rates) are monitored to detect regressions before a full rollout |
| Shadow testing | Runs a new model in parallel with the current production model in real time, sending the same requests to both without affecting live responses. It compares new and old models on live data in a controlled, low-risk way and can uncover performance regressions and data drift before full deployment |
| Model conversion testing | The model keeps acceptable predictive accuracy, consistent behavior and operational efficiency (e.g., inference speed, memory usage) after conversion from its training format to a deployment format for the target production environment |
| Cross-device testing | The MLS performs correctly across its intended range of deployment targets, from mobile and edge devices to cloud servers |
| API testing | The MLS exposes well-defined, standards-compliant interfaces. It validates handling of inputs and outputs, error messages, and integration workflows with external systems (data feeds, clients, the pipeline) |
mermaid
flowchart TB
PRE["Before go-live<br/>installability · model conversion · cross-device ·<br/>API · rollback readiness"] --> SH["Shadow testing<br/>users see the OLD model only;<br/>the new model's responses are only logged"]
SH --> CA["Canary testing<br/>a small % of users see the NEW model;<br/>latency, accuracy, error rates watched"]
CA --> FULL["Full rollout"]Shadow testing carries no user risk because its responses are only logged; canary testing deliberately exposes a small group of real users; rollback has to be proven before either begins. (Running shadow before canary is a common practice, not a syllabus rule.)
Model conversion testing: A test type to determine whether an ML model converted for deployment retains acceptable predictive accuracy, consistent behavior, and operational efficiency.
Cross-device testing: A test type to determine whether a component or system performs correctly across its intended range of devices.
API testing: A test approach performed by submitting requests to a test object using its application programming interface.
Installability: The degree to which a component or system can be successfully installed and/or uninstalled in a specified environment.
Source: ISTQB® Glossary
Cross-device testing: A test type to determine whether a component or system performs correctly across its intended range of devices.
API testing: A test approach performed by submitting requests to a test object using its application programming interface.
Installability: The degree to which a component or system can be successfully installed and/or uninstalled in a specified environment.
Source: ISTQB® Glossary
Exam Distinction: shadow vs. canary
Both use live production data, and both are online. The difference is whether the new model's response reaches the user. Shadow: never; the new model runs in the background and its responses are only compared. Canary: yes, for a small subset of users, while metrics are watched.
Unit Quiz
Unit Quiz: Chapter 7
Q1. Chapter 7 addresses risks introduced by what, rather than by the ML model itself?
AThe training data
BEnd users' expectations
CML development tools, configuration choices and deployment mechanisms
DRegulators
Q2. A team cannot reproduce a single evaluation result: the ML functional performance varies slightly every time the training is re-run. Which mitigation does the 7.1.1 table give for "incorrect interpretation of test results due to the stochastic nature of the learning process"?
AML functional performance testing (multiple runs with statistical analysis)
BSmoke testing
CUsability testing
DFramework documentation review
Q3. Which deployment test type runs a new model in parallel with the production model on the same live requests, WITHOUT affecting the responses users receive?
ACanary testing
BRollback testing
CInstallability testing
DShadow testing
Q4. When must rollback testing be performed, according to the syllabus?
AOnly after a failed deployment
BBefore deployment, to confirm rollback readiness
COnly in production, on live traffic
DNever; rollbacks are tested by monitoring
Q5. A model trained in a full-precision desktop format is converted to a compact format for a mobile app. Which deployment test type checks that it keeps acceptable predictive accuracy, consistent behavior and operational efficiency?
ACross-device testing
BAPI testing
CModel conversion testing
DCanary testing
0 / 5 answered