Appearance
Ch 6: Metamorphic, Drift, Fitting, A/B and Back-to-Back Testing
Official Reference
ISTQB® CT-AI Syllabus v2.0, Chapter 6 (225 minutes), sections 6.1.5–6.1.10, Learning Objectives (LOs) AI-6.1.5 (K3), AI-6.1.7, AI-6.1.8, AI-6.1.9, AI-6.1.10 (K2), and Hands-on Objective HO-6.1.6 (H2). Official syllabus and documents
Exam weight: five questions from this page. AI-6.1.5 is a 2-point K3 question: given a source test case and follow-up inputs, pick the follow-up outputs that are consistent with the metamorphic relation. The four K2 LOs are classic "distinguish" questions: data vs. concept drift, static vs. dynamic drift testing, over- vs. underfitting, A/B vs. back-to-back.
All five techniques on this page help with the test oracle problem (4.1.3) or with changes over time. None of them needs an exact expected result for every input.
6.1.5 Metamorphic Testing (K3)
Metamorphic testing: A test technique in which test conditions are metamorphic relations.
Metamorphic relation: A description of how a change to an input for a test case affects an expected result based on the required behavior of a test item.
Source test case: A test case that passed and is used as the basis of follow-up test cases in metamorphic testing.
Follow-up test case: A test case generated by applying a metamorphic relation to a source test case during metamorphic testing.
Source: ISTQB® Glossary
Metamorphic relation: A description of how a change to an input for a test case affects an expected result based on the required behavior of a test item.
Source test case: A test case that passed and is used as the basis of follow-up test cases in metamorphic testing.
Follow-up test case: A test case generated by applying a metamorphic relation to a source test case during metamorphic testing.
Source: ISTQB® Glossary
Metamorphic testing (MT) derives new follow-up test cases from a previously passed source test case by changing ("metamorphizing") it with a metamorphic relation (MR). The MR is based on a property of a required function of the test object and describes how a change in a test case's inputs is reflected in its expected results.
mermaid
flowchart TB
S["Source test case<br/>input X → output Y<br/>(passed)"] -->|"apply MR to the input"| F["Follow-up test case<br/>input X' → output Y'"]
F --> C{"Does Y' relate to Y<br/>as the MR requires?"}
C -->|"yes"| OK["More confidence"]
C -->|"no"| D["Failure: investigate<br/>(model defect or wrong MR)"]You never need the exact value of Y': you only check whether Y' stands in the required relation to Y, which is why MT works where no affordable oracle exists.
| Aspect | What the syllabus says |
|---|---|
| Applicability | Most test objects; functional and non-functional testing |
| Test objectives | Consistency (outputs align across related inputs) · Monotonicity (outputs change in a direction as the input changes) · Invariance (outputs stay stable under perturbations) |
| Particularly useful when | Generating expected results is problematic because no affordable test oracle exists, e.g., MLS using big data, or when testers are unclear how the model derives its predictions |
| Used in AI/ML for | Image recognition, search engines, route optimization, voice recognition |
| Chosen over oracle-based testing when | (1) no reliable expected outputs (model opacity or data scale), (2) the system is a black box, (3) relational properties (not absolute values) are enough for confidence |
| Example | A risk-assessment MLS predicting age at death: more cigarettes smoked → lower prediction (monotonicity) |
| Deriving MRs | From domain knowledge, requirements, or domain properties (e.g., laws of physics) |
| Validating MRs | Expert review, running them on reference models, checking coverage of edge cases |
| Limitations | Incorrect MRs (e.g., overlooking complex interactions between variables) or incomplete sets of MRs give false confidence. MT finds relational flaws but not all absolute errors, so combine it with other techniques |
Worked example: deriving metamorphic test cases (K3)
System (illustrative): a food-delivery app's ETA predictor. Inputs: distance (km), restaurant prep time (min), rain (yes/no), customer name. Source test case (passed): distance 3 km, prep 10 min, no rain, customer "Ana" → ETA 28 min.
| MR | Property (objective) | Follow-up input (change from source) | Expected relation | Observed | Verdict |
|---|---|---|---|---|---|
| MR1 | Monotonicity (domain: further = longer) | distance 6 km | ETA' ≥ 28 | 41 | ✓ |
| MR2 | Monotonicity (requirement: rain slows couriers) | rain yes | ETA' ≥ 28 | 33 | ✓ |
| MR3 | Invariance (name must not matter) | customer "Bilal" | ETA' = 28 (± agreed tolerance) | 28 | ✓ |
| MR4 | Monotonicity (prep adds time) | prep 25 min | ETA' ≥ 28 | 26 | ✗ failure |
| MR5 | Consistency (same input, same output) | identical order repeated | ETA' = 28 (± tolerance) | 28 | ✓ |
The MR4 violation (longer preparation but a shorter ETA) is a failure to investigate. Either the model learned a spurious pattern, or the MR is wrong. Perhaps long-prep restaurants are close to courier hubs? That would be a "complex interaction between variables", so review the MR with a domain expert before raising a defect.
Another common pattern: refining a filter (partition relation). For a search or recommendation system, if the source query says "pool: don't care" and returns set S, then the follow-up queries "pool: yes" and "pool: no" must each return a subset of S, the two results must not overlap, and together they must cover S. A follow-up that adds new items, drops items entirely, or lists the same item under both is a violation. (Supplementary pattern, built on the consistency objective.)
Reusable MR template (supplementary):
text
MR id / name ..........: MR-__ ____________________________
Objective .............: [ ] consistency [ ] monotonicity [ ] invariance
Basis .................: requirement ___ | domain knowledge ___ | domain property (e.g., physics) ___
Source test case ......: input = ______________________ output = ________ (passed? Y/N)
Input transformation ..: ______________________________________________
Expected relation .....: output' (=, ≥, ≤, ⊆, disjoint, ...) output tolerance: ______
Validated by ..........: expert review | reference model | edge-case coverage
Known limits ..........: interactions that could make this MR invalid: ______________Exam technique for the K3 question
- Read the source test case and find what changed in each follow-up input.
- Translate the change into the relation the outputs must satisfy: more restrictive filter → subset; a mutually exclusive split → no overlap, union = source; "more X" → not less Y; an irrelevant change → same output.
- Eliminate every option that breaks the relation. Exact values are irrelevant; the relationship is what is tested.
Hands-on exercise HO-6.1.6 (H2): Apply metamorphic testing
Goal (syllabus): derive several MRs for a given MLS (some where source and follow-up expected results are the same, some where they differ), generate source test cases (they need not be guaranteed to pass, but remember MT's limits when passed source tests are unavailable), derive the follow-up test cases, and run them.
Setup: pip install numpy pandas scikit-learn. The script trains a small delivery-time model and checks four MRs:
python
import numpy as np
import pandas as pd
from sklearn.ensemble import RandomForestRegressor
# A small "ML system under test": predicts a delivery time (minutes) from order features.
rng = np.random.default_rng(42)
n = 3000
train = pd.DataFrame({
"distance_km": rng.uniform(0.5, 12, n),
"prep_min": rng.uniform(5, 30, n),
"raining": rng.integers(0, 2, n),
})
train["eta_min"] = (8 + 3.2 * train["distance_km"] + 0.9 * train["prep_min"]
+ 6 * train["raining"] + rng.normal(0, 3, n))
features = ["distance_km", "prep_min", "raining"]
model = RandomForestRegressor(n_estimators=100, random_state=0).fit(train[features], train["eta_min"])
predict = lambda df: model.predict(df[features])
# Source test cases: 200 realistic orders (we do NOT know their exact expected ETA)
src = pd.DataFrame({
"distance_km": rng.uniform(1, 10, 200),
"prep_min": rng.uniform(5, 25, 200),
"raining": 0,
})
src_out = predict(src)
def check(name, follow_up, relation):
out = predict(follow_up)
violations = (~relation(src_out, out)).sum()
print(f"{name:<62} violations: {violations}/{len(src)}")
# MR1 (monotonicity): longer distance, all else equal -> ETA must not decrease
check("MR1 distance +2 km -> ETA not lower",
src.assign(distance_km=src["distance_km"] + 2), lambda s, f: f >= s)
# MR2 (monotonicity): rain instead of no rain -> ETA must not decrease
check("MR2 raining = 1 -> ETA not lower",
src.assign(raining=1), lambda s, f: f >= s)
# MR3 (invariance): shuffling the order of the batch must not change any single prediction
shuffled = src.sample(frac=1, random_state=1)
out_shuffled = pd.Series(predict(shuffled), index=shuffled.index).sort_index().to_numpy()
print(f"{'MR3 shuffle batch -> identical predictions':<62} violations: "
f"{(~np.isclose(out_shuffled, src_out)).sum()}/{len(src)}")
# MR4 (monotonicity, small step): distance +0.1 km -> ETA must not decrease
check("MR4 distance +0.1 km -> ETA not lower (fine-grained)",
src.assign(distance_km=src["distance_km"] + 0.1), lambda s, f: f >= s)One run gave: MR1 1/200 violations, MR2 0/200, MR3 0/200, and MR4 76/200. Discuss: a random forest's prediction is "stepped" and noisy, so tiny changes can lower the ETA. Is that a model defect, or is the MR too strict and in need of a tolerance? This is exactly the "incorrect MR → false conclusions" judgement the syllabus warns about.
6.1.7 Drift Testing (K2)
Drift testing: A type of maintenance testing that evaluates the degree to which data drift, concept drift or both occur.
Data drift: A change in the distribution of input data over time that negatively impacts the ML functional performance of an operational ML model.
Concept drift: A change in user expectations, behavior or the operational environment over time that negatively impacts the ML functional performance of an operational ML model.
Source: ISTQB® Glossary. Section 6.1.7 frames concept drift as a change in the relationship between input and correct output; see below.
Data drift: A change in the distribution of input data over time that negatively impacts the ML functional performance of an operational ML model.
Concept drift: A change in user expectations, behavior or the operational environment over time that negatively impacts the ML functional performance of an operational ML model.
Source: ISTQB® Glossary. Section 6.1.7 frames concept drift as a change in the relationship between input and correct output; see below.
| Data drift | Concept drift | |
|---|---|---|
| What changes (6.1.7) | The statistical properties of the operational input data | The relationship between the input data and the correct output |
| Why | Shifts in user behavior, seasonality… | The model's learned patterns or rules no longer reflect current reality; its decision boundaries become outdated |
| Syllabus example | A spam filter meets new types of phishing attacks that did not exist during training | New financial regulations turn a "low-risk" transaction type into "high-risk"; the meaning of the data changed |
Two ways to test for drift:
| Dynamic drift testing | Static drift testing | |
|---|---|---|
| Needs ground truth? | Yes: user feedback provides the current ground truth | No |
| How | Compare the model's output with the current ground truth and compare the difference with a threshold | Compare the statistical properties of input and predicted output distributions, e.g., with Kolmogorov-Smirnov; a significant difference indicates drift |
| Feedback examples (film recommender) | Direct: the user rates a recommendation. Indirect: data on which films the user actually watched | n/a |
Exam Distinction
No feedback or ground truth available → static drift testing (distributions only). "Comparing live performance metrics with a threshold" requires ground truth, so it is dynamic. And data drift = the inputs change; concept drift = what counts as the correct answer changes.
6.1.8 Testing for Overfitting and Underfitting (K2)
Overfitting: The generation of an ML model that corresponds too closely to the training dataset, resulting in a model that finds it difficult to generalize to new data.
Underfitting: The generation of an ML model that does not reflect the underlying trend of the training dataset, resulting in a model that makes inaccurate predictions.
Source: ISTQB® Glossary
Underfitting: The generation of an ML model that does not reflect the underlying trend of the training dataset, resulting in a model that makes inaccurate predictions.
Source: ISTQB® Glossary
There are three possible outcomes: overfitting, underfitting, and a "right-fitting" model. Testing for them happens during training, evaluation and tuning.
| Overfitting | Underfitting | |
|---|---|---|
| Cause (syllabus) | The model learns the training data too well, capturing noise rather than the underlying pattern, so it generalizes poorly to new data | The model is too simple for the structure of the data, or the training data lacks features that reflect an important input–output relationship |
| Symptom | Good on seen data, poor on unseen data | Poor ML functional performance on both training and validation datasets |
| How to detect | Evaluate on a separate test dataset not used in training, including some less common examples. Significantly worse on test than on validation → possibly overfitting | Metrics (accuracy, precision, recall, F1) consistently low on both training and validation. Learning curves: training and validation errors stay high and close together, with no significant improvement |
mermaid
flowchart TB
Q1{"Performance on<br/>training/validation data?"} -->|"low on both"| U["Underfitting<br/>(too simple / missing features)"]
Q1 -->|"high"| Q2{"Performance on independent<br/>test data (incl. rare cases)?"}
Q2 -->|"significantly lower"| O["Overfitting<br/>(learned noise)"]
Q2 -->|"similar"| R["Right-fitting"]Underfitting shows up immediately on the data the model already knows; overfitting only shows up when you test it on data it has never seen.
6.1.9 A/B Testing (K2)
A/B testing: A test approach in which two variants of a test object are statistically evaluated to determine which performs better for specified characteristics. (ISTQB® Glossary)
| Point | What the syllabus says |
|---|---|
| What | The responses of two variants (A and B) to the same inputs are compared to decide which is better. A statistical approach that typically needs results from multiple test runs |
| Classic example | Two promotional offers emailed to two halves of a marketing list; e-commerce companies divert consumers to different functionality in production |
| Oracle problem | Addresses it by using the existing system as a partial test oracle |
| What it does NOT do | It does not generate test cases and gives no guidance on test design, though operational inputs are often used |
| Testing AI updates | With agreed acceptance criteria (e.g., ML functional performance metrics), it checks that the updated variant performs as well as, or better than, the previous one |
| Complex systems | E.g., a smart city transport routing update, comparing average commute times for the two variants on consecutive weeks |
| Self-learning systems | When the system changes, automated tests run and the results are compared with those before the change: improved → accept; otherwise → revert |
| Statistical techniques | t-test, z-test, chi-squared test, Mann-Whitney U test |
6.1.10 Back-to-Back Testing (K2)
Back-to-back testing: A test approach in which a pseudo-oracle is used.
Pseudo-oracle: An independently derived variant of the test item used to generate results, which are compared with the results of the original test item based on the same test inputs.
Source: ISTQB® Glossary
Pseudo-oracle: An independently derived variant of the test item used to generate results, which are compared with the results of the original test item based on the same test inputs.
Source: ISTQB® Glossary
| Point | What the syllabus says |
|---|---|
| How | An alternative version of the system serves as a pseudo-oracle. Its outputs are compared with the system under test's for the same inputs |
| Source of the pseudo-oracle | An existing system, or one developed specifically for testing (at a cost) |
| Independence | Ideally no shared software components, otherwise both may contain the same defect and agree while both are wrong (a real issue with reusable open-source AI components). So it is often built by a different, ideally independent team, with different frameworks, algorithms or model settings. Conventional software can be a pseudo-oracle if it solves the same problem |
| Cost saver | For functional back-to-back testing, the pseudo-oracle only has to match functional behavior, not the non-functional requirements |
| Test design | Only test inputs are needed, no expected results. Inputs can come from existing tests (e.g., regression suites) or be generated automatically from training data, so very large numbers are possible with automation |
| Where it shines | Migrating an MLS to a new environment (e.g., development → production) and comparing results across environments; finding subtle defects, especially with edge cases or unusual inputs |
Exam Distinction: A/B vs. back-to-back
A/B testing compares two variants of the same MLS using ML functional performance metrics and statistics to decide which is better. Back-to-back testing compares the SUT with an independent pseudo-oracle to detect defects (any disagreement is investigated). A pseudo-oracle made by tweaking hyperparameters, fine-tuning or adding RAG to the same model is not independent enough.
Unit Quiz
Unit Quiz: Chapter 6, page 2 (6.1.5–6.1.10)
Q1. A model predicts a used car's market value. Source test case (passed): a 5-year-old car with 60,000 km is valued at €14,000. Using the metamorphic relation "more mileage, all else equal, never increases the value", which follow-up result would REVEAL a failure?
ASame car with 90,000 km → €12,500
BSame car with 60,000 km → €14,000
CSame car with 90,000 km → €15,200
DSame car with 65,000 km → €13,900
Q2. When does the syllabus say metamorphic testing is typically selected over traditional oracle-based testing?
AWhen no reliable expected outputs exist (model opacity or data scale), the system is a black box, or relational properties are enough for confidence
BWhen full specifications with exact expected results are available
COnly for non-functional testing
DOnly when the source test case failed
Q3. After new financial regulations, a transaction type previously considered "low-risk" must now be classified "high-risk", although the transaction data itself looks the same. What kind of drift is this?
AData drift
BOverfitting
CModel API defect
DConcept drift
Q4. A churn model has no ground truth available in operation (no user feedback, no record of who actually left). Which drift testing approach is possible?
ADynamic drift testing
BStatic drift testing, comparing the statistical properties of input and predicted output distributions, e.g., with a Kolmogorov-Smirnov test
CA/B testing against the training data
DNeuron coverage analysis
Q5. Training and validation errors both stay high and close together, and they hardly improve as training continues. What does this indicate?
AOverfitting
BData drift
CUnderfitting
DA right-fitting model
Q6. What does A/B testing determine for an updated ML system, according to the syllabus?
AWhether the updated variant performs as well as, or better than, the previous variant against agreed acceptance criteria
BWhether the internal algorithm structure is correct
CWhich test cases should be designed
DWhether components interact correctly
Q7. Which pseudo-oracle is MOST suitable for back-to-back testing of an ML model?
AThe same model with slightly different hyperparameters
BThe same model, fine-tuned on more data
CThe same model, augmented with RAG
DAn alternative version developed by an independent team using a different ML development framework
0 / 7 answered
Topic Drills: 6.1.5 · 6.1.7 · 6.1.8 · 6.1.9 · 6.1.10 · Next: Chapter 7 →