Appearance
Ch 3: Data and Performance Metrics
Official Reference
ISTQB® CT-AI Syllabus v2.0, Chapter 3 (375 minutes), sections 3.2–3.3, Learning Objectives (LOs) AI-3.2.1 (K2), AI-3.2.3 (K2), AI-3.3.1 (K3), and Hands-on Objectives HO-3.2.2, HO-3.3.2, HO-3.3.3 (H2). Official syllabus and documents
Exam weight: AI-3.3.1 always gets its own 2-point K3 question: calculate (or pick the formula for) a metric from a confusion matrix. The two K2 LOs here share the pooled Chapter 3 questions. Bring the four formulas to the exam in your head; there is no formula sheet.
Data preparation is one of the most crucial and resource-intensive activities in the ML workflow. It typically takes a much larger share of the effort than model selection and building. If operational data differs significantly from training data, ML functional performance and safety assumptions may no longer hold. Data preparation is closely linked to the data pipeline, which processes raw data into a usable format for both training and prediction.
Data preparation: The activities of data acquisition, data preprocessing and feature engineering in the ML workflow. (ISTQB® Glossary; an AI-specific keyword of Chapter 3)
ML functional performance metric: A measure that relates to the functional correctness of an ML model. (ISTQB® Glossary; keyword)
ML functional performance metric: A measure that relates to the functional correctness of an ML model. (ISTQB® Glossary; keyword)
3.2.1 Activities in Data Preparation (K2)
mermaid
flowchart TB
A["Data acquisition<br/>identify types · gather · label"] --> P["Data preprocessing<br/>clean · transform · augment · sample"]
P --> F["Feature engineering<br/>select · extract features"]
E["Exploratory data analysis (EDA)<br/>trends, patterns, anomalies, visualization"] -.-> A
E -.-> P
E -.-> FThe three preparation activities run roughly in this order, often iteratively, while EDA runs alongside all of them to give insight into the data.
| Activity | Includes (syllabus) |
|---|---|
| Data acquisition | Identifying relevant data types (numerical, categorical, images, text…); gathering data from databases, APIs or real-time sensors; labeling data for supervised learning, verifying accuracy and consistency. Forms: numerical, categorical, image, tabular, text, time series, sensor, geospatial, video, audio |
| Data preprocessing | Cleaning: removing defects, duplicates and outliers; imputing missing values (mean, median, mode); anonymizing or removing personal information. Transforming formats, scaling, normalizing. Augmenting: increasing sample size, incorporating adversarial examples for robustness, generating synthetic data. Sampling subsets to reduce training times and costs |
| Feature engineering | Selecting relevant features by their contribution to model performance. Extracting a subset of informative, non-redundant features to reduce training times and costs |
| Exploratory data analysis (EDA), in parallel | Discovering trends, patterns and anomalies; visualizing data with plots and charts |
More points from the syllabus:
- Preparing training data is typically iterative and often manual; activities may be reordered or omitted depending on the project.
- Operational data should match the characteristics of the training data (distributions, feature ranges) so the model performs as expected in production. The preparation steps themselves may be adjusted for production efficiency and scalability.
Data preprocessing: The activities of data cleaning, data transformation, data augmentation, and data sampling in the ML workflow.
Feature engineering: The activity in which those attributes in the raw data that best represent the underlying relationships that should appear in the ML model are identified for use in the training data.
Exploratory data analysis: An interactive, visual, and hypothesis-driven process to summarize, explore, and understand the main characteristics and patterns of data.
Source: ISTQB® Glossary
Feature engineering: The activity in which those attributes in the raw data that best represent the underlying relationships that should appear in the ML model are identified for use in the training data.
Exploratory data analysis: An interactive, visual, and hypothesis-driven process to summarize, explore, and understand the main characteristics and patterns of data.
Source: ISTQB® Glossary
Exam Distinction
Augmentation adds examples and sampling reduces them; both are preprocessing. EDA is not exploratory testing: it is an analytical activity that informs data preparation. Feature engineering happens before or during training, never after.
Hands-on exercise HO-3.2.2 (H2): Data preparation
Goal (syllabus): for a given dataset, perform the applicable data preparation steps to produce a dataset for a supervised classification model. This is the first step for the later exercises.
Setup (once): pip install pandas seaborn scikit-learn. The Titanic dataset is downloaded by seaborn on first use.
python
import seaborn as sns
import pandas as pd
from sklearn.model_selection import train_test_split
df = sns.load_dataset("titanic") # data acquisition (already labeled: "survived")
# EDA: understand the data before changing it
print(df.shape)
print(df.isna().sum()) # missing values per column
print(df["survived"].value_counts(normalize=True)) # class balance
# Feature engineering: drop columns that duplicate or leak the label, or are mostly empty
# - "alive" is the label again in words (a leak!), "class"/"embark_town" duplicate other columns,
# - "who"/"adult_male"/"alone" are derived from other columns, "deck" is mostly missing
df = df.drop(columns=["alive", "class", "embark_town", "who", "adult_male", "alone", "deck"])
# Preprocessing: cleaning
df = df.drop_duplicates() # remove duplicates
df["age"] = df["age"].fillna(df["age"].median()) # impute missing values (median)
df = df.dropna(subset=["embarked"]) # drop the few rows with no port
# Preprocessing: transformation (categorical -> numeric)
X = pd.get_dummies(df.drop(columns="survived"), columns=["sex", "embarked"], drop_first=True)
y = df["survived"]
# Split into training / validation / test datasets (60 / 20 / 20), stratified by the label
X_trainval, X_test, y_trainval, y_test = train_test_split(
X, y, test_size=0.2, stratify=y, random_state=42)
X_train, X_val, y_train, y_val = train_test_split(
X_trainval, y_trainval, test_size=0.25, stratify=y_trainval, random_state=42)
print(len(X_train), len(X_val), len(X_test))Tester's takeaway: the alive column would have given a perfect but useless model. Spotting label leakage is a data defect that input data testing (Chapter 5) is designed to find.
3.2.3 Training, Validation, and Test Datasets (K2)
Logically, three sets of equivalent data (e.g., randomly selected from one representative dataset) are needed:
| Dataset | Used to… |
|---|---|
| Training dataset | Train the model |
| Validation dataset | Evaluate and then tune the model |
| Test dataset (also called the holdout dataset) | Test the tuned model |
Training dataset: A dataset used to train an ML model.
Validation dataset: A dataset used to evaluate a trained ML model with the purpose of tuning the model.
Source: ISTQB® Glossary. The test dataset has no glossary entry; the wording in the table above is from syllabus section 3.2.3.
Validation dataset: A dataset used to evaluate a trained ML model with the purpose of tuning the model.
Source: ISTQB® Glossary. The test dataset has no glossary entry; the wording in the table above is from syllabus section 3.2.3.
How much data for each? If suitable data is abundant, the amounts depend on:
- the expected complexity of the model;
- the algorithm used to train it;
- available resources (RAM, disk space, computing power, network bandwidth, time);
- the desired confidence in the resulting model.
When data is limited, a three-way split can leave too little training data, which increases the risk of underfitting. Strategy:
mermaid
flowchart TB
ALL["All available data"] --> HO["Small final hold-out test set<br/>(if feasible)"]
ALL --> POOL["Training + validation pool"]
POOL --> KF["k-fold cross-validation (k often 5 or 10)<br/>train on k-1 folds, validate on 1, repeat k times<br/>folds often built by stratified sampling"]
KF --> AVG["Average the metrics of the k folds<br/>→ choose hyperparameters"]
AVG --> FIN["Train final model on the WHOLE pool"]
FIN --> T1["Evaluate ONCE on the hold-out test set<br/>→ unbiased final assessment"]
HO --> T1Every record is used for validation exactly once during cross-validation, but the hold-out test set is touched only at the very end; that is what keeps the final estimate unbiased.
- Folds are usually assigned randomly, often with stratified sampling so each fold is representative, especially with imbalanced data or small datasets.
- Averaging the metrics (e.g., accuracy, F1-score) over the folds gives a more reliable estimate of generalization.
- If a hold-out test set is not feasible because of extreme data scarcity, the average cross-validation performance is optimistically biased and cannot serve as an unbiased estimate.
- Other resampling methods for limited data: leave-one-out cross-validation (k = number of samples) and bootstrap techniques.
Bootstrap technique: A resampling technique that repeatedly draws samples with replacement from a training dataset to estimate the ML functional performance criteria of an ML model. (ISTQB® Glossary)
Exam Distinction
The training dataset fits the model; it is not used for hyperparameter tuning. The validation dataset tunes the model; it is not used for deployment. The test dataset gives the final assessment and is never used for tuning or for generating training data.
3.3.1 Calculation of ML Functional Performance Metrics (K3)
In classification, a model rarely predicts correctly every time, partly because of the probabilistic nature of ML models and data noise. A confusion matrix sorts every prediction into one of four outcomes:
| Actual: Positive | Actual: Negative | |
|---|---|---|
| Predicted: Positive | True Positive (TP) | False Positive (FP) |
| Predicted: Negative | False Negative (FN) | True Negative (TN) |
Layout of Figure 2 in the syllabus: predicted values in rows, actual values in columns.
Watch the layout
The syllabus notes that a confusion matrix may be presented differently (e.g., predicted and actual swapped). The official sample exam uses actual in rows, predicted in columns. Before calculating, find the cell where predicted and actual are both positive (TP), then label the others.
The four formulas (syllabus 3.3.1):
Accuracy = (TP + TN) / (TP + TN + FP + FN) × 100%: the percentage of all classifications that were correct.
Precision = TP / (TP + FP) × 100%: the proportion of predicted positives that were correct; "how sure can I be about a positive prediction?"
Recall (also called sensitivity) = TP / (TP + FN) × 100%: the proportion of actual positives that were found; "how confident can I be that no positives are missed?"
F1-score = 2 × (Precision × Recall) / (Precision + Recall): the harmonic mean of precision and recall, from 0 to 100. Close to 100 means both are high. Low means the model misses true cases or raises many false alarms.
Accuracy = (TP + TN) / (TP + TN + FP + FN) × 100%: the percentage of all classifications that were correct.
Precision = TP / (TP + FP) × 100%: the proportion of predicted positives that were correct; "how sure can I be about a positive prediction?"
Recall (also called sensitivity) = TP / (TP + FN) × 100%: the proportion of actual positives that were found; "how confident can I be that no positives are missed?"
F1-score = 2 × (Precision × Recall) / (Precision + Recall): the harmonic mean of precision and recall, from 0 to 100. Close to 100 means both are high. Low means the model misses true cases or raises many false alarms.
Memory aid (supplementary): the Precision denominator is everything Predicted positive (TP + FP). The Recall denominator is everything that is Really positive (TP + FN).
Worked example 1: a spam filter
200 emails: TP = 45 (spam caught), FP = 5 (good mail flagged as spam), FN = 15 (spam missed), TN = 135 (good mail passed).
| Step | Calculation | Result |
|---|---|---|
| 1. Check the total | 45 + 5 + 15 + 135 | 200 ✓ |
| 2. Accuracy | (45 + 135) / 200 | 90.0% |
| 3. Precision | 45 / (45 + 5) = 45 / 50 | 90.0% |
| 4. Recall | 45 / (45 + 15) = 45 / 60 | 75.0% |
| 5. F1-score | 2 × (0.90 × 0.75) / (0.90 + 0.75) = 1.35 / 1.65 | 81.8 |
Interpretation: when the filter says "spam", it is right 9 times out of 10 (precision), but it lets a quarter of all spam through (recall). F1 (81.8) sits below the simple average (82.5) because the harmonic mean penalizes the weaker metric.
Worked example 2: why accuracy can mislead (imbalanced data)
1,000 card transactions, of which 10 are fraudulent (positive).
| Model | TP | FP | FN | TN | Accuracy | Precision | Recall | F1 |
|---|---|---|---|---|---|---|---|---|
| A: "everything is legitimate" | 0 | 0 | 10 | 990 | 99.0% | undefined (0/0) | 0% | n/a |
| B: a real fraud model | 8 | 40 | 2 | 950 | 95.8% | 16.7% | 80.0% | 27.6 |
Model A has the higher accuracy but catches no fraud at all. When the positive class is rare, accuracy is dominated by the true negatives. Use recall (missed fraud is costly) and precision (alerts that turn out false waste investigators' time), or F1 as a balance. This is why acceptance criteria name the metric that matters, e.g., "recall of 90% for a defect prediction system" (2.2.1). The interpretation is supplementary; the formulas are the syllabus's.
Reusable checklist for the K3 exam question
- Orient the matrix: find "predicted positive and actual positive" (TP), then label FP, FN and TN.
- Write the formula the question asks for. Don't calculate all four unless you need them.
- Plug in and simplify. Many questions ask only for the formula with numbers, e.g., (78/100) × 100.
- Sanity-check: every metric is between 0 and 100%. Precision and recall share the numerator TP. Accuracy uses all four cells.
- For F1, use the harmonic mean, not the average, and remember the result is on a 0–100 scale.
Hands-on exercise HO-3.3.2 (H2): Evaluate an ML model with selected metrics
Goal (syllabus): using the classification model from the previous exercise, calculate and display accuracy, precision, recall and F1-score, using the framework's library functions where applicable.
python
from sklearn.metrics import (confusion_matrix, accuracy_score,
precision_score, recall_score, f1_score)
y_pred = final.predict(X_test)
tn, fp, fn, tp = confusion_matrix(y_test, y_pred).ravel() # scikit-learn: rows = actual, cols = predicted
print(f"TP={tp} FP={fp} FN={fn} TN={tn}")
# By hand, with the syllabus formulas ...
print("accuracy ", (tp + tn) / (tp + tn + fp + fn))
print("precision", tp / (tp + fp))
print("recall ", tp / (tp + fn))
# ... and with the library, which should give the same numbers
print(accuracy_score(y_test, y_pred), precision_score(y_test, y_pred),
recall_score(y_test, y_pred), f1_score(y_test, y_pred))Hands-on exercise HO-3.3.3 (H2): Impact of different models and dataset combinations
Goal (syllabus): use different ML models and dataset combinations, and observe their effect on training (training time) and on the final behavior of the model (ML functional performance metrics).
python
import time
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.metrics import f1_score
models = {
"logistic regression": make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000)),
"decision tree": DecisionTreeClassifier(max_depth=4, random_state=42),
"random forest": RandomForestClassifier(n_estimators=200, random_state=42),
"SVM": make_pipeline(StandardScaler(), SVC()),
}
for share in [0.1, 0.5, 1.0]: # dataset combinations: 10%, 50%, 100% of the training data
n = max(20, int(len(X_train) * share))
for name, model in models.items():
start = time.perf_counter()
model.fit(X_train[:n], y_train[:n])
secs = time.perf_counter() - start
f1 = f1_score(y_val, model.predict(X_val))
print(f"{share:>4.0%} data | {name:<20} | train {secs*1000:6.1f} ms | validation F1 {f1:.3f}")Observe: which model gains most from more data? Which is slowest to train? Does the ranking of models change with the amount of data?
Unit Quiz
Unit Quiz: Chapter 3, page 2 (3.2–3.3)
Q1. Which list contains ONLY data preprocessing activities, as categorized in the syllabus?
ACleaning data, transforming and scaling data, augmenting data, sampling subsets
BGathering data from APIs, labeling data, identifying data types
CSelecting relevant features, extracting non-redundant features
DTraining, evaluating and tuning the model
Q2. A team replaces missing "age" values with the median age of all records. Which data preparation activity is this?
AFeature engineering
BData acquisition
CExploratory data analysis
DData preprocessing (cleaning: imputing missing values)
Q3. Which dataset is used to evaluate a model and then tune it?
AThe training dataset
BThe validation dataset
CThe test (holdout) dataset
DThe operational dataset
Q4. Data is scarce, so a team keeps a small hold-out test set and uses 5-fold cross-validation on the remaining data. How many times is a model trained during the cross-validation, and on how much of the pool each time?
A5 times, each time on 4/5 of the pool, validating on the remaining fold
BOnce, on the whole pool
C5 times, each time on 1/5 of the pool
D4 times, each time on the hold-out test set
Q5. A classifier produced: TP = 40, FP = 10, FN = 20, TN = 130. What is its PRECISION?
A66.7%
B85.0%
C80.0%
D20.0%
Q6. Same classifier: TP = 40, FP = 10, FN = 20, TN = 130. What is its RECALL?
A80.0%
B66.7%
C33.3%
D92.9%
Q7. Same classifier: precision = 80%, recall ≈ 66.7%. What is its F1-score (rounded)?
A73.3
B85.0
C66.7
D72.7
0 / 7 answered
Topic Drills: 3.2.1 · 3.2.3 · 3.3.1 · Next page: Neural Networks →