Appearance
Ch 5: Representativeness, Constraints and Labels
Official Reference
ISTQB® CT-AI Syllabus v2.0, Chapter 5 (180 minutes), sections 5.1.4–5.1.6, Learning Objectives (LOs) AI-5.1.4 (K2), AI-5.1.5 (K3), AI-5.1.6 (K2), and Hands-on Objective HO-5.1.7 (H2). Official syllabus and documents
Exam weight: three questions from this page. AI-5.1.5 is a 2-point K3 question: given a dataset and its business rules, pick the constraint type that validates a specific rule. Learn the three categories and their nine named constraints.
5.1.4 Testing for Data Representativeness (K2)
Data representativeness testing: A test type to determine if the datasets used for training an AI model are fair representations of the data expected to be encountered by the operational model. (ISTQB® Glossary)
It determines how closely the training, validation and test datasets match the real-world data the model will meet in operation. It addresses skewed datasets, missing data, disproportionate feature distributions, inadequate coverage of operational scenarios, and imbalanced class representation (see 5.1.1).
mermaid
flowchart TB
S1["1. Define the target population<br/>use cases · users · environments ·<br/>expected distributions + edge cases<br/>→ reference dataset (stratified)"] --> S2["2. Analyze data characteristics<br/>EDA of training/test data AND reference data ·<br/>feature relationships · anomalies, gaps"]
S2 --> S3["3. Apply statistical assessment<br/>Chi-squared, Kolmogorov-Smirnov ·<br/>class imbalance · typical + edge cases"]
S3 --> W["Before training: prevents models built on<br/>unrepresentative data<br/>After deployment: monitor operational<br/>inputs for data drift (6.1.7)"]The reference dataset is the yardstick: everything in steps 2 and 3 compares the training and test data against it.
| Step | Activities (syllabus) |
|---|---|
| 1. Define the target population | Understand the intended use cases and operational context. Analyze the characteristics of end users and operational environments. Identify the expected operational data distributions and critical edge cases by consulting domain experts, analyzing data from existing or similar systems, and using benchmark datasets from trusted sources (e.g., NIST, industry databases). Apply stratified sampling to the representative reference data to create a baseline covering all relevant subgroups |
| 2. Analyze data characteristics | Apply EDA to the training/test datasets and to the reference dataset; visualize distributions (histograms, scatter plots, other graphs). Examine feature relationships, especially correlations, to find patterns that should be preserved. Identify anomalies, gaps or unusual concentrations |
| 3. Apply statistical assessment techniques | Formal tests such as Chi-squared and Kolmogorov-Smirnov to compare distributions. Check for data imbalances, especially in classification. Verify adequate coverage of typical scenarios and edge/boundary cases |
Stratified sampling: A technique to confirm if a sample proportionally represents different sub-populations within the overall population. (ISTQB® Glossary)
When: before model training, to avoid building models on unrepresentative data. After deployment, monitor the properties of operational input data continuously to detect data drift (6.1.7).
Exam Distinction
The reference dataset is a statistical baseline representing expected operational data. It is used to compare the training data's distributions, correlations and coverage. It is not a universal benchmark, not the dataset for final model testing, and stratified sampling is applied to it, not directly to the training data.
5.1.5 Dataset Constraint Testing (K3)
Dataset constraint testing: A type of static testing to determine if logical or specified constraints are followed on a dataset. (ISTQB® Glossary)
Dataset constraint testing verifies whether the data adheres to predefined rules or constraints, to confirm its integrity and consistency. Just as a database schema defines structure, types and relationships, a set of constraints acts as a logical model of the dataset that correct data should satisfy.
| Category | Constraint | Tests that… (syllabus) |
|---|---|---|
| Single-value: one attribute value in a single instance | Missing | there are no missing values or missing attributes |
| Range | a value is within a given range | |
| Type | the value matches the specified type (e.g., an integer attribute holds no string or real number) | |
| Multi-value: typically one attribute across several instances | Sum | the sum of all values equals, exceeds or does not exceed a value (e.g., total Formula 1 race points cannot exceed 102 and must exceed 50.5) |
| Count | the count of non-null values for attributes or instances equals, exceeds or does not exceed a value | |
| Duplicate | identical or near-identical values or instances stay within a limit (often zero) | |
| Useful | an attribute contains some repeated values. If every entry is unique (like an ID or timestamp), it typically gives no useful patterns for the model to learn | |
| Outlier | identifies values that could be statistical outliers | |
| Comparison: a special multi-value form comparing different values | Greater Than | one attribute's value is greater than a second attribute's (e.g., lines of code > lines of code with defects) |
| Correlate | values of one attribute correlate with values of a second attribute (e.g., students ≥ 1.33 standard deviations above the mean mark also have grade "A") |
Manual or automated? Possible manually, but the scale and typically large datasets mean it is automated as part of the data pipeline. The tool can report to data scientists (training data anomalies) or to operations staff (operational data problems).
Worked example: applying dataset constraint testing (K3)
Dataset (illustrative): hospital admissions used to train a length-of-stay predictor. Attributes: patient_id (integer), age (integer, 0–120), ward (one of CARD, ORTH, NEUR, GERI), admitted (date), discharged (date), stay_days (integer, = discharged − admitted).
| # | patient_id | age | ward | admitted | discharged | stay_days |
|---|---|---|---|---|---|---|
| 1 | 1001 | 34 | CARD | 2026-03-01 | 2026-03-04 | 3 |
| 2 | 1002 | 131 | ORTH | 2026-03-01 | 2026-03-02 | 1 |
| 3 | 1003 | 58 | (empty) | 2026-03-02 | 2026-03-07 | 5 |
| 4 | 1004 | 47 | NEUR | 2026-03-03 | 2026-03-02 | −1 |
| 5 | 1001 | 34 | CARD | 2026-03-01 | 2026-03-04 | 3 |
| 6 | 1006 | "forty" | CARD | 2026-03-04 | 2026-03-06 | 2 |
| 7 | 1007 | 66 | ORTH | 2026-03-04 | 2026-03-09 | 9 |
| 8 | 1008 | 72 | GERI | 2026-03-05 | 2027-09-01 | 545 |
Step 1: turn each business rule into a named constraint, then step 2: run it.
| Business rule | Constraint (category → type) | Rows failing |
|---|---|---|
| Age must be between 0 and 120 | Single-value → range | 2 |
| Every admission has a ward | Single-value → missing | 3 |
| Age is an integer | Single-value → type | 6 |
| No record appears twice | Multi-value → duplicate (limit 0) | 1 & 5 |
| ≥ 99% of rows have a ward | Multi-value → count | fails (7 of 8 = 87.5%) |
| Patients present on any day in a ward ≤ its 30 beds | Multi-value → sum | (check per ward and day) |
patient_id should not be used as a feature | Multi-value → useful | column flagged: all values unique per patient |
| Unusual stays must be reviewed | Multi-value → outlier | 8 (545 days) |
| Discharge after admission | Comparison → greater than | 4 |
stay_days = discharged − admitted | Comparison → correlate | 7 (9 ≠ 5). Row 4 passes this check because −1 matches its wrong dates, so you need the greater-than constraint as well |
Step 3: report to the data scientists (training data). Row 8 may be a genuine long stay, so an outlier is a flag for review, not automatically a defect.
Reusable decision guide for the K3 question
- Does the rule concern one attribute in one record? → single-value: missing / range / type.
- Does it concern one attribute across many records? → multi-value: sum / count / duplicate / useful / outlier.
- Does it relate two different attributes? → comparison: greater than for an ordering; correlate for a functional or statistical relationship (e.g., monthly payment = loan / 324, stay = discharge − admission).
Common trap: a calculated field ("A = B / 324") looks like a range check, but range only bounds one attribute. It cannot check the relationship between two attributes.
5.1.6 Label Correctness Testing (K2)
Label correctness testing: A test type to determine the accuracy and consistency of labels assigned to examples in a dataset.
Multiple annotation: An approach in which labelled data points from multiple annotators are compared.
Inter-annotator agreement: The degree of consensus or similarity among the annotations made by different annotators on the same data.
Source: ISTQB® Glossary
Multiple annotation: An approach in which labelled data points from multiple annotators are compared.
Inter-annotator agreement: The degree of consensus or similarity among the annotations made by different annotators on the same data.
Source: ISTQB® Glossary
Label correctness is essential in supervised learning. Inaccurate or inconsistent labels directly undermine performance and generalization. There are seven common approaches:
| Approach | How it works (syllabus) |
|---|---|
| Expert review | Domain experts or trained annotators manually review a sample of labelled data using domain knowledge and guidelines |
| Multiple annotation | Data points are independently labelled by several annotators and compared using a form of back-to-back testing (6.1.10). Disagreements highlight defects. Inter-annotator agreement (IAA) is measured with Cohen's Kappa or simple percentage agreement. Low IAA can indicate defects in labelling guidelines, ambiguous data, or poor annotation |
| Risk-based prioritization for review and annotation | Focus expert reviews and multiple annotation on samples that are likely to be mislabeled (ambiguous, near a class boundary) and on data that most affects success or safety |
| Data distribution analysis | When comparable datasets exist, compare label distributions to reveal anomalies and possibly incorrect labels |
| Automated rule-based tests | Check labels against predefined rules, e.g., bounding boxes must not overlap or extend beyond the image |
| Model loss analysis | Data points with high loss during training (the model struggles to learn the assigned label) may be mislabeled |
| Model confidence score analysis | Data points with low prediction confidence may be mislabeled, ambiguous, or outside the training distribution |
A combination is often most effective: expert reviews early to set clear labelling guidelines, IAA scores from multiple annotation to refine the labelling process, then model-based approaches to find remaining label defects as the model develops.
mermaid
flowchart LR
E["Early<br/>expert reviews<br/>→ clear labelling guidelines"] --> M["During labelling<br/>multiple annotation + IAA<br/>→ refine the process"]
M --> B["As the model develops<br/>model loss and confidence<br/>score analysis → remaining defects"]Human-centered approaches come first to fix the process; model-based approaches come later, when there is a model whose loss and confidence can point at suspicious labels.
Exam Distinction
Multiple annotation compares annotators' labels with each other. Data distribution analysis compares label distributions across datasets. Model loss analysis flags high-loss points. Confidence score analysis flags low-confidence points. Questions often swap these descriptions.
Hands-on exercise HO-5.1.7 (H2): Input data testing
Goal (syllabus): for a given dataset (e.g., structured, tabular data), perform input data testing for things like missing data, duplicate data and outliers.
Setup: pip install pandas seaborn. This script turns dataset constraints into automated checks on the Titanic dataset:
python
import seaborn as sns
import pandas as pd
df = sns.load_dataset("titanic")
def report(name, failed_rows):
n = len(failed_rows)
print(f"{'FAIL' if n else 'pass'} {name:<55} {n} row(s)")
# --- Single-value constraints -------------------------------------------
report("missing: age must be present", df[df["age"].isna()])
report("missing: embarked must be present", df[df["embarked"].isna()])
report("range: age within 0-100", df[(df["age"] < 0) | (df["age"] > 100)])
report("range: fare >= 0", df[df["fare"] < 0])
report("type: pclass is an integer 1, 2 or 3", df[~df["pclass"].isin([1, 2, 3])])
# --- Multi-value constraints --------------------------------------------
dups = df[df.duplicated(keep=False)]
report("duplicate: no identical rows allowed", dups)
non_null_age = df["age"].notna().sum()
print(f"{'FAIL' if non_null_age < 0.9 * len(df) else 'pass'} count: >= 90% of rows have an age{'':<24} {non_null_age}/{len(df)}")
q1, q3 = df["fare"].quantile([0.25, 0.75])
iqr = q3 - q1
report("outlier: fare outside Q1-1.5*IQR .. Q3+1.5*IQR", df[(df["fare"] < q1 - 1.5 * iqr) | (df["fare"] > q3 + 1.5 * iqr)])
for col in ["sex", "embarked", "deck"]:
print(f"info useful: '{col}' has {df[col].nunique()} distinct values in {len(df)} rows")
# --- Comparison constraints ---------------------------------------------
report("correlate: 'alive' must agree with 'survived'", df[(df["alive"] == "yes") != (df["survived"] == 1)])
report("correlate: 'class' must agree with 'pclass'",
df[df["class"].astype(str).map({"First": 1, "Second": 2, "Third": 3}) != df["pclass"]])Expected findings (one run): 177 missing ages, 2 missing ports, 160 rows involved in duplicates, about 116 fare outliers (IQR rule), and a count constraint that fails (714/891 ages). Discuss: which failures are data defects (duplicates?), which need domain review (fares of 500 are real first-class tickets), and which you would hand to data preparation (imputing ages).
Unit Quiz
Unit Quiz: Chapter 5, page 2 (5.1.4–5.1.6)
Q1. In data representativeness testing, what is the purpose of applying stratified sampling to the representative reference data?
ATo shrink the training data so training is faster
BTo create a baseline that covers all relevant subgroups in the target domain
CTo generate synthetic data for missing classes
DTo label the reference data automatically
Q2. When should data representativeness testing be performed, according to the syllabus?
ABefore model training, with continuous monitoring of operational input data after deployment to detect data drift
BOnly after the model has been deployed
COnly during acceptance testing
DOnly when the model's accuracy drops
Q3. A dataset rule says: "discharge_date must be later than admission_date". Which type of dataset constraint is this?
ASingle-value range constraint
BMulti-value sum constraint
CComparison constraint (greater than)
DMulti-value duplicate constraint
Q4. Every value in the "transaction_id" column is unique. Which dataset constraint would flag this column as unlikely to help the ML model learn?
ADuplicate constraint
BCount constraint
CRange constraint
DUseful constraint
Q5. Why does the syllabus say dataset constraint testing is usually automated as part of the data pipeline?
ABecause manual constraint testing is forbidden by ISO/IEC 25059
BBecause the scale of the activity and the typically large size of datasets make manual testing impractical; the tool can then report training-data anomalies to data scientists and operational data problems to operations staff
CBecause constraints can only be expressed in code
DBecause automation removes the need to define constraints
Q6. Three annotators independently label the same 500 medical images, and their labels are compared with Cohen's Kappa. The score is low. What does this MOST likely indicate?
AThe model is overfitted
BData drift has occurred
CDefects in the labelling guidelines, ambiguous data, or poor annotation
DThe images are duplicated
Q7. During training, a few data points keep a very high loss: the model cannot learn their assigned labels. Which label correctness approach uses this signal?
AModel loss analysis
BData distribution analysis
CExpert review
DAutomated rule-based tests
0 / 7 answered
Topic Drills: 5.1.4 · 5.1.5 · 5.1.6 · Next: Chapter 6 →