Skip to content

Topic Drill 5.1.6: Label Correctness Testing ​

About this drill

LO AI-5.1.6 (K2): Explain label correctness testing.
Lesson: Ch 5, section 5.1.6

This drill covers every point in section 5.1.6:

  • Why label correctness matters in supervised learning
  • Expert review; multiple annotation (back-to-back comparison, IAA, Cohen's Kappa, low IAA causes); risk-based prioritization
  • Data distribution analysis; automated rule-based tests (bounding boxes); model loss analysis; model confidence score analysis
  • Combining approaches over time

Topic Drill: 5.1.6

Q1. Two radiologists check a random 5% of the labelled X-ray images against the labelling guidelines. Which label correctness approach is this?

AMultiple annotation
BExpert review
CModel loss analysis
DData distribution analysis

Q2. In multiple annotation, how are the labels from different annotators compared, according to the syllabus?

AWith A/B testing
BWith metamorphic testing
CWith neuron coverage
DUsing a form of back-to-back testing

Q3. Which metrics does the syllabus name for measuring inter-annotator agreement (IAA)?

ACohen's Kappa or simple percentage agreement
BPrecision and recall
CChi-squared and Kolmogorov-Smirnov
DMargin of error and confidence level

Q4. Budget allows only 2,000 of 200,000 images to be multiply annotated. Using risk-based prioritization, which images should be chosen FIRST?

AA random sample of 2,000
BThe 2,000 most recently added images
CAmbiguous images, such as those near a boundary between classes, and images most likely to affect the application's success or safety
DThe images with the highest model confidence

Q5. A public dataset for the same task has about 5% "defective" labels, but the team's own dataset has 30%. Which approach revealed this anomaly?

AAutomated rule-based tests
BData distribution analysis
CExpert review
DModel confidence score analysis

Q6. A script flags every annotation whose bounding box extends beyond the edges of the image. Which approach is this?

AAutomated rule-based tests
BMultiple annotation
CModel loss analysis
DExpert review

Q7. After training, the model gives very low prediction confidence for a group of data points. According to the syllabus, what may this indicate?

AThat those points are definitely labelled correctly
BThat the model is overfitted
CThat the data pipeline is too slow
DThat those points may be mislabeled, ambiguous, or outside the model's training data distribution

Q8. What sequence of label correctness approaches does the syllabus give as an example of an effective combination?

AModel-based approaches first, then expert reviews, then multiple annotation
BOnly multiple annotation, throughout the project
CExpert reviews early to set clear labelling guidelines, then IAA scores from multiple annotation to refine the labelling process, then model-based approaches as the model develops
DAutomated rule-based tests only, because they scale

Q9. Which TWO statements about label correctness testing are CORRECT?

Select 2 options.

ALabel correctness only matters for unsupervised learning
BInaccurate or inconsistent labels directly undermine the performance and generalization of ML models
CA high inter-annotator agreement score indicates defects in the labelling guidelines
DMultiple annotation is a fully automated technique
EData points with high loss during training can indicate mislabeling

0 / 9 answered

← 5.1.5 Dataset Constraint Testing · Next drill: 6.1.1 Model Risks →

Unofficial study guide for the ISTQB® Certified Tester AI Testing (CT-AI) exam. Not affiliated with International Software Testing Qualifications Board (ISTQB®).