Skip to content

Topic Drill 4.1.2: Rationale for a Statistical Approach ​

About this drill

LO AI-4.1.2 (K2): Explain why a statistical approach is often needed when testing AI-based systems.
Lesson: Ch 4, section 4.1.2

This drill covers every point in section 4.1.2:

  • Non-determinism: the same inputs give different outputs; one test proves nothing; statistically significant suites
  • Distributional performance evaluation: training vs. operational distributions
  • Handling uncertainty and bias: confident but wrong; confidence intervals, hypothesis tests, error analysis
  • Regulatory and safety context: thresholds with high confidence; claims across broad scenarios; the pointer to 6.1.3

Topic Drill: 4.1.2

Q1. A tester runs the same 20 prompts through an AI document classifier twice and gets 2 different classifications in the second run. What does this imply for the test approach?

AThe classifier is defective and must be rejected
BBecause of non-determinism, the test suite must be sufficiently large to give statistically significant results
CThe second run should be ignored
DOnly one run of each test should ever be made

Q2. A pedestrian-detection model was trained mostly on daytime images, but it will operate day and night. Which reason for statistical testing does this BEST illustrate?

ARegulatory and safety context
BNon-determinism
CHandling uncertainty and bias
DDistributional performance evaluation

Q3. Which statistical techniques does the syllabus name for quantifying and analyzing accuracy, fairness and AI robustness?

AConfidence intervals, hypothesis testing and error analysis
BBoundary value analysis and equivalence partitioning
CNeuron coverage and k-multisection neuron coverage
DCode reviews and static analysis

Q4. A rail-signalling regulator asks for evidence that an AI component is reliable across a broad range of scenarios, not just a few demonstrations. Which reason for a statistical approach applies MOST directly?

ANon-determinism
BDistributional performance evaluation
CRegulatory and safety context
DHandling uncertainty and bias

Q5. Which of the following is NOT one of the reasons the syllabus gives for a statistical approach to testing AI-based systems?

ANon-determinism
BDistributional performance evaluation
CRegulatory and safety context
DAI-based systems are too large for test automation

Q6. An AI triage model gives high-confidence answers that are nevertheless wrong for some patient groups. Under which reason for statistical testing does the syllabus discuss "confident but incorrect predictions"?

ADistributional performance evaluation
BHandling uncertainty and bias
CNon-determinism
DRegulatory and safety context

Q7. Where does the syllabus describe a practical statistical approach (margin of error, confidence level, number of tests) for testing probabilistic ML systems?

ASection 6.1.3, ML functional performance testing of probabilistic ML systems
BSection 3.4.3, coverage measures for neural networks
CSection 5.1.5, dataset constraint testing
DSection 7.1.2, ML system deployment testing

Q8. Which TWO statements about statistical testing of AI-based systems are CORRECT?

Select 2 options.

AA single failed test proves that the model fails its accuracy requirement
BStatistical testing removes the need for a separate test dataset
CThe test suite must be sufficiently large to provide statistically significant results
DOnly adaptive systems need statistical testing
EStatistical methods support claims about reliability across a broad range of scenarios, rather than only for specific examples

0 / 8 answered

← 4.1.1 Locked and Adaptive · Next drill: 4.1.3 Test Oracles →

Unofficial study guide for the ISTQB® Certified Tester AI Testing (CT-AI) exam. Not affiliated with International Software Testing Qualifications Board (ISTQB®).