Appearance
Topic Drill 6.1.3: ML Functional Performance Testing of Probabilistic ML Systems
About this drill
LO AI-6.1.3 (K2): Explain how ML functional performance testing is carried out for probabilistic machine learning systems.
Lesson: Ch 6, section 6.1.3
This drill covers every point in section 6.1.3:
- What it measures; moving beyond pass/fail because of non-determinism
- Statistical acceptance criteria (metric, MoE, CL); minimum number of tests; fixing CL vs. fixing MoE
- The worked numbers (601 / 589 / ±1.1% / sequential testing ~170 / 299 for safety), understood, not calculated
- An independent, representative test dataset; a framework with statistical analysis; reporting with confidence
Topic Drill: 6.1.3
Q1. What is a PRECONDITION for ML functional performance testing of a probabilistic ML system, according to 6.1.3?
AA model with 100% accuracy on the training data
BAcceptance criteria defined in statistical terms: a performance metric, a margin of error and a confidence level
CA pseudo-oracle built by an independent team
DA complete set of metamorphic relations
Q2. The margin of error is kept FIXED while more tests are run. According to the syllabus, what happens?
AThe confidence level decreases
BThe measured accuracy must increase
CThe margin of error increases
DThe confidence level increases, so the result becomes more certain
Q3. A model was measured at 91% accuracy ±5% at a 90% confidence level. Many more tests are run, the observed accuracy stays at 91%, and the confidence level is kept at 90%. Which result is MOST likely now?
A91% ±3% at 90% confidence level
B91% ±7% at 90% confidence level
C91% ±5% at 85% confidence level
D95% ±5% at 90% confidence level
Q4. In the syllabus example, why is a sample of 601 test cases described as "conservative" for ±4% at the 95% confidence level?
ABecause the model is assumed to be 98% accurate
BBecause it includes a safety factor of 50%
CBecause it assumes the true accuracy could be anywhere between 0% and 100%, which gives the largest possible uncertainty but guarantees the ±4% MoE under all conditions
DBecause only 300 of the tests are executed
Q5. What is the purpose of sequential testing in this context?
ATo run the test cases one after another in a fixed order
BTo analyze results as they accumulate and stop early when there is sufficient evidence that supports (or rejects) the accuracy target
CTo test the model sequentially in each environment
DTo retrain the model after each test
Q6. For a safety-critical system, the requirement is "99% reliability with 95% confidence". What does the syllabus say this needs?
A601 test cases, of which at least 589 must pass
B170 test cases
C99 test cases, all of which must pass
D299 test cases, all of which must pass
Q7. Which test dataset does ML functional performance testing of a probabilistic model require?
AA large test dataset that is completely independent of the training and validation datasets and is a representative sample of the operational input domain
BThe validation dataset, reused for speed
CA small set of hand-picked edge cases only
DThe training dataset, to confirm what the model learned
Q8. Will candidates have to compute sample sizes or margins of error with statistical formulas in the CT-AI exam?
AYes, for every K2 question in Chapter 6
BYes, but only for safety-critical examples
CNo: the syllabus states the formulas and numerical calculations are for illustration and understanding only
DNo, because statistics is not part of the syllabus
Q9. Which TWO statements about ML functional performance testing of probabilistic ML systems are CORRECT?
Select 2 options.
AThe final result is reported as a single pass/fail verdict
BRunning more tests increases the margin of error
CResults are reported with statistical confidence, e.g., "accuracy of 94% ±4% at the 95% CL"
DTest cases are executed with the ML model within an ML development framework capable of statistical analysis
EThe validation dataset can be reused as the test dataset
0 / 9 answered
← 6.1.2 Model Documentation · Next drill: 6.1.4 Adversarial Testing →