Appearance
Topic Drill 4.1.3: Test Oracles for AI-Based Systems
About this drill
LO AI-4.1.3 (K2): Explain the challenges and solutions relating to test oracles for AI-based systems.
Lesson: Ch 4, section 4.1.3
This drill covers every point in section 4.1 / 4.1.3:
- The five causes: probabilistic and non-deterministic (thresholds, tolerances), exploratory development and incomplete specs, complexity, subjectivity, self-learning
- The five solutions: output boundaries, environmental boundaries, expert consultation (and its limits), specialized testing (A/B, back-to-back, metamorphic), proxy oracles
Topic Drill: 4.1.3
Q1. An AI system labels every object in each frame of a 4K video stream. It is impractical for testers to manually check the expected output for thousands of frames. Which cause of the test oracle problem is this?
ASubjectivity of behavior
BSelf-learning systems
CComplexity of tasks
DExploratory development and incomplete specifications
Q2. A start-up builds an AI feature through quick experiments; the requirements keep changing and parts are not written down at all. Which cause of the test oracle problem is this?
AExploratory development and incomplete specifications
BProbabilistic and non-deterministic nature
CComplexity of tasks
DSubjectivity of behavior
Q3. A music recommender keeps updating its internal model from listening behavior. Expected results the testers prepared last month are already invalid. Which cause is this?
AComplexity of tasks
BSubjectivity of behavior
CExploratory development and incomplete specifications
DSelf-learning systems
Q4. A supervised classifier has a single correct target class for each input, but its outputs are probabilistic. What does the syllabus say defining a strict pass/fail oracle often requires?
AAsking end users to vote on each output
BSetting thresholds or tolerance ranges
CRetraining the model before every test
DRemoving the probabilistic layer from the model
Q5. Testers agree that an autonomous lawn mower must stop within 30 cm of any obstacle. Which test oracle solution is this?
ADefining output boundaries
BDefining environmental boundaries
CProxy oracle
DExpert consultation
Q6. For a camera-based quality inspection system, testers fix the lighting level, the room temperature and the network latency in the test lab. Which solution is this, and why?
AProxy oracles, to judge outputs with a second model
BExpert consultation, to get expected results
CDefining environmental boundaries, so that outputs are predictable and repeatable
DSpecialized testing, to avoid expected results
Q7. What limitation does the syllabus mention for expert consultation as a test oracle solution?
AExperts are not allowed to review AI outputs
BExperts can only judge numerical outputs
CExpert consultation is too fast to be reliable
DThe experts' opinions might differ or be fallible
Q8. A team has a small labeled dataset and uses it to train a separate model, which then predicts expected outputs for a large set of unlabeled test inputs. Which solution is this?
AMetamorphic testing
BProxy oracle
CDefining output boundaries
DA/B testing
Q9. Which TWO of the following are named in the syllabus as "specialized testing" approaches that address the test oracle problem?
Select 2 options.
AA/B testing
BSmoke testing
CMetamorphic testing
DInstallability testing
EUsability testing
Q10. Which of the following is NOT one of the solutions to the test oracle problem listed in the syllabus?
ADefining output boundaries
BProxy oracles
CRemoving all non-determinism from the AI model before testing
DExpert consultation
0 / 10 answered
← 4.1.2 Statistical Approach · Next drill: 4.2.1 Testing GenAI →