Appearance
Topic Drill 6.1.10: Back-to-Back Testing
About this drill
LO AI-6.1.10 (K2): Explain how back-to-back testing is used in the context of machine learning systems.
Lesson: Ch 6, section 6.1.10
This drill covers every point in section 6.1.10:
- Pseudo-oracle: existing or purpose-built (at a cost); independence and shared components (open source); independent teams, frameworks, algorithms, settings; conventional software as a pseudo-oracle
- Functional back-to-back testing: the pseudo-oracle needn't meet non-functional requirements
- Only inputs needed (regression suites, generated from training data); migration and edge cases
- Back-to-back vs. A/B testing
Topic Drill: 6.1.10
Q1. Where can the pseudo-oracle for back-to-back testing come from, according to the syllabus?
AOnly from the training data
BOnly from end users
CAn existing system, or one developed specifically for testing (which comes at a cost)
DThe validation dataset
Q2. Why should the pseudo-oracle and the system under test ideally NOT share common software components?
ABecause both systems might contain the same defect, so their outputs could match even when both are wrong
BBecause shared components make the tests run slower
CBecause licensing forbids it
DBecause shared components make the outputs always differ
Q3. Both the system under test and its pseudo-oracle use the same popular open-source preprocessing library. Why is this a concern?
AOpen-source libraries cannot be used in ML systems
BIt makes the pseudo-oracle too expensive
CIt prevents automation of the tests
DA defect in the shared library could affect both systems identically, hiding it from back-to-back testing; the syllabus notes this is a particular problem given the widespread reuse of open-source AI components
Q4. Can conventional (non-AI) software serve as the pseudo-oracle for an ML model?
ANo, the pseudo-oracle must always be another ML model
BYes, if it solves the same problem
COnly for unsupervised models
DOnly if it uses the same framework
Q5. For FUNCTIONAL back-to-back testing, the team builds a pseudo-oracle that is ten times slower than the production model. Is this acceptable?
ANo, the pseudo-oracle must meet all non-functional requirements of the SUT
BNo, slow pseudo-oracles give wrong functional results
CYes: for functional back-to-back testing the pseudo-oracle only needs to match functional behavior, not the non-functional requirements, which can make it cheaper to build
DOnly if the SUT is also slowed down
Q6. What does a team need to prepare for back-to-back testing, and where can it come from?
AOnly test inputs, no expected results; they can come from existing test cases such as regression suites, or be generated automatically from training data
BTest inputs and exact expected results for every case
COnly expected results
DMetamorphic relations
Q7. In which situation does the syllabus say back-to-back testing provides SIGNIFICANT value?
AWhen choosing between two marketing campaigns
BWhen labelling training data
CWhen measuring statistical significance of a fairness metric
DWhen migrating an ML system to a new environment (e.g., from development to production) and comparing test results across environments
Q8. Which other technique in the syllabus is described as using "a form of back-to-back testing"?
ADisparate impact analysis
BMultiple annotation in label correctness testing
CStatic drift testing
DRed teaming
Q9. Which TWO statements CORRECTLY distinguish back-to-back testing from A/B testing?
Select 2 options.
ABack-to-back testing requires exact expected results
BBack-to-back testing is used to detect defects
CBack-to-back testing should use a fine-tuned copy of the SUT as pseudo-oracle
DA/B testing compares two variants of the same MLS using ML functional performance metrics and statistical techniques
EBack-to-back testing can only be used in production
0 / 9 answered
← 6.1.9 A/B Testing · Next drill: 7.1.1 ML Development Risks →