Appearance
Topic Drill 6.1.9: A/B Testing
About this drill
LO AI-6.1.9 (K2): Explain how A/B testing is used in the context of machine learning systems.
Lesson: Ch 6, section 6.1.9
This drill covers every point in section 6.1.9:
- Two variants, the same inputs, statistical comparison over multiple runs; the marketing and e-commerce examples
- Partial test oracle; no test case generation; operational inputs
- Testing updates against agreed acceptance criteria; complex systems (routing); self-learning systems (accept or revert)
- t-test, z-test, chi-squared, Mann-Whitney U
Topic Drill: 6.1.9
Q1. Which description of A/B testing matches the syllabus?
AComparing the system under test with an independent pseudo-oracle to detect defects
BComparing the responses of two variants of a program to the same inputs to determine which is better, typically statistically over multiple test runs
CDeriving follow-up test cases from a source test case
DRunning a new model in parallel without affecting users
Q2. How does A/B testing help with the test oracle problem?
AIt generates the expected results automatically
BIt uses domain experts as the oracle
CIt replaces the need for acceptance criteria
DIt typically uses the existing system as a partial test oracle
Q3. What does the syllabus say A/B testing does NOT do?
AIt does not generate test cases and gives no guidance on how the tests should be designed
BIt does not use statistics
CIt does not work in production
DIt does not compare two variants
Q4. A city updates its AI transport-routing system. How does the syllabus suggest testing whether the update is more effective?
AWith back-to-back testing against a pseudo-oracle
BWith metamorphic relations on route length
CWith A/B testing, e.g., comparing average commute times for the two variants on consecutive weeks
DWith static drift testing
Q5. How can A/B testing be applied to a SELF-LEARNING system?
AIt cannot be applied to self-learning systems
BWhen the system makes a change, automated tests are run and the resulting characteristics are compared with those before the change; if improved, the change is accepted, otherwise the system reverts
CBy freezing the system permanently
DBy comparing it with a different vendor's system
Q6. Which statistical techniques does the syllabus name as the most popular for A/B testing?
AKolmogorov-Smirnov only
BCohen's Kappa and percentage agreement
CMargin of error and confidence level
Dt-test, z-test, chi-squared test and Mann-Whitney U test
Q7. What is needed before A/B testing can be used to test updates to an AI-based system?
AAgreed acceptance criteria, such as ML functional performance metrics
BA pseudo-oracle built by an independent team
CA complete formal specification of expected outputs
DNeuron coverage of 100%
Q8. Which TWO statements about A/B testing are CORRECT?
Select 2 options.
AA/B testing detects defects by comparing the SUT with an independent pseudo-oracle
BA/B testing analyzes the internal structure of the algorithm
CMany e-commerce and web-based companies use A/B testing in production, diverting different consumers to different functionality
DA/B testing generates diverse test cases covering all inputs
EA/B testing can be used for a simple classifier as well as for far more complex systems
0 / 8 answered
← 6.1.8 Overfitting and Underfitting · Next drill: 6.1.10 Back-to-Back Testing →