Skip to content

Topic Drill 7.1.2: ML System Deployment Testing ​

About this drill

LO AI-7.1.2 (K2): Explain the various forms of ML system deployment testing.
Lesson: Ch 7, section 7.1.2

This drill covers every point in section 7.1.2:

  • Installability (install, configure, uninstall; GPU drivers; framework compatibility; scripts)
  • Rollback (model or complete system; must be tested before deployment)
  • Canary (small % of traffic, real-time metrics) vs. shadow (parallel, same requests, no user impact; regressions and drift)
  • Model conversion (accuracy, behavior, efficiency after format conversion); cross-device; API testing

Topic Drill: 7.1.2

Q1. An MLS installs correctly on new servers but fails on older servers with an outdated GPU driver, and its uninstall script leaves files behind. Which deployment test type covers these issues?

ACanary testing
BShadow testing
CInstallability testing
DModel conversion testing

Q2. When must rollback testing be performed?

ABefore deployment, to confirm rollback readiness
BOnly after a deployment has failed
COnly during canary testing
DOnce a year, as part of maintenance

Q3. What can rollback testing cover, according to the syllabus?

AOnly the user interface
BOnly the training data
COnly the hardware
DOnly the model, or the complete system (e.g., including the data pipeline)

Q4. An updated model is released to 5% of users while latency, accuracy and error rates are monitored in real time, so that regressions can be caught before a full rollout. What is this?

AShadow testing
BCanary testing
CA/B testing of two marketing offers
DCross-device testing

Q5. Which defects does the syllabus say SHADOW testing can uncover before full deployment?

AInstallation script failures
BInconsistent annotator labels
CDefects such as performance regressions and data drift, by comparing the new and old models on live data
DFramework documentation gaps

Q6. A speech model trained in full precision is quantized and converted for an on-device runtime. Testers check its accuracy, behavior consistency, inference speed and memory use afterwards. Which test type is this?

AModel conversion testing
BRollback testing
CAPI testing
DInstallability testing

Q7. An MLS will run on smartphones, edge gateways and cloud servers. Which test type verifies that it performs correctly across all these deployment targets?

ACanary testing
BShadow testing
CRollback testing
DCross-device testing

Q8. What does API testing verify as part of MLS deployment testing?

AThat the model's weights are correct
BThat the MLS exposes well-defined, standards-compliant interfaces, handling inputs, outputs and error messages correctly and integrating with external systems such as data feeds, clients and the pipeline
CThat users like the predictions
DThat the model can be uninstalled

Q9. Which TWO statements CORRECTLY describe shadow and canary testing?

Select 2 options.

ACanary testing compares models offline, without live data
BShadow testing does not use live data
CShadow testing routes the same live requests to both models without affecting the responses users receive
DCanary testing is used mainly for component integration testing
ECanary testing deliberately serves a small subset of real users with the new model

Q10. A bank wants to compare a new credit model with the current one on real applications, with ZERO risk that any customer receives a decision from the new model. Which test type fits BEST?

AShadow testing
BCanary testing
CA/B testing that splits customers between the models
DRollback testing

0 / 10 answered

← 7.1.1 ML Development Risks · Back to Chapter 1 drills

Unofficial study guide for the ISTQB® Certified Tester AI Testing (CT-AI) exam. Not affiliated with International Software Testing Qualifications Board (ISTQB®).