Skip to content

Topic Drill 7.1.1: ML Development Risks and Mitigations ​

About this drill

LO AI-7.1.1 (K2): Give examples of test approaches used for risk mitigation of ML development.
Lesson: Ch 7, section 7.1.1

This drill covers every point in section 7.1 / 7.1.1:

  • The chapter's focus: development tools, configuration choices and deployment mechanisms, not the model itself
  • Every row of the risk table: APIs, framework selection, unfairness, installation/build, evaluation implementation, performance, usability, library and algorithm defects, security, documentation, algorithm and hyperparameter selection, data allocation, evaluation approach, stochastic results, deployment defects, incompatibility, no improvement

Topic Drill: 7.1.1

Q1. A developer passes arguments to a PyTorch loss function in the wrong order, so training silently optimizes the wrong thing. Which mitigation does the 7.1.1 table map to "incorrect or unintended use of library or framework APIs"?

AUsability testing
BAPI testing
CData allocation review
DShadow testing

Q2. After installing a new version of the ML development framework, the team needs a quick test to confirm that its essential services are running. Which mitigation fits "defective framework installation or build"?

AML functional performance testing
BBack-to-back testing
CAlgorithm suitability review
DSmoke testing

Q3. The framework's built-in F1-score function gives values that do not match a hand calculation. Which mitigations does the table give for "defective implementation of the evaluation by the framework"?

AReviews of the framework evaluation code, and cross-checking the framework's evaluation results (e.g., against manual benchmarks)
BSmoke testing and usability testing
CShadow testing and canary testing
DData allocation review only

Q4. The ML development framework becomes very slow and sometimes unresponsive when processing large batches of data. Which mitigation applies?

AML functional performance testing
BFramework documentation review
CPerformance testing
DTesting for bias

Q5. Which TWO mitigations does the table list for "defect in a library used by the framework (e.g., defect in PyTorch)" and "defective algorithm implementation"?

Select 2 options.

AML functional performance testing
BUsability testing
CBack-to-back testing
DSmoke testing
EFramework documentation review

Q6. A project leader must decide between a gradient-boosting algorithm and a neural network for the project's goals. Which mitigation does the table give for "sub-optimal algorithm selection" (besides A/B testing)?

ASmoke testing
BAlgorithm suitability review
CSecurity testing
DModel conversion testing

Q7. A team suspects its learning rate and network structure are poorly chosen. Besides ML functional performance testing, which mitigation does the table list for "sub-optimal hyperparameter selection"?

AData allocation review
BUsability testing
CShadow testing
DA/B testing

Q8. A data scientist discovers that some records appear in both the training and the test datasets. Which mitigation does the table map to "defective allocation of data to training, validation, and testing datasets"?

AData allocation review
BData provenance testing
CRollback testing
DUsability testing

Q9. A single evaluation run shows a model meeting its target, but re-running the training gives slightly different results each time. Which mitigation addresses "incorrect interpretation of test results due to the stochastic nature of the learning process"?

ASmoke testing
BFramework suitability review
CML functional performance testing, using multiple runs and statistical analysis
DInstallability testing

Q10. After deployment to a new server cluster, the model fails because of an incompatible GPU driver version. Which mitigations does the table give for "deployed model is incompatible with the operational environment"?

AA/B testing and usability testing
BSmoke testing and MLS deployment testing (7.1.2)
CData allocation review and framework documentation review
DTesting for bias and security testing

Q11. Which TWO mitigations does the table list for a "deployment defect (e.g., from generating a modified version for a target platform)"?

Select 2 options.

AUsability testing
BSmoke testing
CData allocation review
DFramework documentation review
EA/B testing

Q12. A newly trained model is ready, but nobody knows whether it is actually better than the model currently in production. Which mitigation does the table map to "deployed model is no improvement over the current model"?

ASmoke testing
BFramework suitability review
CData allocation review
DShadow testing

Q13. Which mitigation does the table give for "poor user documentation for the framework"?

AFramework documentation review
BModel documentation review
CUsability testing
DAPI testing

0 / 13 answered

← 6.1.10 Back-to-Back Testing · Next drill: 7.1.2 Deployment Testing →

Unofficial study guide for the ISTQB® Certified Tester AI Testing (CT-AI) exam. Not affiliated with International Software Testing Qualifications Board (ISTQB®).