Appearance
Topic Drill 5.1.2: Testing for Bias
About this drill
LO AI-5.1.2 (K2): Explain how to test for bias.
Lesson: Ch 5, section 5.1.2
This drill covers every point in section 5.1.2:
- What bias in an MLS is; sensitive attributes; why it matters (unlawful, discriminatory)
- Sources: data bias (unrepresentativeness, historical skews, poisoning) and algorithmic bias (credit-score threshold)
- The seven approaches: workflow reviews, dataset documentation reviews, static analysis, EDA, dynamic testing, label correctness testing, disparate impact analysis (four steps, attribute combinations, realistic counterfactuals)
Topic Drill: 5.1.2
Q1. How does the syllabus describe bias in an ML system?
AAny error in the model's predictions
BThe bias value added inside each neuron
CNon-random, unfair differences in treatment based on sensitive attributes such as gender, age or race, which is often unlawful and makes the system discriminatory
DRandom noise in the training data
Q2. A recruitment model is trained on ten years of past hiring decisions, during which very few women were hired for engineering roles. What is the PRIMARY source of bias here?
AData bias: a historical skew in the training data
BAlgorithmic bias: a defect in the algorithm
CNeuron bias
DSampling for training speed
Q3. A tester examines how a dataset was collected, who annotated it, and which populations it represents, to find potential sources of unfairness. Which bias test approach is this?
AStatic analysis of model code
BDynamic testing with a representative dataset
CDisparate impact analysis
DReview of dataset documentation
Q4. A tool scans the data preparation scripts and finds that applicants' postcodes are used in a way that mishandles a sensitive attribute. Which bias test approach is this?
AExploratory data analysis
BStatic analysis of data preparation programs and model implementation code
CLabel correctness testing
DMultiple annotation
Q5. Visualizing and clustering the training data reveals that one age group forms an unusually small, isolated cluster with very different outcome labels. Which bias test approach found this?
ADisparate impact analysis
BReview of the ML workflow
CEDA of the training data
DDynamic testing
Q6. A tester feeds a known unbiased, representative dataset through a deployed model and finds statistically significant differences in approval rates between age groups. What can be concluded about where the bias came from?
AThe approach detects bias in the model outputs regardless of whether it was introduced in the training data or during model development, so further analysis is needed to find the source
BIt must be data bias
CIt must be algorithmic bias
DIt cannot be bias, because the input dataset was unbiased
Q7. In disparate impact analysis, what does the final step involve?
ARemoving the sensitive attributes from the dataset
BIdentifying the sensitive attributes
CRetraining the model on counterfactuals
DAnalyzing the results of multiple tests to reach a statistically significant result about whether changing a sensitive attribute changes the model's output
Q8. For disparate impact analysis, a tester generates the counterfactual "a 16-year-old applicant with 25 years of employment history". What is the problem, according to the syllabus?
ACounterfactuals must never change age
BThe counterfactual is unrealistic, so the model may respond to the unrealistic example rather than to any underlying bias
CCounterfactuals can only change one attribute per test
DThere is no problem: any counterfactual is valid
Q9. Which TWO statements about testing for bias are CORRECT?
Select 2 options.
ABias in an ML system is usually lawful
BDisparate impact analysis can be applied to combinations of sensitive attributes to reveal hidden bias
CDynamic testing can only detect data bias
DLabel correctness testing can find mislabeling that teaches incorrect associations between attributes and outcomes for sensitive attributes
EStatic analysis executes the model with test data
Q10. A team wants to detect potential bias in its training data EARLY, before any model has been trained. Which approach is MOST suitable?
ADisparate impact analysis on the deployed model
BDynamic testing with a representative dataset
CReviewing the ML workflow, especially data preparation, and the dataset documentation
DA/B testing of two model versions
0 / 10 answered
← 5.1.1 Input Data Risks · Next drill: 5.1.3 Data Pipeline Testing →