Skip to content

Topic Drill 3.2.3: Training, Validation, and Test Datasets ​

About this drill

LO AI-3.2.3 (K2): Contrast the use of training, validation, and test datasets in the development of an ML model.
Lesson: Ch 3, section 3.2.3

This drill covers every point in section 3.2.3:

  • The role of each dataset; equivalent data; the test dataset = holdout dataset
  • The four factors that decide data amounts when data is abundant
  • Limited data: underfitting risk, a small hold-out set, k-fold cross-validation, stratified folds, averaging, final training on the pool, one-time test
  • The optimistic bias without a hold-out set; leave-one-out cross-validation and bootstrap

Topic Drill: 3.2.3

Q1. A team tunes its model's hyperparameters by repeatedly checking the score on the TEST dataset. What is the main problem?

AThe training dataset becomes too small
BThe test dataset is no longer independent, so it cannot give an unbiased final assessment; the validation dataset should be used for tuning
CHyperparameters cannot be tuned at all
DThe model will automatically underfit

Q2. When suitable data is abundant, which of the following does the syllabus list as a factor in deciding how much data to use for training, evaluation and testing?

AThe number of stakeholders
BThe colour depth of the images
CThe number of testers available
DThe desired confidence in the resultant model

Q3. When data is limited, what risk does a strict three-way split (train / validation / test) create?

AInsufficient data for effective training, increasing the risk of underfitting
BOverfitting to the test dataset
CConcept drift
DExcessive training time

Q4. A training-and-validation pool of 1,000 records is used for 10-fold cross-validation. In each round, how many records are used for training and how many for validation?

A100 for training, 900 for validation
B500 for training, 500 for validation
C900 for training, 100 for validation
D1,000 for training, 0 for validation

Q5. Why are cross-validation folds often built using stratified sampling?

ATo make training faster
BTo make each fold representative, especially with imbalanced data or small datasets
CTo remove outliers
DTo avoid needing a test dataset

Q6. After the best hyperparameters have been found through cross-validation, what does the syllabus say typically happens next?

AThe best fold's model is deployed directly
BCross-validation is repeated on the hold-out test set
CThe hyperparameters are tuned again on the test set
DA final model is trained on the entire training and validation pool and evaluated ONCE on the hold-out test set

Q7. Data is so scarce that NO hold-out test set can be kept, so only cross-validation is used. What does the syllabus say about the resulting performance estimate?

AThe average cross-validation performance is optimistically biased and cannot serve as an unbiased estimate
BIt is unbiased, because every record was used for validation once
CIt is pessimistic, because each model saw less data
DIt is identical to a hold-out test result

Q8. What is leave-one-out cross-validation?

AA bootstrap technique that samples with replacement
BRemoving one feature at a time to test its importance
CA special case of k-fold cross-validation where k equals the number of samples
DLeaving the test dataset out of the project

Q9. Which TWO statements about training, validation and test datasets are CORRECT?

Select 2 options.

AThe validation dataset is used to train the model
BThe bootstrap technique is k-fold cross-validation with k = 2
CCross-validation metrics from each fold are summed, not averaged
DLogically, three sets of equivalent data are needed, e.g., randomly selected from a single representative dataset
EThe test dataset is also known as the holdout dataset

0 / 9 answered

← 3.2.1 Data Preparation · Next drill: 3.3.1 Performance Metrics →

Unofficial study guide for the ISTQB® Certified Tester AI Testing (CT-AI) exam. Not affiliated with International Software Testing Qualifications Board (ISTQB®).