Skip to content

Ch 3: Forms of ML and the ML Workflow ​

Official Reference

ISTQB® CT-AI Syllabus v2.0, Chapter 3 (375 minutes), section 3.1, Learning Objectives (LOs) AI-3.1.1, AI-3.1.2, AI-3.1.4 (all K2), and Hands-on Objective HO-3.1.3 (H2). Official syllabus and documents

Exam weight (whole Chapter 3): ~18% of points (7 questions, 8 points). Six K2 questions are shared across seven K2 LOs, and the K3 metrics calculation (AI-3.3.1) is worth 2 points. This page covers three of the K2 LOs.

Chapter 3 is the "ML for testers" chapter. The syllabus says why it matters: different test activities apply at different stages of the ML workflow, and failures often originate in earlier steps such as data preparation or model selection. If you know the workflow, you know where to look for defects. Chapter 3 is split over three pages: this one (3.1), Data and Performance Metrics (3.2–3.3), and Neural Networks (3.4).

AI-specific keywords on this page (know name and definition, K1): machine learning, ML algorithm, ML model, ML workflow, supervised learning, unsupervised learning, reinforcement learning, classification, ML regression, clustering, association, ML development framework, pretrained model. All definitions below: ISTQB® Glossary.
ML algorithm: An algorithm used to create an ML model from a training dataset.
ML model: An implementation of machine learning that generates a prediction, classification or recommendation based on input data.
ML workflow: The set of activities used to develop, deploy and operate an ML model.

Testing analogy

Algorithm vs. model: think of the ML algorithm as a compiler and the training data as the source code. The output, the ML model, is the program that actually runs in production. A defect can come from any of the three.

3.1.1 Different Forms of Machine Learning (K2) ​

mermaid
flowchart LR
  ML["ML algorithms"] --> SL["Supervised learning<br/>labeled data"]
  ML --> UL["Unsupervised learning<br/>unlabeled data"]
  ML --> RL["Reinforcement learning<br/>agent + environment<br/>rewards / penalties"]
  SL --> C["Classification<br/>predefined classes"]
  SL --> R["ML regression<br/>continuous values"]
  UL --> CL["Clustering<br/>group similar points"]
  UL --> AS["Association<br/>relationships between attributes"]

First ask "is there a label?" (supervised), "no label?" (unsupervised) or "is there an agent getting rewards?" (reinforcement); then ask whether the output is a class, a number, a group or a relationship.

FormHow it learnsTypical problem types (syllabus examples)
Supervised learningFrom labeled data: each set of inputs has a corresponding output label (e.g., images labeled "dog" or "cat"); learns to map inputs to outputsClassification: assign inputs to predefined classes, e.g., spam or not spam, image recognition. ML regression: predict continuous numerical values, e.g., a person's age from lifestyle data, stock prices
Unsupervised learningFrom unlabeled data: infers patterns or structures, grouping similar inputs by shared featuresClustering: group data points by similarity, e.g., customer segmentation for targeted marketing. Association: identify relationships or dependencies among data attributes, e.g., purchasing patterns used to recommend products
Reinforcement learningAn intelligent agent interacts with its environment and receives rewards or penalties, so it learns from experience rather than from a datasetRobotics, autonomous vehicles, adaptive systems like chatbots. Challenges: setting up the environment, designing the reward function, selecting the best strategy

The choice of approach depends on the nature of the available data and the specific task.

Classification: An ML function that predicts the output class for a given input.
ML regression: A type of ML function that results in a numerical or continuous output value for a given input.
Clustering: An ML function that groups similar data points together.
Association: An unsupervised ML learning technique that identifies relationships and dependencies between samples.
Reinforcement learning: An approach in which an ML model, known as an agent, learns by using a trial-and-reward feedback loop with its environment to achieve specific objectives.

Exam Distinction: "regression"

In this syllabus, ML regression means predicting a continuous value. It is not regression in the other ISTQB® syllabi, where it means software changes causing change-related defects (regression testing). The syllabus points this out explicitly.

Exam Distinction: classification vs. clustering

Both put things into groups. Classification needs predefined, labeled classes (supervised). Clustering discovers groups that were not defined in advance (unsupervised). Look for whether labels exist in the scenario.

3.1.2 Machine Learning Workflow (K2) ​

mermaid
flowchart TB
  O["Understand the objectives<br/>agree purpose + acceptance criteria"] --> F["Select a framework"]
  F --> A["Select & build the algorithm"]
  A --> D["Prepare & test data"]
  subgraph G["Model generation & test"]
    direction TB
    TR["Train the model"] --> EV["Evaluate the model<br/>(validation dataset)"]
    EV --> TU["Tune the model<br/>(hyperparameters)"]
    TU --> TR
    EV --> TE["Test the model<br/>(independent test dataset)"]
  end
  D --> TR
  TE --> DE["Deploy the model"]
  DE --> U["Use the model"]
  U --> M["Monitor & tune the model"]
  M -.->|"retrain / new model"| D

Training, evaluation and tuning loop together as "model generation"; only an acceptable model goes on to testing with an independent test dataset, and the workflow as a whole is applied iteratively.

ActivityWhat happens (syllabus)
Understand the objectivesStakeholders agree the purpose of the model, aligned with business priorities. Acceptance criteria (including ML functional performance metrics, see 3.3) are defined
Select a frameworkChoose an ML development framework (see 1.1.7) based on objectives, acceptance criteria (2.2) and business priorities
Select & build the algorithmChosen based on objectives, acceptance criteria and available data. Often retrieved from a software library rather than hand-coded; compiled if required
Prepare & test dataData acquisition, data preprocessing, feature engineering (3.2); exploratory data analysis (EDA) alongside. Data must be representative of operational data. Data and automated preparation steps are tested (Chapter 5)
Train the modelThe algorithm uses training data. Model hyperparameters (model structure, e.g., number of layers in a neural network, depth of a decision tree) and algorithm hyperparameters (control the training, e.g., number of iterations) are passed to the algorithm
Evaluate the modelAgainst the agreed ML functional performance metrics, using the validation dataset. In practice several models are created (e.g., random forests, SVM, neural networks, with various training datasets) and the best combination is chosen
Tune the modelEvaluation results are used to adjust model and algorithm hyperparameters, then the model is retrained
Test the modelThe acceptable model is tested with an independent test dataset against the agreed ML functional performance criteria, and results are compared with evaluation. Significantly lower results mean going back to model generation or even to data preparation. Non-functional tests (e.g., time to provide a prediction) may also be run. Typically done by data engineers or scientists, but testers with domain knowledge and access can do it too
Deploy the modelThe tuned model is usually re-engineered for deployment along with its data pipeline, generally through the framework. Targets include embedded systems and the cloud (web API). The re-engineered model is tested to verify it still meets its acceptance criteria
Use the modelIntegrated into a larger operational system; scheduled batch predictions or real-time on request
Monitor & tune the modelThe model may drift (6.1.7), so it is regularly evaluated against its acceptance criteria. A new model (new data, new hyperparameters, or both) may be compared to the existing one with A/B testing (6.1.9)

Two more points the syllabus makes:

  • The workflow is a logical sequence, but in practice it is applied iteratively.
  • It does not show integration with the non-ML parts of the system. ML models usually cannot be deployed in isolation (e.g., a vision data pipeline cleans data before it reaches the model). When the model is part of a larger system, integration, system and acceptance test levels may be performed.
Hyperparameter: A parameter used to either control the training of an ML model or to set the configuration of an ML model. (ISTQB® Glossary)
ML functional performance criteria: Criteria based on ML functional performance metrics used as a basis for model evaluation, tuning and testing. (ISTQB® Glossary)

Exam Distinction: evaluate vs. test

Evaluate uses the validation dataset and feeds tuning; it is part of model generation. Test uses an independent test dataset and checks the agreed acceptance criteria once an acceptable model exists. Questions that match descriptions to workflow activities (e.g., "the model is tested on the target platform" → Deploy the model) are common.

Hands-on exercise HO-3.1.3 (H2): Create an ML model

Goal (syllabus): select, train and test a classification model using supervised learning, and explain the difference between evaluating/tuning and testing by comparing the accuracy on the validation and test datasets.

This exercise builds on the data preparation in HO-3.2.2. Run that code first; it gives you X_train, X_val, X_test, X_trainval and the matching y_ variables.

python
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score

# Model generation: train + evaluate + tune the hyperparameter max_depth (a model hyperparameter)
best_depth, best_acc = None, 0.0
for depth in [2, 3, 4, 5, 8, None]:
    model = DecisionTreeClassifier(max_depth=depth, random_state=42).fit(X_train, y_train)
    acc = accuracy_score(y_val, model.predict(X_val))          # evaluate on VALIDATION data
    print(f"max_depth={depth}: validation accuracy={acc:.3f}")
    if acc > best_acc:
        best_depth, best_acc = depth, acc

# Test the model: retrain the chosen configuration, then test ONCE on the independent test dataset
final = DecisionTreeClassifier(max_depth=best_depth, random_state=42).fit(X_trainval, y_trainval)
print(f"chosen max_depth={best_depth}  validation={best_acc:.3f}  "
      f"test={accuracy_score(y_test, final.predict(X_test)):.3f}")

Reflect: compare the chosen model's validation and test accuracy. On a small dataset they can differ in either direction (one run gave validation 0.756 and test 0.795). The point is their role. The validation data was used to choose the model, so it is not an independent measurement. The untouched test dataset is what you report against the acceptance criteria. A significantly lower test result would send you back to model generation or data preparation.

3.1.4 Pretrained Models, Fine-Tuning, and Retrieval-Augmented Generation (K2) ​

Pretrained model: An ML model that has already been trained on a large, general purpose dataset and can be reused or fine-tuned for specific tasks.
Retrieval-augmented generation (RAG): An approach where a GenAI system retrieves relevant external information to supplement its internal knowledge and generate more accurate, contextually relevant responses.
Source: ISTQB® Glossary

Training a new model from scratch is costly and time-consuming. Two ways to reuse a pretrained model:

Fine-tuningRAG
What it isTake a pretrained neural network and adapt it to a new, different task with additional training on task-specific dataProvide the LLM with task-specific data sources, transformed into a searchable format and compared with the subject of the prompt
What changesThe model's weights: the entire network, specific layers (typically near the output end) or additional layersNothing in the model. Relevant documents are added to an enhanced prompt
Data neededMuch less training data (and effort) than training from scratchRelevant documents, identified and made searchable up front
Success factorSimilarity of original and new task: cat breeds → dog breeds works well; image classifier → spoken accents does not. Fine-tuning an LLM for ISTQB® boundary value analysis is a small change, achievable with good training dataMore pertinent information in the prompt, so the response is likely more precise
ProcessTrain, evaluate ML functional performance, fine-tune further until acceptance criteria are metRetrieve, enhance prompt, generate

A pretrained model can use RAG, fine-tuning, or both. Whichever is used, biases and vulnerabilities in the pretrained model typically carry over to the new model, so testing is needed to confirm that it performs reliably and fairly in the new task.

mermaid
flowchart LR
  Q["User prompt"] --> S["Search task-specific<br/>data sources"]
  S --> E["Enhanced prompt =<br/>prompt + relevant documents"]
  Q --> E
  E --> L["Pretrained LLM<br/>(unchanged)"]
  L --> A["More precise response"]

RAG improves the answer by changing the input, never the model, which is why "RAG requires no change to the pretrained model" is a reliable exam fact.

Unit Quiz ​

Unit Quiz: Chapter 3, page 1 (3.1)

Q1. An airline wants to predict how many minutes each flight will be delayed, using historical flight records labeled with the actual delay. Which form of ML is this?

AClassification
BClustering
CML regression
DAssociation

Q2. Which statement about reinforcement learning is CORRECT?

AAn intelligent agent learns from rewards and penalties received from its environment, rather than from a dataset
BIt learns from a dataset in which every input has an output label
CIt groups unlabeled data points by similarity
DIt is the same as ML regression

Q3. In the ML workflow, which activities together form "model generation"?

APrepare & test data, train, and test the model
BSelect a framework, select & build the algorithm, and deploy the model
CDeploy, use, and monitor & tune the model
DTrain, evaluate, and tune the model

Q4. A model scores 94% accuracy on the validation dataset but only 81% on the independent test dataset. What does the syllabus say may be necessary?

ADeploy the model, because the validation result is what counts
BReturn to the model generation activities, or even to the prepare & test data activity, to train a new model
CRe-run the test with the validation dataset instead
DLower the acceptance criteria to 81%

Q5. Which statement about retrieval-augmented generation (RAG) is CORRECT?

ARAG retrains the pretrained model on task-specific documents
BRAG adds extra layers to the neural network to store documents
CRAG provides task-specific data sources to the LLM through an enhanced prompt, with no change to the pretrained model
DRAG removes any bias inherited from the pretrained model

Q6. When is fine-tuning a pretrained model MOST likely to be effective, according to the syllabus?

AWhen the new task is very similar to the original task, e.g., adapting a cat-breed classifier to identify dog breeds
BWhen the new task is very different, e.g., adapting an image classifier to recognize spoken accents
CWhen no training data is available for the new task
DOnly when all layers of the network are retrained

0 / 6 answered

Topic Drills: 3.1.1 · 3.1.2 · 3.1.4 · Next page: Data and Performance Metrics →

Unofficial study guide for the ISTQB® Certified Tester AI Testing (CT-AI) exam. Not affiliated with International Software Testing Qualifications Board (ISTQB®).