Skip to content

Topic Drill 4.2.2: Red Teaming (K3) ​

About this drill

LO AI-4.2.2 (K3): Implement red teaming for GenAI systems.
Lesson: Ch 4, section 4.2.2, including the worked example and plan template

This drill covers every point in section 4.2.2:

  • What red teaming is and where it comes from (wargaming, tiger teams); typical harmful outputs; strengthening the system
  • Any AI system, but especially GenAI (attack space); end-to-end vs. model only; harm types beyond security and safety; regulation (EU AI Act)
  • Adaptive vs. static (benchmarking); security RT (indirect prompt injection, RAG documents) vs. safety RT (ordinary use)
  • Timing (after internal QA, before deployment); multi-turn prompting; the five steps
  • Scaling: crowd-sourcing, LLM-generated attacks with an LLM judge, hybrid; blue teaming

Topic Drill: 4.2.2 (K3)

Q1. Which description of red teaming (RT) matches the syllabus?

AA static benchmark of model capabilities
BReal-time monitoring and filtering of inputs to an operational system
CA systematic, often black-box form of fault attack that probes an AI-based system to identify harmful capabilities, particularly in its outputs
DA white-box review of the model's source code

Q2. Why is red teaming ESPECIALLY critical for GenAI systems?

AThe vast range of possible inputs and outputs creates an extensive "attack space"
BGenAI systems cannot be tested in any other way
CGenAI systems are always adaptive
DRegulations only apply to GenAI

Q3. A travel-booking chatbot is due to go live in a month. Internal quality assessments are scheduled to finish next week. When is red teaming MOST effective?

AImmediately, before the internal quality assessments, so that red teamers find the simple defects
BAfter go-live, once real users have generated traffic
COnly after the first security incident
DAfter the internal quality assessments are complete, but before deployment

Q4. Red teamers upload a PDF containing hidden text that tells the assistant, which summarizes documents using RAG, to reveal its system prompt, and the assistant complies. What kind of finding is this?

AA safety finding under ordinary use
BA security finding: an AI-specific vulnerability such as indirect prompt injection or malicious content hidden in documents used by RAG
CA benchmark regression
DA fairness finding

Q5. A user with no malicious intent asks a health chatbot whether two common medicines can be taken together and receives dangerous advice. Which kind of red teaming evaluation is designed to find this?

ASecurity red teaming focused on malicious inputs
BBlue teaming
CSafety (and other non-security) red teaming, which looks for harmful outputs under ordinary use
DBenchmarking

Q6. A company has only three expert red teamers but needs far broader coverage of attack prompts. Which approach does the syllabus describe for this?

AAutomated red teaming, where an LLM generates numerous attack prompts and another LLM checks the outputs, possibly combined with human testers (hybrid)
BReducing the scope to a single harm type
CSkipping red teaming and relying on blue teaming
DRunning the same few prompts repeatedly

Q7. Which statement CORRECTLY contrasts red teaming with blue teaming?

ARed teaming is ongoing defensive monitoring; blue teaming is a pre-deployment attack
BThey are two names for the same activity
CBlue teaming replaces red teaming once a system is deployed, and its insights are not shared
DRed teaming is proactive and pre-deployment focused; blue teaming is ongoing, real-time defensive monitoring and filtering of inputs to an operational system, and red teaming insights improve those monitors and filters

Q8. How does red teaming relate to benchmarking, according to the syllabus?

ARed teaming is a type of benchmark
BRed teaming is an adaptive, dynamic evaluation in which prompts can be updated in response to outputs, complementing static approaches like benchmarking by testing under extreme and unexpected conditions
CBenchmarking makes red teaming unnecessary
DRed teaming is only used to calculate benchmark scores

Q9. A red team has finished its interactive sessions and analyzed the failures it found. What is the NEXT step in the syllabus's approach?

ADeploy the system immediately
BDelete the failing prompts to protect the system
CCreate datasets from the identified threats to support mitigations and system improvements
DStart a new red team with different members

Q10. A city council will launch a GenAI assistant that answers citizens' questions about benefits, using RAG over council documents. Which plan is the MOST effective implementation of red teaming?

ABefore launch, assemble a diverse team (security tester, benefits caseworker, accessibility advocate, legal adviser), give access in a sandbox with synthetic citizen data, run open-ended and checklist-based multi-turn sessions covering security (incl. poisoned RAG documents), privacy, bias and misinformation, analyze the failures, and build datasets for mitigations and blue-team filters
BLaunch on schedule, then have the two developers who built the assistant run an automated LLM attack generator against the public endpoint for prompt injection only, judge the outputs with a second LLM, and patch the system prompt for each finding
CBefore launch, run the assistant against several well-known public LLM benchmarks for reasoning, safety and bias, compare its scores with competing products, and approve the launch if it scores above average in every category
DBefore launch, configure real-time input filters and output monitoring for the production endpoint, set alert thresholds for harmful content, and let the operations team block and analyze suspicious requests as they occur

Q11. Which TWO statements about red teaming are CORRECT?

Select 2 options.

ARed teaming can only be applied to GenAI systems
BRed teaming is only concerned with security vulnerabilities
CRed teaming replaces benchmarking
DRed teaming typically covers the complete end-to-end AI-based system, but can be applied to only the AI model
EThe core activity is often interactive prompting in multi-turn dialogues (e.g., 15–20 turns)

0 / 11 answered

← 4.2.1 Testing GenAI · Next drill: 4.3.1 Test Levels →

Unofficial study guide for the ISTQB® Certified Tester AI Testing (CT-AI) exam. Not affiliated with International Software Testing Qualifications Board (ISTQB®).