Appearance
Topic Drill 4.2.1: Testing Generative AI
About this drill
LO AI-4.2.1 (K2): Explain how generative AI can be tested.
Lesson: Ch 4, section 4.2.1
This drill covers every point in section 4.2 / 4.2.1:
- What is evaluated: correctness, coherence, creativity (originality, novelty), functional and non-functional requirements, several media types
- Black-box testing: inputs, assessment factors, manual or automated, end-user applications
- The input explosion problem: system and user prompts, API, temperature, maximum tokens, context window
- Pass/fail: manual review against qualitative criteria, or a second GenAI used with care
- Non-functional measures; benchmark suites
Topic Drill: 4.2.1
Q1. Which set of aspects does the syllabus say testing GenAI outputs evaluates?
ASpeed, cost and memory only
BCorrectness, coherence and creativity, including the originality and novelty of results, plus whether functional and non-functional requirements are met
CExact match against predefined expected outputs
DGrammar and spelling only
Q2. In black-box testing of a GenAI system, which factors does the syllabus say are considered when assessing the results?
ANeuron coverage and kMNC
BThe number of layers in the model
CTraining time and dataset size
DClarity, originality and adherence to domain-specific rules
Q3. A tester notices that the same question gets a different answer depending on what was asked earlier in the same chat. Which part of the input explosion problem does this illustrate?
AThe context window, which retains previous parts of a conversation
BThe maximum tokens parameter
CAPI access
DBenchmark suites
Q4. A tester raises the temperature setting of an LLM and runs the same prompt several times. What should they expect, based on the definition of temperature?
AIdentical outputs every time
BShorter outputs
CLess predictable, more creative outputs
DFaster response times
Q5. When GenAI output is reviewed manually, what does the syllabus say determines whether a test has passed or failed?
AWhether the tester personally likes the output
BThe qualitative evaluation criteria defined in the requirements
CWhether the output is longer than the prompt
DWhether a benchmark score improved
Q6. Which measures does the syllabus list for NON-functional testing of GenAI systems?
APrecision, recall and F1-score
BClarity, originality and coherence
CNeuron and boundary coverage
DCPU and GPU usage, memory consumption, network bandwidth and response times, during inference and training
Q7. What is the purpose of benchmark suites for GenAI, according to the syllabus?
ATo give standardized evaluation through curated datasets and tasks, allowing consistent comparison across models and systematic, reproducible identification of areas for improvement
BTo replace red teaming
CTo train the model on new data
DTo filter user prompts in production
Q8. For which kind of GenAI application does the syllabus say black-box testing is particularly useful?
AInternal batch data pipelines
BFirmware for embedded controllers
CEnd-user applications such as chatbots or design tools, where user satisfaction depends on the usefulness and plausibility of the content
DModel training frameworks
Q9. Which TWO of the following are listed as contributors to the input explosion problem when testing GenAI?
Select 2 options.
AThe number of testers in the team
BThe maximum tokens parameter
CThe screen resolution of the tester's laptop
DOptional system prompts in addition to the user prompt
EThe test management tool in use
0 / 9 answered