ISTQB CT-AI: Testing Machine Learning and Generative AI Systems

The ISTQB Certified Tester AI Testing certification has entered a major transition. CT-AI v2.0 is now the latest version and refocuses the syllabus on testing AI-based systems themselves: data quality, machine-learning behavior, neural networks, AI-specific quality characteristics, model testing, and the testing of generative-AI systems such as large language models. The previous v1.0 also included using AI to support testing, but ISTQB has separated that topic more clearly in newer specialist material.

By late September 2026, CT-AI v2.0 is the current version while v1.0 remains in a defined sunset period. English v1.0 training and exams, including retakes, are available until April 21, 2027; non-English versions remain available until October 21, 2027. The v2.0 exam has 40 questions, 44 available points, a passing score of 29, and a standard duration of 60 minutes. The Certified Tester Foundation Level is a prerequisite.

The ISTQB certification pathway helps clarify the scope. CT-AI is about building confidence in AI-based systems. The separate CT-GenAI focuses on applying generative AI across testing work. Candidates should decide whether they need to test AI products, use AI as a testing capability, or eventually understand both.

AI quality cannot be reduced to ordinary deterministic correctness

Traditional software often allows a direct oracle: for a given input, the expected result is known. AI-based systems can be probabilistic, data-dependent, adaptive, or difficult to explain. The same prompt may produce different acceptable responses, and a model can perform well on aggregate while failing badly for a particular subgroup or edge case.

CT-AI therefore emphasizes AI-specific quality characteristics and acceptance criteria. Accuracy is only one dimension. Robustness, transparency, explainability, controllability, bias, safety, privacy, and other characteristics may determine whether a system is fit for purpose. The relevant mix depends on the application and the consequences of error.

Testers need to translate these qualities into observable evidence. “The model should be fair” is not testable until the team defines affected groups, relevant outcomes, metrics, thresholds, and data. “The assistant should not hallucinate” requires a method for distinguishing grounded answers, unsupported claims, uncertainty, and acceptable variation.

Data is part of the system under test

Data separation is another source of testing risk. Training, validation, and test data serve different purposes, and accidental leakage between them can make a model appear stronger than it is. The tester should understand where datasets came from, how they were sampled, whether labels are trustworthy, and whether the held-out evaluation data is sufficiently independent. After deployment, changes in the incoming population can also make previously good evidence less representative, so operational monitoring and re-evaluation matter.

Input-data tests can examine structure, ranges, missing values, duplicates, class balance, prohibited values, and relationships between fields before model behavior is considered. These checks are not merely data-cleaning tasks. They protect the validity of later model tests because performance numbers derived from corrupted or unrepresentative inputs can produce a confident but misleading quality conclusion.

Machine-learning behavior depends heavily on data. Training, validation, test, and production data can contain missing values, label errors, duplicates, leakage, imbalance, stale distributions, or sampling bias. A strong AI test strategy therefore examines data quality and lineage rather than treating datasets as passive inputs.

Representativeness matters because models learn patterns from what they see. A model trained on one population, device type, language, season, or business process may degrade when deployed into another. Testers should ask whether the evaluation data reflects the intended operating environment and whether important rare cases are present.

Data leakage is another critical risk. If information from the target or future state accidentally enters training features, reported performance can look excellent while real-world behavior collapses. AI testing needs enough understanding of the development pipeline to recognize when the evaluation setup itself produces misleading confidence.

Machine-learning metrics need interpretation, not worship

Classification metrics illustrate why context matters. Accuracy can look impressive when one class dominates the data, while precision and recall expose different costs around false positives and false negatives. A medical screening system, fraud detector, or content filter may tolerate those errors differently. Testers do not choose a metric because it is popular; they connect the metric and threshold to acceptance criteria and stakeholder harm, then examine whether performance is consistent across relevant segments rather than only in aggregate.

Classification systems are often evaluated with metrics such as precision, recall, specificity, accuracy, F-scores, and confusion matrices. No single metric is universally best. The right choice depends on what errors cost. A fraud detector may value catching true fraud differently from avoiding false alerts; a medical classifier may place a different weight on missed positives.

Aggregate metrics can also hide subgroup weaknesses. A model can achieve high overall accuracy while underperforming for a smaller population. Testers should evaluate performance across meaningful slices, operating conditions, and thresholds rather than stopping at one headline number.

The machine-learning lifecycle context helps explain why evaluation continues after model training. Deployment, monitoring, data drift, model updates, and feedback loops can change the risk picture even when the initial test results were strong.

Model and neural-network testing require different kinds of evidence

Model testing should also distinguish repeatability from usefulness. Some AI systems produce probabilistic outputs, so an exact string or value may be the wrong oracle. Test design can instead define tolerances, invariant properties, expected ranking behavior, reference sets, or statistical acceptance bands. The chosen oracle must still be defensible: replacing deterministic assertions with vague human judgment would simply move the quality problem rather than solve it.

CT-AI v2.0 includes model testing and the fundamentals of neural networks because testers need enough technical understanding to reason about AI failures. The objective is not to turn every tester into a data scientist. It is to understand what is being evaluated, where uncertainty comes from, and how development choices affect testability.

Model testing may examine functional performance, robustness to perturbations, sensitivity to input changes, behavior near decision boundaries, or performance across subgroups. Neural-network systems raise additional concerns because internal behavior can be opaque and large models can contain complex interactions that are difficult to trace with conventional coverage measures.

Testing therefore combines metrics, representative data, adversarial or stress cases, metamorphic relationships, back-to-back comparison, exploratory investigation, and domain-specific oracles. The test approach should match the model type, risk, and available evidence.

Generative AI introduces open-ended outputs, hallucinations, and new attack surfaces

For large-language-model features, a useful evaluation set should represent the tasks, languages, user populations, edge conditions, and failure modes that matter to the product. Teams can combine automated measures with structured human review, but both need clear criteria. A model change, prompt change, retrieval change, or safety-policy change can alter behavior, so the evaluation set also acts as a regression asset that lets testers compare versions instead of relying on memorable anecdotes.

CT-AI v2.0 explicitly includes testing generative AI and large language models. These systems create outputs rather than selecting only from predefined classes, which makes evaluation more difficult. Quality may involve factuality, relevance, completeness, safety, consistency, tone, instruction following, citation behavior, and resistance to malicious input.

The generative AI fundamentals behind tokens, context, inference, and application design help testers understand why prompt wording and context windows can change behavior. Retrieval, tools, system instructions, and grounding layers also become part of the system that must be tested.

Reducing hallucinations requires more than asking a model to “be accurate.” Test strategies can include grounding, retrieval validation, source checks, response constraints, uncertainty handling, and evaluation sets containing known failure patterns. The tester must measure whether these controls work under realistic and adversarial conditions.

The v2.0 transition changes the boundary between testing AI and using AI for testing

That boundary is useful when planning a study path. CT-AI asks how to evaluate a system whose product behavior depends on data and AI models; CT-GenAI asks how a tester can use generative AI as part of testing work. The same organization may need both capabilities, but the risks, test objects, and evidence are different. Keeping the distinction clear prevents candidates from over-studying prompt productivity when the CT-AI objective is model quality, input data, or AI-system test strategy.

CT-AI v1.0 covered both testing AI-based systems and using AI to support testing. CT-AI v2.0 narrows the focus toward testing AI systems across their lifecycle. ISTQB’s CT-GenAI specialist certification now provides a clearer route for people who want to apply generative AI to requirements analysis, test design, automation, reporting, and other testing activities.

This separation is useful because the risks are different. Testing an AI product requires evidence about the model, data, quality characteristics, and deployment behavior. Using an LLM to generate test cases raises questions about prompt quality, hallucinated requirements, sensitive data, reproducibility, review, and human accountability.

Candidates already far into v1.0 preparation can still take that exam during the transition period, but anyone starting fresh should compare objectives carefully. The latest version is the better default for professionals whose main goal is to test AI-based systems with a current lifecycle-oriented framework.

CT-AI preparation should combine testing judgment with basic AI literacy

Foundation Level knowledge remains necessary because CT-AI still depends on core ideas such as risk, test design, acceptance criteria, defects, and lifecycle alignment. Candidates should be comfortable with CTFL v4.0 before adding AI-specific complexity.

Study should then move from concepts to scenarios. Given a biased dataset, what test evidence is needed? Given a classifier with strong accuracy but poor recall, what business risk is being hidden? Given an LLM with grounded retrieval, how would the team test unsupported claims? Given a model update, what needs regression testing and monitoring?

The certification is most useful when candidates stop treating AI as magic. AI systems have architecture, data, assumptions, interfaces, failure modes, and operational contexts that can be analyzed. CT-AI provides a vocabulary and test framework for making that analysis systematic enough to support real quality decisions.

ExamSnap's ISTQB CT-AI Practice Test Questions and Exam Dumps, study guide, and video training course are complicated in premium bundle. The Exam Updated are monitored by Industry Leading IT Trainers with over 15 years of experience, ISTQB CT-AI Exam Dumps and Practice Test Questions cover all the Exam Objectives to make sure you pass your exam easily.