LLM applications often work well in demos using standard prompts alone, but in complex real-world production environments, they frequently encounter unexpected failures. To catch this gap prior to deployment, you need an evaluation framework established during the development stage.
In Part 1, we covered what LLM evaluation is and what methods and metrics are used to measure quality. This installment focuses on how to integrate that evaluation into the actual development process. It summarizes how to build datasets to validate performance before deployment, how to compare changes through experiments, and how to connect everything to a CI/CD pipeline.
This article is based on the Pre-Production and CI/CD sections of Arize AI's "The definitive guide to LLM evaluation."
☑️ What is Development-Stage Evaluation?
Development-stage evaluation is the process of verifying performance using curated or generated datasets and comparing changes through experiments to filter out regressions before deploying an application to production. The offline evaluation and CI/CD checks covered in Part 1 fall under this category. Offline evaluation tests models using pre-prepared inputs in a controlled environment rather than live production data. The goal is to verify before deployment whether a prompt change or model swap improved results, and whether it broke responses that were previously handled well.
☑️ Why a Golden Dataset is the Starting Point for Development-Stage Evaluation
A golden dataset is a carefully constructed and verified collection of examples that serves as the ground truth baseline for evaluating model performance. This dataset forms the foundation of many evaluation strategies, ensuring that evaluation results are reliable and consistent. If a dataset is too narrow or overly refined, it may score high on evaluation metrics but underperform under real user traffic. A golden dataset should represent the tasks supported by the application, frequent incoming queries, and challenging edge cases.
Handcrafted Datasets
The most basic form is where domain experts or dataset designers create examples manually. They handwrite various inputs that the model must process along with their corresponding correct outputs. The strength of this approach lies in its ability to craft nuanced, tricky examples tailored to specific use cases or edge cases.
Ground Truth Annotation
This is the step where humans assign correct answers to the handcrafted dataset. Ground Truth refers to the correct answers or labels that serve as the evaluation benchmark. For instance, if an annotator labels a language model's output as correct or incorrect based on its alignment with expected behavior, this value becomes the baseline for comparing subsequent model performance.
Multi-Annotator Verification
To ensure the quality and reliability of Ground Truth, multiple annotators independently label and verify each example. A common approach is to accept a label as finalized Ground Truth if two out of three annotators agree. This multi-annotator strategy mitigates individual biases or errors and enhances dataset reliability by measuring inter-annotator agreement.
Human annotation is also utilized to attach custom labels to traces or spans. Annotating production responses directly allows them to be used for filtering and analysis, or categories can be assigned to specific parts of a conversation to perform granular analysis on where the model succeeds or fails. Furthermore, human-labeled data serves as high-quality datasets for CI/CD testing, few-shot prompting, and targeted evaluation.
☑️ Synthetic Dataset Generation and Validation
A synthetic dataset is artificially generated data designed to mimic real-world information. Created through algorithms or rules, it is engineered to represent specific patterns or scenarios developers and researchers wish to test. Examples include customer support chats, Q&A pairs, product reviews across various tones, and code snippets containing deliberate bugs.
Synthetic data allows teams to establish controlled experimental environments, capture a broad array of edge cases, and protect privacy by avoiding the use of actual user data. It is particularly useful as golden data for consistent experimentation when developing and testing applications that have not yet been launched.
Best Practices for Synthetic Datasets
Synthetic datasets are not a set-it-and-forget-it asset; they require ongoing maintenance. It is recommended to establish regular update schedules to reflect model improvements and data drift, document the prompts, models, and post-processing procedures used during generation to maintain transparency, and periodically evaluate performance against real-world data and the latest models.
Relying solely on synthetic data can lead to missing key cases or input types that humans would naturally consider. Incorporating a small number of human-labeled examples can dramatically enhance the quality and effectiveness of synthetic datasets—this is particularly effective in few-shot learning scenarios where models must generalize from minimal examples. However, when adding human-crafted examples, it is best to maintain balance so they do not overpower the proportion of synthetic data.
☑️ Criteria for Selecting Evaluation Models
The model selected for evaluation depends on iteration frequency, cost-versus-accuracy trade-offs, and flexibility requirements. Many teams start evaluating with the same base model used in their application to establish a baseline. Later, they transition to Small Language Models (SLMs) to reduce costs; small models offer significantly lower operational expenses and run faster when serving as guardrails. Going a step further, fine-tuned models tailored to specific evaluations can also be deployed. The key to selection is balancing cost and accuracy under the premise that evaluations will be iterated frequently.
☑️ How to Validate Changes Using CI/CD Experiments
An experiment is a systematic procedure to test and validate changes to an LLM application using curated datasets. By defining a dataset, creating a task that generates outputs, and configuring evaluators to score those outputs, you can measure how effectively the modified pipeline operates. An experiment consists of three main components:
- Dataset: A collection of examples providing inputs—and optionally expected outputs—to evaluate the application. It is used to track improvements resulting from changes to prompts, models, or other components.
- Task: A function or process that produces JSON-serializable outputs. It reproduces the LLM functionality being tested; for example, if you change a prompt, the task executes the examples using the new prompt to generate outputs.
- Evaluator: A function that takes the task's output and executes the assessment. It serves as the criteria for determining experimental success, with multiple evaluators—ranging from LLM-based to code-based—being customizable.
Experiments can test not only prompt or model changes, but also alterations in how the application is orchestrated—such as new integrations, API interactions, or external function calls. This allows teams to verify whether response latency improves, errors decrease, and the overall user experience is enhanced.
Methods for Building Evaluation Datasets
Evaluation datasets can be constructed through several avenues. The most recommended starting point is manual curation—a method where you directly compile the inputs the application must handle along with examples of good responses. Having as few as 20 high-quality examples can provide significant value. After launching the application, historical logs become a rich resource. Because they reflect actual user behavior, you can select underperforming cases—such as instances that received negative feedback or runs that took long to complete—and add them to the dataset to test against future regressions. Once a base set of examples is collected, you can rapidly scale up volume using synthetic data; however, because synthetic data tends to mirror source examples, it is best to secure a solid set of handcrafted examples first.
☑️ Experimental Results Aren't Black and White
When interpreting experiment results, it is essential to understand that outcomes are often mixed. Certain metrics may improve while others regress. For example, restructuring a prompt might boost consistency scores while slightly reducing factual accuracy.
Many teams manually select validation examples and then overlay metrics such as average score, F1, recall, and precision onto the evaluation output to assess performance. While statistical checks are not yet universally adopted, they are increasingly becoming a standard best practice. The key is to weigh trade-offs and prioritize improvements aligned with the application's goals.
☑️ CI/CD Automation and Promoting Changes
Building a CI/CD pipeline enables you to maintain control even as your application evolves. Just like in traditional software development, catching issues early via automated testing is crucial. By creating experiments that automatically validate changes using curated datasets and your preferred evaluation methods, you can integrate them with GitHub Actions to trigger automated tests every time code is pushed.
Experiments run in a CI/CD pipeline aim to replicate production conditions as closely as possible. Ground Truth evaluations should run regardless of what is being tested; a failure here signals that something fundamental is broken, and the change should not be promoted. Threshold-based experiments detect whether metrics like hallucination or accuracy cross defined baselines. For example, if the hallucination rate spikes significantly compared to the baseline, it serves as a red flag that requires investigation prior to promotion.
Determining whether an evaluation failure should block a Pull Request (PR) is handled similarly to traditional unit testing. Critical failures in Ground Truth comparisons or major metric spikes should block the PR, but not all evaluation failures need to block a release—some are better suited for post-deployment monitoring. This represents a fundamental paradigm shift: LLM evaluations are the new unit tests. Even if teams do not block changes for every minor failure, these evaluations act as real tests designed to catch critical application flaws.
There is another critical shift to consider: LLM applications are impacted not only by code modifications, but also by model updates and drift in production inputs. Consequently, even when there are no open PRs or active development changes, experiments need to be executed on a regular schedule to detect shifts in performance.
☑️ Implementing Development-Stage Evaluation in Practice
The dataset construction, experimentation, and CI/CD integration discussed so far represent methodology. To operationalize this framework within an actual development workflow, you need a tool that consolidates dataset management, experiment execution, and pipeline integration into a single platform.
Arize AX is an AI observability and evaluation platform designed to support this stage of development. It enables teams to curate datasets from data gathered across production, staging, and evaluation workflows; compare prompt, model, and orchestration changes via experiments; and build experiments that automatically validate changes using curated datasets and preferred evaluation methods. Integrated with GitHub Actions, these experiments run automatically upon code push, allowing you to verify change stability prior to deployment without relying on manual testing.
▶ Read more about Arize AX here
☑️ Frequently Asked Questions
What is a Golden Dataset?
A golden dataset is a carefully constructed and verified collection of examples that serves as the ground truth baseline for evaluating model performance. It combines handcrafted examples, Ground Truth annotations, and multi-annotator verification to ensure reliability and diversity.
How do you validate synthetic datasets?
Synthetic datasets should be validated through regular update schedules, detailed documentation of generation processes to ensure transparency, and periodic performance benchmarking against real-world data and state-of-the-art models. Incorporating a small set of human-labeled examples can further improve overall quality and generalization capability.
What components make up an experiment?
An experiment consists of three components: a Dataset, a Task, and an Evaluator. The dataset provides inputs and expected outputs, the task reproduces the function being tested to generate outputs, and the evaluator scores those outputs to determine success.
Should evaluation failures always block deployment?
Not necessarily. Severe failures in Ground Truth comparisons or dramatic metric spikes should block PRs, but certain evaluation failures are better monitored post-deployment. Finding the right balance to catch critical flaws without creating unnecessary bottlenecks is key.
☑️ Conclusion
Development-stage evaluation serves as a safety net to verify quality prior to deployment. The core objective is to establish evaluation baselines using golden and synthetic datasets, compare changes through experiments, and integrate with CI/CD to catch regressions early. In this installment, we examined dataset construction, experiment setup, result interpretation, and CI/CD automation along with change promotion. The next installment will cover post-deployment evaluation—specifically, production guardrails, self-improvement, and governance.
As the official Korean partner of Arize AI, Cloud Networks supports the adoption and implementation of Arize AX, an AI observability and evaluation platform. If you are interested in establishing a development-stage evaluation framework for your LLM applications, please contact Cloud Networks.
[Source: Arize AI, "Pre Production LLM Evaluation", https://arize.com/llm-evaluation/pre-production-llm-evaluation/ , Arize AI, "CI/CD for LLM Apps", https://arize.com/llm-evaluation/ci-cd-for-llm-apps/ ]
