Unstable generative AI? How to build tests with Ragas

Why does generative artificial intelligence feel so unstable?

When we first deploy a generative language model to production, the initial impression is usually excellent. However, as users ask complex questions or underlying context data changes, inaccurate or completely fabricated answers begin to appear. In practice, this happens because probability-based models do not follow rigid programming rules, but instead calculate the most likely next word based on learned patterns.

To make matters worse, changing a simple temperature parameter or updating the model version can completely transform the assistant's behavior. Without a structured testing process, the team remains blind, discovering failures only when the final customer complains. It is precisely to solve this lack of predictability that we need to adopt automated testing pipelines, comparable to traditional software development unit tests.

What is Ragas and how does it solve the validation problem?

Ragas, an acronym for Retrieval Augmented Generation Assessment, is an open-source library specifically designed to evaluate artificial intelligence systems that retrieve external data before responding. In practice, imagine your assistant consulting an internal technical manual to answer an employee. Ragas acts as an automated auditor that checks whether the generated response actually used the manual data, whether it genuinely answers the question asked, and whether the text makes logical sense.

To perform this audit intelligently, the tool itself uses smaller, analysis-focused language models as judges. This means that instead of relying on time-consuming manual checks, the system executes hundreds of test questions in seconds. This automation turns continuous improvement into a viable process, allowing teams to test dozens of prompt variations before pushing any changes to production.

Key metrics to evaluate response quality

Evaluating text generated by artificial intelligence requires looking at different angles of the response. The first essential metric is faithfulness, which measures whether the response relies strictly on the facts provided in the context or if the model invented additional information. When faithfulness is low, we face a classic hallucination scenario, a topic we discuss in detail in our article about how to build strict validation guardrails.

Another fundamental metric is response relevancy, which analyzes whether the generated text truly resolves the user's original query without fluff. We also measure context recall precision, checking whether the search system found the correct documents in the database before feeding the main model. Together, these metrics form a complete assistant health dashboard, pointing out exactly where the flow is failing when quality drops.

How to implement your first automated test pipeline

Creating an automated evaluation pipeline with Python and Ragas is a straightforward process that can be integrated into your continuous integration tools. The first step involves defining a test dataset containing real questions, expected contexts, and reference answers considered correct. Next, we configure the script to send these questions to your AI assistant, collecting the generated responses for later analysis.

Below is a basic Python code example demonstrating how to structure a test dataset and run faithfulness and relevancy evaluations using the library:

from datasets import Dataset
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy

# Example data collected from your assistant
data = {
    "question": ["How do I reset the system password?"],
    "answer": ["To reset, go to profile settings and click on forgot my password."],
    "contexts": [["Password reset instructions are available in the user profile screen."]]
}

dataset = Dataset.from_dict(data)

# Running the automated evaluation
result = evaluate(
    dataset=dataset,
    metrics=[faithfulness, answer_relevancy]
)

print(result)

This script runs in a few seconds and returns numerical scores from zero to one for each evaluated metric. If you combine this testing routine with local execution environments, such as those we describe in the guide on how to run local AI with Llama.cpp and Ollama in a homelab, you can validate model changes without incurring external API costs. To explore these architectural concepts further in production, see our related technical analysis on Slow MySQL queries? How to optimize composite indexes.

Final considerations on generative system stability

Ensuring the reliability of generative artificial intelligence applications is no longer a differentiator; it has become a basic engineering requirement. Adopting structured testing frameworks like Ragas allows technical teams to replace subjective uncertainty with concrete quality data. By continuously monitoring faithfulness and relevance metrics, you catch regressions before they affect end users, keeping the system safe and predictable.

Ultimately, the stability of an intelligent assistant depends directly on the discipline with which its workflows are tested and audited. As project complexity grows, investing in evaluation automation saves hours of manual debugging and protects product reputation. Start by integrating small test suites into your current workflow and gradually expand coverage as new needs arise.