Ragas and DeepEval: two libraries for scoring RAG, and why their scores deserve suspicion

Ragas can score a RAG pipeline without anyone writing the correct answers in advance. The original paper by Shahul Es and co-authors, first published in September 2023, calls this “reference-free” evaluation: no human-labelled ground truth is required.

When a client asks a forward deployed engineer “what percentage of the time is this bot right?”, that feature is both a help and a trap.

It helps when the client has no ready-made set of questions with human-labelled answers, because Ragas does not need one to get started. It is a trap because, without human labels, the thing doing the scoring is usually another LLM.

A number with two decimal places looks precise, but it is only as reliable as the model that produced it.

The right way to use Ragas and DeepEval, then, is for fast signals and for catching regressions. Do not use them to put a number on a slide for the board before anyone has checked it.

Ragas measures, DeepEval tests

Ragas describes itself as a library for evaluating LLM applications. The repository now sits under the vibrantlabsai organisation on GitHub and is licensed under Apache-2.0. It offers both LLM-based and traditional metrics, lets you write custom metrics with a decorator, and can generate synthetic test datasets covering a range of scenarios.

Microsoft’s playbook splits Ragas’s RAG metrics into two groups. Faithfulness and answer relevancy measure generation. Context relevancy and context recall measure retrieval. Faithfulness in particular penalises every claim in an answer that cannot be inferred from the context.

DeepEval takes a different approach. Its developers position it as “Pytest for LLM applications”: open source, local-first, Apache 2.0 licensed, with more than 50 built-in metrics (including LLM-as-a-judge), and able to evaluate both end to end and component by component.

The core idea is to write Pytest-style assertions against LLM output and run them in CI.

Where to start with an internal knowledge bot

Imagine you are deploying a bot that answers questions about internal procedures at a bank, and you have 50 real questions submitted by staff. Do not run faithfulness straight away. Measure hit rate first. According to Evidently AI, it is a binary metric that simply checks whether at least one relevant passage made it into the top K, and it needs no LLM judge.

Suppose 35 of the 50 questions have the right passage in the top 5: a hit rate of 70%. For the other 15, no prompt, however good, will let the bot answer correctly, because the documents it needs never reach the model. The fix lies in chunking, embeddings or search.

Only once retrieval is sound should you measure generation. Suppose an answer contains four claims, one of which is not in the context; faithfulness will penalise exactly that claim. This is where Ragas earns its keep: it separates “retrieved the wrong documents” from “made things up”, two failures that call for different fixes.

Ragas’s synthetic test sets can extend those original 50 questions. The final step is to turn the questions the bot once got wrong into DeepEval tests with score thresholds. If the client changes the chunk size in week three, CI will show immediately which questions broke, before users start complaining.

Who grades the grader?

DeepEval’s faithfulness metric checks whether the output is factually consistent with the retrieval context. G-Eval lets you score against custom criteria. Both are scored by an LLM, and Microsoft warns that judges of this kind have reliability problems, including position bias, length bias and self-enhancement bias.

Evidently AI is more specific: a judge may favour answers that are longer, more polished, or closer to its own style. A judge is only as good as its prompt and underlying model. A faithfulness score of 0.85 is therefore not a fact but the opinion of one particular model with one particular prompt.

Microsoft also says plainly that human review is still needed, especially for domain-specific tasks. In practice: take 20 to 30 questions, ask a client-side expert to grade them by hand, and compare the results with the judge’s scores. If the two diverge significantly, fix the criteria in G-Eval before trusting any dashboard.

The most common mistakes

The first common mistake is fixing the prompt when the problem is retrieval. In the bank example above, a week of prompt rewrites will not rescue the 15 questions whose correct documents never made the top 5. If hit rate is low, go back to chunking and search first.

The second is trusting a synthetic question set that nobody has reviewed. Machine-generated questions may not resemble how bank staff actually ask things, so have someone on the client side skim them and remove the off-target ones before they go into the test set.

The third is letting the same model both generate the answers and score them. Given the self-enhancement bias that Microsoft and Evidently AI describe, this tends to produce flattering scores. The last is reporting judge scores to the client without checking them against any hand-graded sample.

What to learn first, and what to put on your CV

A sensible learning order: retrieval metrics that need no judge, then faithfulness and context recall, and finally wiring evaluation into CI. When a job description mentions “evaluation”, “eval harness” or “LLM-as-a-judge”, have a concrete example of an eval suite you have built ready to discuss in the interview.

On your CV, write something like “built a 50-question eval suite, separated retrieval errors from generation errors, checked judge scores against human graders” rather than “experience with Ragas”. The first shows the reader you know where to be sceptical; the second only shows you have installed a library.

Calling a library takes an afternoon to learn. Explaining to a client why a score of 0.85 is not enough to go live is the hard part, and the part clients most need an FDE for.