Quality and Safety for LLM Applications: four lessons on LLM risk, best studied alongside the tools that measure it

The longest lesson in Quality and Safety for LLM Applications is not about hallucinations or jailbreaks. It is “Passive and active monitoring”, at 28 minutes longer than any other. That running time alone suggests a point of view: the safety of an LLM application has to be tracked continuously, not tested once and forgotten.

For a forward deployed engineer this is daily work. The demo runs smoothly on sample data. Then, in the first week of real client data, someone asks: “How do we know it isn’t making things up?” The best answer is not reassurance but a number.

Who teaches it, and what are the four lessons?

The course is part of DeepLearning.AI’s short-course series, made with WhyLabs. The instructor is Bernease Herman, a Data Scientist at WhyLabs. The company makes LangKit, an open-source toolkit that computes text metrics from a language model’s inputs and outputs for monitoring.

The course is beginner level and runs 2 hours 16 minutes. Its stated goals are fairly concrete: detect and prevent hallucinations, jailbreaks and data leakage, then monitor and improve the safeguards over time.

The core is four videos: “Hallucinations” at 25 minutes, “Data Leakage” at 19 minutes, “Refusals and prompt injections” at 26 minutes, and the monitoring lesson at 28 minutes. A graded quiz closes the course. The whole thing fits into an afternoon.

To turn each lesson into a skill you can use straight away, read three things alongside it: the SelfCheckGPT paper, the LangKit source code, and Openlayer’s guide to PII detection. The measurement techniques below come from those sources.

A model that is making things up will contradict itself

The hallucinations lesson pairs well with SelfCheckGPT, a method published by Manakul, Liusie and Gales in March 2023. Its strength is that it detects hallucinations without an external reference database. It relies only on consistency across sampled answers.

The idea is simple. If the model really knows something, repeated samples will give the same answer. If it is inventing, the answers tend to diverge.

Imagine you are deploying an internal chatbot for an insurance company. A user asks for the deadline to file a claim. You call the model 5 times and all 5 say “30 days”: that answer is more trustworthy.

If instead you get 30, 45, 30, 60 and 14 days, that is a warning sign, even though each answer on its own sounds entirely confident.

On a client site you rarely have a ready-made answer key, so this technique is usable immediately. The first thing to do: take a small sample of real questions, measure the share of answers that diverge, and bring that number to the meeting in place of a feeling that “it seems fine”.

Prompt injection can be measured as a number too

LangKit scores prompt injection risk by how similar a prompt is to known attacks. A vague risk becomes a number you can set thresholds on, log and chart.

Because the score is based on similarity to known attacks, an entirely new kind of attack may score low. Even so, for enterprise clients a measurable filter is far better than a system prompt telling the model “don’t do anything bad”.

Data leakage starts with a regular expression

“Data Leakage” is the shortest of the four core lessons, but it touches the client’s biggest worry when real data goes into the system. The simplest way to turn that worry into a number is to count.

LangKit has a patterns metric that counts strings matching a set of regexes you define yourself. WhyLabs’ LangKit container detects PII such as email addresses, phone numbers and credit cards by default, in both prompts and responses.

Go back to the insurance client above, and suppose their policy numbers have a fixed format. You write a regex for that format, add email and phone numbers, and run it over the first week’s 1,000 responses. Three match, so the leakage rate is 0.3%.

The WhyLabs container also lets you set a threshold for each metric, so that number becomes a pass or fail report.

Regex has blind spots. Openlayer notes that it misses free-form PII, such as a person’s name in the middle of a sentence or a truncated identifier, so it needs to be combined with other detectors. Red Hat OpenShift AI also uses regex detectors in its guardrails for inputs and outputs. Regex is the first line of defence, not the only one.

Why is the monitoring lesson worth the closest study?

Real users always differ from the people who write the tests. A test suite written in week one can hardly anticipate everything the client’s staff will type in month three.

Openlayer observes that PII leakage rates change over time, and production monitoring is where you see that change. So the 0.3% figure should be measured every week, not just once.

There is one more boundary to make clear to clients. According to Openlayer, logging is only observation; blocking at inference time is enforcement. The advice for FDEs: agree from the start which metrics only need an alert and which must block the response once they cross a threshold.

Who should take it, and in what order?

The course suits engineers who have already built an LLM application, such as a simple RAG system, and now have to defend it in front of a client. If you have never called a model API, build a small project first so you have your own data to try the techniques on.

Take the hallucinations lesson first, then prompt injection and data leakage. Leave monitoring for last: by then you will have measured a few metrics yourself from the first three lessons, so you will have something concrete to monitor. Take the quiz after you have tried at least one technique on your own data.

On the career side, look for FDE job postings that mention evals, guardrails or monitoring for LLMs. Don’t write “completed the course” on your CV. Write what you measured, for example “tracked weekly PII leakage rate and self-contradiction rate on an internal chatbot”.

Clients won’t ask which course you took. They will ask how many times the model leaked data last week, and a good FDE is the one who can answer with a number.