The DeepLearning.AI course that makes your LLM hallucinate first
In under two hours, it builds a text-to-SQL agent and teaches the order for fixing it, which every forward deployed engineer should know by heart.
In brief
- A short intermediate course, 1 hour 51 minutes long, taught by Sharon Zhou (Lamini) and Amit Sangani (Meta).
- The lessons open by deliberately making a text-to-SQL agent running on Llama 3-8b hallucinate, which starts the evaluation process.
- Fine-tuning with LoRA and MoME comes last, after prompting and self-reflection.
This 1-hour-51-minute course asks you to do something almost nobody does while preparing a demo: make your own agent hallucinate on purpose. You build an agent that turns questions into SQL, then deliberately set up situations where it answers wrongly, so that evaluation can begin.
That is how Improving Accuracy of LLM Applications opens. It is an intermediate-level short course from DeepLearning.AI, with 7 videos and 4 code examples. It was announced on DeepLearning.AI’s community forum on August 14, 2024, and was made with two partners, Lamini and Meta.
For an FDE, the course is worth an evening. The problem is set out cleanly: build a SQL agent, then attach evaluation metrics to find out how well it performs. If a client has ever complained that their agent is “right some of the time”, this is the skill you need.
Two instructors, one very realistic problem
Sharon Zhou is co-founder and CEO of Lamini. Amit Sangani is Senior Director of Partner Engineering at Meta. Coursera also offers the course as a guided project.
The project running through the course uses Llama 3-8b to build a text-to-SQL application on a custom schema. It is the kind of problem FDEs regularly meet on site with a client.
Client databases tend to have their own table and column names that the model has never seen. Practicing on a custom schema is close training for that situation.
Break it first, measure it carefully after
The course starts by simulating hallucinations, producing the first failure cases as a starting point for evaluation. Only then comes the main sequence: building and wiring up the full application, prompt engineering with self-reflection, then rigorous evaluation.
At first glance evaluation seems to come after the prompt fixes, but there are really two layers of it. The hallucinations at the start are a quick test to see where the agent breaks. The evaluation framework later is something the course page asks you to build yourself, to measure performance systematically.
A hypothetical example shows why the second layer cannot be skipped. Say you collect 20 real questions from the sales team along with the correct SQL, and the agent gets 14 right, or 70%.
After a prompt change it gets 17 right, or 85%, but now fails two questions it used to answer correctly. Without that set of 20, all you see is a few tries that look “a bit better”, and you carry a new bug into the client meeting.
So the first thing to do at a client is ask for real questions and correct answers confirmed by the users themselves. That small dataset becomes the yardstick for every later change.
Fine-tuning comes last, for good reason
In the course, fine-tuning is the next step, after prompting and self-reflection. The order makes sense from a deployment point of view: changing a prompt takes minutes, while fine-tuning needs data, infrastructure and time waiting for results.
When fine-tuning’s turn comes, the course uses an LLM itself to generate the dataset, then introduces instruction fine-tuning and PEFT techniques such as LoRA and Mixture of Memory Experts (MoME). It also covers Lamini’s memory tuning, a way of embedding facts into the model to reduce hallucinations.
The course’s makers call memory tuning a new direction for factual accuracy. That praise comes from the company that built the technique, so verify it with evals on your own problem. Synthetic data generated by an LLM and left unchecked can also teach the model exactly the errors you are trying to fix.
Who should take it, and in what order
The course is rated intermediate and taught through code, so it suits people who write Python regularly and have built at least a prototype LLM application. If you are new to LLMs, get comfortable with basic prompting first.
If time is short, watch the section on building the evaluation framework first, apply it straight away to a current project, then go back to prompting and fine-tuning. The fine-tuning material is most useful once you have a baseline to compare against.
On the career side, when reading FDE job ads, treat mentions of evals or accuracy measurement as a sign the role needs exactly this chain of skills. In interviews, explaining the order you followed, from failure cases to prompts to evals and only then fine-tuning, and why, is more persuasive than listing technique names.
The same goes for a CV: a line stating how many questions your eval set had and how far accuracy rose says more than “experience with LLMs”. Use real scores from your own project; the 70% and 85% above are only illustrative.
The course’s greatest value lies not in LoRA or MoME but in the order: cheap fixes first, expensive ones later, and fine-tuning only when the numbers show prompting has hit its ceiling. An FDE who knows that order will rarely promise a client something they have not measured.
Was this article useful?
Thanks for the feedback!
5 sources
- Improving Accuracy of LLM Applications - DeepLearning.AI
- New Course! Enroll in Improving Accuracy of LLM Applications · 2024-08-14
- ksm26/Improving-Accuracy-of-LLM-Applications (GitHub, student repo of the course)
- Improving Accuracy of LLM Applications - Lesson 1: Introduction (DeepLearning.AI Learn)
- Improving Accuracy of LLM Applications - Coursera