# Full Stack Deep Learning's LLM Bootcamp 2023: skip the tools, study the LLMOps

> The course was made when LangChain was still new, so most of the tools it names are dated. Its lessons on testing, managing prompts and taking LLM apps to production still match an FDE's job today.

Original: https://fdetimes.net/en/books-courses/llm-bootcamp-2023-llmops-review/

This is an LLM course from April 2023, yet its most memorable advice is mundane: keep your prompts in Git. In the LLMOps lecture, Josh Tobin argues that most teams should do this, because Git is easy to use and already sits in their workflow.

The course's official name is "LLM Bootcamp - Spring 2023", run by The Full Stack (Full Stack Deep Learning). Charles Frye, Sergey Karayev and Josh Tobin taught it in person in San Francisco, then put all the recordings online for free.

Their stated aim was to bring learners up to the state of the art and enable them to build and deploy LLM applications themselves, even without ML experience.

If your job is turning a demo into a system with real users, this course covers exactly that journey, from prompt to production. The catch is that many of the tools it names are dated, so you need to know which lectures to study closely and which to skim.

## What is dated, and what still works?

The prompt engineering lecture teaches problem decomposition and chain-of-thought, getting the model to critique its own answers, combining several runs (ensembling), and common pitfalls.

"Launch an LLM App in One Hour" introduces a toolkit for prototyping and exploration: ChatGPT, LangChain, Colab. All three were 2023 choices, so treat them as examples, not recommendations.

The more lasting value is in the next two parts. The Augmented Language Models lecture covers retrieval with a vector index and how to manage embeddings, followed by tool use.

The askFSDL project walkthrough builds a complete application in production style: tooling for a Python project, data ETL, deployment on Modal, monitoring with Gantry.

The vendor names may change, but the steps stay the same.

| Lecture | Study closely | Skim |
|---|---|---|
| LLMOps | Evals, prompt management, TDD and CI | 2023 tool survey |
| Augmented LMs, askFSDL | Retrieval, ETL, deploy, monitoring | Specific vendor names |
| UX | Copilot and Bing Chat case studies | — |
| Agents (guest) | List of production challenges | AutoGPT, BabyAGI |

The guest lecture on agents mentions AutoGPT and BabyAGI, both now dated. Even so, its list of challenges still holds: controlling tool calls, parsing output, handling long context.

## Idea one: production data always differs from test data

Tobin explains why testing LLMs is hard: the data distribution in production always differs from the data at training time. Imagine building a Q&A system over internal documents for a logistics company. The project team writes 30 sample questions in full, clear sentences, and the system answers almost all of them correctly.

In the first week of real use, warehouse staff type things like "HN batch late why", mixing abbreviations with internal codes. The 30-question test set contains nothing like that, so a high score on it says nothing about real quality. The first thing to do at a client is to log real questions from day one, then add them to the eval set every week.

## Idea two: prompts are code

The LLMOps lecture applies test-driven development and continuous integration to LLMs. Imagine a prompt living in the file `prompts/answer_v3.txt`. Each change to an instruction is a commit, and CI reruns the eval set above. If accuracy drops, you know exactly which line caused it and can roll back to the previous version.

Tobin also observed that production tooling for LLMs was still immature at the time, and his advice for tracking prompts was still Git. The reason the choice is worth keeping: it demands nothing beyond the workflow the team already has, so you can start in the first week of a project.

**Key point:** A prompt without a commit and an accompanying test set is not ready for production.

## Idea three: the language interface is part of the product

The UX lecture discusses how to design language interfaces, using GitHub Copilot and Bing Chat as case studies. Backend engineers often skip this part, but FDEs should not, because you will be the one sitting next to the user when the model gets it wrong.

Before your next demo, sketch the screen the user will see when an answer is wrong, and how they can report it.

## Who should take it, and in what order?

If you are a software engineer moving into FDE work, watch LLMOps first, then Augmented LMs and askFSDL, then UX. Skim prompt engineering and the "one hour" lecture if you have already called an LLM API.

This order puts the hardest part of the job, making the system work correctly on real data, ahead of the more exciting parts.

On your CV, don't just write "completed LLM Bootcamp". Build a small project with retrieval, an eval set drawn from real questions, and CI that runs the evals whenever a prompt changes, then describe exactly those three things.

When reading FDE job descriptions, look for the words "evaluation" and "production": those are what this course teaches best.

The tools in the course will keep ageing. The habit of measuring a system against real data does not, and that is the most valuable thing to take from it.

**Try this week:**

- Watch the LLMOps lecture first, then write down five real questions that users on your current project have typed into the system.
- Move one prompt that lives in code or a shared document into its own file in Git, with a small eval set that runs on every commit.
- Skim the agents lecture, copy its list of production challenges, and check it against the application you are building.

## Sources

- [LLM Bootcamp - Spring 2023 - The Full Stack](https://fullstackdeeplearning.com/llm-bootcamp/spring-2023/)

- [Free Full Stack LLM Bootcamp (KDnuggets)](https://www.kdnuggets.com/2023/06/free-full-stack-llm-bootcamp.html)

- [LLMOps (The Full Stack)](https://fullstackdeeplearning.com/llm-bootcamp/spring-2023/llmops/)