# A support-ticket classifier that trains just 0.07% of the model

> LoRA needs no GPU cluster; what decides the result is consistent labels and a held-back test set that keeps the final score honest.

Original: https://fdetimes.net/en/guides/lora-fine-tune-support-ticket-classifier/

In the healthcheck example in the PEFT library, a Llama-3.2-1B model has 1,236,666,368 parameters, but only 851,968 of them are trained: 0.0689%. All the others stay frozen.

The exact figure depends on the model and the configuration. Still, it shows why an engineer working on a client's site can fine-tune a model for a narrow problem without a GPU cluster.

Support ticket classification is that kind of narrow problem: the input is a piece of text and the output is one label from a fixed list. If you are moving into a Forward Deployed Engineer role, doing this end to end and being able to explain every decision gives you a complete project to present in interviews.

This guide shows how to build a ticket classifier with LoRA on a small language model. You finish with a compact adapter and an evaluation number you can trust.

## What do you need before running anything?

You need Python, Hugging Face's `peft` library with the `transformers` ecosystem that comes with it, and a small base model. Opt-350m and Llama-3.2-1B, the two models the PEFT documentation uses in its examples, are both sensible choices for a laptop with a GPU or a small cloud machine.

Labelled data matters more than the GPU. Supervised fine-tuning requires every training example to come with the correct answer, called the target. For tickets, the target is the category: for example "thanh toán" (payments), "đăng nhập" (login), "giao hàng" (delivery) or "khác" (other). The examples below use these Vietnamese labels.

Fine-tuning means adapting a pre-trained model to a specific task rather than training one from scratch. So you do not need millions of tickets. You need consistent labels.

## Step 1: Format tickets as input–label pairs

Take the simple route and treat classification as short text generation: the model reads a ticket and generates the label name. This matches `TaskType.CAUSAL_LM` in PEFT's official example. Each JSONL line might look like this (the prompt format is up to you). The first ticket reads "I was charged twice for order #4821", the second "I didn't receive the OTP code when logging in":

```json

    {"text": "Ticket: I was charged twice for order #4821.\nLabel:", "label": "billing"}

    {"text": "Ticket: I didn't receive the OTP code when logging in.\nLabel:", "label": "login"}

```

**Check:** count tickets per label. If one label accounts for most of the data, note it now, because it will make the accuracy figure in the evaluation step look better than it really is.

## Step 2: Split the data and lock away the test set

This step decides whether your result can be trusted. Split the data three ways: train for learning, validation for tuning hyperparameters, and test, used exactly once after every decision has been made.

```python
import random
random.seed(42)
random.shuffle(tickets)
n = len(tickets)
train = tickets[: int(n * 0.70)]
val   = tickets[int(n * 0.70): int(n * 0.85)]
test  = tickets[int(n * 0.85):]
```

Picture 2,000 tickets: that gives you 1,400 for training, 300 for validation and 300 for testing. Write the test file to disk and do not open it again until step 5.

**Check:** all three sets contain every label. If tickets share a customer or an email thread, consider putting them in the same set so that information does not leak between sets.

## Step 3: Configure LoRA and wrap the base model

The snippet below uses exactly the values from the PEFT quicktour, with the arguments reordered for readability. The variable `base_model` is the model you have already loaded (the loading code is left out for brevity):

```python
from peft import LoraConfig, TaskType, get_peft_model

config = LoraConfig(
    r=8,
    lora_alpha=32,
    lora_dropout=0.1,
    target_modules=["q_proj"],
    task_type=TaskType.CAUSAL_LM,
    inference_mode=False,
)
model = get_peft_model(base_model, config)
```

The two parameters you will adjust most are `r` and `target_modules`.

Raising `r`, the rank, means more parameters to train and more capacity to learn, at the cost of more compute and a greater risk of overfitting. `target_modules` decides which layers LoRA attaches to. Here it is only `q_proj`. To cover every linear layer, as QLoRA does, set `target_modules="all-linear"`, and the share of trainable parameters will rise.

`lora_alpha=32` and `lora_dropout=0.1` come from the official example. On a first run, leave both alone so that each experiment changes only one variable (`r` or `target_modules`) and the comparison table in step 4 stays readable.

**Check:** print the number of trainable parameters against the total. The result should be a very small fraction, of the same order as the 0.0689% in the Llama-3.2-1B example. If it is close to 100%, LoRA has not been applied correctly and you are fine-tuning the whole model.

**Key point:** The original weights are frozen, so what you produce is only the difference, and that difference can be detached, swapped in or assigned to a single task.

## Step 4: Train, but read validation, not training loss

With `CAUSAL_LM`, the simplified picture is that the model learns to predict the next piece of text. So each training example is the input string joined to the label. At prediction time you supply only the "Ticket: ...\nNhãn:" part (Nhãn means label) and read what the model generates.

The sketch below covers only the data and scoring. For the training loop, use the familiar trainer from the Hugging Face ecosystem; `predict` is a function you write yourself to have the model generate a label:

```python
# Simplified sketch
def to_training_text(t):
    return t["text"] + " " + t["label"]

train_texts = [to_training_text(t) for t in train]

def accuracy(rows, predict):
    correct = sum(1 for t in rows if predict(t["text"]).strip() == t["label"])
    return correct / len(rows)

def majority_baseline(rows):
    labels = [t["label"] for t in rows]
    top = max(set(labels), key=labels.count)
    return labels.count(top) / len(rows)
```

The last function is the one many people skip. Suppose 150 of the 300 validation tickets are labelled "thanh toán": a lazy "model" that always answers "thanh toán" already scores 50%. If your LoRA gets 240 out of 300, or 80%, what matters is the 30-point gap over the baseline, not the 80% on its own.

The most common trap is a training loss that falls steadily to a very low level, followed by a hasty report of success. A model can reach low training loss simply by memorising the data. With 1,400 tickets and many epochs, that is entirely possible.

Picture this: after epoch 3, training loss keeps falling but accuracy on the 300 validation tickets stalls or even drops. That is the signal to stop. At this point you are free to try alternatives, such as fewer epochs, a different `r` or switching to `all-linear`, as long as every comparison is made on validation.

**Check:** keep a small table of configuration, validation accuracy and baseline. When the client asks "why r=8?", that table is your answer.

## Step 5: Open the test set exactly once, then save the adapter

Once the configuration is fixed, run `accuracy(test, predict)` on the 300 test tickets and record the result. That is the number you report. If it disappoints, do not go back, tweak and rerun the test: doing so turns the test set into a second validation set and the number loses its value.

Save the adapter and you will find it surprisingly small. The PEFT documentation says the opt-350m adapter is only about 6MB, against roughly 700MB for the full weights.

## Where do first attempts usually go wrong?

A common reflex is to raise `r` as soon as results look weak. A higher rank gives the model more capacity, but with a few thousand tickets it also makes memorisation easier. Check the labels before raising the rank.

Inconsistent labels are the next source of error. Two support agents may file the same ticket, "charged but can't log in", under different categories, and the model can only learn as well as the labels allow.

The hardest to spot is peeking at the test set. The number still looks normal; it just can no longer be trusted.

## What does this skill look like on a client site?

Hugging Face explains the reason for PEFT plainly: fine-tuning every parameter again for each task becomes increasingly expensive and impractical. On a deployment project this is very concrete. The client has one ticket queue, later adds a second, and then a business unit wants its own set of labels.

Because the base model is frozen and the adapter is only the difference, you can keep one base model and swap in several LoRAs by task. Picture three classifiers for three departments: the total cost is a few small adapters, not three full copies of the model. That is the answer the client's infrastructure team wants to hear.

The first thing to do on arrival at a client is not training. Sit down with the support lead to agree a definition for each label, then pull a sample of tickets to check whether the existing labels are consistent. Most of the project's risk lies in that step, not in `LoraConfig`.

If you are aiming for an FDE role, look for keywords such as PEFT, LoRA, fine-tuning and evaluation in job descriptions. On your CV, do not just write "fine-tuned an LLM". Spell out the share of trainable parameters, the adapter size, the baseline you compared against, how you split the data and the fact that the test set was run only once.

Anyone can run `get_peft_model`. What clients pay for is an evaluation number they can believe.

**Try this week:**

- Collect 1,000–2,000 public or self-made tickets, label them by category, split 70/15/15 and lock away the test file
- Run the LoRA configuration from the PEFT quicktour (q_proj, r=8), then try target_modules="all-linear" and compare on validation
- Push the repo to GitHub with a README stating the number of trainable parameters, the adapter size, the most-frequent-label baseline and the test result from a single run

## Sources

- [What is fine-tuning? | IBM](https://www.ibm.com/think/topics/fine-tuning)

- [Quicktour (PEFT documentation)](https://huggingface.co/docs/peft/main/en/quicktour)

- [LoRA (PEFT documentation)](https://huggingface.co/docs/peft/main/en/package_reference/lora)

- [What Is Machine Learning Model Training?](https://domino.ai/blog/what-is-machine-learning-model-training)

- [Training ML Models - Amazon Machine Learning](https://docs.aws.amazon.com/machine-learning/latest/dg/training-ml-models.html)
