# Counting a client's error logs is useless until you normalize them

> A few dozen lines of Python with re and Counter can turn a messy log file into a ranked table of errors you can take to the client meeting.

Original: https://fdetimes.net/en/guides/normalize-error-logs-before-counting/

According to Python's logging documentation, `Logger.exception()` writes the stack trace alongside the message. That is why the log files clients send are rarely tidy, one line per error: a single error can drag ten lines of traceback behind it, and none of those lines starts with a timestamp or a log level.

There is a second problem. Error messages usually contain order numbers, millisecond values and memory addresses. Count the raw messages directly and the same timeout error splits into hundreds of groups that each appear once, so the table tells you no error stands out.

For a forward deployed engineer, the first question when a client sends logs is almost always "which error happens most?". This guide builds a small Python script that answers it using only the standard library: `re` and `collections.Counter`. You need Python 3 and a log file to try it on.

## Start with a sample log file

The format below is a made-up, simplified example. A client's real logs will differ, but the structure is usually similar: structured fields at the start of the line, free text at the end. Save it as `app.log`:

```text
2026-11-03 09:14:22 [ERROR] [payments] Timeout calling bank API order=84213 after 3000ms
2026-11-03 09:14:25 [INFO] [api] GET /orders/84213 200
2026-11-03 09:14:31 [ERROR] [payments] Timeout calling bank API order=90117 after 3012ms
2026-11-03 09:15:02 [ERROR] [worker] Failed to process job 5521
Traceback (most recent call last):
  File "worker.py", line 42, in run
KeyError: 'customer_id'
2026-11-03 09:15:40 [ERROR] [worker] Failed to process job 5530
Traceback (most recent call last):
  File "worker.py", line 42, in run
KeyError: 'customer_id'
```

Note the six lines that do not begin with a timestamp. They are the tracebacks for the two failures on jobs 5521 and 5530, and each block has to travel with the line above it.

If the client uses Python's default configuration, the format is even simpler: `severity:logger name:message`, according to the logging documentation. The standard levels run from DEBUG to CRITICAL, so you already have a list to filter on.

## Step 1: a regex you can still read in three months

Create a file called `logtop.py`. Write the regex as a raw string (the `r` prefix) so Python does not process backslash sequences such as `\d` before the regex engine sees them. Use `re.VERBOSE` to break the pattern into commented pieces:

```python
import re
from collections import Counter

LINE = re.compile(r"""
    ^(?P<ts>\d{4}-\d{2}-\d{2}\ \d{2}:\d{2}:\d{2})\s+  # timestamp
    \[(?P<level>.+?)\]\s+                             # log level
    \[(?P<logger>.+?)\]\s+                            # logger name
    (?P<msg>.*)$                                      # free-text message
""", re.VERBOSE)
```

Each `(?P...)` is a named group, a Python-specific syntax that lets you pull out a field by name rather than by position. The pattern is compiled once with `re.compile()` at the top of the file; the `re` documentation states that this is more efficient when the same expression runs many times, which here means millions of lines.

The detail that is easy to miss is the `?` after `.+`. It makes the quantifier non-greedy, so it matches as few characters as possible. On the sample lines above, the regex still splits correctly without the `?`, because the matcher backtracks until the rest of the pattern can match.

The trouble starts when the message itself contains square brackets. Try a line where the part after the timestamp is `[ERROR] [api] Rejected [role] [admin] for user 7`. The greedy version stops at the furthest `]` that still allows a match, giving `level` as `ERROR] [api] Rejected [role` and `logger` as `admin`. The non-greedy version stops at the first `]` and correctly returns `ERROR` and `api`.

One more caveat: in VERBOSE mode whitespace is ignored, so the space between the date and the time must be written as `\ `.

**Check:** run `LINE.match(...)` on the first line in the REPL, call `.groupdict()` and confirm that `level` is `ERROR`. Then try the line with brackets in the message above.

## Step 2: attach tracebacks to the right error

`groupdict()` returns every named group as a dictionary, which is just what you need to turn a line into a record. Any line that does not match the pattern is treated as a continuation of the previous record:

```python
def parse(path):
    records = []
    with open(path, encoding="utf-8") as f:
        for raw in f:
            line = raw.rstrip("\n")
            m = LINE.match(line)
            if m:
                rec = m.groupdict()
                rec["trace"] = []
                records.append(rec)
            elif records:
                records[-1]["trace"].append(line)
    return records
```

**Check:** the sample file has 11 lines, but you should get 5 records, and the records for jobs 5521 and 5530 should each have a `trace` three lines long. If you get more than 5, the regex is wrongly matching traceback lines.

## Step 3: normalize first, then count

This step decides the whole result table. `re.sub` replaces every occurrence of a pattern unless you pass the `count` argument, so each rule needs only one call:

```python
RULES = [
    (re.compile(r"0x[0-9a-fA-F]+"), "<HEX>"),
    (re.compile(r"\d+"), "<NUM>"),
]

def normalize(text):
    for pat, placeholder in RULES:
        text = pat.sub(placeholder, text)
    return text

def signature(rec):
    sig = normalize(rec["msg"])
    if rec["trace"]:
        sig += " | " + normalize(rec["trace"][-1].strip())
    return sig
```

The order of the rules matters: replace digits first and `0x7f3a` becomes `xfa`, leaving the HEX rule nothing to catch. After this step, both timeout lines in the sample file become `Timeout calling bank API order= after ms`.

This is where the tracebacks attached in step 2 earn their place. In the sample file, the message "Failed to process job" does not say why the job failed; the last line of the traceback does: `KeyError: 'customer_id'`.

Append that line to the signature and the two worker errors become `Failed to process job  | KeyError: 'customer_id'`. If the same message later arrives with a different exception, it forms its own group instead of being lumped in.

Do the arithmetic on the sample file. Counted raw, the four ERROR lines produce four groups of one each, which tells you nothing. After normalization there are two groups, each appearing twice. On a real log tens of thousands of lines long, that gap separates a useless table from a list of things to fix.

**Key point:** Don't count error messages. Count their shape once every variable part has been stripped out.

## Step 4: rank, with where and when each error started

`Counter` is a dict subclass for counting hashable objects, and signature strings are hashable. `most_common(n)` returns the n most frequent elements with their counts, ordered from most to least. Alongside the counting, the script keeps the earliest timestamp for each signature and the logger of that occurrence:

```python
if __name__ == "__main__":
    records = parse("app.log")
    errors = [r for r in records if r["level"] in ("ERROR", "CRITICAL")]
    counts = Counter()
    first = {}
    for r in errors:
        sig = signature(r)
        counts[sig] += 1
        if sig not in first or r["ts"] < first[sig][0]:
            first[sig] = (r["ts"], r["logger"])
    for sig, n in counts.most_common(10):
        ts, logger = first[sig]
        print(f"{n:>6}  {ts}  [{logger}]  {sig}")
```

Comparing `r["ts"]` as a string works only because the sample format runs from year to second; if the client's logs record time differently, convert to `datetime` before comparing. This is also a simplified version: each signature keeps only the logger of its first occurrence, so if the same error comes from several loggers you will need a separate `Counter` for loggers.

Run `python3 logtop.py`. **Check:** the output should look like this:

```text
     2  2026-11-03 09:14:22  [payments]  Timeout calling bank API order=<NUM> after <NUM>ms
     2  2026-11-03 09:15:02  [worker]  Failed to process job <NUM> | KeyError: 'customer_id'
```

If you see four lines, the normalization rules did not run or are in the wrong order. If the worker error splits in two, check that step 2 attached the tracebacks to the right records.

## Mistakes that make the numbers mislead you

Forgetting the raw string is the most common trap. Without the `r` prefix, some escape sequences such as `\b` are interpreted by Python first, and the regex silently matches the wrong thing without raising an error. Leaving bracketed fields greedy is dangerous in the same way, as the `[role] [admin]` line in step 1 shows.

Harder to spot is over-normalization. The `\d+` rule turns `HTTP 502` and `HTTP 404` into the same signature, though for the client they are very different problems. When the signature at the top of the table looks too generic, open a few of the original records and read them before drawing conclusions.

## When hand-written regex is no longer enough

A hand-written rule set works well when the client has one or two services. When logs come from dozens of services in different formats, the `RULES` list grows faster than you can maintain it. That is the point to look at Drain3, an open-source library that learns log templates from raw logs as they stream in.

The idea behind Drain is close to what you have just written. The algorithm stores learned patterns in a parse tree and replaces tokens that do not match with the wildcard `<*>`, just as `` stands in for an order number.

The Drain3 documentation also advises feeding in only the free-text part, without timestamps or hostnames, so the regex from step 1 remains useful for cutting out the `msg` field first.

## Take the top-errors table into the meeting

Having a top-10 error table at the first session gives you a concrete story to open the meeting with: this error occurs most, it comes from this logger, it started at this time, and the root cause sits in the last line of the traceback. Write the script so that only the file path and the line regex need changing, because every client's log format is different.

If you are aiming for an FDE role, put the script on GitHub with an anonymized sample log, and describe it on your CV in terms of results: the number of log lines processed, and the number of error groups before and after normalization.

When reading job ads, watch for lines about handling production incidents or customer issues: that is where to bring up this tool in an interview.

The next time a client sends a log file of a few hundred megabytes with the note "the system is really slow", you will have an answer in numbers before the second call begins.

**Try this week:**

- Take a real log file from a project you are working on, run the script from this guide and record the top 10 error signatures.
- Add a new normalization rule (for UUIDs or email addresses, say) and compare the number of groups before and after.
- Package the script as a command-line tool that takes a file path, and put it on GitHub with an anonymized sample log.

## Sources

- [Regular expression HOWTO — Python 3.15.2 documentation](https://docs.python.org/3/howto/regex.html)

- [re — Regular expression operations — Python 3.15.2 documentation](https://docs.python.org/3/library/re.html)

- [collections — Container datatypes — Python documentation](https://docs.python.org/3/library/collections.html)

- [Logging HOWTO — Python 3.15.2 documentation](https://docs.python.org/3/howto/logging.html)

- [logpai/Drain3 (GitHub)](https://github.com/logpai/Drain3)

- [Drain: simgple but effective log parsing algorithm](https://martinlwx.github.io/en/log-parsing-algorithm-drain/)
