Hypothesis in practice: run a customer data parser through 500 generated inputs

Imagine a customer sends an orders file containing a refund line that reads -3.50. Your parser reads it as -2.50. The hand-written tests pass, and so does the round-trip test. The bug only surfaces when the customer’s accountants reconcile at month end.

A customer data parser deserves more testing than its simple appearance suggests. You may write it from a handful of sample files, but afterwards it has to run every month on data you have never seen.

Hypothesis is a Python library for property-based testing. Instead of guessing each example in advance, you describe a whole range of scenarios and let the machine explore the possibilities.

What follows sticks to a single example from start to finish. You will write an order-line parser, generate a few hundred structured lines, see how a property that looks airtight still misses a bug, and then pin the customer’s real broken line into the test.

What do you need before you start?

You need Python 3, pytest and Hypothesis installed in a virtualenv (usually pip install hypothesis pytest). The data format in this article is hypothetical: each line holds an order ID, a date and an amount, separated by semicolons, for example ORD-00042;2024-03-07;12.50. The code is simplified for illustration and is not a production parser.

Step 1: a parser that looks correct

# parser.py — simplified for illustration
from dataclasses import dataclass

@dataclass
class Order:
    order_id: str
    date: str
    amount_cents: int

def parse_line(line: str) -> Order:
    order_id, date, amount = line.strip().split(";")
    whole, _, frac = amount.partition(".")
    cents = int(whole) * 100 + int((frac + "00")[:2])
    return Order(order_id, date, cents)

def format_line(o: Order) -> str:
    return f"{o.order_id};{o.date};{o.amount_cents // 100}.{o.amount_cents % 100:02d}"

For 12.50, the function returns 1250. For 12, it returns 1200. Unit tests on a few sample lines will all pass, and that is exactly where the trap lies.

Step 2: describe the data with strategies, not examples

The hardest part of property-based testing is generating complex inputs. Hillel Wayne has observed that the PBT literature rarely discusses this part.

To combine several strategies, use @composite: a function with this decorator takes draw as its first argument and calls it to pull values from each smaller strategy.

Each field is generated with st.from_regex and fullmatch=True, so the whole generated string matches the pattern.

# test_parser.py
from hypothesis import given, settings, strategies as st
from parser import Order, parse_line, format_line

@st.composite
def orders(draw):
    order_id = draw(st.from_regex(r"ORD-[0-9]{5}", fullmatch=True))
    date = draw(st.from_regex(r"20[0-9]{2}-[01][0-9]-[0-3][0-9]", fullmatch=True))
    cents = draw(st.integers(min_value=-10_000_000, max_value=10_000_000))
    return Order(order_id, date, cents)

@settings(max_examples=500)
@given(orders())
def test_round_trip(order):
    assert parse_line(format_line(order)) == order

The @given decorator turns an ordinary function into a Hypothesis test. By default each test runs 100 examples; max_examples=500 raises that number. Run pytest and the test passes.

If you want a ready-made test skeleton to edit, the hypothesis write command (the Ghostwriter) will generate one for you. The Ghostwriter documentation calls round-trip a very strong property, especially when configuration options vary along with the object.

Step 3: why does the test pass when the parser is wrong?

Send -250 cents round the loop. Python’s // rounds down, so -250 // 100 gives -3, while % takes the sign of the divisor, so -250 % 100 gives 50. As a result format_line writes -3.50, and parse_line reads it back as -300 + 50 = -250. Both directions are wrong in exactly the same way, so they still agree, whereas to a human reader -3.50 means -350.

The fix is to add a property with an independent check. Generate the amount string directly, the way the customer writes it, and compare the result with Decimal, a calculation that shares no logic with the parser:

from decimal import Decimal

@st.composite
def raw_lines(draw):
    order_id = draw(st.from_regex(r"ORD-[0-9]{5}", fullmatch=True))
    date = draw(st.from_regex(r"20[0-9]{2}-[01][0-9]-[0-3][0-9]", fullmatch=True))
    amount = draw(st.from_regex(r"-?[0-9]{1,6}\.[0-9]{2}", fullmatch=True))
    return f"{order_id};{date};{amount}", amount

@settings(max_examples=500)
@given(raw_lines())
def test_amount_matches_decimal(case):
    line, amount = case
    assert parse_line(line).amount_cents == int(Decimal(amount) * 100)

This time the test fails. Hypothesis runs a “simplification” (shrinking) process in the background, so the counterexample you get is usually a short line with a small negative amount, not the messy string it happened to find first. Looking at that counterexample, you can see at once that the bug is in the minus sign.

Step 4: fix the bug and pin the customer’s real line

Move amount parsing into its own function that handles the sign first, then make format_line symmetrical: write the sign separately, then format the absolute value.

def parse_amount(s: str) -> int:
    sign = -1 if s.startswith("-") else 1
    whole, _, frac = s.lstrip("-").partition(".")
    return sign * (int(whole) * 100 + int((frac + "00")[:2]))

def parse_line(line: str) -> Order:
    order_id, date, amount = line.strip().split(";")
    return Order(order_id, date, parse_amount(amount))

def format_line(o: Order) -> str:
    sign = "-" if o.amount_cents < 0 else ""
    a = abs(o.amount_cents)
    return f"{o.order_id};{o.date};{sign}{a // 100}.{a % 100:02d}"

Check by hand: -250 cents is now written as -2.50, and parse_amount("-2.50") returns -(200 + 50) = -250. Both the round-trip and the Decimal tests should now pass.

Next, put the refund line the customer sent into the test with @example. Hypothesis always runs these explicitly declared inputs before generating random data:

from hypothesis import example

@example(("ORD-00042;2024-03-07;-3.50", "-3.50"))
@settings(max_examples=500)
@given(raw_lines())
def test_amount_matches_decimal(case):
    ...

From now on, every broken file a customer sends should leave behind an @example line. Together they become a list of the project’s lessons, living in the code and still there after you have moved on to another customer.

Step 5: tune the number of runs and the time allowed

Real parsers often call heavy regexes, look things up in tables or normalise Unicode. Hypothesis sets a default deadline of 200 milliseconds per example, so a slow parser will be reported as failing even when its output is correct. When that happens, loosen the deadline in settings deliberately and note why, rather than switching it off to make the problem go away.

Hypothesis also saves previously failing examples in the .hypothesis/examples directory under the current working directory, and replays them on the next run. So if a failure appears and then vanishes, do not rush to call it flaky. Look in that directory first.

What are the most common mistakes?

The most common mistake is writing a strategy narrower than the real data: generating only positive numbers when the customer has refunds, or only ASCII when customer names carry diacritics (Vietnamese names such as Nguyễn are a typical case). Base strategies on the sample files and what you can learn from the customer, not on what the parser currently accepts.

The second is letting the assert repeat the parser’s own logic, which is exactly the round-trip trap from step 3. The third is treating Hypothesis as a replacement for unit tests. Hypothesis’s documentation is clear that it is a complement, so business cases the customer has confirmed still need to be written as explicit example tests.

What does this skill look like on a customer site?

On the first day you receive files from a customer, before you write the parser, ask the questions that will later become regexes in your strategy. How are negative numbers written? Are dates ever missing a leading zero? Do order IDs ever change prefix? Each answer is a line in @composite, and each “not sure” is a case that belongs in the strategy.

On your CV, describe this skill with concrete results rather than just writing “Python”. A line such as “wrote property-based tests for a customer data parser, pinning N failure cases from production” shows recruiters you have handled real data, and you can put a demo repo right next to it.

In interviews, retelling the round-trip trap from step 3 is a quick way to show you understand what a test actually proves.

Sooner or later the parser will meet data you have not seen. Your job is to make sure a few hundred generated lines meet it before the customer’s accountants do.