# The data format you choose for a client is a contract you must keep

> The wrong format rarely breaks on day one; it breaks when real data or a changed schema drifts from what both sides quietly assumed.

Original: https://fdetimes.net/en/guides/client-data-format-csv-protobuf-contract/

RFC 4180, the document many teams still call "the CSV standard," classifies itself as informational: a reference, not an official standard. Every time you nod along to "just send us a CSV," you are signing a contract whose terms nobody has written down.

For a forward deployed engineer, this is one of the longest-lived technical decisions of an entire deployment. Integration code can be rewritten, but a data format already running on the client's side still has to be honored after you leave the project.

So the question worth asking when choosing a format is what you are committing to, and how that commitment survives as the data and the schema change.

## A format is a contract; the file extension is just a label

Every format answers three questions in its own way. How are the bytes encoded? Where do the name and type of each field live, inside the data or in a separate document? And when you add or drop a field, can the client still read it?

CSV answers all three vaguely. RFC 4180 requires fields containing commas, double quotes or line breaks to be wrapped in double quotes. It also notes that common usage of CSV is US-ASCII, and that other character sets must be declared through a separate parameter.

Take a hypothetical case: in the first week at a logistics client, the head of data asks only for a daily CSV export.

Three weeks later, a delivery address contains an unquoted comma, a customer name contains non-ASCII characters with no agreed encoding, and the client's report shifts columns without a single error line.

Both breakages sit exactly where RFC 4180 issued its warnings.

## Protobuf keeps field names on the reader's side

Protobuf is far stricter than CSV, but it puts the contract somewhere else. Its encoding documentation states that on the wire each field is identified only by its number; names and types can be determined only on the decoding side, from the message definition.

The first consequence is that sending a client Protobuf messages without the `.proto` file is like sending a locked archive and keeping the key.

The second is more forgiving. Martin Kleppmann explains that when a parser meets an unfamiliar field number, it does not know the field's name but can still work out how many bytes to skip from the type code.

The Protobuf documentation confirms that old parsers ignore new fields they do not understand, which is why adding a field is usually safe.

Enums are treated leniently too. In proto3, an unrecognized enum value is preserved in the message on decoding rather than failing the whole read, so a client's old system can carry along a value it does not yet know. Client code should therefore have a branch for values it does not recognize.

**Key point:** The best format is the one the client can still read after you have left the project.

## Example: an order feed when the schema changes

Back to the logistics client. Say that after two months, operations wants to add a `delivery_window` field to each order and drop the unused `legacy_zone` field.

With CSV, you add a column. If the client's script reads by column position rather than by name, every column after it shifts one cell without any error. If the file has no header, the client has no way to notice at all.

With Protobuf, adding `delivery_window` under a new number is all it takes. Removing a field is where the trap lies. A field number is the field's permanent identity, and both the Protobuf documentation and Kleppmann stress that an assigned number must never be reused. When you drop `legacy_zone`, you put its number on the reserved list:

```proto
message Order {
  reserved 3;
  reserved "legacy_zone";
  string order_id = 1;
  string address = 2;
  string delivery_window = 4;
}
```

Consider what happens if, six months later, another engineer assigns number 3 to a new field, `priority`. The client's old system still decodes number 3 using the `legacy_zone` definition, and the Protobuf documentation concedes that the wire format has no way to detect a field encoded with one definition and decoded with another.

The result may be a parse error, which the Protobuf documentation itself calls the best case. But the list of consequences also includes corrupted data and leaked personal data, problems nobody notices until the client finds them.

One `reserved` line blocks that whole class of incident: the protoc compiler raises an error the moment someone tries to reuse a reserved number.

## When the schema changes, who upgrades first?

This question belongs in the first meeting. Confluent Schema Registry, the schema management tool in the Kafka ecosystem, defines BACKWARD as meaning that consumers using the new schema can read data produced with the immediately preceding schema. For Protobuf, Confluent's documentation recommends the BACKWARD_TRANSITIVE mode.

It follows from that definition that the reading side moves to the new schema first, and only then does the writing side change. If you control the reader, or the client upgrades quickly, that order is easy to keep.

But if the client is a large enterprise where every code change goes through an approval committee, your side will want to change the schema long before theirs does. Then you have to ask whether the client's old systems can read the new data, and with Protobuf the answer depends on whether you only added fields or also changed a type or reused a number.

## What to do from the first discovery session

Start by asking who reads the data and with what tools. An analyst opening a file in a spreadsheet and a service calling your API continuously need very different contracts.

Next, ask how often the schema will change and how long the client's upgrade cycle is. A schema still in flux combined with a slow-upgrading client is a signal to choose a format with an explicit schema and clear evolution rules, then settle the compatibility direction you commit to.

Then write the spec on a single page and send it to the client for confirmation, even if all they said was "just send a CSV." For CSV, that page records the delimiter, quote character, encoding, whether there is a header and the type of each column.

For Protobuf, it is the `.proto` file kept in a repository the client can access, along with the reserved rules. Finally, build a test that reads old data with the new schema and run it on every release.

## Common mistakes

The most common mistake is treating CSV as a standard everyone understands the same way, when the very document describing it is only a reference. The second is sending Protobuf messages with no process for delivering the `.proto` file, leaving the client to guess what field 4 means.

The third is deleting a Protobuf field without adding it to reserved. The fourth is harder to spot: changing a field's type. Protobuf's best-practices documentation warns that this breaks deserialization, even though some old data will still parse under the new type, so testing a few records can give a false sense of safety.

The last is choosing a format based on your own team's preferences instead of the client's tools. However efficient a format is, it is useless if the recipient cannot open it.

## A schema story is worth more than a skills line

On a CV, do not just list "CSV, Protobuf, Schema Registry." Describe a time the schema changed and the client's systems kept running normally.

When an interviewer asks about data integration, a strong answer needs only three points, as in the order example above: you agreed with the client which side upgrades first, you put number 3 into `reserved` when dropping `legacy_zone`, and every release ran a test that reads old data with the new schema. Those three points show you have actually done the work.

The next time a client says "just send a CSV and we're fine," treat it as the moment a negotiation opens, not the moment it closes.

**Try this week:**

- Take a CSV file you currently send a client and write a one-page spec for it: delimiter, quoting, encoding, whether there is a header, and the type of each column.
- Open the most recent .proto file you touched, find any deleted field that is not in reserved, and fix it.
- Write a test that reads data produced by an older schema version using the current schema, then add it to the release pipeline.

## Sources

- [Language Guide (proto 3) | Protocol Buffers Documentation](https://protobuf.dev/programming-guides/proto3/)

- [Encoding | Protocol Buffers Documentation](https://protobuf.dev/programming-guides/encoding/)

- [Proto Best Practices | Protocol Buffers Documentation](https://protobuf.dev/best-practices/dos-donts/)

- [Schema evolution in Avro, Protocol Buffers and Thrift](https://martin.kleppmann.com/2012/12/05/schema-evolution-in-avro-protocol-buffers-thrift.html)

- [RFC 4180: Common Format and MIME Type for Comma-Separated Values (CSV) Files](https://www.rfc-editor.org/rfc/rfc4180)

- [Schema Evolution and Compatibility for Schema Registry on Confluent Platform](https://docs.confluent.io/platform/current/schema-registry/fundamentals/schema-evolution.html)
