FDE Times dataFDE jobs open 873▼9 Change in open roles over 7 days, like-for-like: roles from companies we started tracking this week are left outNew in 7 days 55Companies hiring 181Remote-friendly 25%Median US pay $200kTop hirer Databricks 98
中文VI

The newspaper of the Forward Deployed Engineer

Guides

The data format you choose for a client is a contract you must keep

The wrong format rarely breaks on day one; it breaks when real data or a changed schema drifts from what both sides quietly assumed.

ShareLinkedInFacebookX

In brief

  • RFC 4180 is only a reference: CSV quoting, encoding and headers all have to be agreed in writing.
  • Protobuf identifies fields by number, so a used number must never be reassigned and a deleted field must go into reserved.
  • The deciding question is which side upgrades first when the schema changes, and it belongs in the first meeting.
Two Protobuf code panels side by side. On the left, without reserved: the line string legacy_zone = 3; is struck through, then int32 priority = 3; is highlighted red because it reuses number 3, so old clients misread it as legacy_zone. On the right, with reserved: the line reserved 3; is highlighted orange, legacy_zone is struck through, and when someone tries to reuse number 3 for priority, protoc raises an error immediately.
A field number is a Protobuf field's permanent identity. Delete legacy_zone and assign number 3 to another field, and the client's older systems will decode it wrongly without any warning. A single reserved line makes protoc block any reuse of the number.

RFC 4180, the document many teams still call “the CSV standard,” classifies itself as informational: a reference, not an official standard. Every time you nod along to “just send us a CSV,” you are signing a contract whose terms nobody has written down.

For a forward deployed engineer, this is one of the longest-lived technical decisions of an entire deployment. Integration code can be rewritten, but a data format already running on the client’s side still has to be honored after you leave the project.

So the question worth asking when choosing a format is what you are committing to, and how that commitment survives as the data and the schema change.

A format is a contract; the file extension is just a label

Every format answers three questions in its own way. How are the bytes encoded? Where do the name and type of each field live, inside the data or in a separate document? And when you add or drop a field, can the client still read it?

CSV answers all three vaguely. RFC 4180 requires fields containing commas, double quotes or line breaks to be wrapped in double quotes. It also notes that common usage of CSV is US-ASCII, and that other character sets must be declared through a separate parameter.

Take a hypothetical case: in the first week at a logistics client, the head of data asks only for a daily CSV export.

Three weeks later, a delivery address contains an unquoted comma, a customer name contains non-ASCII characters with no agreed encoding, and the client’s report shifts columns without a single error line.

Both breakages sit exactly where RFC 4180 issued its warnings.

Protobuf keeps field names on the reader’s side

Protobuf is far stricter than CSV, but it puts the contract somewhere else. Its encoding documentation states that on the wire each field is identified only by its number; names and types can be determined only on the decoding side, from the message definition.

The first consequence is that sending a client Protobuf messages without the .proto file is like sending a locked archive and keeping the key.

The second is more forgiving. Martin Kleppmann explains that when a parser meets an unfamiliar field number, it does not know the field’s name but can still work out how many bytes to skip from the type code.

The Protobuf documentation confirms that old parsers ignore new fields they do not understand, which is why adding a field is usually safe.

Enums are treated leniently too. In proto3, an unrecognized enum value is preserved in the message on decoding rather than failing the whole read, so a client’s old system can carry along a value it does not yet know. Client code should therefore have a branch for values it does not recognize.

Example: an order feed when the schema changes

Back to the logistics client. Say that after two months, operations wants to add a delivery_window field to each order and drop the unused legacy_zone field.

With CSV, you add a column. If the client’s script reads by column position rather than by name, every column after it shifts one cell without any error. If the file has no header, the client has no way to notice at all.

With Protobuf, adding delivery_window under a new number is all it takes. Removing a field is where the trap lies. A field number is the field’s permanent identity, and both the Protobuf documentation and Kleppmann stress that an assigned number must never be reused. When you drop legacy_zone, you put its number on the reserved list:

message Order {
  reserved 3;
  reserved "legacy_zone";
  string order_id = 1;
  string address = 2;
  string delivery_window = 4;
}

Consider what happens if, six months later, another engineer assigns number 3 to a new field, priority. The client’s old system still decodes number 3 using the legacy_zone definition, and the Protobuf documentation concedes that the wire format has no way to detect a field encoded with one definition and decoded with another.

The result may be a parse error, which the Protobuf documentation itself calls the best case. But the list of consequences also includes corrupted data and leaked personal data, problems nobody notices until the client finds them.

One reserved line blocks that whole class of incident: the protoc compiler raises an error the moment someone tries to reuse a reserved number.

When the schema changes, who upgrades first?

This question belongs in the first meeting. Confluent Schema Registry, the schema management tool in the Kafka ecosystem, defines BACKWARD as meaning that consumers using the new schema can read data produced with the immediately preceding schema. For Protobuf, Confluent’s documentation recommends the BACKWARD_TRANSITIVE mode.

It follows from that definition that the reading side moves to the new schema first, and only then does the writing side change. If you control the reader, or the client upgrades quickly, that order is easy to keep.

But if the client is a large enterprise where every code change goes through an approval committee, your side will want to change the schema long before theirs does. Then you have to ask whether the client’s old systems can read the new data, and with Protobuf the answer depends on whether you only added fields or also changed a type or reused a number.

What to do from the first discovery session

Start by asking who reads the data and with what tools. An analyst opening a file in a spreadsheet and a service calling your API continuously need very different contracts.

Next, ask how often the schema will change and how long the client’s upgrade cycle is. A schema still in flux combined with a slow-upgrading client is a signal to choose a format with an explicit schema and clear evolution rules, then settle the compatibility direction you commit to.

Then write the spec on a single page and send it to the client for confirmation, even if all they said was “just send a CSV.” For CSV, that page records the delimiter, quote character, encoding, whether there is a header and the type of each column.

For Protobuf, it is the .proto file kept in a repository the client can access, along with the reserved rules. Finally, build a test that reads old data with the new schema and run it on every release.

Common mistakes

The most common mistake is treating CSV as a standard everyone understands the same way, when the very document describing it is only a reference. The second is sending Protobuf messages with no process for delivering the .proto file, leaving the client to guess what field 4 means.

The third is deleting a Protobuf field without adding it to reserved. The fourth is harder to spot: changing a field’s type. Protobuf’s best-practices documentation warns that this breaks deserialization, even though some old data will still parse under the new type, so testing a few records can give a false sense of safety.

The last is choosing a format based on your own team’s preferences instead of the client’s tools. However efficient a format is, it is useless if the recipient cannot open it.

A schema story is worth more than a skills line

On a CV, do not just list “CSV, Protobuf, Schema Registry.” Describe a time the schema changed and the client’s systems kept running normally.

When an interviewer asks about data integration, a strong answer needs only three points, as in the order example above: you agreed with the client which side upgrades first, you put number 3 into reserved when dropping legacy_zone, and every release ran a test that reads old data with the new schema. Those three points show you have actually done the work.

The next time a client says “just send a CSV and we’re fine,” treat it as the moment a negotiation opens, not the moment it closes.

Was this article useful?

Use with your AI assistantAsk Claude ↗Ask ChatGPT ↗
6 sources
Read next on the roadmap · Stage 2: Broad engineering'Latest wins' quietly overwrites your client's CRM dataThe usual culprit is a write based on a stale read, so give every field one owning system and let a human settle unclear cases.