# Why FDEs should stop copying YAML for every customer

> One shared base, a small overlay per customer that records only what differs, and upstream fixes that need making just once.

Original: https://fdetimes.net/en/tools/kustomize-base-overlays-for-fdes/

On 29 May 2018, when Jeff Regan and Phil Wittrock of Google introduced kustomize on the Kubernetes blog, they opened with a situation every FDE has met: you copy a few YAML files of API objects and edit them to fit your needs.

At that moment everything still works. The trouble comes later, when the original gets improvements and you have no easy way to pull them into your copy.

For an FDE, this repeats with every new customer. Picture it: the first customer needs 3 replicas, the second needs more memory, the third runs in its own namespace. By the fifth customer the repo holds five almost identical YAML directories, and nobody can say for certain on which lines they differ.

Kustomize is a way out of that loop without learning a templating language. If you deploy, or are about to deploy, the same product to several customers, it is worth learning before your tenth Helm chart.

## What does the tool actually do?

The Kubernetes documentation describes Kustomize as a standalone tool that customises Kubernetes objects through a kustomization file. It is also built into kubectl: since version 1.14, kubectl can read a directory containing a kustomization file. As long as your kubectl is that version or later, you can render and apply overlays without installing anything else.

The difference lies in the philosophy. The README of the kubernetes-sigs/kustomize project stresses that the tool works on raw YAML, with no templates, and leaves the original files untouched. The very title of the 2018 introduction called it template-free configuration customisation. Your YAML stays valid YAML that anyone can read, with no curly braces mixed in.

## Three concepts, three real things at a customer site

The official glossary defines three concepts, and they map directly onto FDE work. The base is the common part. An overlay, according to the SIG CLI documentation, is simply another kustomization that points to the base and to the patches to be applied to it.

A variant is the result in the cluster after an overlay is applied to the base. In FDE terms, a variant is one customer's deployment. The SIG CLI documentation uses one shared base for dev, staging and production variants, but the same mechanism works equally well for customer A, customer B and customer C.

**Key point:** The base holds what is shared, the overlay records only what differs, and the variant is what the customer actually runs.

## A minimal example: two customers, one base

Suppose your product has a Deployment called api. Customer Acme needs 3 replicas; customer Beta keeps the default. The directory tree might look like this:

```
base/
  deployment.yaml
  kustomization.yaml
overlays/
  acme/
    kustomization.yaml
    replicas.yaml
  beta/
    kustomization.yaml
```

Acme's overlay only needs to point at the base and declare the patch:

```yaml
# overlays/acme/kustomization.yaml
resources:
  - ../../base
patches:
  - path: replicas.yaml
```

```yaml
# overlays/acme/replicas.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 3
```

Everything that distinguishes Acme from the standard product sits in the six lines of that patch file. The Kubernetes documentation also recommends small patches, each with one purpose, such as a separate patch for replicas. When upstream adds a liveness probe to the base, you edit exactly one file, then `kubectl apply -k overlays/acme` and `kubectl apply -k overlays/beta` carry the change to both customers.

That is the answer to the problem the 2018 introduction described: upstream fixes no longer have to be copied by hand into every copy.

There is a side benefit. When a customer asks how their version differs from the standard one, the answer is their overlay directory, which is also the first thing worth reviewing when handing over to the customer's operations team.

## What do you lose without templates?

Kustomize's strength and its limit come from the same choice. Without templates you cannot write loops or conditions such as "if the customer enables feature X, generate three more objects". Every difference has to be expressed as a patch on a real base.

So patches are tightly bound to the structure of the base. The Kubernetes documentation states that the target resource is matched by the group, version, kind and name in the patch file. When the base renames a resource, the overlays patching it drift too, so treat the base like an internal API: change it carefully and re-render every overlay before merging.

If a customer's configuration varies heavily by parameter, a templating tool may suit better. But for the most common FDE case, the same product with a few small differences per customer, having no templates is a plus: anyone who opens an overlay sees at once what was changed for that customer.

## A mistyped patch does not always raise an error

The easiest mistake is a typo in a patch, followed by the belief that the fix is done. The consequences differ sharply depending on whether the typo is in a resource name or a field name.

With a wrong resource name, for instance a patch targeting `my-app` when the base has no Deployment by that name, you usually see an error straight away. A report on the Kubernetes forum records the build stopping with the message "failed to find unique target for patch".

But a community wiki warns that some kinds of patch that match no resource are skipped with no error and no warning. This behaviour depends on the patch type and the kustomize version.

With a wrong field name, say `replica` instead of `replicas`, the API server does not recognise the field, and the outcome depends on validation. kubectl enables strict server-side validation by default, so `kubectl apply` usually rejects unknown fields.

When validation is turned off with `--validate=false`, on an older cluster, or through a tool that sets its own defaults, the bad field may produce only a warning or be dropped silently. Acme then still runs with the default replica count, and the render or diff still shows replicas unchanged.

## Read the render, not the patch file

The second mistake is applying straight to a customer's cluster before seeing the final YAML. The safeguard is simple: before every apply, especially right after the base has changed, run two commands.

```
kubectl kustomize overlays/acme
kubectl diff -k overlays/acme
```

The first prints every resource after the overlay is applied; look for the line `replicas: 3` in the api Deployment. The second compares the cluster's current state with the state it would have after applying. The same community wiki also recommends running this build-and-diff step in CI for every overlay, and in a multi-customer repo you should.

## When an overlay grows into a second base

The third mistake is harder to spot: one customer's overlay swells until it becomes a second base. At that point you are back to the copied-YAML problem, just with renamed directories.

When two overlays carry the same block of patches, that is the signal that the block belongs in the base.

## What to learn first, and how to show it

First, get solid on base, overlay and variant, and on the habit of rendering and diffing before applying. Then practise writing patches for the fields that commonly differ between customers, such as replicas, resources and env. At a customer site, the first job worth doing is to find the YAML that has been copied and edited, and consolidate it into one base.

When interviewing for an FDE role that deploys the same product to many customers, ask directly how the team manages each customer's configuration; the answer tells you whether this skill is needed from day one.

On a CV, the line that carries weight is not "knows Kustomize" but a number: how many customers share one base, how many hand-copied files were deleted.

## A 20-minute exercise with the Acme example

Recreate the directory tree above on your machine. Step one: change `name: api` in `replicas.yaml` to `name: apii`, run `kubectl kustomize overlays/acme` and note whether you get an error or a render with no change.

Step two: restore the correct name, change `replicas` to `replica`, render again and look at the api Deployment in the output: which field appears, and what value `replicas` has. If you have a test cluster, also run `kubectl diff -k overlays/acme` to see how the server reacts to the unknown field.

Step three: fix it, render, and stop only when you see `replicas: 3` in the output.

Those three renders teach something a patch file never tells you on its own: whether a patch is right or wrong shows up only in the output. Trust an overlay only once you have seen the customer's change in the render with your own eyes.

**Try this week:**

- Pick a service that has two near-identical YAML copies, extract the shared part into base/ and move the differences into two overlays
- Change one thing in the base, run kubectl kustomize for both overlays, then kubectl diff -k to confirm the change reached both customers
- Add a line with numbers to your CV: how many customers share one base, how many YAML files you removed

## Sources

- [Declarative Management of Kubernetes Objects Using Kustomize](https://kubernetes.io/docs/tasks/manage-kubernetes-objects/kustomization/)

- [Introducing kustomize; Template-free Configuration Customization for Kubernetes](https://kubernetes.io/blog/2018/05/29/introducing-kustomize-template-free-configuration-customization-for-kubernetes/)

- [GitHub - kubernetes-sigs/kustomize: Customization of kubernetes YAML configurations](https://github.com/kubernetes-sigs/kustomize)

- [Kustomize (SIG CLI documentation)](https://kubectl.docs.kubernetes.io/guides/introduction/kustomize/)

- [Glossary (SIG CLI, Kustomize Reference)](https://kubectl.docs.kubernetes.io/references/kustomize/glossary/)

- [Kustomize - “failed to find unique target for patch …”](https://discuss.kubernetes.io/t/kustomize-failed-to-find-unique-target-for-patch/10582)

- [Kustomize Footguns (GrokDevOps Wiki)](https://grokdatum.github.io/grokdevops-wiki/library/topics/kustomize/footguns)

- [Kubernetes API Concepts (Field validation)](https://kubernetes.io/docs/reference/using-api/api-concepts/)
