# Running AI in a factory with no internet: packaging, updating and monitoring edge machines

> With no network, one file missing from the USB stick means a drive back to the plant, and a broken update can leave the line running with nobody checking for defects.

Original: https://fdetimes.net/en/guides/air-gapped-edge-ai-deployment-factory/

Picture this: you are standing next to an edge machine on a defect-inspection line, with a USB stick in your pocket, and the only network in the room is the plant's internal one. You cannot `docker pull`, you cannot `pip install`, and you cannot open a cloud dashboard. Whatever is not on that USB stick does not run today.

Some customers cut their production network off from the internet for security reasons; some sites simply have no connection.

The hard part here is not training the model. It is three operational jobs: packaging everything you need, updating in a way that can be undone when it fails, and monitoring so you know what the machine is actually doing.

All three rest on the same principle. In an air-gapped environment, every assumption about the network has to be written down as a file, a verification step or a limit with a number attached. The three sections below follow a single scenario.

## A release must be self-contained

The concrete scenario: a single-node K3s cluster running Triton Inference Server, serving a scratch-detection model, plus a collector that sends metrics back to headquarters. According to the K3s air-gap installation docs, images can come from an upstream registry, a registry mirror or an airgap image tarball.

On a factory floor where nobody runs a private registry, the third option is the most practical. Without a private OCI registry, K3s lets you copy images to each node by hand. That is laborious across many machines, but for a handful of edge boxes it is far simpler than standing up and maintaining a registry.

The first trap is versioning: K3s requires the binary and the airgap image archive to come from the same release. Download the binary on Monday and the tarball on Thursday, and they can easily drift apart without you noticing. You usually find out only once you are standing in the plant.

So every delivery should be a release package built by a script, not assembled by hand. The package carries a manifest recording the version and the checksum of every file:

```
usb/
  release-2026.10.1.tar              # everything below
  release-2026.10.1.sigstore.json    # signature bundle for the tarball
# inside the tarball:
  manifest.sha256                    # checksum of every file
  k3s/                               # binary + airgap images, SAME version
  images/app-images.tar              # application images
  models/scratch-detector/3/model.onnx
```

The signature is the part people tend to skip. Sigstore's bundle format allows signatures to be verified offline, so the edge machine can check by itself that the tarball on the USB stick is exactly what your team signed, without calling out. When the customer's security team asks how you know the stick has not been swapped, you already have the answer.

## Updates: design for the failure

In the cloud, a bad deploy is usually rolled back within minutes. In a factory, it can leave the line running with no defect inspection until someone drives out. So when choosing an update mechanism, the question to ask is not whether it installs, but what state the machine is in when it breaks.

At the operating-system layer, Mender splits the disk into two A/B partitions. If the new version fails to boot, the bootloader detects the problem and returns to the previous version. Mender also has a standalone mode for fully offline sites, which lets you apply an update locally, for example from USB. That gives you automatic rollback without a server.

At the model layer, there are two Triton behaviours you need to understand. In the default mode, Triton loads models at startup and then ignores any changes to the model repository while the server is running. Copy a new model into the directory and Triton keeps serving the old one; for the new one to take effect, you have to restart the server or switch to explicit model control mode.

POLL mode is dangerous in the opposite direction. According to NVIDIA's documentation, Triton can see a repository in the middle of a change, before copying has finished, and behave unexpectedly. Copying files from USB into a directory that is being polled is the easiest way to land in exactly that situation.

The safe approach is explicit mode: copy the complete model version into a new directory, check it, and only then issue the load command. Triton has one more point in its favour: if a reload fails, the model currently running is kept. An update script might look like this:

```bash
#!/usr/bin/env bash
set -euo pipefail
USB=/media/usb
REL=release-2026.10.1

# 1. Verify the signature offline before touching anything
cosign verify-blob --key /etc/fde/release.pub \
  --bundle "$USB/$REL.sigstore.json" "$USB/$REL.tar"

# 2. Extract to a separate directory and check checksums against the manifest
mkdir -p "/opt/releases/$REL"
tar -xf "$USB/$REL.tar" -C "/opt/releases/$REL"
(cd "/opt/releases/$REL" && sha256sum -c manifest.sha256)

# 3. Add the new model version without overwriting the old one
cp -r "/opt/releases/$REL/models/scratch-detector/3" /models/scratch-detector/

# 4. Triton runs with --model-control-mode=explicit: request a reload
curl -fsS -X POST localhost:8000/v2/repository/models/scratch-detector/load
```

If step 4 fails, the old model version keeps serving and you have logs to take back for analysis. Every step in the process should share this property: when it breaks, it stays put in the last known good state.

## Does green on the portal mean the machine is healthy?

This is the part that surprises people most. Azure IoT Edge needs to connect to IoT Hub once after the runtime is installed. After that, the device and the devices behind it can run with an intermittent connection or none at all. But when the device loses internet access, its status on the Azure portal can still show 200 OK.

This is not unique to Azure. Any monitoring system that only waits for bad news will go quiet exactly when you need it most. Monitor with active signals instead: a timestamped heartbeat, with an alert when the heartbeat stops, rather than waiting for the machine to report an error.

**Key point:** Where there is no network, silence from your monitoring system is not good news.

The second trap is the buffer. When offline, IoT Edge stores telemetry locally, bounded by a TTL and by disk space, and the default TTL is just two hours. Prometheus in Agent mode also writes data temporarily to disk when the receiving end loses its connection, but only for a limited window.

The 2021 post introducing that feature put the limit at two hours; the figure may have changed in later versions, so check the documentation for the exact version you run.

Do the arithmetic. Suppose the link from the plant to headquarters goes down from Friday evening to Monday morning, roughly 60 hours.

With IoT Edge's default two-hour TTL, the buffer holds only 2 of those 60 hours, and 58 hours of data are lost. Those are precisely the night shifts you need to see to know whether the model drifted as the lighting in the plant changed.

The OpenTelemetry Collector offers a better option. With a storage extension configured, the telemetry queue lives on disk and fully replaces the in-memory queue, so data survives a restart.

If you size the disk for the longest offline period the customer will accept, the retention period becomes a number you and the customer agreed on, not a tool's default.

## Mistakes that keep recurring

The most common mistake is treating a networked lab as a dress rehearsal for the factory. If you have never installed the full release on a machine with its network cable pulled, you do not know whether the release is complete.

Another is leaving defaults in place: Triton's default model-loading mode, the two-hour TTL, the in-memory queue. Each default is sensible in the cloud and wrong on the factory floor.

The next is never rehearsing a rollback, even though an A/B scheme is only trustworthy once you have watched the machine fall back to the old version on the customer's actual hardware.

The last is trusting the cloud dashboard. If finding out whether the machine is running requires going through the internet, you need an additional local screen for the operations engineers on site.

## Turning this skill into an edge in your job search

When reading FDE job descriptions, look for phrases such as "air-gapped", "on-prem", "edge" or "disconnected environments". They signal that the role needs exactly these skills.

On your CV, do not just write "experience with Kubernetes". Write that you designed a signed offline update process with rollback, and how many hours of lost connectivity your buffer can absorb.

To practise before an interview, answer this question out loud: "The plant has no internet, and every month you have to replace the model on a few edge machines. How do you bring the new version in, how does the machine know the package has not been swapped, and what happens to the line if the new model fails to load?"

A convincing answer covers four points: a same-version release package with checksums, a signature verified offline, model loading in explicit mode so the old version keeps running on failure, and a heartbeat to confirm the update succeeded. Any point you miss is where you need more practice.

In a factory, few people remember your model's accuracy to the percentage point. They remember the update that broke while the line kept running as normal.

**Try this week:**

- Pull the network cable on a virtual machine or a Raspberry Pi, then install K3s air-gapped using only a release directory built by a script; write down everything that turns out to be missing.
- Run Triton with --model-control-mode=explicit, deliberately load a broken model version and confirm the old version keeps serving requests.
- Ask for (or set yourself) a number: the maximum hours the plant can be offline. Then configure the OpenTelemetry Collector's on-disk queue and calculate enough disk space to cover that many hours.

## Sources

- [Air-Gap Install | K3s](https://docs.k3s.io/installation/airgap)

- [Operate Azure IoT Edge devices offline | Microsoft Learn](https://learn.microsoft.com/azure/iot-edge/offline-capabilities)

- [Model Management, NVIDIA Triton Inference Server](https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/user_guide/model_management.html)

- [Introduction | Mender documentation](https://docs.mender.io/overview/introduction)

- [Introducing Prometheus Agent Mode, an Efficient and Cloud-Native Way for Metric Forwarding](https://prometheus.io/blog/2021/11/16/agent/)

- [Exporter Helper: Persistent Queue, Storage, and Retry Settings (OpenTelemetry Collector)](https://github.com/open-telemetry/opentelemetry-collector/blob/main/exporter/exporterhelper/README.md)

- [Verifying Signatures | Sigstore documentation](https://docs.sigstore.dev/cosign/verifying/verify/)