Field debugging: a hypothetical case of an autonomous vessel veering off course while docking

Picture an uncrewed vessel that holds its planned course offshore all morning. In the afternoon, as it comes into the harbour, it starts to drift off heading, corrects itself, then drifts again. You take the logs back to the office, run them again, and everything looks fine.

Dealing with this kind of bug is part of the job description for FDEs who work on autonomous systems. Saronic, which builds autonomous surface vessels, describes its FDEs as the people who integrate, deploy and improve autonomous systems in real-world environments.

According to the job description, FDEs handle software, hardware and networking issues together during deployments. They also spend 50% or more of their time travelling, domestically and internationally.

You may never set foot on a boat. But nearly every FDE has met a bug that only happens at the customer’s site: no debugger attached, a slow network, and nobody able to replay the exact moment for you. The sea is the harshest version of this problem, which also makes it the clearest place to learn it.

In the field, the hard part is reproducing the bug

A moment at sea happens only once. The waves, the wind, the surrounding traffic and the satellite signal will never line up the same way again. If you rush a fix on the spot without recording anything, you cannot prove the fix is right, and if the bug comes back you won’t recognise it.

At Saronic, the maritime test planning role tracks defects through a structured process that records reproduction steps, severity and corrective action for each one. The principle worth taking for yourself: until you can reproduce a bug, you have nothing solid to fix.

Reproduction requires data. Saronic says the endurance trial of its Corsair vessel produced about 500 hours of video from multiple cameras, roughly 17 TB in total.

The company says it will use this dataset for model development, replay and regression testing. The trial was designed to run continuously to assess reliability and endurance, which means exposing the bugs that short tests miss.

A slow link forces you to choose what to send home

A study of autonomous vessels notes that at sea they are limited by low-bandwidth satellite links. Nobody streams 17 TB over a link like that. So you split the job in two: record everything on the vessel’s own machine, and send ashore just enough for people on land to know something has happened.

Below is a simple sketch of that idea. The thresholds, cooldown and field names are only examples:

import collections, json, time

def send_over_satlink(msg):
    """Stub: replace with a real satellite client that has a queue and a size limit."""
    payload = json.dumps(msg).encode()
    assert len(payload) < 512, "summary must be small"
    print("SATLINK", payload)

class FieldRecorder:
    """Record everything on the local machine, send only a summary over the slow link."""
    def __init__(self, window_s=120, rate_hz=50, cooldown_s=60):
        self.buf = collections.deque(maxlen=window_s * rate_hz)
        self.cooldown_s = cooldown_s
        self.last_snap = {}  # reason -> time of the most recent snapshot

    def record(self, s):  # s: dict with t, heading_err_deg, gps_fix, link_ok, sw_version
        self.buf.append(s)
        if abs(s["heading_err_deg"]) > 5:
            self.maybe_snapshot("heading_error")

    def maybe_snapshot(self, reason):
        now = time.monotonic()
        if now - self.last_snap.get(reason, float("-inf")) < self.cooldown_s:
            return  # still the same heading-error episode, no new snapshot
        self.last_snap[reason] = now
        path = f"/data/snap_{int(time.time())}_{reason}.jsonl"
        with open(path, "w") as f:
            for s in self.buf:
                f.write(json.dumps(s) + "\n")
        # Only a few hundred bytes go over the satellite; the full file stays on board
        send_over_satlink({"reason": reason, "file": path, "n": len(self.buf)})

The key is the ring buffer: when an anomaly is detected, you already have the previous two minutes of data, the part that led up to the bug. The cooldown matters just as much. Without it, a heading error that lasts a few minutes would generate thousands of snapshots, filling the disk and clogging the satellite link.

Also record the state of all three layers in the same record: software version, GPS status and link status. That way, when you isolate the bug, you have the evidence in hand.

Example: the heading error while docking

Back to the vessel at the start of this article. This is an illustrative scenario for practising the method, not an incident Saronic has published. On paper there are three groups of hypotheses: the navigation logic mishandles a situation that only occurs near shore, a sensor reports different values in a harbour environment, or a lost connection makes the system switch modes.

For the positioning layer, the hypothesis has grounds. Research on autonomous vessels warns that when GPS loses signal, is jammed or is spoofed, a ship with no crew on board can end up in a dangerous situation. That is why the gps_fix field in the record matters so much.

The decisive test is replay. You feed the exact recorded snapshot into the navigation stack on an office machine. If the virtual vessel drifts off heading just as it did at sea, the bug is in the logic, and you now have a test case to fix against.

If the replay comes out perfect, there are two possibilities. The first is that the data you fed in differs from what the real system received at sea, so suspect timing, hardware or the network.

The second is that the stack is non-deterministic: thread ordering, processing latency or a random step can make the same input produce different outputs.

So run the replay several times, fix the seed and use a simulated clock. If the results vary between runs, fix the stack’s determinism first, before drawing any conclusion about the hardware.

The result goes into the bug report like this:

Field	How to fill it in for the heading error
Symptom	Heading error exceeds the threshold, repeatedly, while approaching the harbour
Conditions	Time, position, GPS status, link status, software version
Reproduction steps	Replay a specific snapshot; record the number of runs and how many reproduced the bug
Severity	Whether it affects operational safety
Suspected layer	Software, hardware or network, with evidence from the replay
Corrective action	The fix, plus a regression test built from this very snapshot

The last row is the one many people skip. The snapshot that triggered the bug should become a permanent test, so the next trial doesn’t have to go hunting for the old bug at sea.

Trigger failures before the real environment does

The best way to avoid surprises at sea is to cause failures yourself in a controlled trial. In 2013, during a sea trial, GPS was deliberately jammed and a prototype system automatically switched to eLoran.

That test targeted a navigation system rather than an autonomous vessel, but the method carries over intact.

A recent demonstration of autonomous capability included an emergency response to a simulated machinery failure, while the vessel was monitored remotely from shore.

Tests like these are only worth something when the criteria are set in advance. Saronic organises its testing around test matrices, in which each requirement is tied to an objective pass/fail criterion.

“The fallback works fine” is just an impression. “On GPS loss, switch to the backup system without exceeding the allowed deviation” is a criterion you can check.

Then there is safety. At Saronic, the on-site test director runs the trial in real time and is responsible for its safety. If you want to trigger a failure deliberately, you must agree it with that person beforehand. Nobody gets to cut off a running system on their own initiative.

Common mistakes

The most common mistake is debugging only in the layer you know. Software engineers dive into the code while the evidence points at the antenna or the link. That is also why the FDE job description puts software, hardware and networking on a single line.

The second mistake is trying to send everything back over the slow link, clogging it exactly when it is needed most. The third is fixing on the spot without saving a snapshot, so the fix can never be proven right. The fourth is testing without pass/fail criteria, then arguing afterwards about whether the results were good.

Shorten the loop between the field and the code

Saronic’s test planner works daily with the software team, vessel operations and suppliers to drive each issue to resolution quickly. A good bug report keeps this loop moving: people in the office receive the snapshot and the reproduction steps, so they don’t need to go into the field to understand what happened.

On the career side, when you read FDE job ads, look for phrases such as “software, hardware, and networking” or “real-world environments”: they signal a job that demands field debugging. On your CV, don’t just write “handled production incidents”.

Tell the story of one specific bug: how you recorded the data, how you reproduced it, which layer you isolated it to and what regression test you turned it into. In an interview, a story like that is far more convincing than a list of tools.

The sea doesn’t give you a second attempt. A good engineer is one who can bring that single moment of failure back to the office and replay it as many times as needed.