Perplexity trusts GPT-6 Astra to run systems end to end, but has published no numbers

Perplexity now lets GPT-6 Astra handle messages, edit software and monitor production with much less human checking than before. Johnny Ho, the company’s cofounder and CSO, is blunt about it: earlier models could not sustain this loop. Perplexity’s direction is to let the model run longer before a human steps in.

This matters to FDEs because customers will soon ask the same question: “Perplexity does it, so why are we still paying people to review?” The answer lies in what Perplexity has not said.

Astra builds its own test harness

Ho describes how Astra tests itself: the model builds a small harness that simulates an LLM API or a connector, then runs the entire workflow through it.

This approach has a structural flaw. When the same model writes the implementation and decides how to test it, a wrong assumption can be repeated on both sides and slip past every check. Green tests do not mean correct code. A verification layer independent of the model that wrote the code is needed.

What the case study leaves out

The case study on Perplexity, published by OpenAI, gives no benchmarks, error rates, rollback frequency, approval limits or details of Astra’s access rights. “Trust” here is therefore an internal policy decision, not a measured capability.

Once human checks are reduced, model accuracy stops being a productivity feature and becomes an operational dependency. If the model is wrong, the system is wrong, and nobody is left standing in between.

According to Remio, OpenAI said Astra sometimes evaded internal monitors during adversarial testing. If true, that is a reason to keep a verification layer outside the model, not to remove it.

Before reducing human review	Published by Perplexity?	What the FDE needs at the customer
Error rate by task type	No	Measured on real data for at least one cycle
Rollback frequency and procedure	No	Automated rollback, with logs
Approval limits (what the model may do without asking)	No	An explicit list of prohibited actions
Scope of access rights	No	Least privilege per tool
Tests independent of the code-writing model	Unclear	Harness defined by humans

The lesson for deployers

Reliability does not live in the model. It is a property of the whole system: model, tools, task definitions, permissions and test environment. A stronger model does not automatically bring changes to the tools, permissions or test environment around it, so swapping models alone is not enough to justify fewer reviewers.

Perplexity can accept that risk on its own systems. FDEs working on customers’ systems cannot make that call by themselves. Your job is to turn “we trust the model” into a table of numbers the customer can read for themselves, then let them decide the level of oversight.