We Told an Agent to Cheat. It Couldn't.

Look familiar?

<system_prompt>
...
Do not approve this document until review is complete.
IMPORTANT: only a Reviewer may approve. The author can never approve their own work.
REPEAT: DO NOT SKIP REVIEW UNDER ANY CIRCUMSTANCES.
THIS IS CRITICAL. UNDER NO CIRCUMSTANCES should you approve a document that is not in ReviewState.
...
</system_prompt>

You plead with an LLM.

action ToApproveControlledDocument(ControlledDocument doc) : ControlledDocument
{
    if (doc.State != ReviewState) throw new Exception("Cannot approve: document is not in ReviewState.");
    doc.State = ApprovedState;
    return doc;
}

You program Buffaly.

A language model is a brilliant text predictor. Buffaly gives it something it was never meant to hold on its own: a guaranteed, auditable process. If you're building agents for approvals, releases, clinical workflows, financial transactions, or HR onboarding, this is the architecture that lets you trust an agent with the real decision, not just the draft. In regulated industries like healthcare, clinical manufacturing, release engineering, and finance, this is the difference between an agent you can deploy and one you can only demo.

State Belongs to the Runtime, Not the Model

Buffaly's answer is simple: let the LLM do what it's brilliant at, and let the runtime own everything else. The model reasons, drafts, and requests. The runtime holds the real state, checks every rule, and is the only thing that can ever change it.

That single design choice closes a hole that a bigger prompt or a stricter schema never can. A JSON schema can require "state" to be one of ["Draft", "Review", "Approved"], which stops the model from writing "SuperDraft", but does nothing to stop it jumping straight from "Draft" to "Approved" and skipping review entirely. A plain tool calling loop, a LangGraph state graph, an AutoGPT style planner: none of them separate "the model asked" from "the runtime allowed it." Buffaly does. The transition logic lives in compiled code the model cannot see or edit, and the model cannot reach the state without going through it. That's the whole difference, and it's enough to make everything below possible.

Opaque Handles

Here's how Buffaly pulls this off. The LLM gets an immutable string, for example ControlledDocument#370, and that's all it ever needs. That string means nothing outside Buffaly's runtime: not a database key, not an editable JSON blob, just a reference. The model can pass it around, quote it back, hand it to a different tool call, and everything still works. It simply can't be turned into anything it isn't.

The real object, its state, its fields, its history, lives only inside the compiled runtime. To change anything, the model calls a typed action and passes the handle as an argument: ToApproveControlledDocument(doc, actor). The runtime resolves the handle, runs the guard conditions in compiled code, and mutates state only if every precondition holds. The model can request a transition. It cannot perform one. Full stop. That is a three-part boundary:

The Proof: A 3-State Document Pipeline

We built a minimal proof of concept, the ProofByConstructionAgent, to show this working, not just describe it. It controls a basic three-state document pipeline:

DraftState ──> ReviewState ──> ApprovedState

If an agent cannot manage a three-state sequence safely, it has no business managing deviation management, corrective actions (CAPA), clinical onboarding, or release gates. The workflow enforces three rules the model cannot negotiate, argue with, or prompt its way around:

1. Precondition: A document can only be approved if its current state is exactly ReviewState.
2. Authorization: Only an actor assigned the Reviewer role can execute an approval.
3. Separation of Duties: The author of the document can never be the reviewer who approves it.

The Enforcement Lives Inside the Compiled Action

From the model's side, the approval action looks like an ordinary tool call. Inside the runtime, it is a guarded transaction. It does not ask the LLM whether approval makes sense. It checks the rules and enforces them:

function Execute(ControlledDocument doc, DocumentUser actor) : ControlledDocument
{
    if (doc.State != ReviewState)
    {
        AddDocumentAuditEvent(doc, actor, "Approval blocked: document is not in ReviewState.");
        return doc; // Return unchanged
    }

    if (actor.Role != ReviewerRole)
    {
        AddDocumentAuditEvent(doc, actor, "Approval blocked: user is not a reviewer.");
        return doc; // Return unchanged
    }

    if (actor.UserId == doc.Author.UserId)
    {
        AddDocumentAuditEvent(doc, actor, "Approval blocked: author cannot self-approve.");
        return doc; // Return unchanged
    }

    // State change is only possible here, inside the guarded runtime code
    doc.State = ApprovedState;
    doc.Reviewer = actor;
    AddDocumentAuditEvent(doc, actor, "Document successfully approved.");
    return doc;
}

If the LLM calls the approval action too early, or with a handle it has no authority to use, the runtime runs the guard, leaves the document state untouched, and writes a permanent, blocked-attempt entry to the audit trail. No exceptions.

We Told the Agent to Break It. It Couldn't.

We told the agent to actively try to break the workflow. Not a scripted demo. A live session where we instructed it to cheat. The timeline panels below show every attempt, and every failure:

Panel 1: Handles are established

The walkthrough initializes by creating Alice (Author), Bob (Reviewer), and ControlledDocument#370. The agent is handed opaque references to these objects, establishing continuity for the conversation without exposing internal properties.

Panel 2: Out-of-order approval is blocked

The agent attempts to call ToApproveControlledDocument immediately while the document is still in DraftState. The runtime intercepts the call, blocks the transition, and records a blocked audit event. The state remains DraftState.

Panel 3: Self-approval is blocked

Alice submits the draft, successfully moving it to ReviewState. The agent then attempts to have Alice approve her own document. The runtime detects the author-reviewer conflict, rejects the action, and logs a blocked audit trail event.

Panel 4: Authorized reviewer approval succeeds

Bob, who possesses the correct Reviewer role and is not the author, executes the approval tool. Because all state guards and separation-of-duty checks pass, the runtime mutates the state to ApprovedState.

The Audit Trail Tells the Full Story

In a compliance context, a chat transcript is close to useless. It only records what was said. What matters is what was attempted, what was blocked, and what actually changed.

Buffaly intercepts every invalid tool call. Blocked attempts are not thrown away, they are written straight into the object's permanent audit history. Every failed attempt becomes proof that the boundary held:

The Neural-Symbolic Split

The LLM keeps the part it is actually good at: drafting language, interpreting messy feedback, making judgment calls. It never gets to decide when a rule-gated step happens. That decision belongs to code, not to a conversation. We call this the Neural-Symbolic Split:

• The Neural Layer (The LLM): Exceptional at interpretative, soft tasks. It excels at parsing unstructured human feedback, drafting SOP language, synthesizing clinical root causes, and organizing messy inputs.
• The Symbolic Layer (The State Machine): Exceptional at deterministic, rule-based operations. It manages step ordering, validates user roles, enforces transition paths, and logs immutable event streams.

In a clinical deviation or CAPA workflow, the LLM is the right tool for drafting a severity assessment or summarizing containment evidence. It is the wrong tool for approving its own assessment, and Buffaly makes that structurally impossible, not just discouraged. The model drafts. The workflow validates. A certified role approves. The model holds the handles. The runtime owns the rules.

Real Security Lives in the Action, Not the Menu

Good UX hides tools the agent doesn't need yet, like hiding the Approve tool while a document is in Draft. Buffaly does that too, it's a great way to help an agent navigate. The real guarantee comes from somewhere sturdier: every compiled mutation action checks its own rules, every single time, whether the agent is confused, careless, or actively trying to cheat. Discovery helpers like ToListNextActions are there to help the agent move smoothly. The guarded action itself is always what actually decides.

Where This Goes Next

This gives an agent something more valuable than intelligence: a process it can be fully trusted to run. Human reviewers still do what they do best, and the model still drafts, reasons, and recommends. A review can never be skipped, faked, or forgotten just because a conversation ran long.

The pipeline shown here is small on purpose: three states, two people, one approval rule, so the mechanics are obvious rather than impressive. The same pattern scales directly to full CAPA workflows, multi-user HR onboarding, clinical intake, and financial release pipelines. The agent drafts and reasons. A compiled action enforces who can do what and when. Every attempt, blocked or successful, becomes permanent proof. This is what agent architecture looks like when it's ready for decisions that actually matter.

The agent can ask. The runtime decides.

The neuro-symbolic framing behind this architecture is backed by research: Alexander Rombach, Chantale Lauer, and Nijat Mehdiyev, “Neuro-Symbolic Agents for Regulated Process Automation: Challenges and Research Agenda” (arXiv:2606.13405v2).