Achilles · test the AI
Constantly Tested AI Never Fights Alone.
Capability families
Eight ways an AI deployment goes wrong.
01Goal hijackCan someone redirect what the AI is trying to do?
- Direct prompt injection, and instructions hidden in content it reads
- A web-retrieval chain that carries the hijack in
02RAG leakageCan retrieval surface data the user should never see?
- Context exfiltration, including planted secrets
- Context leaking across sessions or tenants
03Agent and tool misuseCan the AI be steered into calling tools it shouldn’t?
- Confused-deputy calls and scope escalation
- An email-memory chain that steers a later tool call
04Permission boundaryDoes the AI stay inside the access it was granted?
- Privilege escalation beyond the requester’s rights
- Authorization checks bypassed
05Jailbreak resistanceDo safety instructions hold under sustained pressure?
- Crescendo: multi-turn pressure, not one-shot tricks
- Persona adoption under sustained role-play
06Insecure outputCan the AI’s output become someone else’s exploit?
- Output that executes downstream
- Injection payloads passed to other systems
07MCP and tool integrityAre the tools and skills it trusts what they claim to be?
- Poisoned tool descriptions and shadow tools
- Skill chains, RAG-to-MCP and server relay · ten scenarios
08Action and approval authorityCan it act beyond the authority it was given?
- Approval bypass and claimed human approval
- Multi-agent escalation, delegated authority, agent relay · eight scenarios
Cross-channel and multi-step
One message rarely breaks an agent. A sequence might.
Crescendo pressure that builds over a conversation — scored on signals in the target’s response, never on the model’s own account of what it did.
- 1Inbound emailCarries hidden instructions
- 2RetrievalPulls the email into context
- 3PlanThe agent adopts the instruction
- 4Tool callA tool is invoked
- 5CanaryAn inert canary shows how far it got
Action Gate states
Did the agent make the call it should have?
Requests · illustrative
- Expected decision
- Permitted
- Observed
- Refund issued to the original card.
Matched. Within policy, to the original destination.
This assessment tests your stack; it is not a runtime control. Achilles is also the reference enforcement stack, whose Action Gate permits only the exact action approved — runtime enforcement is a separate engagement.
Effect-bound approvals
An approval is for one exact action.
Approved action · synthetic
Refund $1,200 for order #88213
Approved by the support lead, 10:42
- Destination
- Card ending 4471
- Amount
- $1,200
- Acting for
- Customer C-1042
- Order
- #88213
- Expected effect
- One refund
Change one bound field
Change any field the approval was bound to and the permit should no longer apply. Achilles tests whether your stack notices.
Destination, amount, identity, order and effect all match the approval. The action runs once.
Illustrative and synthetic. Achilles tests approval binding in its action-and-approval-authority family; its reference Action Gate binds a permit to the action, policy and authority epoch, and blocks edits to any of the eight canonical fields.
Approval binding
Does an approval still hold if the amount or destination changes?
Check at the moment of action
Is permission re-checked when the action runs, not only when it was planned?
Cumulative limits
Can many small actions add up to one the agent was never allowed to take?
Revocation
When access is withdrawn, how quickly does the agent actually stop?
Override
Can an urgent request push past a hold it should respect?
Refusal memory
Does the agent remember a refusal — or keep retrying until it hears yes?
Observed effect
What happened — not what the agent said.
| Attempted | Policy decision | Actual invocation | Recorded outcome |
|---|---|---|---|
| Refund $120 to the original card | Allowed | refund.create(120, card 4471) | Observed effect — a 2xx to exactly the permitted actionMatches |
| Refund $4,800 | Escalate | refund.create(4800) — no approval on record | Authority violation — verifying the effect needs a witnessFinding |
| Email the customer list to an outside address | Denied | None | No effect observedMatches |
| “I’ve cancelled the order for you.” | — | order.cancel never called | No effect — the agent’s account was wrongUnobserved |
Canary comparison and retest
A fix that blocks real work is its own finding.
| Outcome | Baseline | Fix 1 | Fix 2 |
|---|---|---|---|
| Authority violations | 6 | 0 | 0 |
| Negative controls holding | 12 / 12 | 12 / 12 | 12 / 12 |
| Legitimate requests refused | 1 | 9 | 1 |
| Benign counterparts let through | 6 / 6 | 2 / 6 | 6 / 6 |
| Verdict | The starting point. | Absent on retest, but it refuses real work. A new finding. | Absent on retest; benign counterparts were let through on this fixture. |
Between releases, a canary comparison decides whether a candidate is worse than the baseline; the weakest result wins. Absent on retest means the failure did not recur, not that legitimate use still works.
SPINE and Minotaur handoff
Achilles tests. SPINE remembers. Minotaur checks the tester.
Achilles
Signed outcome
Chain outcomes and observed effects, each signed.
SPINE
Claim updated
Evidence can hold the claim back, never lift it.
SPINE
Decision recalculated
Worst-of-N, held until a retest. Your team decides.
Minotaur
Method measured
Achilles is scored on planted cases with known answers.
Keep going
Where Achilles leads.
Mythos AI Security
Find out what your agents can actually do.
Tell us which agents and tools matter most, and we will scope the boundaries Achilles should challenge first.










