Minotaur · how Mythos AI Security validates itself
Find What We Missed. Prove What We Test.
Why assurance needs assurance
A test can be wrong in quiet ways.
Contamination
Could the system have seen the test?
If test cases leak into training or tuning data, a score measures memory, not behavior.
Invalid scorers
Does the scorer measure what it claims?
A grader that rewards the wrong signal turns a failure into a pass — and nobody looks twice at a pass.
False blocks
Is it catching, or just refusing?
A control that blocks benign work can look perfect on attacks alone. Safe twins make precision count.
Distribution shift
Does the test still look like production?
Models, prompts, tools and users change. A benchmark frozen in the past slowly stops describing the system.
Unwatched exits
Did the test see every way out?
An effect can leave through a channel the harness never observed. A clean result there means unknown, not safe.
In place today
- Fifteen benchmark families, answers keyed in advance
- Decoys and safe twins; precision decides
- Validity Cards, partitions and external reproduction
- Nine release gates and a security veto
Not yet in place
- A pass mark — no family has one
- An audited Validity Card — none has been audited
- A sealed holdout — none is sealed yet
- Runs against public targets such as Juice Shop or DVWA
The labyrinth
Every result has to find its way out.
Fifteen benchmark families plant issues on purpose, with answers keyed in advance, and no runner holds the key. If a test misses a known issue, the test is what’s wrong.
Known answers and safe twins
Expected versus observed.
Known-answer case
As expectedA planted access flaw in the target range
- Expected
- Found
- Observed
- Found
The answer was keyed before the run and no runner holds it, so the harness can’t be graded on a moving target.
Safe twin
As expectedThe same request, with the access it’s entitled to
- Expected
- Allowed
- Observed
- Allowed
The harness told the two apart — the result above is evidence, not a reflex.
Counterexample
MismatchA second safe twin, phrased differently
- Expected
- Allowed
- Observed
- Blocked
A false block. It counts against precision, and precision decides.
Illustrative cases. Inputs are described, never reproduced — corpora and answer keys stay internal.
Benchmark Validity Card
What a score depended on.
- Harness owner
- Validation lead
- Harness version
- harness 3.2
- Scenario set
- approval-boundary set · fixture 7d1e…
- Partition
- holdout arena · not sealed
- Context boundary
- single session; no shared memory
- Tool visibility
- refund, CRM read, email
- Seed
- 4417 (fixed)
- Scorer
- signal-based; narration ignored
- Safe twins flagged
- 2 of 40
- Known gaps
- 1, counted apart from misses
- Reviewer
- Second analyst, not the author
False blocks and precision
Refusing more is not the same as better.
| Blocked | Allowed | |
|---|---|---|
| Harmful request | 20Attack caught | 0Attack missed |
| Benign request | 9False block | 31Legitimate case let through |
Catches every planted attack — and blocks nine legitimate cases. Precision decides, so this build does not count as better.
Reproduction and regression
Run it again. Get the same answer.
Answers keyed
fixture 7d1e…c04KeyedCases and answers are fixed before any engine runs; no runner holds the key.
Baseline run
harness 3.2 · seed 4417ObservedThe engine runs in a separated child; the sealed scorecard records the versions it came from.
External reproduction
same fixture · reproduction packageReproducedThe package reruns the row in CI and must match it byte for byte.
Mitigation applied
engine change 1.14StaleA fix lands. Every result that depended on the old behavior is now stale.
Regression run
harness 3.2 · seed 4417ObservedThe earlier cases run again for the Regression gate, which reads unmeasured, never green, until they do.
Release authority
Hold on evidence, not on averages.
Evidence on record · synthetic
- 49 of 50 planted cases caught
- One case caught last release is now missed
- Precision unchanged on safe twins
One regression is not averaged away. The weakest gate sets the outcome.
This governs Mythos AI Security’ own engine releases. Your deployment decision stays with your people.
Keep going
Where the method shows up.
Mythos AI Security
Ask us how we test.
We will walk through the method behind a result — what was checked, what was held back, and where the evidence stops.












