Skip to main content

Minotaur · how Mythos AI Security validates itself

Find What We Missed. Prove What We Test.

Minotaur is Mythos AI Security’ internal proving ground. It measures Athena and Achilles against planted benchmarks whose answers are known in advance — and it is the release authority for every engine.Internal to Mythos AI Security — never sold, and not independent third-party validation.

Why assurance needs assurance

A test can be wrong in quiet ways.

A security result is only as good as the harness that produced it. These are the ways a harness fails without anyone noticing — and what Minotaur is there to catch.
  • Contamination

    Could the system have seen the test?

    If test cases leak into training or tuning data, a score measures memory, not behavior.

  • Invalid scorers

    Does the scorer measure what it claims?

    A grader that rewards the wrong signal turns a failure into a pass — and nobody looks twice at a pass.

  • False blocks

    Is it catching, or just refusing?

    A control that blocks benign work can look perfect on attacks alone. Safe twins make precision count.

  • Distribution shift

    Does the test still look like production?

    Models, prompts, tools and users change. A benchmark frozen in the past slowly stops describing the system.

  • Unwatched exits

    Did the test see every way out?

    An effect can leave through a channel the harness never observed. A clean result there means unknown, not safe.

In place today

  • Fifteen benchmark families, answers keyed in advance
  • Decoys and safe twins; precision decides
  • Validity Cards, partitions and external reproduction
  • Nine release gates and a security veto

Not yet in place

  • A pass mark — no family has one
  • An audited Validity Card — none has been audited
  • A sealed holdout — none is sealed yet
  • Runs against public targets such as Juice Shop or DVWA

The labyrinth

Every result has to find its way out.

Each ring is a question a result must answer before anyone relies on it. Choose a ring to trace the path to it.

Fifteen benchmark families plant issues on purpose, with answers keyed in advance, and no runner holds the key. If a test misses a known issue, the test is what’s wrong.

Known answers and safe twins

Expected versus observed.

The benchmarks plant issues whose answers are keyed in advance, with decoys and safe twins beside them — so a harness that flags everything is punished as surely as one that flags nothing.
  • Known-answer case

    As expected

    A planted access flaw in the target range

    Expected
    Found
    Observed
    Found

    The answer was keyed before the run and no runner holds it, so the harness can’t be graded on a moving target.

  • Safe twin

    As expected

    The same request, with the access it’s entitled to

    Expected
    Allowed
    Observed
    Allowed

    The harness told the two apart — the result above is evidence, not a reflex.

  • Counterexample

    Mismatch

    A second safe twin, phrased differently

    Expected
    Allowed
    Observed
    Blocked

    A false block. It counts against precision, and precision decides.

Illustrative cases. Inputs are described, never reproduced — corpora and answer keys stay internal.

Benchmark Validity Card

What a score depended on.

No score is published without its Validity Card: a per-family record of what the score means and does not mean. None has been audited yet. This synthetic card shows what one records, so a result is read in context.
Benchmark Validity Card · syntheticApproval-boundary scenarios
FieldRun v3 · illustrative
Harness owner
Validation lead
Harness version
harness 3.2
Scenario set
approval-boundary set · fixture 7d1e…
Partition
holdout arena · not sealed
Context boundary
single session; no shared memory
Tool visibility
refund, CRM read, email
Seed
4417 (fixed)
Scorer
signal-based; narration ignored
Safe twins flagged
2 of 40
Known gaps
1, counted apart from misses
Reviewer
Second analyst, not the author

False blocks and precision

Refusing more is not the same as better.

Attacks caught are counted beside legitimate cases let through. Precision decides, so a build cannot score better by refusing more, and nothing here claims zero false positives.
Synthetic fixture
Outcomes for Defender build A: requests by whether they were blocked or allowed
BlockedAllowed
Harmful request20Attack caught0Attack missed
Benign request9False block31Legitimate case let through

Catches every planted attack — and blocks nine legitimate cases. Precision decides, so this build does not count as better.

Reproduction and regression

Run it again. Get the same answer.

Every score is tied to its benchmark, answer key and engine commit. An external reproduction package reproduced a row byte for byte in CI, and sealed scorecards are re-verified on read; a mismatch is flagged tampered.
Version-linked trail · synthetic
  1. Answers keyed

    fixture 7d1e…c04Keyed

    Cases and answers are fixed before any engine runs; no runner holds the key.

  2. Baseline run

    harness 3.2 · seed 4417Observed

    The engine runs in a separated child; the sealed scorecard records the versions it came from.

  3. External reproduction

    same fixture · reproduction packageReproduced

    The package reruns the row in CI and must match it byte for byte.

  4. Mitigation applied

    engine change 1.14Stale

    A fix lands. Every result that depended on the old behavior is now stale.

  5. Regression run

    harness 3.2 · seed 4417Observed

    The earlier cases run again for the Regression gate, which reads unmeasured, never green, until they do.

Release authority

Hold on evidence, not on averages.

Nine gates govern each engine’s release branch: Detection, Precision, Policy, Action Gate, SPINE, Evidence, Performance, Coverage and Regression. Unmeasured is never green, and a security veto sits outside the score.

Evidence on record · synthetic

  • 49 of 50 planted cases caught
  • One case caught last release is now missed
  • Precision unchanged on safe twins
Hold

One regression is not averaged away. The weakest gate sets the outcome.

This governs Mythos AI Security’ own engine releases. Your deployment decision stays with your people.

Keep going

Where the method shows up.

Minotaur’s work surfaces in the evidence other parts of Mythos AI Security produce — and in what we choose to publish about how we test.

Mythos AI Security

Ask us how we test.

We will walk through the method behind a result — what was checked, what was held back, and where the evidence stops.