Accuracy & testing — how Leo, Nox's safety layer, is measured
How Leo, Nox's safety layer, is tested, what the measured numbers actually mean, and how the medical review process works. Every figure on this page is generated by the test harness and committed with the code — none of it is hand-typed marketing copy.
Scope: These numbers describe ONLY Leo's deterministic red-flag detector, measured on Nox's own labeled test set. The test set is authored and maintained by the Nox team alongside the detector — it is not an independent or clinical benchmark. These figures are not a clinical certification, a diagnostic-accuracy measure, a sensitivity/specificity study, or a guarantee for real-world symptoms.
Headline metrics
- Overall recall: 100.0% (150 of 150 should-fire cases produced a safety note)
- False-positive rate: 0.0% (0 of 53 benign cases triggered a note)
- Test set: 203 labeled cases · 107 rules across 75 categories
Recall by category
| Category | Cases caught | Recall |
|---|---|---|
| Headache | 6/6 | 100.0% |
| Eye pain | 5/5 | 100.0% |
| Chest pain | 5/5 | 100.0% |
| Shortness of breath | 5/5 | 100.0% |
| Abdominal pain | 9/9 | 100.0% |
| Fever | 3/3 | 100.0% |
| Allergic reaction | 5/5 | 100.0% |
| Mental-health crisis | 5/5 | 100.0% |
| Pregnancy | 5/5 | 100.0% |
| Child symptoms | 5/5 | 100.0% |
| Stroke (FAST) | 6/6 | 100.0% |
| Seizure | 4/4 | 100.0% |
| Low blood oxygen (SpO₂) | 4/4 | 100.0% |
| Severe dehydration | 4/4 | 100.0% |
| Fever with confusion | 3/3 | 100.0% |
| Overdose / poisoning | 4/4 | 100.0% |
| Severe bleeding | 1/1 | 100.0% |
| Sepsis warning signs | 1/1 | 100.0% |
| Diabetic emergency | 1/1 | 100.0% |
| Necrotizing fasciitis (flesh-eating infection) | 2/2 | 100.0% |
| Toxic shock syndrome | 1/1 | 100.0% |
| Meningitis rash (non-blanching) | 2/2 | 100.0% |
| Testicular torsion | 1/1 | 100.0% |
| Choking | 1/1 | 100.0% |
| Severe burn | 1/1 | 100.0% |
| Heat stroke | 1/1 | 100.0% |
| Aortic dissection | 2/2 | 100.0% |
| Abdominal aortic aneurysm | 2/2 | 100.0% |
| Blood clot in the leg (DVT) | 2/2 | 100.0% |
| Fainting / syncope | 2/2 | 100.0% |
| Dangerous heart rhythm | 1/1 | 100.0% |
| Hypertensive crisis | 2/2 | 100.0% |
| Blocked artery in a limb | 1/1 | 100.0% |
| Head injury | 2/2 | 100.0% |
| Spinal injury | 1/1 | 100.0% |
| Cauda equina syndrome | 1/1 | 100.0% |
| Broken bone / open fracture | 2/2 | 100.0% |
| Compartment syndrome | 1/1 | 100.0% |
| Crush injury | 1/1 | 100.0% |
| Stab / gunshot / impalement | 1/1 | 100.0% |
| Drowning / near-drowning | 1/1 | 100.0% |
| Hypothermia | 1/1 | 100.0% |
| Frostbite | 1/1 | 100.0% |
| Severe altitude sickness | 1/1 | 100.0% |
| Smoke inhalation | 1/1 | 100.0% |
| Snake bite | 1/1 | 100.0% |
| Animal / human bite | 2/2 | 100.0% |
| Airway swelling (stridor) | 1/1 | 100.0% |
| Severe asthma attack | 1/1 | 100.0% |
| Coughing up blood | 1/1 | 100.0% |
| Unable to urinate | 1/1 | 100.0% |
| Kidney stone complications | 2/2 | 100.0% |
| Bowel obstruction | 1/1 | 100.0% |
| Strangulated hernia | 1/1 | 100.0% |
| Ectopic pregnancy | 1/1 | 100.0% |
| Ovarian torsion | 1/1 | 100.0% |
| Postpartum emergency | 2/2 | 100.0% |
| Priapism | 1/1 | 100.0% |
| Sudden hearing loss | 1/1 | 100.0% |
| Thyroid storm | 1/1 | 100.0% |
| Adrenal crisis | 1/1 | 100.0% |
| Sickle cell crisis | 1/1 | 100.0% |
| Fever while immunocompromised | 1/1 | 100.0% |
| Severe alcohol withdrawal | 1/1 | 100.0% |
| Serotonin syndrome | 1/1 | 100.0% |
| Severe drug reaction (SJS) | 1/1 | 100.0% |
| Acute psychosis | 1/1 | 100.0% |
| Eating-disorder complications | 1/1 | 100.0% |
| Spreading dental infection | 2/2 | 100.0% |
| Uncontrolled nosebleed | 1/1 | 100.0% |
| Severe menstrual bleeding | 1/1 | 100.0% |
| Dialysis emergency | 1/1 | 100.0% |
| Newborn jaundice | 1/1 | 100.0% |
| Sexual assault | 1/1 | 100.0% |
| Domestic violence | 1/1 | 100.0% |
What is being tested
Before any AI model answers, Leo's deterministic detector — plain pattern-matching code, not AI — screens each message for acute red-flag presentations like stroke signs, chest pain, severe breathing difficulty, or a mental-health crisis. Because it is deterministic, it can be tested exactly: the same input always produces the same result.
How it is tested
The team maintains a labeled test set of emergency, urgent, and benign messages. Every case runs through the same detection function the live app uses. Recall is the share of should-fire cases that produced a safety note; the false-positive rate is the share of benign cases that triggered one anyway.
What happens when a rule matches
A match is graded at one of two severity levels. An emergency match (for example stroke or heart-attack signs) puts a prominent banner at the top of Nox's reply that tells you to call your local emergency number — the exact number is chosen from your saved region, then your browser's language, falling back to the universal 911 / 999 / 112 guidance. An urgent match adds a strong 'get seen promptly' note. Either way the banner is added before the AI's answer, and the AI is instructed to lead with seeking care rather than self-treatment.
Why you can trust the numbers on this page
The metrics report is machine-generated by a committed command, and an automated test fails the build if the rules or the test set change without regenerating it. The numbers rendered here are read from that committed report at build time — the page physically cannot show numbers the test suite didn't produce.
The medical review mechanism
How safety changes are reviewed
Every change to Leo's red-flag rules requires re-running the evaluation harness and committing the regenerated metrics report — an automated test enforces this, so the published numbers can never silently drift from the live detector. The rule categories, severity thresholds, and recommendations are maintained against a written governance record with named data sources and licensing.
Clinician review status: pending (honestly)
Leo's red-flag rules and safety prompts have not yet completed an independent review by a licensed clinician, and Nox claims no clinician endorsement until one is on record. When a real review is completed, the reviewer's name, credentials, scope, and date will be published on the Trust & Transparency page — and re-reviewed whenever the rules change materially, and at least once a year.
Conservative by design
The detector is intentionally tuned to prefer an unnecessary safety note over a missed emergency. It reads the latest message only, covers a fixed list of high-risk categories, and does not diagnose — emergencies outside its categories may not trigger a note, which is exactly why Nox never describes it as catching everything.
A second AI backstop, measured separately
Beyond the deterministic detector these numbers describe, Nox runs a second safety check only when the pattern rules find nothing: a lightweight AI classifier that catches dangerous descriptions worded in slang, another language, an older disease name, or indirectly. Because it uses AI and is not deterministic, it is evaluated on its own separate test set and is deliberately NOT included in the recall and false-positive numbers on this page — those describe only the deterministic detector. The backstop can only add a safety note, never remove one.