Benchmarks

Every number on this page comes from replaying real speech through the exact pipeline the app ships, with the date, the sample size and the method attached. Nothing is extrapolated. The story of why is on the Accuracy page; this page is the numbers.

The challenge

Building a voice-to-text app with an AI tool? Point it here. Tell your AI to run the benchmarks on this page. When it's done, you'll see why usable everyday accuracy takes hundreds of hours of development — otherwise words go missing, words get invented, words get mangled, and numbers, SKUs, model codes, IDs and dashes come out wrong.

Just a consumer? Try it yourself. Dictate any example on this page — or in the try-these table on the Accuracy page — into the app you're using now, maybe paying for, and count your manual corrections. That's where Fabled Flow's hundreds of hours went.

The standard score

Measured 2026-08-08 on build 10.0 (b734), on the COMPLETE public test sets — every utterance, zero excluded. We score with the industry-standard methodology (AssemblyAI's published WER method) against our own stricter standard: fully formatted output, nothing normalized away. The other products in the table publish no dataset-named numbers at all. One footnote we volunteer ourselves: LibriSpeech — one leg of the benchmark's multi-source suite, and the one our complete-set numbers above come from — is 19th-century audiobook prose, and our own error audit attributes roughly 40–48% of our charged "errors" to benchmark noise — US/UK spellings, archaic proper nouns, our designed number formatting being penalized for being right. The full explanation is on the Accuracy page.

What is WER?

WER stands for Word Error Rate. It's the score behind the industry's public leaderboards, including the Open ASR Leaderboard where the engines are ranked, and it counts one thing: against a reference transcript, how many words got substituted, invented, or lost. Lower is better.

So what does a good WER score prove? That the model didn't miss your words. Which is very useful — it's the first thing you need. If the words aren't there, nothing else matters.

Here's what the score does not prove. Before any errors are counted, the methodology deliberately strips both texts down: capitals gone, punctuation gone, sentence boundaries gone, written-out numbers flattened into digits. The leaderboard's own paper says it plainly: "The pipeline includes number normalization (e.g., 'zero' to '0')" (arXiv 2510.06961). The score is computed on text no one would ever hand to another person.

That's the honest ladder. An engine that doesn't miss words is the foundation. Then you need something on top of it that turns what the engine heard into text you can actually use — otherwise it's gibberish. It's chaos. It's messy. Every dictation product is an engine plus something on top. The question is what the something is. Ours is the Editor.

What the score cannot see

Every pair below scores identically under the standard methodology — zero measured difference. Left is the correct text. Right is what an engine can hand you instead:

Words didn't go missing in any of these examples — so the score sees no difference. But none of them are usable to the person dictating: they'd make every fix by hand, because formatting isn't taken into consideration.

Sit with the first row for a second. A dog's weight became a price. That sentence is not benchmark data — it comes from our own book-length test runs, reading Call of the Wild through the app, where we caught a build formatting the dog's weight as currency. The scoring arithmetic is the benchmark's own, though: its normalizer converts "140 pounds" into "£140" before counting, so weight and money are the same word to it — same score, opposite meaning. No standard score would ever have seen it.

And the benchmark's own data does the same thing with money. The benchmark is a multi-source suite — audiobooks, talks, parliament, meetings, earnings calls — and the rows above come from two of its test sets. The fine row is from its LibriSpeech set: the reference spells out "a fine of three hundred dollars", our app shipped "a fine of $300", and the score cannot tell them apart. The two earnings rows are real reference sentences from its earnings-call set, Earnings-22 — modern speech, dense with the numbers people actually dictate — where "86 million up 60%" and the fully spelled-out form score identical. The scorer even deletes the reference's own "Uh," before counting.

Why identical? Before the first error is counted, the methodology lowercases everything, deletes punctuation and symbols, and converts written-out numbers to digits — on both sides. Case, sentences, codes, addresses, currency-versus-weight: gone before counting starts.

And one case that is nearly free, so we say so instead of rounding it to zero: dictate the buzzer code 1297822 and get one hundred 297822 — a mangled, unusable code that costs just two word errors under the standard score. Nearly free. Not zero. The page doesn't overstate, even in our favor.

Examples drawn from the standard benchmark's own test data (its LibriSpeech and Earnings-22 sets) and our recorded test runs. Rerun any pair yourself — the scoring rules are linked in the table header.

Speed

Accuracy and speed together is the product's premise, so speed is a benchmark here, not a slogan:

• Key release → text landed: 0.5–0.8 s typical with AI formatting off — the shipped claim, measured across real daily use; median full round-trip to pasted text ~1.0 s (MacBook Neo, July 2026).
• Engine decode: ≈55 ms per second of audio, measured warm on the shipping build on an 8 GB machine (2026-08-08): a 4-second dictation decodes in about 0.22 s, a 14-second one in about 0.77 s.
• Optional AI formatting adds 1–2 s — still entirely on-device, and off by default.
• Long-form delivery is flat with length (Rolling Decode). Fabled Flow transcribes while you speak, so release-to-text stays seconds-class at any length. Measured: a 74.1 s dictation delivered in 867 ms where a competitor product on the same engine family took 3.8–5.0 s on the same machine (2026-08-10); and 78.3 minutes of continuous speech in ONE dictation, delivered 4.2 s after release (2026-08-11). The fastest measured long-form delivery among the products we've tested — and the best measured combination of accuracy and speed we've recorded.

Resource and head-to-head figures measured July 2026 on the same MacBook Neo (8 GB), one app at a time, same recordings; Spokenly's timing isn't measurable end-to-end.

The torture test

Public test sets contain none of the things real dictation is made of. So we built our own battery — designed by the maker from real, daily dictation — and run it end-to-end through the app on the same machine:

• Real recorded speech — talks, fast casual speech, technical dictation — scored word-by-word against human transcripts.
• Real dictation shapes — identifier codes and SKUs, spoken email and web addresses, phone numbers, currencies, dates and times, unit addresses and street names, spoken punctuation and brackets, deliberate stumbles and thinking pauses, mixed content in one breath.
• Competitor legs — the same audio replayed through competitor products' real end-to-end pipelines, one app at a time, on the same Mac.

Current baseline (July 2026 runs): 95–100% word accuracy on real recorded English speech across news-style, conversational, and fast casual sources — about 30 minutes of unique audio, roughly 7,900 scored words — and the deterministic formatting battery renders codes, addresses and numbers identically on every repeat, where the leading cloud competitor was recorded rendering identical spoken codes five different ways.

Honest status: the full torture-test re-run on the current build is scheduled. Its complete results — accuracy, formatting, and speed, beside the July baseline — will publish on this page when the run finishes. Nothing here is extrapolated in the meantime.

How we measure

Three things matter in dictation, and we score all three:

• Speed — from key release to text ready, and to text pasted.
• Accuracy — words transcribed correctly versus a reference transcript of the same audio.
• Formatting — numbers, punctuation, lists, codes and addresses rendered the way a person would type them.

Accuracy scoring follows the industry-standard WER methodology — both texts are normalized, then every substitution, insertion, and deletion counts as one error. Our runner script pulls the samples straight from the public datasets and feeds them through the shipping app; anyone with the app can reproduce our numbers. Comparisons against other dictation apps replay the same audio through each app's real end-to-end pipeline on the same Mac, one app at a time.

Per-language reports

Benchmarks are the results page — and results are per-language. As our per-language dictation standards and native-speaker verification complete, each language's accuracy, speed and formatting results will slot in here as its own section. Available now:

• English — preliminary report
• 中文（简体）/ Chinese (Simplified) — preliminary report
• German, French, Spanish, Italian, Portuguese, Japanese, Korean, Cantonese — in verification, reports coming

Numbers on this page are updated as new measured rounds finish — never cherry-picked, never blended across builds. Where a result is preliminary or a re-run is pending, it says so next to the number.