← Blog

Measurement · Speech recognition

KB-Whisper vs Pianissimo — we measured four Swedish speech models on 1,804 dialect clips

September 2026 Fredrik Carlsson 7 min read

On 24 September 2026, Klang AI in Helsingborg released Pianissimo, an open Swedish speech model under CC BY 4.0. It went straight up against KB-Whisper from KBLab, the research lab at the National Library of Sweden. Both are fine-tunes trained on tens of thousands of hours of Swedish speech, KB-Whisper from OpenAI’s Whisper and Pianissimo from NVIDIA’s Parakeet. But nobody outside had compared them.

So we did it. Four models, 1,804 clips, the same method for all of them. This article is the result, and the raw data is at the bottom so you can recompute the table without taking our word for any of it.

Why the measurement was needed

When Klang released Pianissimo they also published a comparison against KB-Whisper. That is an honest thing to do, and their numbers turned out to hold up. But two things were missing.

The first is kb-whisper-small. It was not included, and it is the model most Swedish dictation apps actually run, because it is the only one small enough to feel fast. A comparison that leaves it out does not answer the question most users have.

The second is that nobody outside had run the numbers. A model maker’s own table is not wrong for being theirs, but it is unchecked until somebody else arrives at the same thing.

We also had a concrete reason: we had to decide which model would be the Swedish default in Sæga. That decision should not rest on a table somebody else wrote, nor on a couple of sentences read out by a speech synthesiser.

The method

The measurement runs on Klang Dialects, an open benchmark for Swedish speech recognition that Klang AI released alongside the model, under CC BY 4.0. It is 1,804 clips, 5.1 hours of speech and 656 speakers, deliberately varied in dialect and speaking style. It is the hardest freely available Swedish material there is.

Using their benchmark is a deliberate choice. Had we built our own, we would also have built in its blind spots, and nobody could have checked us. This way it cuts both ways: their published numbers are a check on our setup, and our setup is a check on their numbers.

The normalisation is theirs too, taken verbatim from the model card: lowercase, punctuation replaced with spaces, whitespace collapsed, and word error rate computed at corpus level rather than averaged per clip. Same method for all four models, same machine, one M-series Mac.

The result

Model We measured Klang published Clips with no errors Severe failures
kb-whisper-large 2.52% 2.30% 83.9% 8
kb-whisper-medium 3.81% 3.58% 77.5% 13
Pianissimo 4.94% 4.85% 70.1% 12
kb-whisper-small 5.72% unpublished 70.1% 20

Corpus-level word error rate, lower is better. “Severe failures” = clips where more than half the words came out wrong.

The check that makes the table credible

Our numbers sit consistently a little above Klang’s published ones, for all three models they reported: +0.22 percentage points for kb-large, +0.23 for kb-medium and +0.09 for Pianissimo. The deviation points the same way throughout and is small, which is exactly what you want to see: some detail of preprocessing differs, not the judgement of which model is best.

That means the ranking holds, and that the fourth number — kb-whisper-small, which nobody has published — can be trusted to the same degree as the other three.

The whole table above is also recomputed from exactly the files linked at the bottom, against reference texts pulled straight from the dataset, rather than copied over from the run. The reproducibility is tested, not just asserted.

What the numbers actually say

kb-whisper-large is substantially better, not marginally. 83.9% of clips came out entirely correct against medium’s 77.5, and clip for clip it wins 255 to 110. It also has the fewest severe failures of all four. If the only thing that matters is that the text is right, this is the one to run.

kb-whisper-small is dominated by both kb-medium and Pianissimo. Worse quality than medium, and worse than Pianissimo on both quality and speed. It is the one unambiguous finding in the whole measurement, and it is a problem for the field rather than for KBLab: small was a reasonable choice for as long as the alternative was waiting, and it no longer is.

Pianissimo’s advantage over small is slim. Both get 70.1% of clips entirely right, and head to head Pianissimo wins 319 clips to 289. Across the whole corpus that is 920 wrong words against small’s 1,066, a difference of 146. A little over half of that difference, 82 words, comes from the 20 clips where small fails severely, against Pianissimo’s 12; the rest is spread across ordinary clips.

Against kb-medium, Whisper is still 1.1 percentage points better. Pianissimo is not the most accurate Swedish model, and Klang does not claim it is. It is the fastest, by a wide margin.

Speed, and why it stands apart

Latency cannot go in the same table without misleading, for two reasons. It depends on the machine, and our figures come from two different runs in which kb-medium gave 680 milliseconds in one and 785 in the other on exactly the same audio. It is the ratio between the rows that means something, not the figures themselves.

On 5.9 seconds of Swedish speech, warm engine, median of three runs on the same Mac:

A figure we deliberately do not borrow

Klang themselves quote 3,600 times realtime for Pianissimo, meaning an hour of audio in a second. That is a real figure, but it describes their own batch run on server hardware. Our measured path on an ordinary Mac, clip by clip, lands at around 100 to 165 times realtime. Both numbers are true about different things, and borrowing theirs for our own marketing would have been false.

What we did with the answer

Sæga runs both model families and lets the user choose, but something has to be the default. We picked kb-whisper-medium, which is neither the fastest nor the most accurate.

The reason for not taking large is memory, not quality. The speech model stays loaded for as long as the app is running, and large would mean roughly 1.5 GB from login to shutdown for a menu bar utility that mostly sits still. A default has to be safe for someone who never opens a setting. Large is one click away and free.

The reason for not defaulting to Pianissimo is that an error is silent while a wait is visible. Using the figures above: the gap is 620 milliseconds on six seconds of speech. It grows more slowly than the audio does, because Whisper’s encoder always works against a fixed thirty-second window however short the recording is, so twice the dictation does not cost twice the wait. A hundred ordinary dictations therefore save on the order of one to two minutes, with the same caveat as above: the absolute figures differ between runs. Those same hundred dictations of thirty words cost about 34 extra wrong words, which is roughly the same amount of time to fix. When the arithmetic is a draw, other things decide, and a misspelled name that travels into an email costs more than a second of waiting.

It is worth saying plainly: the model we promoted most in the release this was for is the least accurate of three, and the one we did not make the default is the best. A benchmark where the author’s own choices happen to win every column is not a benchmark.

Recompute it yourself

The hypotheses from the run are published, one JSONL file per model, with the clip id and the text the model produced:

The reference transcripts and the audio come from KlangAI/klang-dialects on Hugging Face. We republish neither their audio nor their ground truth, only our own model output. Normalise as the model card describes, compute word error rate at corpus level, and you should land on the table above.

Thanks

To KBLab at the National Library of Sweden for KB-Whisper, and to Klang AI for Pianissimo and for releasing their benchmark openly instead of only publishing the result. This article could not have been written otherwise.

If you would rather choose a model in practice than read about the numbers: how to choose a speech model in Sæga → (in Swedish)

Run them on your own Mac

Sæga downloads and runs KB-Whisper and Pianissimo locally. Every speech model is free, and the audio never leaves your Mac.

Download Sæga — free to start
macOS 14 Sonoma or later · Apple Silicon

Same article in Swedish →