Whisper's Word Timestamps Hide Your Pauses. Here's How We Fit Subtitles to the Real Speech

ClipUtils transcribes with Whisper large-v3-turbo through a hosted API, asking for word-level timestamps. On real speech with pauses, three things happen.

First, each word ends where the next one starts. A 1.1-second pause after "guys," is counted as part of "guys,", so a caption showing that word stays up through the pause. In our 17-second test, burned-in captions sat over 6.3 seconds of silence.

Second, fillers vanish but their time doesn't. Whisper usually leaves "um" out of the text, and the time it took gets added to a neighbouring word. In our clip, the word "Utils." was stamped as lasting 2.7 seconds because an "um" and a pause were folded into it.

Third, short clips often come back as a single segment. The whole 17-second video was one line.

None of this is a bug in Whisper: its timestamps come from the model's attention, not from listening for silence. But anything built on top of them, captions, cuts or dubbing, inherits the problem: every pause becomes invisible.

Before touching the words, we needed ground truth: where in the audio is someone actually making sound? A simple energy detector is enough. The audio (16 kHz mono) is cut into 10-millisecond frames, and each frame's loudness is measured in decibels.

A frame counts as sound when it is at least 12 dB louder than the room's own noise (the quietest tenth of the recording) and within 30 dB of the loudest speech. Gaps shorter than 120 milliseconds are inside a word, because consonants like "t" and "k" make tiny silences, so they are bridged. Bursts shorter than 60 milliseconds are clicks and are dropped.

On the test clip this found 12 stretches of sound and pauses of up to 1.1 seconds, exactly what you hear when you play it.

Each word is then re-timed against that sound. If its stamped time contains a quiet stretch of 250 milliseconds or more, the word ends there: the rest of its time was really a pause. Its start and end then snap to the edges of its sound, as long as no other word claims that sound.

One case took a whole evening. One word, "Hi,", was stamped over two separate sounds: a flat hum at 4.3 seconds and a short rising sound at 5.2 seconds. One is the word, the other is probably "hmm". Pitch analysis couldn't tell them apart, because the speaker's earlier, drawn-out "Hiii" was just as flat. Even Windows' built-in speech recognizer heard a real "Hi" as "hmm".

So the rule became: when a word starts after a pause and holds two separate sounds, it covers both. For captions that's harmless. For an editor that deletes audio, it's the safe choice: it might leave a stray "hmm" in, but it will never cut a real word.

With honest word times, splitting into caption lines becomes easy. A line breaks at any pause of 0.6 seconds or more, and after a sentence ends once the line holds at least a second of speech.

If a line would run past 5.5 seconds or 42 characters, it breaks at the most natural point inside it: after a comma, or at its longest pause. Never in the middle of a name: our first attempt split "Clip | Utils" across two lines.

The 17-second clip went from one subtitle line to five, each starting and ending with the speech.

This bug was hiding everything else in the editor. "Shorten long pauses" decided what counts as quiet with a rule like "25 dB below the speech". On a studio recording that works. On a phone recording, the speech sat around -22 dB, so quiet meant below -47 dB, while the room's own noise from a fan or the street sat at -42 dB. The room was never quiet enough to count as a pause, so the feature silently did nothing on phone recordings, which is most of what people upload.

The new rule: quiet means 25 dB below the speech or 6 dB above the room's noise, whichever is higher, but never closer than 12 dB to the speech. The editor now shortens both 1.1-second pauses in the test clip.

Two more rules make cuts precise. A cut stops where a kept word's sound begins, because timestamps can be 0.3 seconds late: "So," was stamped as running into the "Hi" after it, so cutting "So," used to clip the start of "Hi". And an unwritten "um" alone in a long gap goes with the pause: if the gap between two words holds at most a second of voice, it's a filler or a breath, and it's cut with the silence.

On the same 17-second video, the subtitle file went from one line covering 0:00 to 0:16 to five lines that start and end with the speech. The AI video editor went from finding no pauses to shortening both 1.1-second pauses and removing the unwritten "um", taking the video from 17.2 seconds to 13.9 seconds. Every kept word stayed 100% intact, and test cuts of a filler, a retake and a single word each removed exactly their own sound.

The same fitted timing now feeds AI dubbing too, so the dubbed voice is paced to the time the speaker actually spoke, not to times padded with pauses.

Check times against the audio, not the transcript: the words can be perfect while the timing is off by a second.

Measure the room, not just the speech: thresholds that ignore the noise floor work in tests and fail on phones.

When a mistake would destroy something, pick the rule that can't: leaving an "um" in is a small miss, while cutting a word is a broken video.