captionbob.transcript.speech_coverage

Speech coverage: the places where the speech of a video has no word.

Whisper writes the text of 30 s of audio in one window of at most 448 tokens. Fast speech in a language with many tokens per word, such as Hindi, needs more. Whisper then stops early in the window. It stretches the last word that it wrote to the end of the window, or it leaves a gap. The captions then show one word for many seconds while the person speaks.

This module finds such places, so that the transcriber can run Whisper again on each of them. A place is flagged when it holds at least MIN_UNCOVERED_SECONDS of speech that no word covers. Each word covers its own time up to WORD_COVER_SECONDS, so a stretched word covers only its start. A word longer than MAX_WORD_SECONDS is flagged too, for speech that the voice detector misses.

The module has no model. The speech parts come from a voice detector, and the words come from the transcriber.

Source file: captionbob/transcript/speech_coverage.py

TimeSpan

@dataclass(frozen=True)
class TimeSpan

A part of the audio.

Args:

• start_seconds: Time at which the part starts.
• end_seconds: Time at which the part ends.

Raises:

• ValueError: When a time is negative or the end is before the start.

TimeSpan.length_seconds

@property
def length_seconds(self) -> float

The length of the part in seconds.

UncoveredSpeech

@dataclass(frozen=True)
class UncoveredSpeech

A place of the audio where speech has no word.

Args:

• span: The place, from the first speech with no word to the last.
• speech_seconds: The seconds of speech in the place that no word covers.

uncovered_speech

def uncovered_speech(
        words: Iterable[Word],
        speech: Iterable[TimeSpan]) -> tuple[UncoveredSpeech, ...]

Give the places where speech has no word.

Pieces of speech with no word that are at most MERGE_GAP_SECONDS apart count as one place. Only a place with at least MIN_UNCOVERED_SECONDS of such speech is given.

Args:

• words: The words of the transcript.
• speech: The parts of speech that the voice detector found, in time order, with no overlap.

Returns:

The places in time order.

long_words

def long_words(words: Iterable[Word]) -> tuple[Word, ...]

Give the words that last longer than MAX_WORD_SECONDS.

Args:

• words: The words.

Returns:

The long words in their order.

repair_places

def repair_places(
        words: Iterable[Word],
        speech: Iterable[TimeSpan]) -> tuple[TimeSpan, ...]

Give the places that Whisper must run on again.

A place is a place of speech with no word or a long word. Each place starts PLACE_LEAD_SECONDS earlier, and it is widened to the start and the end of the words that overlap it, so that a stretched word is heard again with the words before it and the speech that it hides. Places that overlap or touch become one place.

Args:

• words: The words of the transcript.
• speech: The parts of speech that the voice detector found, in time order, with no overlap.

Returns:

The places in time order, with no overlap.

replace_place_words

def replace_place_words(
        words: Iterable[Word], place: TimeSpan,
        new_words: Iterable[Word]) -> tuple[Word, ...]

Put the words of a new run in place of the words of a place.

The words that overlap the place go, and the new words inside the place come, each cut to the place. A new word that starts in the last PLACE_END_MARGIN_SECONDS of the place is left out, because Whisper wrote it past the end of the cut audio. A new word at an edge of the place that is the word next to the place heard again is left out. When the new words are not more than the old words of the place, the old words stay, because the new run found nothing that the first run missed.

Args:

• words: The words of the transcript, in the order of their starts.
• place: The place of the new run.
• new_words: The words of the new run.

Returns:

The words in the order of their starts.

cut_long_words

def cut_long_words(words: Iterable[Word]) -> tuple[Word, ...]

Cut each word that is longer than MAX_WORD_SECONDS to that length, so that its caption does not stay on the screen. Each cut is logged as a warning.

Args:

• words: The words.

Returns:

The words in their order, with the long words cut.