captionbob.transcript.whisper_transcriber

Whisper transcriber: the words of a video with their times.

The transcriber runs a Whisper model with faster-whisper. When ctranslate2 finds an NVIDIA GPU, the model runs on the GPU with float16, and the transcriber opens the NVIDIA libraries of the Python environment before it makes the model. When ctranslate2 finds no NVIDIA GPU, the model runs on the CPU with int8, and the transcriber opens no NVIDIA library. The model must be in the local Hugging Face cache already. The transcriber never downloads a model.

Whisper writes 30 s of audio in one window of at most 448 tokens. Fast speech in a language with many tokens per word, such as Hindi, needs more, and Whisper then skips the rest of the window. After the first run, the Silero voice detector of faster-whisper marks the speech. Each place where speech has no word, or where a word is too long, is transcribed again on its own, as speech_coverage.py describes.

Source file: captionbob/transcript/whisper_transcriber.py

WhisperSettings

@dataclass(frozen=True)
class WhisperSettings

Which Whisper model the transcriber runs, and how.

Args:

• model_name: Name of the faster-whisper model, for example "large-v3".
• compute_type: Number type of the model, for example "int8", or None for the number type of the device that runs the model: float16 on the GPU and int8 on the CPU.

Raises:

• ValueError: When the model name is empty, or when the compute type is an empty text.

whisper_device

def whisper_device() -> str

Choose the device that runs the Whisper model.

Returns:

GPU_DEVICE when ctranslate2 finds an NVIDIA GPU, and CPU_DEVICE when it finds none.

device_compute_type

def device_compute_type(settings: WhisperSettings, device: str) -> str

Give the number type of the model on a device.

Args:

• settings: The settings of the transcriber.
• device: GPU_DEVICE or CPU_DEVICE.

Returns:

The compute type of the settings, or the compute type of the device when the settings name none.

words_from_segments

def words_from_segments(segments: Iterable[Any]) -> tuple[Word, ...]

Turn the segments of faster-whisper into one list of words.

Args:

• segments: The segments that the model gives. Each segment holds its words, and each word has the values word, start, end and probability.

Returns:

The words of all segments in their order. Each text has no space around it. A word with no text is left out.

Raises:

• ValueError: When a segment holds no word list, or when a word has an impossible time or probability.

segments_with_progress

def segments_with_progress(
        segments: Iterable[Any], total_seconds: float,
        report_progress: ProgressReporter) -> Iterator[Any]

Give the segments of faster-whisper one by one, and report the progress before each one.

The model works while the segments are read, so the end time of a segment is the part of the audio that is done. The reported time is never more than the total. Audio with no length reports nothing.

Args:

• segments: The segments that the model gives. Each segment has its end time in seconds.
• total_seconds: The seconds of audio in all.
• report_progress: Takes the seconds done and the seconds in all.

Yields:

Each segment in its order.

read_audio

def read_audio(media_path: Path) -> numpy.ndarray

Read the audio of a video or audio file as Whisper takes it.

Args:

• media_path: The video or audio file.

Returns:

The audio as one channel of float samples at SAMPLE_RATE.

speech_spans

def speech_spans(audio: numpy.ndarray) -> tuple[TimeSpan, ...]

Give the parts of the audio where someone speaks, from the Silero voice detector of faster-whisper.

Args:

• audio: The audio at SAMPLE_RATE.

Returns:

The parts of speech in time order, with no overlap.

WhisperTranscriber

class WhisperTranscriber
    def __init__(self, settings: WhisperSettings) -> None

Gets the words of a video from its audio with a Whisper model.

The model is loaded when the transcriber is made, on the GPU when ctranslate2 finds an NVIDIA GPU and on the CPU otherwise, and it stays there while the transcriber lives. One transcriber can transcribe many videos.

Args:

• settings: The model name and the compute type.

Raises:

• FileNotFoundError: When the model, or on the GPU an NVIDIA library of the environment, is not on this machine.
• OSError: When an NVIDIA library cannot be opened.

Choose the device, open the NVIDIA libraries for the GPU and load the model.

WhisperTranscriber.transcribe

def transcribe(
        self, media_path: Path,
        report_progress: ProgressReporter | None = None) -> Transcript

Get the words of one video or audio file.

After the first run, each place where speech has no word, or where a word is too long, is transcribed again, as _repair_skipped_speech describes. The progress covers the first run.

Args:

• media_path: The video or audio file to transcribe.
• report_progress: Takes the seconds of audio done and the seconds in all as each segment is done, or None for no progress.

Returns:

The transcript with the language, the audio duration and each word with its times. The word list is empty when the audio has no speech.

Raises:

• FileNotFoundError: When the file does not exist.
• ValueError: When the model gives an impossible word.