captionbob.transcript.word_kinds

Word kinds of a transcript, from Stanza.

The emoji keywords of a caption are its nouns, its proper nouns and its adjectives. The tagger gives each word of a transcript its word kind with the Stanza model of the language of the transcript. The model tool captionbob/devtools/stanza_models.py writes the models into the folder stanza of the emoji index folder.

The tagger gives Stanza the words of Whisper as the tokens, so each kind belongs to exactly one word. Stanza may split one token into more words, such as "del" into "de el" in Spanish; the token is a keyword when one of its words is a keyword. The words go to Stanza in sentences that end at a full stop, a question mark, an exclamation mark or a danda, and a sentence is cut at MAX_SENTENCE_WORDS, because a long sentence takes much longer.

A language with no model in the folder has no tagger. The caller then uses the letter limit of the keyword pickers. A language with no spaces between its words, such as Chinese or Japanese, gets pieces of words from Whisper, so its tags may be poor.

Stanza tags on the GPU when torch finds one, and on the CPU otherwise. The CUDA build of torch finds an NVIDIA GPU, and the CPU build of torch finds none. A loud log line names the device.

Source file: captionbob/transcript/word_kinds.py

SentenceTagger

class SentenceTagger(Protocol)

Gives the word kinds of the tokens of sentences, as Stanza does.

SentenceTagger.kinds

def kinds(
        self, sentences: Sequence[Sequence[str]]
        ) -> list[list[tuple[str, ...]]]

Give for each token of each sentence the kinds of its words.

stanza_language

def stanza_language(
        resources: Mapping[str, Any], language: str) -> str | None

Give the Stanza name of a language of Whisper.

Args:

• resources: The resources of Stanza, as its resources file holds them.
• language: The language code of the transcript, such as hi.

Returns:

The language code of Stanza, which is the code itself or the code that it is an alias of, such as zh-hans for zh. None when Stanza has no such language.

language_processors

def language_processors(
        resources: Mapping[str, Any],
        language: str) -> tuple[str, ...] | None

Give the processors of the tagger that a Stanza language has.

Args:

• resources: The resources of Stanza, as its resources file holds them.
• language: The language code of Stanza, with no alias.

Returns:

The processors of WORD_PROCESSORS that the default pipeline of the language has, in their order. None when the language has no default model for the tokens or for the word kinds.

read_resources

def read_resources(models_folder: Path) -> dict[str, Any]

Read the resources file of a folder of Stanza models.

Args:

• models_folder: The folder of the Stanza models.

Returns:

The resources of Stanza.

Raises:

• FileNotFoundError: When the folder has no resources file.
• ValueError: When the resources file does not hold a JSON object.

ends_sentence

def ends_sentence(word_text: str) -> bool

Say if a word ends a sentence.

Args:

• word_text: The text of the word. A closing mark, such as a quote, may stand behind the end mark.

Returns:

True when the last mark in front of the closing marks is one of SENTENCE_END_MARKS.

tagger_sentences

def tagger_sentences(words: Sequence[Word]) -> list[list[int]]

Cut the words of a transcript into the sentences of the tagger.

A word with no letter, no joining mark and no number, such as a dash, is in no sentence.

Args:

• words: The words of the transcript in time order.

Returns:

The places of the words of each sentence in the words, in order. A sentence ends at a word that ends a sentence, or when it has MAX_SENTENCE_WORDS words.

WordKindTagger

class WordKindTagger
    def __init__(self, tagger: SentenceTagger, language: str) -> None

Names the emoji keyword words of a transcript by their word kinds.

Args:

• tagger: Gives the word kinds of the tokens of sentences.
• language: The language code of the transcript, for the log.

WordKindTagger.load

@classmethod
def load(
        cls, models_folder: Path,
        language: str) -> "WordKindTagger | None"

Load the Stanza model of a language from a folder of models.

Args:

• models_folder: The folder of the Stanza models, which the model tool wrote.
• language: The language code of the transcript, such as hi.

Returns:

The tagger, or None when the folder has no model of the language or Stanza has no model of its tokens and word kinds.

Raises:

• FileNotFoundError: When the folder has no resources file, or when a model file of the language is missing.
• ValueError: When the resources file does not hold a JSON object.

WordKindTagger.keyword_words

def keyword_words(self, words: Sequence[Word]) -> frozenset[Word]

Give the words of a transcript that are emoji keywords.

Args:

• words: The words of the transcript in time order.

Returns:

The words whose token has a noun, a proper noun or an adjective.

Raises:

• RuntimeError: When the tagger gives another count of sentences or of tokens than it got.