captionbob.emoji.emoji_matcher - captionbob docs

Emoji matcher: the emoji closest in meaning to a word.

The emoji index tool, captionbob/devtools/emoji_index.py, writes an emoji index folder once. The folder holds the index file, one vector for each name and one vector for each keyword of each emoji, the emoji images and the model. The matcher loads the folder. It compares a word with the names of an emoji, such as pill, and with its keywords, such as medicine. The index tool leaves out the keywords that are English stop words, such as you or like, because a caption word that is a keyword gets a high peak.

For each word the matcher compares the meaning vector of the word with the vector of each name and keyword. The score of a text is the cosine of the two vectors. The model gives a high cosine to almost every pair of short texts, so the score alone does not tell a keyword from a word such as "the". The peak of a word tells them apart: it is how far the score of its best text stands above the mean of its scores of all the texts, counted in standard deviations of those scores. A word such as "dog" has one name far above the rest. A word such as "the" has no name far above the rest.

A word is a keyword when its peak reaches the minimum peak. The words of a caption go through no list of stop words, so the matcher works in every language of the model. The text of a word keeps its letters, its joining marks, such as the vowel signs of Hindi, and its numbers, by the rule of captionbob/transcript/keywords.py. The emoji of a caption is the emoji of its keyword with the best peak. A caption with no keyword gets no emoji.

Source file: captionbob/emoji/emoji_matcher.py

EmojiEntry

class EmojiEntry(NamedTuple)

One emoji of the index.

Args:

codepoint: The Unicode codepoints of the emoji in hex, joined by _, such as 1f600.
names: The names of the emoji, such as pill.
image_path: The image of the emoji that an effect shows.
keywords: The keywords of the emoji, such as medicine. The matcher compares the names and the keywords with words.

EmojiMatch

class EmojiMatch(NamedTuple)

The closest emoji of a keyword.

Args:

emoji: The emoji.
word: The keyword, as the matcher compares it.
name: The name or the keyword of the emoji that is closest to the keyword.
score: The cosine of the vectors of the keyword and the name.
peak: How far the score stands above the mean of the scores of the keyword with all the names, in standard deviations.

Embedder

class Embedder(Protocol)

Gives the meaning vectors of texts, as TextEmbedder does.

Embedder.embed

def embed(self, texts: Sequence[str]) -> np.ndarray

Give one vector of length 1 for each text, as float32.

match_text

def match_text(word_text: str) -> str

Give the text of a word as the matcher compares it.

The marks at the two ends that are not part of the word, such as a comma, a full stop or a quote, are taken off, and the letters are made small.

Args:

word_text: The text of one word of the transcript.

Returns:

The text to compare. It is empty when the word has no letter, no joining mark and no number.

joined_vectors

def joined_vectors(
        names: Sequence[Sequence[str]], keywords: Sequence[Sequence[str]],
        name_vectors: np.ndarray, keyword_vectors: np.ndarray) -> np.ndarray

Join the vectors of the names and of the keywords of each emoji.

Args:

names: The names of each emoji.
keywords: The keywords of each emoji, in the same order.
name_vectors: One vector for each name, in the order of the emojis and of their names.
keyword_vectors: One vector for each keyword, in the order of the emojis and of their keywords.

Returns:

One vector for each text, in the order of the emojis and, for each emoji, of its names and then its keywords.

Raises:

ValueError: When the lists of names and keywords do not have one item for each emoji, or when the vectors do not have one row for each name or keyword.

EmojiMatcher

class EmojiMatcher
    def __init__(
            self, entries: Sequence[EmojiEntry], vectors: np.ndarray,
            embedder: Embedder,
            minimum_peak: float = DEFAULT_MINIMUM_PEAK) -> None

Finds the closest emoji of words, and the emoji of a caption.

Args:

entries: The emojis.
vectors: One vector of length 1 for each name and keyword, in the order of the emojis and, for each emoji, of its names and then its keywords, in the shape (texts, vector size).
embedder: Gives the vectors of the words.
minimum_peak: The lowest peak of a keyword, 0 or above.

Raises:

TypeError: When the minimum peak is not a number.
ValueError: When there are fewer than FEWEST_TEXTS names and keywords, when the vectors do not have one row for each of them, or when the minimum peak is not finite or is below 0.

EmojiMatcher.load

@classmethod
def load(
        cls, index_folder: Path,
        minimum_peak: float = DEFAULT_MINIMUM_PEAK) -> "EmojiMatcher"

Load an emoji index folder that the emoji index tool wrote.

Args:

index_folder: The emoji index folder.
minimum_peak: The lowest peak of a keyword, 0 or above.

Returns:

The matcher, with the model of the folder.

Raises:

FileNotFoundError: When a file of the index, an emoji image or a file of the model is not there.
ValueError: When the index has another form or another model, or when the vectors do not agree with the names, the keywords or the model.

EmojiMatcher.entries

@property
def entries(self) -> tuple[EmojiEntry, ...]

The emojis of the index.

EmojiMatcher.match_words

def match_words(
        self, word_texts: Sequence[str]) -> tuple[EmojiMatch | None, ...]

Give the closest emoji of each word that is a keyword.

Args:

word_texts: The texts of the words.

Returns:

For each word, the emoji of its closest name or keyword with the score and the peak, or None when the peak is below the minimum peak or the word has no letter. A name wins a tie with a keyword. A word whose scores of all the texts are the same has no peak.

EmojiMatcher.best_emoji

def best_emoji(self, words: Sequence[Word]) -> Path | None

Give the emoji of the words of one caption.

Args:

words: The words of the caption.

Returns:

The image of the closest emoji of the keyword with the best peak. The first such keyword wins a tie. None when no word is a keyword.