Attention Mechanisms : Information Retrieval :: Contextual Embeddings : ?

Attention mechanisms help systems find the most relevant pieces of information, much like a search engine pulls up the best documents. Contextual embeddings do a similar job but for language: they turn each word into a vector that depends on its surroundings, so the model knows exactly what that word means in that sentence. These embeddings are then used for a wide range of language tasks, such as translating a sentence, answering a question, or deciding if a review is positive or negative. For example, the word “bank” in “river bank” gets a different embedding than in “bank account,” allowing the model to pick the right meaning.