๐Ÿ“  word2vec

= a technique for learning word embeddings using shallow neural network


General approach:

  1. Train on binary prediction task
    • Is w likely to show up near v?
  2. Use learned classifier weights as word embeddings

Skip-Gram task

  • Approach: Predict if candidate word c is a neighbor
  • Positive examples: target word t and context word c (+/- 2 word window)
  • Negative examples: Random words from lexicon
  • Train a ๐Ÿท ML Classifiers to distinguish the two cases (โ†’ ๐Ÿšถ Logistic Regression)
    • Iteratively make the embeddings for a word:
      • more like the neighbors embeddings
      • less like the embeddings of other words
  • Use the learned weights as the embeddings