🔪 Tokenization §

= the process of splitting up text into an array of tokens (e.g. words/symbols)

How? §

• 🔡 Regular Expressions
  • e.g. s/\s+/\n/ or s/(\w[\w\d]+)\./\1\. /

📖 Example: §

• “I like cookies” → “I” “like” “cookies”