🧮 tf-idf

=== a statistical measure that evaluates how relevant a 🈁 Word is to a document in a collection of documents==

Formula

==$w_{t,d}=t{f}_t\times id{f}_t$==

• $t$: term
• $d$: document
• $t{f}_t$: log transformed term frequency:
  • ${\mathrm{tf}}_{t,d}=\{\begin{array}{ll}1+{\log }_{10}\mathrm{count}(t,d)& \text{ if }\mathrm{count}(t,d)>0\\ 0& \text{ otherwise }\end{array}$
  • → greater when term is frequent in a document
• $id{f}_t$: inverse document frequency
  • ${\mathrm{idf}}_i=\log \left(\frac{N}{{\mathrm{df}}_i}\right)$
  • $N$ = collection size, $d{f}_i$ = # num of docs with word i
  • → greater when the term is rare in the collection

To compare 2 documents instead of 2 words:

$d=\frac{w_1+w_2+\dots +w_k}{k}$