Anoncheg on Nostr: Part2: #dailyreport #negativesampleing #sampling #llm #recsys samples Selecting more ...

Part2: #dailyreport #negativesampleing #sampling #llm #recsys
samples

Selecting more frequent words from noise
distribution. P(w) = (f(w)^3/4)/Z, where f(w) frequency
of word, Z - normalization constant.

Example "The dog is playing with a bone," and assume a
window size of 2 positive samples for the target word
"dog" would include:
- ("dog", "The")
- ("dog", "is")
- ("dog", "playing")
- ("dog", "with")
- ("dog", "a")
- ("dog", "bone")

Negative Samples: ("dog", "car"), ("dog", "apple"),
("dog", "house"), ("dog", "tree")

calc:
logσ(vdog​⋅vbone​)+logσ(−vdog​⋅vcar​)+logσ(−vdog​⋅vapple​)+logσ(−vdog​⋅vhouse​)

蠡