Pretty fast word2vec with Numba
At a high level, the skip-gram flavor of this algorithm looks at a word and its surrounding words, and tries to maximize the probability that the word's vector representation predicts those actual words occurring around it. If it is trained on the phrase the quick brown fox jumps, the word2vec input representation of the word brown would yield a high dot product with the output vectors for the words the, quick, fox and jumps. And if the algorithm is trained on a lot more text, there's a good chance that it will start to learn that the words brown and red appear in similar contexts, so that their vector representations will be pretty close to one another's. I'm constantly trying to navigate the trade-off from simultaneously maximizing my laziness and the speed of my code. Aiming for the lazy side, I originally wanted to experiment with using autograd to write my gradient updates for me, and as nice as it was to not have to write out the gradient calculation, this pretty quickly bubbled to the top as one of the major bottlenecks, leaving me to write the gradients manually by faster means.
May-11-2016, 20:01:32 GMT
- Technology: