The objective here is to learn word embeddings, that are then evaluated on analogy-based word-similarity tasks. Leveraging the fast training afforded by NCE and using a sligthly simpler model than Minh and Teh ICML2012 they are able to outperform state-of-the-art method of Mikolov et al. 2013, using four times less data and an order of magnitude less computing time.

Word2Vec is a recent breakthrough in the world of NLP. Tomas Mikolov a Czech computer scientist and currently a researcher at CIIRC ( Czech Institute of Informatics, Robotics and Cybernetics) was one of the leading contributors towards the research and implementation of word2vec. Word embeddings are an integral part of solving many problems in NLP. They depict how humans understand language to a machine. You can imagine them as a vectorized representation of text.

In the previous post, we introduced NLP. To find out word meanings with the Python programming language, we used the NLTK package and worked our way into word embeddings using the gensim package and Word2vec. Since we only touched the Word2Vec technique from a 10,000-feet overview, we are now going to dive deeper into the training method to create a Word2vec model. The Word2vec (Mikolov et al. 2013)[1][2] is not a singular technique or algorithm. It's actually a family of neural network architectures and optimization techniques that can produce good results learning embeddings for large datasets.

On 21th June 2018 at Buildo, Data Science Milan has organized an event on a fashion topic: Natural Language Processing (NLP). Nowadays we found many applications of NLP, such as machine translation (Google translator), question answering (chatbot), web and application search (Amazon), lexical semantics (Thesaurus), sentiment analysis (Cambridge Analytica), natural language generator (Reddit bot). What is the mean of natural language processing? Natural language processing is a branch of artificial intelligence representing a bridge between humans and computers; it can be broadly defined as the automatic manipulation of natural language, like speech and text, by software. There are many ways to represents words in NLP and you cannot use text data directly on machine learning algorithms.

Embeddings of medical concepts such as medication, procedure and diagnosis codes in Electronic Medical Records (EMRs) are central to healthcare analytics. Previous work on medical concept embedding takes medical concepts and EMRs as words and documents respectively. Nevertheless, such models miss out the temporal nature of EMR data. On the one hand, two consecutive medical concepts do not indicate they are temporally close, but the correlations between them can be revealed by the time gap. On the other hand, the temporal scopes of medical concepts often vary greatly (e.g., \textit{common cold} and \textit{diabetes}). In this paper, we propose to incorporate the temporal information to embed medical codes. Based on the Continuous Bag-of-Words model, we employ the attention mechanism to learn a "soft" time-aware context window for each medical concept. Experiments on public and proprietary datasets through clustering and nearest neighbour search tasks demonstrate the effectiveness of our model, showing that it outperforms five state-of-the-art baselines.

Editor's note: This post is only one part of a far more thorough and in-depth original, found here, which covers much more than what is included here. The Skip-gram model architecture usually tries to achieve the reverse of what the CBOW model does. It tries to predict the source context words (surrounding words) given a target word (the center word). If we used the CBOW model, we get pairs of (context_window, target_word)where if we consider a context window of size 2, we have examples like ([quick, fox], brown), ([the, brown], quick), ([the, dog], lazy) and so on. Now considering that the skip-gram model's aim is to predict the context from the target word, the model typically inverts the contexts and targets, and tries to predict each context word from its target word.

Natural Language Processing or NLP combines the power of computer science, artificial intelligence (AI) and computational linguistics in a way that allows computers to understand natural human language and, in some cases, even replicate it. The ultimate goal of NLP is to analyze human language, 'understand' it and then derive a coherent meaning in a way that is beneficial for the user. In ideal scenarios, NLP completes a task faster and far more efficiently and effectively than any human can. One of the main challenges in NLP is'understanding' the language. We use speech or text as our main communication medium.

Before we start, have a look at the below examples. So what do the above examples have in common? You possible guessed it right – TEXT processing. All the above three scenarios deal with humongous amount of text to perform different range of tasks like clustering in the google search example, classification in the second and Machine Translation in the third. Humans can deal with text format quite intuitively but provided we have millions of documents being generated in a single day, we cannot have humans performing the above the three tasks. It is neither scalable nor effective. So, how do we make computers of today perform clustering, classification etc on a text data since we know that they are generally inefficient at handling and processing strings or texts for any fruitful outputs?

Neural network language models (NNLMs) have attracted a lot of attention recently. In this paper, we present a training method that can incrementally train the hierarchical softmax function for NNMLs. We split the cost function to model old and update corpora separately, and factorize the objective function for the hierarchical softmax. Then we provide a new stochastic gradient based method to update all the word vectors and parameters, by comparing the old tree generated based on the old corpus and the new tree generated based on the combined (old and update) corpus. Theoretical analysis shows that the mean square error of the parameter vectors can be bounded by a function of the number of changed words related to the parameter node. Experimental results show that incremental training can save a lot of time. The smaller the update corpus is, the faster the update training process is, where an up to 30 times speedup has been achieved. We also use both word similarity/relatedness tasks and dependency parsing task as our benchmarks to evaluate the correctness of the updated word vectors.