A Comparison of Document Similarity Algorithms

Document similarity is an important part of Natural Language Processing and is most commonly used for plagiarism-detection and text summarization. Thus, finding the overall most effective document similarity algorithm could have a major positive impact on the field of Natural Language Processing. This report sets out to examine the numerous document similarity algorithms, and determine which ones are the most useful. It addresses the most effective document similarity algorithm by categorizing them into 3 types of document similarity algorithms: statistical algorithms, neural networks, and corpus/knowledge-based algorithms. The most effective algorithms in each category are also compared in our work using a series of benchmark datasets and evaluations that test every possible area that each algorithm could be used in. NTRODUCTION Document similarity analysis is a Natural Language Processing (NLP) task where two or more documents are analyzed to recognize the similarities between these documents. Document similarity is heavily used in text summarization, recommender systems, plagiarism-detection as well as in search engines. Identifying the level of similarity or dissimilarity between two or more documents based on their content is the main objective of document similarity analysis.