Working with Text -Part 4. Techniques in handling text data

Example: 'I want to read a book' In the above example there are 6 tokens which are- ('I', 'want, 'to', 'read', 'a' and'book') A type is the class of all tokens containing the same character sequence. In the above example, there are only 5 types which are - 'can, 'you', 'a, 'as' and'canner' as'can', 'as' and'a' are being repeated. In the above example, by deleting period and hyphens between the characters and words we are normalizing the type by making it a term. So the term in the above example is: 'USA' and'antiinflammatory' Example: "Hello everyone.Welcome to the course." The tokens for the given sentence will be -- ['Hello','everyone', 'Welcome', 'to', 'the', 'course'] Welcome to the Natural Language Processing course.