import bisect 2 import re

Use any tokenizer you want as long it as the same API.""" In order to convert the dataset to NER format we suggest tokenizing Tweet text and utilizing the character offsets to identify mention tokens. This approach works as long as the tokenizer returned offsets correspond to the offset of the phrase in the original text, i.e. See example code in listing 1. Metric Description strong_mention_match strong_mention_match is a micro-averaged evaluation of entity mentions. A system span must match a gold span exactly to be counted as correct.strong_all_match