1 import bisect 2 import re

In order to convert the dataset to NER format we suggest tokenizing Tweet text and utilizing the character offsets to identify mention tokens. E.g. just setting up my twttrwith offsets 19and 24, and DBpedia category as Organization, can be converted to the NERBIO format as follows: tokens, starts, ends = tokenize_with_offsets("just setting up my twttr")and then assigning Olabels to all tokens outside the phrase start and end offsets and B-ORG and I-ORG label to all tokens within the phrase offsets. This approach works as long as the tokenizer returned offsets correspond to the offset of the phrase in the original text, i.e. tokenization is non-destructive. See example code in listing 1. A system span must match a gold span exactly to be counted as correct.