Natural Language Processing Examples in Government Data

Tom is an analyst at the US Department of Defense (DoD).1 All day long, he and his team collect and process massive amounts of data from a variety of sources--weather data from the National Weather Service, traffic information from the US Department of Transportation, military troop movements, public website comments, and social media posts--to assess potential threats and inform mission planning. While some of the information Tom's group collects is structured and can be categorized easily (such as tropical storms in progress or active military engagements), the vast majority is simply unstructured text, including social media conversations, comments on public websites, and narrative reports filed by field agents. Because the data is unstructured, it's difficult to find patterns and draw meaningful conclusions. Tom and his team spend much of their day poring over paper and digital documents to detect trends, patterns, and activity that could raise red flags. In response to these kinds of challenges, DoD's Defense Advanced Research Projects Agency (DARPA) recently created the Deep Exploration and Filtering of Text (DEFT) program, which uses natural language processing (NLP), a form of artificial intelligence, to automatically extract relevant information and help analysts derive actionable insights from it.2 Across government, whether in defense, transportation, human services, public safety, or health care, agencies struggle with a similar problem--making sense out of huge volumes of unstructured text to inform decisions, improve services, and save lives.