Province of Équateur
How to Use Large Language Models for Text Coding: The Case of Fatherhood Roles in Public Policy Documents
Lupo, Lorenzo, Magnusson, Oscar, Hovy, Dirk, Naurin, Elin, Wängnerud, Lena
Recent advances in large language models (LLMs) like GPT-3 and GPT-4 have opened up new opportunities for text analysis in political science. They promise automation with better results and less programming. In this study, we evaluate LLMs on three original coding tasks of non-English political science texts, and we provide a detailed description of a general workflow for using LLMs for text coding in political science research. Our use case offers a practical guide for researchers looking to incorporate LLMs into their research on text analysis. We find that, when provided with detailed label definitions and coding examples, an LLM can be as good as or even better than a human annotator while being much faster (up to hundreds of times), considerably cheaper (costing up to 60% less than human coding), and much easier to scale to large amounts of text. Overall, LLMs present a viable option for most text coding projects.
News at a glance
SCI COMMUN### Infectious diseases The 11th Ebola outbreak in the Democratic Republic of the Congo (DRC) is officially over, giving the country respite from the disease for the first time in more than 2 years. On 18 November, the World Health Organization (WHO) announced that no new cases had been identified for 42 days, twice the incubation period for the deadly virus. The outbreak, in the western Équateur province, started in late May, just as a bigger one in the eastern DRC was coming to an end. (That outbreak had killed 2200 people.) The Équateur outbreak sickened 130 and killed 55; a campaign that vaccinated more than 40,000 people is credited with helping end it. Special portable coolers that keep the vaccine at −80°C for up to 1 week allowed health workers to administer the shots in communities deep in the rainforest, accessible only by boat or helicopter. The same technology will be useful in efforts to distribute COVID-19 vaccines in Africa, says Matshidiso Moeti, WHO's regional director. The coronavirus pandemic complicated the fight against Ebola, WHO says, but the expertise gained by local health workers in earlier outbreaks in the region was a major advantage. They will remain on the lookout for potential flare-ups. $1,000,000 —Gift from entertainer Dolly Parton in April to support development of Moderna's coronavirus vaccine, which the company last week said showed an efficacy of 94.5%. “I felt so proud to have been part of that little seed money,” Parton told BBC. ### Marine ecology The Allen Coral Atlas, a project to map the world's shallow coral reefs with high-resolution satellites, last week launched a monitoring system to detect coral bleaching events as they occur. When corals face extreme heat, they expel their algal symbionts, leaving them bone white and vulnerable to stress; repeated bleaching episodes, growing more common with global warming, can cause massive die-offs. The system detects the whitening using imagery from the privately owned Planet satellite constellation, processed with machine learning. A pilot has begun in Hawaii to use the data as an early warning system for researchers, to help them identify and study species both vulnerable and resistant to warming extremes. The monitoring of bleaching is expected to expand next year to shallow reefs globally. ### Diagnostics The U.S. Food and Drug Administration (FDA) issued its first emergency use authorization last week for an at-home diagnostic test that can detect the pandemic coronavirus in just minutes. However, the test might not be widely available until spring 2021. Produced by Lucira Health, a biotech company, it is expected to cost less than $50 and require a doctor's prescription. The company says it will soon distribute tests in parts of California and Florida; it says it needs time to scale up manufacturing for national distribution. Lucira's test amplifies viral genetic material, making it nearly as accurate as laboratory tests that use the polymerase chain reaction, the current gold standard. FDA previously approved at-home tests that must be mailed to a laboratory for analysis. Several other companies are working on rapid antigen tests, which detect viral particles, for home use. But concerns remain about antigen tests' reliability. Still, some public health specialists consider widely available, low-cost, at-home testing vital for controlling the pandemic. ### Funding A new U.S. National Institutes of Health (NIH) award will allow early-career investigators who want to shift research directions when applying for their first independent award to submit a proposal without first generating preliminary data to support their idea. Reviewers will instead assess the soundness of the project's approach. The Katz award is named for Stephen Katz, a longtime champion of young researchers who was director of the National Institute of Arthritis and Musculoskeletal and Skin Diseases when he died in 2018. The grant will build on an NIH policy that prioritizes proposals from early-stage investigators—those no more than 10 years from completing their training who are applying for their first research grant. The policy has been credited with raising their numbers from fewer than 600 supported in 2013 to more than 1300 last year. Applications for the first Katz awards are due on 26 January 2021. ### Leadership Democrats in Congress say a political appointee given a senior post at the U.S. National Institute of Standards and Technology (NIST) is unfit for the job because he lacks technical skills and holds pseudoscientific views about racial differences on IQ tests. On 9 November, Jason Richwine, an independent public policy analyst, took up the new position of deputy undersecretary of commerce for standards and technology, and Commerce Secretary Wilbur Ross subsequently issued an order that would put Richwine in charge of the $1 billion research agency if NIST Director Walter Copan leaves or is fired. On 17 November, Representative Eddie Bernice Johnson (D–TX), who leads the science committee in the U.S. House of Representatives, asked Ross to justify the moves. Richwine has advocated for more restrictive immigration policies, and his 2009 doctoral thesis argued that lower IQ scores by Mexican and Hispanic immigrants suggest a genetic component to intelligence that is “likely to persist over several generations.” ### Diversity The editors of Nature Communications say they are reviewing a paper that drew scalding criticism after it suggested that encouraging female junior scientists to work with female mentors could “hinder the careers of women.” The 17 November study, led by data scientist Bedoor AlShebli of New York University, Abu Dhabi, examined 3 million mentor-protégé pairs and how gender influenced the impact of papers later published by the protégés. Female protégés, it concluded, did better if they worked with male mentors. Critics pounced, noting the authors ignored reviewer complaints about the study's methods and arguing the journal was promoting a harmful and unfounded message. The article's authors said they welcome the review. ### Animal diseases European authorities reported on 19 November they have detected highly pathogenic avian influenza in 302 birds in eight countries. Only 18 cases were in poultry; most of the rest were in wild birds, the European Food Safety Authority and its partners said. The number of infected birds is expected to rise with winter migrations. Several flu strains were identified, but no people were reported to be infected, and the risk of that occurring is considered low; researchers studying the viruses found no genetic markers indicating they had adapted to infect mammals. But the threat to poultry is high, and the report's authors recommended bird producers increase precautions against infections. VACCINE APPLICATION Days after making public the final analysis of their 40,000-person COVID-19 vaccine trial, which found 95% efficacy, Pfizer and its German partner BioNTech filed for emergency authorization of the messenger RNA vaccine from the U.S. Food and Drug Administration—the first such request for a vaccine during the pandemic. They plan to seek additional approvals in other countries soon. Pfizer hopes to supply up to 50 million doses this year. REMDESIVIR PANNED A World Health Organization panel recommended against using the antiviral drug remdesivir to treat most hospitalized COVID-19 patients. Its review of four studies of 7000 people found that the drug, which the U.S. Food and Drug Administration approved last month for hospitalized patients, did not reduce mortality or speed recovery. But the panel encouraged further study of it. AMMO BAN Denmark has become the first nation to ban all lead-based hunting ammunition, including bullets and shotgun pellets, to protect wildlife. Hunters annually release about 2 tons of lead into Denmark's environment; waterbirds and other species eat the toxic material and die. European regulators are considering a ban like Denmark's.