Goto

Collaborating Authors

Magna Carta Scientiae

#artificialintelligence

Science is a catalyst for human progress. But a publishing monopoly and funding monopsony have inhibited research. We intend to improve incentives in science by developing smart research contracts. These will collectively reward scientific activities, including proposals, papers, replications, datasets, analyses, annotations, editorials, and more. Peer-to-peer review networks will be designed to help evaluate proposals and publications. Long term, these smart contracts help accelerate research by minimizing science friction, ensuring science quality, and maximizing science variance. Papers are the fundamental asset of the research economy: they serve as proof of work that valuable research has been completed. Funding agencies and research institutions evaluate scientists based on their publications. Principal investigators (PIs) attract prospective students and collaborators via papers. Investors and companies use scientific literature to conduct due diligence on research ranging from basic discoveries to clinical studies. Thus, the evaluation and dissemination of papers are vital to this research economy. Publishers are the sole arbiters of papers today. They assign a value -- denominated in "prestige" -- by accepting a paper into the appropriate journal based on selectivity and domain. To evaluate papers, journals typically outsource it to two or three PIs, who often outsource it further to their students. Reviewers are unpaid for this peer review work, as it is an expected part of their scientific duties. Peer review is believed to be necessary because of the industrialization of science. Research papers and proposals have become too specialized and too numerous, making it difficult to assess merit prima facie. As a result, scientific incentives have become distorted in two major ways: prestige capture and reviewer misalignment. Over half of all research papers in 2013 were published by five companies, who have used their centuries of brand equity to build an economic moat. This results in prestige capture, which akin to regulatory capture, causes public and scientific interest to be directed towards the regulators of prestige. Publishers have exploited prestige capture to become the ultimate rent-seekers, with operating margins between 25-40% and market capitalizations up to $50B.



For Siri's New Competitor, SkyPhrase, Academia Isn't Big Enough for AI

AITopics Original Links

Academia is supposed to be a place where creative types can be free, and with that freedom accomplish great things, whether it be new art, breakthrough treatises, scientific discoveries, or feats of engineering. But academia isn't what it used to be, and to provide some insights into some of its problems, I compared notes with friend and former colleague, Nick Cassimatis, who is associate professor in the Department of Cognitive Science at Rensselaer. In our own ways, he and I have found severe limitations in academia today, limitations that led to my leaving academia to co-found a research institute, 2AI to be funded by intellectual property, and that led Nick to start his own company outside academia, SkyPhrase in order to achieve his ambitions in artificial intelligence. Nick's romantic ambitions started early – he began research into artificial intelligence and natural language at the precocious age of fifteen, and wrote a French-to-English translation program that helped put him on the Top-20 High School Students List by USA Today. More than simply artificial intelligence, his aim is to understand human-level intelligence, and how it can come about via many unintelligent parts.


Hacking Academia: Data Science and the University

#artificialintelligence

A reflection on our SciFoo breakout session, where we discussed issues of data science within academia. Almost a year ago, I wrote a post I called the Big Data Brain Drain, lamenting the ways that academia is neglecting the skills of modern data-intensive research, and in doing so is driving away many of the men and women who are perhaps best equipped to enable progress in these fields. This seemed to strike a chord with a wide range of people, and has led me to some incredible opportunities for conversation and collaboration on the subject. One of those conversations took place at the recent SciFoo conference, and this article is my way of recording some reflections on that conversation. SciFoo is an annual gathering of several hundred scientists, writers, and thinkers sponsored by Digital Science, Nature, O'Reilly Media & Google.


Concrete steps to diversify the scientific workforce

Science

The recent events that precipitated the resurgence of the Black Lives Matter movement and the disproportionately devastating impact of COVID-19 on many communities of color are stark reminders of the pernicious effects of systemic racism on all aspects of our society, including science, medicine, and public health. The lack of diversity in the scientific and health professions—a longstanding manifestation of racism—can no longer be ignored, excused, or attributed to uncontrollable factors. We write at this moment of reckoning to explain what is lost by a lack of diversity; to describe some promising efforts to achieve it; and to propose urgent, larger-scale actions that political and institutional leaders, educators, and scientists can take to redress the inequities that pervade our professions. African American, Latinx, and Indigenous peoples have historically been underrepresented in the research enterprise, with their proportions declining as they progress from undergraduate to graduate school to faculty positions (see the figure, top). The glacial pace of the increase in the percentage of minority PhDs over the past two decades, if extrapolated, suggests that it will take many more decades for the workforce to reflect the makeup of the US population (see the figure, bottom). To wait so long for an equitable outcome should be unacceptable to us all. Some who defend the status quo claim that a lack of diversity does not compromise the quality of science or the likelihood of making discoveries that improve human well-being. We strongly disagree. Why? First, because any barrier to entry into STEM (science, technology, engineering, mathematics) fields weakens science and carries unacceptable opportunity costs. By limiting the pool from which future scientists are drawn, the full range of talent is reduced, and progress is slowed. Second, when science is more inclusive, the range of questions asked will broaden, as happened when women began to enter the biomedical profession in larger numbers in the 1970s and 1980s ([ 1 ][1]). As an example, a more diverse group of geneticists might have prevented the large human genetic databases from becoming so highly skewed toward European ancestry genomes, limiting their power to identify genetic determinants of disease in other groups ([ 2 ][2]). Third, barriers to the inclusion of specific demographic groups limit the potential impact of science on society. Today, the reluctance of minority communities to participate in clinical trials for COVID-19 vaccines or even to receive vaccines that have been rigorously tested and approved ([ 3 ][3]) reflects an understandable skepticism of medical authority that arose from historic injustices toward African American and Hispanic communities. For scientific advances to be widely accepted throughout an increasingly diverse US population, both the composition and leadership of our scientific and medical communities must become much more representative. Last, the US census projects that by 2045, no single group, as defined by the US government, will hold a majority ([ 4 ][4]). In 2018, only 50% of the population under 18 years of age was white—with 25% Hispanic, 14% African American, and 5% Asian American—and the white proportion continues to drop every year. As Congress has recognized, the US will be unable to compete in the global arena in the future if it fails to draw talent from its diverse citizenry. The tendency to prefer and to value people most like oneself is a deeply held human trait, one that needs conscious monitoring to overcome. Scientists are not singularly resistant to the phenomenon of implicit or unconscious bias, which can affect all aspects of professional life: hiring, evaluation, promotion, citation practices, and grant funding ([ 5 ][5]). For example, data suggest that African American grant applicants for funding from the National Institutes of Health (NIH) face racial bias in the awarding of grants ([ 6 ][6]), with African American applicants receiving grant review priority scores that were 10 percentile points lower than scores for white or Asian American applicants, substantially reducing their chances to receive funding. Despite the efforts of the NIH leadership to understand these findings, the discrepancy has never been fully explained, and the difference in success rates (the “funding gap”) has never been closed. The NIH recently announced a new initiative, UNITE, a multipronged effort to end structural racism and its consequences at the NIH, including inequities in evaluations of grant applications ([ 7 ][7]). A commitment of this kind is a first and laudable step to making meaningful progress. In an attempt to rationalize the lack of diversity in the scientific workforce, some have argued that science is a meritocracy, and that the absence of diverse voices, although unfortunate, largely reflects the limited diversity of the pipeline of trainees. This passive view—delegating the problem to a metaphorical pipeline outside of our control—ignores actions that the scientific community can take to address systemic racism and its consequences. During the past few decades, several programs have aimed to increase the inclusion of minorities in science. Although well-meaning, many of these have been either ineffective or not conducted at a scale adequate to substantially change national percentages. We urgently need more and bolder efforts. Fortunately, we can now build on some recent programs that have had notable success in training minority scientists who are now pursuing productive careers in research. These programs appear to have three key features: reducing the sense of isolation by using cohorts to create communities, making strong institutional and individual commitments to mentoring, and removing barriers to research careers by providing full financial support during training. For example, since 1993, hundreds of undergraduates from the University of Maryland Baltimore County's (UMBC) Meyerhoff Scholars Program have gone on to receive MDs and PhDs in STEM fields ([ 8 ][8]). The program relies heavily on a cohort model, in which Meyerhoff scholars form a community that provides mutual support and encouragement; students receive intensive personal advising and counseling by UMBC faculty; and they enjoy access to laboratory opportunities, starting in their freshman year. The program is now being replicated, with promising outcomes, at a number of other research universities ([ 8 ][8]). Another successful model is the Specialized Training and Advanced Research (STAR) program at the University of California, Los Angeles ([ 9 ][9]). This program attracts physicians nearing the end of their clinical training to study for a PhD in a variety of research fields. Although not designed as a minority-serving program, it has capitalized on medical schools having been more successful than graduate programs in attracting minority students. By removing financial burdens that often discourage medical trainees from considering substantial engagement in laboratory research, and by building a strong sense of community that counters the isolation that minority students often experience in the sciences, the program has been successful in directing physicians into research. Another approach that universities have successfully used to diversify their faculty is cluster hiring, in which searches are designed to attract a group of faculty of color over a short period of time. This approach exploits features of the cohort model, including building a community that provides mutual support, encouragement, and peer mentoring. Recognizing the effectiveness of such approaches, in January 2020 the NIH announced the $241 million Faculty Institutional Recruitment for Sustainable Transformation (FIRST) initiative. The FIRST program has allowed approximately a dozen universities and medical schools to expand their faculty in emerging areas of research, with a requirement that every person hired must have a track record of working to promote an inclusive culture in science ([ 10 ][10]). One should not underestimate the role that money plays in the choices that students make to become scientists. The pay gap between whites and African American, Latinx, and Indigenous people, coupled with long-standing and often unwritten discriminatory policies, has prevented generations of minority Americans from accruing appreciable property and other forms of wealth ([ 11 ][11]). Without the security that such family assets provide, it is much more difficult to embark on PhD or MD training that may not lead to an attractive salary for many years—often while carrying substantial student debt. ![Figure][12] Race and ethnicity in the biomedical research workforce (Top) Demographics by career stage in 2016. The red line denotes the proportion of the specified race and ethnicity in the US population in 2016. (Bottom) Growth in Black and Hispanic PhD recipients over time. The dashed line indicates a linear best-fit trendline. Data are from ([ 15 ][13]). GRAPHIC: H. BISHOP/ SCIENCE BASED ON S. TILGHMAN ET AL. We recommend three approaches to redress this situation: a major federal initiative to diversify the scientific and engineering workforce, a reshaping of institutional culture to welcome underrepresented minorities into STEM research, and grant-funding policies that immediately address current inequities. ### A coordinated federal program to diversify the scientific workforce President Biden has strongly signaled his intention to seek remedies for past racial injustice in this country. At the same time, he has forcefully declared his commitment to scientific solutions to the nation's problems, and he has elevated the role of the White House Office of Science and Technology Policy (OSTP) by seeking to place its director, nominee Eric Lander, in his Cabinet, an unprecedented action. We call on President Biden to take an even bolder step in support of both equity and science by proposing legislation that would establish and fund a broad interagency National Science and Engineering Diversity Initiative (NSEDI). And we call on Congress to pass such a bill once submitted for consideration. We recommend that OSTP, reporting directly to the President, organize a programmatic planning process for NSEDI, establish a long-range national strategic plan for diversifying the scientific workforce, and coordinate the distribution of funds to the relevant federal agencies to carry out these plans. A comprehensive effort for diversifying the STEM workforce will require actions that affect and provide support for all components of the scientific enterprise: K-12, college, and graduate education; professional training programs; employment in the public and private sectors; and research grants in many fields. Therefore, NSEDI will need to be developed in conjunction with all of the federal science agencies that participate in the National Science and Technology Council, as well as with the Small Business Administration and the Department of Education. Although it is premature to estimate NSEDI's budget until extensive planning has been undertaken, it seems prudent to expect, on the basis of examples of the anticipated activities (see the box), that an effective program will require an annual new Congressional appropriation of at least 10 billion dollars for several years—a substantial sum but only about 2% of national spending (public and private) on research and development and less than 8% of the federal government science budget. To ensure that NSEDI's programs are working appropriately, Congress should require that an external advisory board be established that develops evidence-based measures to evaluate NSEDI's projects, recommends changes in its portfolio and budget, and reports regularly on the diversity of the nation's scientific workforce. ### Reshaping institutional policies As important as federal funding will be, there are also steps that academic institutions must take to effect an enduring change in the culture of science. The criteria for hiring and promotion of all scientists—from junior faculty to senior administrators—should include evidence of a commitment to diversity, equity, and inclusion. Institutions should also take steps to diminish the “minority tax” that is imposed on faculty of color engaged in diversity efforts by ensuring that such programs are led, at least jointly, by nonminority faculty. Moreover, offices and programs established to enhance the careers of minority scientists should be empowered with clear reporting structures to leadership, as well as with administrative and financial support. #### Possible programs to diversify the workforce Colleges, universities, and research institutions should take steps to educate faculty, students, and staff about the history of racism in the United States and provide training for those who serve as mentors and advisers for minority scientists at all career stages. For example, the Center for Improvement of Mentored Experience in Research at the University of Wisconsin–Madison designs training modules that are used by many institutions across the country (). Mentors need not be sought solely within a single laboratory, department, or institution but can be found within the national mentoring networks being generated by scientific and engineering societies. Experimental approaches for bringing about change in the culture of science should be encouraged, and the impact of programs designed to expand the participation of minority scientists should be regularly assessed to identify the most successful strategies. ### NIH policies to redress structural racism Our first two recommendations are directed to the broad scientific enterprise irrespective of field. Their adoption will take time, and their full impact will not be felt for many years. As biomedical scientists driven by the “fierce urgency of now,” we propose three immediate steps that our major source of funding, the NIH, could take that do not require either Congressional action or a culture change in academia. 1. Address financial barriers faced by minority scientists The NIH Research Supplements to Promote Diversity in Health-Related Research is a targeted mechanism that allocates extramural funds to support scientists from diverse backgrounds before establishing an independent research program ([ 12 ][14]). We recommend that these grant supplements be paired with a student loan repayment program to reduce the financial burdens of advanced education and training. This would repay up to $50,000 annually of qualified educational debt for minority PhD and MD students. In addition, the application process and evaluation criteria for these Research Supplements should be standardized across NIH institutes, and the length of support should be increased to at least 3 years to enable adequate time for securing individual grant funding. 2. Close the gap in NIH funding of grants for minority scientists The funding gap for African American scientists has been estimated by one of us to be equivalent to about 25 research project grants and 25 smaller exploratory grants per year ([ 13 ][15]). We recommend that the NIH director establish, through the Common Fund, a Demonstration Project (DP) designed to eliminate the gap within 5 years. Eligibility should be similar to the requirements for other DPs and follow the guidelines for the NIH Research Supplements to Promote Diversity in Health-Related Research. 3. Expand funding for businesses that employ minority scientists Currently, 4.8% of the NIH research project grant budget is directed toward small businesses through the Small Business Innovation Research (SBIR) and Small Business Technology Transfer (STTR) programs ($1.1 billion annually). In 2019, only 3.5% of these SBIR/STTR grants were awarded to principal investigators from minority groups ([ 14 ][16]). We recommend that the OSTP establish a goal of at least 5% for minority participation in the NIH SBIR/STTR programs. We propose ambitious, concrete steps for political and institutional leaders, educators, and scientists to take in the immediate future. But we acknowledge that even a successful implementation of these policy recommendations will fall short unless society addresses the broader issues of racism that produce the inequities in the first place. 1. [↵][17]1. J. Stone, 2. V. W. Pinn, 3. J. Rudick, 4. M. Lawrence, 5. M. Carlyn , J. Womens Health (Larchmt.) 15, 234 (2006). [OpenUrl][18][PubMed][19] 2. [↵][20]1. A. C. Need, 2. D. B. Goldstein , Trends Genet. 25, 489 (2009). [OpenUrl][21][CrossRef][22][PubMed][23][Web of Science][24] 3. [↵][25]1. B. Farmer , “COVID vaccine trials move at warp speed, but recruiting Black volunteers takes time,” Kaiser Health News, 2020; . 4. [↵][26]1. S. L. Colby, 2. J. M. Ortman , “Current population reports,” P25-1143 (US Census Bureau, 2014). 5. [↵][27]1. K. Dutt , “How implicit bias and lack of diversity undermine science,” Scientific American, 2018; . 6. [↵][28]1. D. K. Ginther et al ., Science 333, 1015 (2011). [OpenUrl][29][Abstract/FREE Full Text][30] 7. [↵][31]1. M. Lauer , “NIH stands against structural racism in biomedical research,” Open Mike, 2021; . 8. [↵][32]1. M. R. Sto. Domingo et al ., Science 364, 335 (2019). [OpenUrl][33][Abstract/FREE Full Text][34] 9. [↵][35]UCLA STAR program, . 10. [↵][36]1. J. Mervis , Science (2020). 10.1126/science.abb1082 11. [↵][37]1. E. Patten , “Racial, gender wage gaps persist in US despite some progress,” Pew Research Center, 2016; [www.pewresearch.org/fact-tank/2016/07/01/racial-gender-wage-gaps-persist-in-u-s-despite-some-progress][38]. 12. [↵][39]National Institutes of Health, “Research supplements to promote diversity in health-related research,” PA-20-222 (NIH, 2020); . 13. [↵][40]1. K. Dzirasa , Cell 183, 576 (2020). [OpenUrl][41] 14. [↵][42]National Institutes of Health, NIH SBIR/STTR award data (NIH, 2020); . 15. [↵][43]National Center for Science and Engineering Statistics, “Women, minorities, and persons with disabilities in science and engineering: 2019,” National Science Foundation, 2019; . Acknowledgments: We are deeply grateful to F. Hrabowski, H. Valantine, O. Ajijola, A. Diaz Vazquez, Y. Fortis Santiago, G. Guerrero-Medina, B. Jones Marlin, K. Milligan-Myhre, C. C. Pinnix, and P. Silveyra for speaking to us about their experiences in bringing diversity and inclusion to biomedical science. The authors also thank C. Pickett for help preparing the manuscript and the members of the Rescuing Biomedical Research Steering Committee for helpful discussions. [1]: #ref-1 [2]: #ref-2 [3]: #ref-3 [4]: #ref-4 [5]: #ref-5 [6]: #ref-6 [7]: #ref-7 [8]: #ref-8 [9]: #ref-9 [10]: #ref-10 [11]: #ref-11 [12]: pending:yes [13]: #ref-15 [14]: #ref-12 [15]: #ref-13 [16]: #ref-14 [17]: #xref-ref-1-1 "View reference 1 in text" [18]: {openurl}?query=rft.jtitle%253DJournal%2Bof%2Bwomen%2527s%2Bhealth%2B%25282002%2529%26rft.stitle%253DJ%2BWomens%2BHealth%2B%2528Larchmt%2529%26rft.aulast%253DStone%26rft.auinit1%253DJ.%26rft.volume%253D15%26rft.issue%253D3%26rft.spage%253D234%26rft.epage%253D247%26rft.atitle%253DEvaluation%2Bof%2Bthe%2Bfirst%2B10%2Byears%2Bof%2Bthe%2BOffice%2Bof%2BResearch%2Bon%2BWomen%2527s%2BHealth%2Bat%2Bthe%2BNational%2BInstitutes%2Bof%2BHealth%253A%2Bselected%2Bfindings.%26rft_id%253Dinfo%253Apmid%252F16620182%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [19]: /lookup/external-ref?access_num=16620182&link_type=MED&atom=%2Fsci%2F372%2F6538%2F133.atom [20]: #xref-ref-2-1 "View reference 2 in text" [21]: {openurl}?query=rft.jtitle%253DTrends%2Bin%2Bgenetics%2B%253A%2B%2BTIG%26rft.stitle%253DTrends%2BGenet%26rft.aulast%253DNeed%26rft.auinit1%253DA.%2BC.%26rft.volume%253D25%26rft.issue%253D11%26rft.spage%253D489%26rft.epage%253D494%26rft.atitle%253DNext%2Bgeneration%2Bdisparities%2Bin%2Bhuman%2Bgenomics%253A%2Bconcerns%2Band%2Bremedies.%26rft_id%253Dinfo%253Adoi%252F10.1016%252Fj.tig.2009.09.012%26rft_id%253Dinfo%253Apmid%252F19836853%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [22]: /lookup/external-ref?access_num=10.1016/j.tig.2009.09.012&link_type=DOI [23]: /lookup/external-ref?access_num=19836853&link_type=MED&atom=%2Fsci%2F372%2F6538%2F133.atom [24]: /lookup/external-ref?access_num=000271717800006&link_type=ISI [25]: #xref-ref-3-1 "View reference 3 in text" [26]: #xref-ref-4-1 "View reference 4 in text" [27]: #xref-ref-5-1 "View reference 5 in text" [28]: #xref-ref-6-1 "View reference 6 in text" [29]: {openurl}?query=rft.jtitle%253DScience%26rft.stitle%253DScience%26rft.aulast%253DGinther%26rft.auinit1%253DD.%2BK.%26rft.volume%253D333%26rft.issue%253D6045%26rft.spage%253D1015%26rft.epage%253D1019%26rft.atitle%253DRace%252C%2BEthnicity%252C%2Band%2BNIH%2BResearch%2BAwards%26rft_id%253Dinfo%253Adoi%252F10.1126%252Fscience.1196783%26rft_id%253Dinfo%253Apmid%252F21852498%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [30]: /lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6Mzoic2NpIjtzOjU6InJlc2lkIjtzOjEzOiIzMzMvNjA0NS8xMDE1IjtzOjQ6ImF0b20iO3M6MjI6Ii9zY2kvMzcyLzY1MzgvMTMzLmF0b20iO31zOjg6ImZyYWdtZW50IjtzOjA6IiI7fQ== [31]: #xref-ref-7-1 "View reference 7 in text" [32]: #xref-ref-8-1 "View reference 8 in text" [33]: {openurl}?query=rft.jtitle%253DScience%26rft.stitle%253DScience%26rft.aulast%253DSto.%2BDomingo%26rft.auinit1%253DM.%2BR.%26rft.volume%253D364%26rft.issue%253D6438%26rft.spage%253D335%26rft.epage%253D337%26rft.atitle%253DReplicating%2BMeyerhoff%2Bfor%2Binclusive%2Bexcellence%2Bin%2BSTEM%26rft_id%253Dinfo%253Adoi%252F10.1126%252Fscience.aar5540%26rft_id%253Dinfo%253Apmid%252F31023915%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [34]: /lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6Mzoic2NpIjtzOjU6InJlc2lkIjtzOjEyOiIzNjQvNjQzOC8zMzUiO3M6NDoiYXRvbSI7czoyMjoiL3NjaS8zNzIvNjUzOC8xMzMuYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9 [35]: #xref-ref-9-1 "View reference 9 in text" [36]: #xref-ref-10-1 "View reference 10 in text" [37]: #xref-ref-11-1 "View reference 11 in text" [38]: http://www.pewresearch.org/fact-tank/2016/07/01/racial-gender-wage-gaps-persist-in-u-s-despite-some-progress [39]: #xref-ref-12-1 "View reference 12 in text" [40]: #xref-ref-13-1 "View reference 13 in text" [41]: {openurl}?query=rft.jtitle%253DCell%26rft.volume%253D183%26rft.spage%253D576%26rft.genre%253Darticle%26rft_val_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Ajournal%26ctx_ver%253DZ39.88-2004%26url_ver%253DZ39.88-2004%26url_ctx_fmt%253Dinfo%253Aofi%252Ffmt%253Akev%253Amtx%253Actx [42]: #xref-ref-14-1 "View reference 14 in text" [43]: #xref-ref-15-1 "View reference 15 in text"