ABSTRACT. We analyze the research outputs of R1 research universities in the United States over the past three decades, comparing three types of universities: public land-grant, public non-land-grant, and private. First, we document the substantial differences in the emphases on research topics across the three types, reflecting the historical roles of the universities. We further observe that universities of all three types have been increasingly diversifying their research portfolios. At the same time, however, the research portfolios of the different types have converged across a number of quantitative measures. As there is a growing reliance of university research on federal funding, we find that publications acknowledging federal funding exhibit a stronger tendency toward individual diversification and collective convergence in topics than those without such acknowledgments, suggesting that federal funding may have an unintended consequence of inducing institutional isomorphism within the diverse ecosystem of American research universities.

ABSTRACT. Some technology companies and media have anointed the metaverse as the future of the internet1. Advances in virtual reality devices and high-speed connections, combined with the acceptance of remote work during the COVID-19 pandemic, have brought considerable attention to the metaverse as more than a mere curiosity for gaming. Despite substantial ambitiously optimistic pronouncements, the future of the metaverse remains uncertain.

ABSTRACT. Market bubbles emerge when asset prices are driven unsustainably higher than asset values and shifts in belief burst them. We demonstrate the same phenomenon for biomedical knowledge when promising research receives inflated attention. We predict deflationary events by developing a diffusion index that captures whether research areas have been amplified within social and scientific bubbles or have diffused and become evaluated more broadly. We illustrate our diffusion approach contrasting the trajectories of cardiac stem cell research and cancer immunotherapy. We then trace the diffusion of unique 28,504 subfields in biomedicine comprising nearly 1.9M papers and more than 80M citations and demonstrate that limited diffusion of biomedical knowledge anticipates abrupt decreases in popularity. Our analysis emphasizes that restricted diffusion, implying a socio-epistemic bubble, leads to dramatic collapses in relevance and attention accorded to scientific knowledge.

ABSTRACT. This paper aims to explore the intersection of large language models (LLMs) and education, particularly focusing on the potential impact of LLMs on the U.S. education system and labor market. We delve into the extent to which the skills taught in U.S. higher education are vulnerable to or can be enhanced by, the capabilities of LLMs like GPT-4.

ABSTRACT. This study proposes a novel framework through which we theorize the dynamic interplay between conceptual boundaries and scientific activities based on the frame of assembly theory. Specifically, we focus on three key pairs of macro-micro dynamics: (1) the emergence of concepts and the shift in knowledge agent attention, (2) the saturation of concepts and the onset of disruptive innovations, and (3) the accumulation of innovation and the emergence of new concepts. We look for evidence of our hypothesis within the field of machine learning. We analyze around four million machine learning publications from 1970 to 2023 documented by OpenAlex, along with its introduced concept graph. We concentrate on three main concepts that gained popularity sequentially: regression, the Monte Carlo method, and deep learning. Our analysis uncovers the evolution and eventual saturation of the regression and Monte Carlo method concepts, and elucidates their role in fostering the subsequent growth of the deep learning concept by looking at the dynamics of knowledge assemblage and agent migration.

ABSTRACT. Scholarly periodicals have been the primary outlets for science publishing, playing a significant role in disseminating research findings to the scientific community. They have also long been used as instruments to probe the structure of science, by representing them as crisp discipline vectors, sparse vectors capturing (co-)citation relationships or online activities, as well as dense vectors based on representation learning methods, yielding several global maps of science that are useful for understanding knowledge flow between fields and supporting decision-makings. However, these maps are static, falling short of capturing rich dynamics of the evolution of science or the development of scholarly communication. Here we build diachronic periodical embeddings based on temporal citation networks between papers.

ABSTRACT. The exponential growth of knowledge presents a paradox: as the collective reservoir expands, knowledge workers increasingly struggle to stay abreast of the latest developments. This predicament, known as the “burden of knowledge”, implies that knowledge workers may disproportionally rely on established concepts over time, particularly as they progress in their careers, which can stifle innovation due to the "knowledge lock-in". Moreover, the reliance on reputation-based signals such as citation counts, reinforced by the Matthew effect, may hinder knowledge workers’ effective engagement with emerging innovation. Recognizing the importance of assimilating new ideas for innovation arises the pivotal question this research seeks to address: How knowledge workers effectively absorb new innovation? Employing a quasi-experimental design, this paper examines how brief, unstructured exposure to new information can drive substantial knowledge acquisition among knowledge workers. This exploration amplifies the power of transient attention in shaping innovation trajectories and provides valuable insights into effective strategies to expedite knowledge absorption.

ABSTRACT. The field of science of science is intertwined with network science. In their various forms, networks are the fundamental building blocks for representing and analyzing the interconnections within the research context. Such systems, be it scientific collaboration, citations, topic formation, or the diffusion of innovations, often exhibit complex topologies consisting of many nodes and edges, along with metadata and other features. Interactive network visualization plays a crucial role in understanding these systems and uncovering hidden patterns. It provides a holistic perspective and enables intuitive interpretations of underlying phenomena. Network visualization can also be applied to understand processes, such as machine learning pipelines or simulations, as well as to disseminate results to a wider audience. However, current tools often fall short in handling large networks with over 10,000 nodes in real-time due to limited rendering capabilities and the absence of continuous layout algorithms.

We introduce Helios-Web, a new network exploration tool capable of visualizing datasets with millions of nodes through GPU-based rendering and continuous force-directed layouts. This is accomplished by reducing the number of calls and information transmitted from the CPU to the GPU and implementing instantiated geometry. Our tool employs billboard rendering, which utilizes a single geometry source to render all the glyphs and lines on the screen. Each instance is then aligned to face the camera through a vertex shader on the GPU (as shown in Figure a). Nodes and edges are rendered using signed distance functions (SDF), which enables the use of various shapes and effects for the glyphs (as illustrated in Figure b).

To overcome the challenge of interaction with nodes and edges in large networks, our tool integrates a high-performance picking system that works seamlessly with the GPU-based rendering pipeline. This technique is based on rendering the scene to an external frame buffer with minor modifications to the fragment shader. This hardware-based picking allows real-time frame-by-frame interactions with nodes or edges, such as hovering, dragging, and filtering (as shown in Figure d).

The GPU-based pipeline of the tool enables the implementation of advanced effects to enhance insights and interactions with data. For instance, kernel density estimation in real-time allows exploring network properties or attributes assigned to nodes (as shown in Figure d). To demonstrate the effectiveness of this tool, we applied it to explore scholarly networks and embeddings derived from large portions of the OpenAlex dataset (as shown in Figure f for the physics field). Figure g shows an embedding visualization with points representing papers from the complete OpenAlex citation network. Density is used to highlight the regions with the highest concentration of papers cited by publications related to COVID-19.

The tool includes an API and interactive features, allowing users to search, filter, and highlight nodes or edges based on their attributes. Our tool outperforms existing open-source solutions for network visualization, including Graphviz, Gephi, Cytoscape, graph-tool, igraph, networkx, Graphia, and 3d-force-graph, being capable of rendering networks with more than 50,000 nodes on modest hardware and more than one million nodes on better hardware. In addition to networks, projections of embeddings can also be explored with the tool, which also provides capabilities to display the neighbors of entities in the original space. In conclusion, our framework provides means to handle large networks with millions of nodes and facilitates real-time visualization and exploration of dynamic complex networks. A preliminary tool to visualize OpenAlex citation networks is also available

ABSTRACT. Between 1850 and 2000, the average size of teams of inventors on U.S. patents increased from 1.3 to 2.3. One explanation for this increase is that more knowledge accumulated in the economy over time, which led inventors to specialize and to coordinate their knowledge assets with partners through collaboration [18, 19]. This explanation, named the ”Burden of Knowledge” hypothesis, is highly influential in the management and economics literatures, serving as a core inspiration for research on the benefits of team-based invention [23, 8], the costs associated with teamwork [6, 9, 26], the importance of knowledge-diverse teams for innovation in firms [16, 1], and the drivers of the long-term decline in R&D productivity [13, 7]. What is striking about the history of technological change, however, is not just how much the knowledge stock expanded, but how quickly it changed. Between the late 19th century and the late 20th century, horses were replaced by cars, candles by light bulbs, and human computers by computational machines. Inventors and firms seeking to innovate during this period did not only have to navigate an increasingly large and complex knowledge environment. They also had to navigate a knowledge environment in which new ideas were being rapidly created, and old ones destroyed. This distinction between static differences in the size of the stock of knowledge, and dynamic differences in its velocity, is critical for developing a strong theory of the microfoundations of organization, because it indicates that there are at least two mechanisms that could have helped to shift the locus of invention from individuals to teams during the 20th century. Specifically, teams may have proliferated not only because of their ability to assemble large bodies of knowledge, but also because of their ability to assemble knowledge quickly.

ABSTRACT. Patents are intended to encourage innovation, but their ability to do so has been debated due to mounting evidence that incumbents use patents to discourage new entrants from innovating. Therefore, competition authorities may compel incumbents to grant royalty-free licenses for their patents in order to foster innovation. The extant empirical literature on royalty-free licensing demonstrates that such licensing arrangements can either foster innovation, have no impact on future innovations, or not adversely affect the licensor. Nevertheless, there is a lack of empirical research examining the utilization of royalty-free licensing as a deliberate corporate strategy by firms. In 2014, Tesla made the decision to provide licenses for all of its patents without charging any royalties. The stated intention behind this move was to stimulate the market for electric vehicles. This study presents the first empirical analysis of the impact of royalty-free licensing as a corporate strategy on both the licensor's innovation and follow-on innovations. The preliminary results indicate that Tesla has raised its innovation intensity by 128%. This aligns with the neck-and-neck competition for technological leaders described by Aghion et al. (2005). Nevertheless, there is no direct impact observed on follow-on innovations or the quality of innovation for Tesla.

ABSTRACT. In the US, UK and elsewhere there's an increasing focus on evidence based policy and on ensuring that policy decisions are made with the best possible evidence or expertise to hand - even if it isn't actually acted on as often as researchers might like.

Drawing on a new database of government policy engagement opportunities and the existing Overton database we create sets of implicit and explicit government evidence needs for the US and UK, grouping them by topic.

We then use an LLM and matching algorithm to see which parts of the research base in the respective countries might contribute useful to those needs, identifying areas of strength as well as potential gaps. We'll discuss the challenges involved in "translating" between the language of government policy documents and academic language when working with embeddings and semantic relatedness.

ABSTRACT. Scientific careers are characterized by a unique yet ubiquitous feature – the existence of hot streaks, which represents a burst of highly cited papers clustered together in sequence. As what is produced during hot streaks receives substantially more use than an individual’s typical performance, hot streaks dominate the main impact of a career. Curiously, hot streaks most likely last for about four years, a duration approximately similar as most scientific grants, prompting us to examine the relationship between the timing of hot streaks and scientific grant funding. Understanding such relationship not only offers novel opportunities to quantitatively probe the inner-workings of science at unprecedented detail and scale, but also helps us uncover fundamental factors that contribute to the success of individuals, and the discoveries that they produce.

ABSTRACT. Firms rely on academic science and actively participate in the production of scientific knowledge. However, the impacts of industry on academic science remain unclear. This study utilizes the site selection decisions of “Million Dollar Plants” to estimate the causal impacts of industry on academic science. We compare the responses of researchers in counties that successfully attracted new firms (“winners”) with those in counties that narrowly missed out on these firms (“runner-ups”). By analyzing the research trajectories of nearly two million scientists, our findings suggest that firm entry significantly shifts scientific research directions, without decreasing the productivity or quality of the researchers' output. This shift in research focus is correlated with increased funding from the entered firms, although the impacts persist even when excluding direct collaborators and grantees of these firms. Furthermore, the changes in research direction appear immediately after the firms' announcements, a time when actual resource provision and interactions are typically minimal. The effects are observed only with the entry of new firms that previously supported research, and are more significant among scientists who are newcomers to firm-related research or those without previous patent applications. We conclude that industry can shape academic science not only through resource provision but also through the redirection of scientists' attention towards more applied and firm-relevant research.

ABSTRACT. The study of paper mills---the organised manufacture of falsified manuscripts that are submitted to a journal for a fee on behalf of researchers, has become an important research topic to address for safeguarding of the integrity of the research process. Moreover, for the Science of Science, paper mills have become part of the data we study.

It is estimated that 2% of all journal submissions across all disciplines originate from paper mills, both creating significant risk that the scholarly literature becomes corrupted, and placing undue burden on the submission process to reject these articles. By understanding the business of paper mills, we can develop strategies that make it harder, (or ideally) impossible for them to operate.

Our key insight in this paper is that paper mills do not just fabricate content. Convincing paper-mill outputs also fabricate co-authorship networks of the researchers that they involve. This latent, collaboration fingerprint will become increasingly important as publishers tighten their processes to reject paper-mill content. Paper-mill papers that successfully make it through the submission and review processes are likely to require a more convincing fabricated author network.

By understanding and identifying the properties of these fabricated co-authorship networks, we propose effective strategies aimed at identifying, and thus preventing, paper-mill activity. By identifying authors that are very likely to be part of fabricated co-authorship networks, we provide techniques to simplify paper mill detection during the submission process; highlight journals that appear to be compromised; and reduce the supply of willing participants by identifying 'at risk' researchers within universities.

ABSTRACT. Securing funding in early stages of one’s research career is crucial.(1,2) Not only it determines whether a researcher can develop their own project and build their own team, but it can define whether they secure a permanent academic job at the university or leave academia. Yet, the more recent debate in science funding of early career researchers is shifting from exploring its effects on future outcomes to understanding who is more likely to benefit from it.

ABSTRACT. In the market for scientists, PhD advisors are potentially important match makers that allocated their graduates to research positions. Theoretically, the use of social networks to allocate workers can be associated with negative or positive outcomes. In this paper, we document investigate the role of the advisor’s co-authorship network for student placement, as well as the association between network hiring and subsequent research productivity of graduates. Answering this research question is important for understanding allocative efficiency in the market for scientists—even more so as the fraction of PhD students placed through their advisor’s network has been rising over time (see figure 1a).

ABSTRACT. We examine the impact of technological competition on voluntary innovation disclosure around the enactment of the Leahy-Smith America Invents Act of 2011 (“AIA”). The AIA moves the U.S. patent system from the first-to-invent to first-inventor-to-file system and induces a patent race that increases technological competition. Firms that are slow to file a patent are disadvantaged in this race. We find that firms file fewer patents in technology areas where they lag behind their peers or completely abandon the laggard domain. Sample firms strategically increase scientific publications in their lagging technology areas in an attempt to block competitors from obtaining a patent. This effect is more pronounced when there is greater inventor mobility, among firms most affected by the AIA, and those facing financial constraints. Furthermore, the strategic disclosure effect is stronger in patent technology classes with fewer experienced attorneys and areas characterized by more intense competition. We find that peers of laggard firms experience greater patent filing rejections for lack of novelty and obviousness reasons after the AIA, suggesting that strategic scientific disclosure is effective. 

ABSTRACT. The availability and usage of data in scientific research have profound implications for datadriven decision-making, technological development, and social equity. Science of science studies have begun to uncover global inequities in the scholarly communication ecosystem, including uneven collaboration practices [4], inequity in peer review [6], disproportionate citation attention to western publications [5] and unequal production of genomics datasets [2].

Research has shown that closer proximity can enhance collaboration between institutions, indicating that researchers may prefer to submit datasets from nearby locations because of local expertise and to facilitate collaboration and data sharing [1]. Yet,“helicopter science” [8] – situations in which the Global North uses data from the Global South without equal benefit – still persists [3]. Previous work has found that the preponderance of the datasets submitted are by the global north and that over 89% are collaborations among scientifically advanced countries [2]. To our knowledge, no studies have examined the influence of helicopter science on dataset use and citation from a science of science perspective. Given this backdrop, this study is guided by the following research questions: (RQ1) To what extent are datasets from the Global South used by researchers in the Global North? (RQ2) What is the proportion of Global South researchers who use Global South datasets?

In this study, we systematically analyzed 3,211 scientific articles across 17 scientific fields: Biology, Business, Chemistry, Computer Science, Economics, Engineering, Environmental Science, Geography, Geology, History, Materials Science, Mathematics, Medicine, Physics, Political Science, Psychology, and Sociology. We use datasets published in the peer-reviewed journal Nature Scientific Data (2014-2024) obtained from Semantic Scholar [10] and OpenAlex [7]. We chose it because it is a rich source of unrestricted, available datasets for researchers across fields and from around the world to use with a diverse sample of fields. We extracted geographic names from the abstracts using ReFineD [9] from Amazon Alexa AI, a zero-shot learning approach that identifies geographic entities in text and links them to Wikidata entities. Importantly, the model performs fine-grained mention detection and disambiguation in a single forward pass through the text and is capable of generalizing to large datasets such as Wikidata, which has 15 times more entities than Wikipedia. The performance of this model in zero-shot applications is precisely what makes this model appropriate for this task - it works out of the box and does not require additional training data to fine-tune the model and improve performance.

Using the World Bank income groups (FY24) and the S&T Capacity Index [11], we classified authors and papers at the country-level. We operationalized Global North as “High income” and “Upper middle income” countries and “Low income” and ”Lower middle income“ countries to Global South. We then compared authors to papers by income level.

Results indicate that author affiliation is identical to the geonames country approximately 25% of the time. With 1,196 (37 %) of the papers containing geoname entities, we analyzed the helicopter science phenomenon by comparing the geographic origin of the dataset used in the paper and the geographic affiliation of the data paper authors. In our sample, authors from high-income countries (i.e., Global North) used data from lower-income countries in 802 papers, whereas scientists from the Global North use datasets in only 12% of papers. Findings emphasize the importance of collaborative research between the Global North and the Global South for data-intensive scientific research. Future work will develop the concept of ‘token collaborators’ to enrich theories of inequity in scientific collaboration.

ABSTRACT. The paradigm of scientific medicine is among the most influential epistemic shifts in the past century, wherein randomized clinical trials (RCTs) represent the impartial arbiter of legitimate medical knowledge, a view prevalent among quantitative social scientists. Nevertheless, not all RCTs agree, and systematic reviews are invoked to reconcile them. These assume the wisdom of crowds, which hinges on diverse perspectives and data, across the distribution of analyzed studies, but socio-epistemic bubbles across them may reduce realized diversity. We theorize how tacit knowledge, beliefs, and expectations accumulate within these ‘socio-epistemic bubbles,’ continuous regions of latent social density that may decrease diversity and increase certainty about healthcare studied by RCTs. To assess our theory, we analyze the Cochrane systematic review repository, covering 20,117 meta-analyses extracted from 1,962 reviews. We find that being closer within ‘social space’ inscribed by scientific collaboration markedly increases agreement regarding RCT effect direction and size. Our analysis suggests that this amplified certainty can drive premature convergence and path-dependency affecting medical practice and population health. Moreover, our findings imply hidden limitations associated with unmeasured social influence across the policy sciences through which conflicting claims perpetuate and highlight the necessity of accounting for them to improve collective certainty.

ABSTRACT. We develop a framework to measure the hierarchical position of countries in the global scientific community, while focusing the rise of Chinese scientists because of their geopolitical significance [2]. To measure Chinese scientists’ leadership position in the international scientific collaboration network, we adapt the neural classifier model developed by [3] to identify the leaders of the 6 million OpenAlex articles that involved bilateral collaboration between global regions (e.g. China-US collaborations, or China-EU collaborations).

ABSTRACT. Advancements in biomedical research are significantly propelled by the development and utilization of diverse biomedical tools, collectively termed ``BioTools''. These encompass laboratory techniques, software applications, and modeling methods essential for experimental investigations and data analyses. Despite their critical roles, the impact of BioTools extends beyond traditional bibliometric indicators like citation counts, rendering their contributions largely invisible within the conventional scientific evaluation framework. Our study challenges the prevailing paradigm by proposing a comprehensive framework to assess the impact of BioTools across multiple dimensions. This framework not only captures the traditional citation-based metrics but also includes indicators reflecting the tools' policy reach, utility in disease research and treatment, and influence on technological innovation. By doing so, it acknowledges and quantifies the often-overlooked contributions of BioTools and their developers to the scientific ecosystem and society at large.

ABSTRACT. The essence of sociology is dynamic. This paper investigates trends or "vogue" in American sociology by constructing semantic networks from a decade (2011-2020) of sociology doctoral dissertations. These networks consist of research terminologies as nodes, interconnected by weighted edges that represent their co-usage frequency. We measure vogue as pairs of words, such as “mental health-stigma,” that move toward the network's backbone over time. We compare vogues with research published by representative sociology journals and we find that they tend to align more closely with journals focusing on American social problems, rather than high-status methodology or theory-oriented journals. We construct a unique dataset of 12,882 school pairs across 114 schools, detailing their dyadic relationships (e.g., geographical co-residence) and the production-adoption of vogue to explore the driving forces of diffusion using dyadic-cluster-robust inference. We find a dense "core" around the Northeast, Mid-West, and Western Coast regions in the exchange of vogue among schools. Adoption is notably influenced by previous research fit, status, institutional classification, and geographical location. Higher-ranked schools are more inclined to adopt vogues while not necessarily producing them. Public schools show a preference for adopting vogues produced by public peers in the same region, while private schools tend to adopt those from private peers of comparable rank. Overall, this work incorporates the once-overlooked cultural dimension into the mechanisms of innovation diffusion.

ABSTRACT. We investigate how audience interconnectedness shapes the use of interdisciplinary research. While research has documented a positive effect of interdisciplinary research on impact, we know relatively little about the role that audiences, who evaluate and use the research for their own purposes, play in that relationship. Focusing on the structural characteristic of audiences a scholar faces – audience interconnectedness – we examine how this characteristic moderates the effects of interdisciplinary research on two outcomes: scholarly impact and disruptiveness. Using 747,259 papers by 67,843 researchers affiliated with 14 large U.S. research universities between 2008 and 2011, we find that the more interdisciplinary the research is, the more impact it has and the more it is likely to be disruptive years later, in 2017. For scholars with a more interconnected audience, the positive effect of interdisciplinary research on scholarly impact is also enhanced, but its effect on disruptiveness is reduced. Our study enhances the literature of interdisciplinary research and social categories by demonstrating how the use of a boundary-spanning work is shaped by both producers who create the work and audiences who evaluate it.

ABSTRACT. Extended Abstract Institutional leaders commonly call for faculty to conduct research that spans more than one area of traditional knowledge; however, the career consequences for people conducting such work are understudied. We present two studies that focus on University faculty whose appointments span more than one area. First, through analyses of 1,696 promotion and tenure cases across six universities, we find a significant interaction whereby faculty with Dual Appointments are evaluated more critically when seeking promotion to associate professor (with tenure) but more favorably when seeking promotion to full professor. Second, we report on a survey of 79 College- and School-level deans from whom we solicited policy ideas for updating and improving faculty evaluation processes. We also found that only 16.5% of deans estimated that interdisciplinary faculty tend to be evaluated unfavorably in current processes. Our studies highlight how universities might hope for – but not necessarily reward – people to conduct interdisciplinary research.

As visible in Figure 1, our analyses show that when dual-appointed faculty apply for promotion, they are less likely to receive unanimous support at the Departmental and College level than faculty who are appointed (more traditionally) in a single department. While the analyses that underlay Figure 1 do not provide direct insight regarding mechanisms that explain the pattern, closer analyses that differentiate between faculty whose dual appointments span relatively large or small distances appear to be important. Specifically, using the Classification of Instructional Programs (CIP) two-level taxonomy of fields, when we differentiate between faculty whose two appointments involve (a) departments whose field of study are in two different top-level categories (e.g., Humanities-and-Arts and Engineering) or (b) departments that are part of the same top-level category (e.g., Social Psychology and Industrial-and-Organizational Psychology), we find that faculty with Dual Appointments that span relatively large distances are more likely to experience non-unanimous votes than faculty with Dual Appointments that span relatively shorter distances (i.e., with fields that exist within the same top-level category).

[Figure visible in uploaded file]

Figure. 1. Relationship between dual appointment status and unanimous votes at the Department Level (Panel A) and College Level (Panel B). Note. Values are model-estimated means from models represented in Table 1’s Models 2 and 4. Error bars represent 95% Confidence Intervals.

In our survey of deans from member institutions of the American Association of Universities (AAU), the vast majority of deans (83.5%) believed that there is either no problem (i.e., interdisciplinarians were treated the same) or, conversely, that interdisciplinarians were actually treated more favorably. Against the backdrop of the P&T study’s findings, the deans’ estimates highlight a partial misalignment between the deans’ perceptions and interdisciplinarians' experience at the promotion to associate level, given that we find evidence of less favorable treatment for that promotion stage.

In response to a prompt to identify updates or improvements to faculty evaluation processes in light of prior research showing that interdisciplinarians tend to be penalized in the academic labor market (Kniffin & Hanks, 2017), deans provided ideas that we classified. The most popular recommendation was for “Policy Adjustment” (71%), followed by “Communicating Value of Interdisciplinary Work” (38%), “Committee Composition” (15%), “Committee Training” (8%), and “Major Overhauls” (4%). It is notable that all but one of the 79 deans provided a substantive answer to our invitation to identify a concrete improvement or update to prevailing P&T processes.

Our study of P&T systems and the survey of deans aims to showcase the value of an evidence-based approach to understanding and improving promotion systems for college and university faculty, particularly in relation to those who take on dual-appointed positions. Despite regularly encouraging students and alums to operate with the benefit of evidence-based approaches, we show – as an illustration of organizational failure (Kerr, 197) – that institutional calls “hoping” for researchers to conduct work that spans traditional areas of knowledge tend to bring unfortunate “folly” for early-career faculty seeking promotion to the Associate rank with tenure as we show evidence of a misalignment in the reward systems for P&T. Rather than rest in such an observation, though, we concurrently present data and analyses that are aimed at stimulating the identification of improvements to update faculty promotion and tenure systems.

References

S. Kerr, On the Folly of Rewarding A, While Hoping for B. Acad. Manage. J. 18, 769–783 (1975).

K. M. Kniffin, A. S. Hanks, Antecedents and near-term consequences for interdisciplinary dissertators. Scientometrics 111, 1225–1250 (2017).

ABSTRACT. Interdisciplinarity is considered essential to solving many of the world’s most pressing problems. However, prior research finds conflicting patterns regarding whether interdisciplinary research advances or harms scientists’ careers. We propose that some of the conflicts can be reconciled by distinguishing between the interdisciplinarity of the knowledge and expertise that inform the research process (“input interdisciplinarity”) and the interdisciplinarity of the framing that markets a work to a particular audience (“output interdisciplinarity”). We test this argument in the setting of journal peer review, using administrative data from a major STEM publishing company. The data comprises reviewer recommendations and editors’ final decisions on 118,412 manuscripts submitted to 62 STEM journals between 2018-2022. We measure text interdisciplinarity, a form of output interdisciplinarity, via title and abstract, and reference interdisciplinarity, which incorporates both input and output interdisciplinarity. We find that greater reference interdisciplinarity is associated with better evaluations, while greater text interdisciplinarity is associated with worse evaluations. This suggests that a manuscript fares best when it acknowledges many disciplines in its references but is otherwise positioned for a monodisciplinary audience. Additionally, journals designated as interdisciplinary do not show penalties against any type of interdisicplinarity, suggesting that existing evaluation penalties are not due to unavoidable cognitive biases.

ABSTRACT. Understanding how specialized knowledge anchors the attention of researchers, and how that anchoring creates boundaries and enables certain interactions is a general problem for various fields of quantitative modeling of science and technology. Here we present a straightforward approach to represent and measure distances between papers or authors based on transition probabilities (TP) in citation networks. The TP in short is the probability that a random walk starting from paper A, and following citation links, will transition through paper B. Aside of its simplicity, this approach has several advantages. First of all, this method does not require a curated classification system. Second it does not involve a clustering step, which comes with certain complications. Third, it provides a continuous measure for distances. Techniques which rely on embedding techniques to create spatial representations have all these capabilities, plus spatial representations offer many analytical possibilities. However, the proposed metric has one clear advantage over these latter approaches, namely that its distance definition follows a clear operationalization, and therefore it is easy to interpret. Our operationalization of research interest distance is tied to information retrieval, the way how a competent expert would go about learning something new in her or his field. More specifically it is deduced from the procedure of snowball literature search (Greenhalgh and Peacock 2005). This is a procedure for learning about new relevant literature by following the forward and backward references starting with some preselected paper(s), and it is a good behavioral model for expert information update.

ABSTRACT. Concerns over research integrity are rising, with increasing attention given to the potential threats from untrustworthy entities within scientific studies. The field of Alzheimer’s Disease (AD) research has been a focal point of these worries [1]. We established a framework to gauge the potential negative influence of researchers, taking AD research as an illustrative example. Analysis of citation network structure can elucidate the potential propagation of misinformation arising at the author level. Our analysis revealed that there aren't any single authors or papers whose citation connections jeopardize a major portion of the field's literature. This indicates a low probability of single entities undermining the majority of works in this area. However, our findings do suggest that attention to the research integrity of the most productive and influential scientists is warranted. Additionally, our study introduces an analytical framework adaptable across various fields and disciplines to evaluate potential risks from fraudulence.

We asked to what extent the most productive and highly cited authors in the field could call into question the integrity of later articles that cited and potentially built on that work. Of the 22,941 authors who published multiple first- or last-author AD papers, the average fraction of the literature citing an author was 0.3% (Figure 1). Less than 1% of these authors were cited by more than 5% of the literature (n = 112,720). In the top 150 authors, the average citation reach of their papers was 7.8% of the literature (n = 112,720). While these highly-cited authors pose a higher risk if they engage in fraudulent activity, on average, 92% of the literature (n = 112,720) in the field does not cite their work. Lower frequencies of misconduct or the presence of ancillary citations [2] would necessarily reduce the meaningful citation reach of such work.

To better understand the potential impact of hypothetical author-level misinformation published in the scientific literature, we visualized the citation reach of randomly selected individual authors cited by approximately 2% of the AD literature (Figure 2). This is a comparable figure to the fraction of the literature citing the author who was called into question in Science (1).

Our study provides quantitative insights into the potential risks of fraudulent research practices in the field of AD research. While the risk of individual scientists jeopardizing the integrity and reliability of the majority of the literature in this field is generally low, certain aspects, such as highly cited papers and authors, warrant attention. By maintaining high standards of research integrity and transparency, the scientific community can minimize the impact of unreliable or flawed research and ensure the advancement of knowledge in the field of AD research.

ABSTRACT. Ground-breaking innovations and scientific discoveries, such as the SARS-CoV-2 virus, represent unique opportunities for researchers to to bolster their productivity and reputation through impactful publications. However, academic success hinges on the accumulation of high-quality publications in prestigious journals. Despite increased female participation in science and comparable scientific quality in publications (Hengel, 2022), women still produce less and have less impact in academia than their male counterparts, and remain under-represented in medical and academic leadership (Huang et al., 2019). Gender productivity gaps may stem from various barriers in scientific academia, including selection into topics (Kozlowski et al., 2022), ability to change research topics (Viglione, 2020), or to women not taking on new publishing opportunities(King and Frederickson, 2020).

ABSTRACT. We reveal findings that question normative beliefs about the gender innovation gap. First, an examination of millions of scientific, technological, and artistic innovations indicates that women innovators encounter special barriers when their innovations push the boundaries of scientific and technical convention. Second, novel data on the innovation review process suggests that the gap in approval rates for women’s boundary-pushing innovations narrows with evaluator experience. Lastly, our findings highlight that women innovators disproportionately match with less experienced, often female, evaluators, exacerbating systemic biases. These findings provide insight into newly identified innovation barriers that impede women innovators and can help inform policies regarding science’s apparent decline in disruptive innovation [3].

Unconventional thinking is crucial for solving major challenges in science and society, yet such innovations often struggle in conservative fields due to their novelty and lack of historical precedents [3]. The recent decline in disruptive ideas signals a concerning trend towards reduced innovation [3, 1]. Increasing women’s participation in innovation could significantly boost patenting, yet their contributions remain underrecognized due to a persistent gender innovation gap across publishing, grant writing, and patenting. Despite known factors that promote innovation, the specific impact of women in evaluative roles on different types of innovations, from incremental to transformative, is still largely unexplored [3, 1].

We leverage five datasets, encompassing 7 million patent applications from the US, the UK, and Canada, alongside over 15,000 Netflix items and 500,000 Food.com recipes. Our approach involves creating category co-classification networks, utilizing the categorization tendencies inherent in patents, film content, and recipes to measure how typical or atypical a combination of categories is [2]. To explore the relationships between inventors and evaluators in the success of boundary-pushing innovations, we use the temporally evolving co-classification network to compute a metric for each innovation that represents the degree to which it features common combinations of categories or spans boundaries by combining areas rarely seen together.

We find that women are penalized and men are rewarded for engaging in boundary-pushing work, with the gender gap widening the more boundary-pushing the innovation is and the more women are on the inventor team (Fig 1A), a finding that generalizes to women patenting internationally, international women screenwriters, and recipe creators. The gender innovation gap, particularly in boundary-pushing patents, is often attributed to gender stereotyping. Contrary to the stereotype, we find that women inventors are frequently evaluated by women examiners more than expected, yet face higher rejection rates by women evaluators, indicating other underlying factors. Evidence points to institutional arrangements as contributors to this gap.

We find that low-experienced examiners are less likely to grant patents than are experienced ones, especially for more boundary-pushing innovations, and that their decisions have higher reversal rates on appeals, highlighting the pivotal role of examiner expertise in decision accuracy. Institutional work crediting policies at the USPTO exacerbate the gender gap. Using data provided following a FOIA request, we examined how many work hours examiners are credited for patent applications in different technical areas. We find areas where women are more likely to invent are given fewer credit hours for examination, particularly for reviewing groundbreaking work. This practice may lead to rushed decisions and mistaken rejections of boundary-pushing work from women. The prevalence of gender homophily in inventor-examiner matches is evident in Fig 1B, with women inventors being 18.6% overrepresented in matches with women examiners, who are less experienced than men examiners (inset). Inexperienced examiners, often women due to higher dropout rates, are likelier to deny boundary-pushing patents, adversely impacting women inventors whose applications they disproportionately review. The resulting institutional barriers show women inventors face systematic disadvantages and underscore the need for equitable evaluation processes to foster inclusivity and equity in innovation.

ABSTRACT. Postdoctoral researchers (hereafter referred to as ‘postdocs’) contribute to scientific, technological, and societal innovations under the supervision of academic faculty and domain experts. Their international movements facilitate the advancement and diffusion of knowledge and thus are important for research and development. To better understand how postdocs move across countries and shape the research workforce, detailed curriculum vitae of postdocs at a global scale are highly needed. However, due to the difficulty of scaling up samples of curriculum vitae, many previous studies focus on specific countries or continents to explore the factors of postdoctoral career trajectories (Arbeit & Yamaner, 2021). Bibliometric data have also been used to reconstruct career trajectories from author affiliations, but it still has limitations because affiliations would not fully capture correct workplaces for postdocs.

ABSTRACT. Despite longstanding concerns about the under-representation of women in science, few studies have approached this issue from the perspective of the changing organization of work in science. Past studies have documented a trend toward increased bureaucratization of scientific work, marked by the growing number of scientists specialized in supporting roles. Using data on the publishing careers of 658,049 scientists from 1951 to 2012 from selected natural and social science fields, we show that these “supporting” career-type scientists have been traditionally associated with women. While we find that the gender difference in career types has converged over the past few decades, this convergence has been largely driven by an increasing share of male scientists taking on supporting roles. We also find that historical gender inequality in career attrition in science is largely attributable to women traditionally occupying “supporting” roles, which suggests that examining work organization is crucial for understanding gender inequality in science. Lastly, using survival analysis, we find that both female “lead” and “supporting” career types face higher attrition rates than their male counterparts. Meanwhile, we find that “lead” career types yield fewer advantages for women compared to men in natural sciences, whereas “supporting” career types are particularly disadvantageous for women in social sciences. Our findings provide science policymakers with insights necessary to tailor support for women scientists by considering the nuances of their production roles in science.

ABSTRACT. Across academic fields in the U.S., tenured and tenure-track faculty are approximately 25 times more likely to have a parent with a PhD than the typical American, and this tendency is higher still at elite universities. The impact of this dramatic lack of socioeconomic diversity on the scientific discoveries and innovations produced by those faculty remains largely unknown. While gender, race, and ethnicity are known to influence researcher specialty, socioeconomic background has not been studied. This project is the first to study and uncover large-scale differences in scholarship across socioeconomic status (SES). To do so, we apply latent Dirichlet allocation (LDA), an unsupervised topic model, to publications of U.S. tenured and tenure-track faculty for which we have self-reported SES. Our results show that some research topics exhibit strong correlations with the SES of faculty.

ABSTRACT. Race and gender disparities in academia are the focus of a voluminous body of academic research, as well as major efforts to tackle them. Socioeconomic background (“SEB”) or class origin, in contrast, is rarely considered in this context, whether in research or as a focus of DEI efforts [5]. Yet recent research has documented a striking underrepresentation of people from less advantaged socioeconomic backgrounds in academia. Less than one third of US PhD recipients 2010-21 are first-generation college graduates, as compared to an estimated 66% of the similar-aged US population [6], and US tenure-track faculty are even more elite than US PhD students [3]. This underrepresentation of lower-SEB tenure-track academics is in large part a function of the pre-PhD pipeline [4]. But it is also possible that – like with gender and race – lower-SEB academics face disproportionate barriers to career advancement during their PhD and tenure-track careers. This has yet to be studied systematically, although interviews with lower-SEB academics suggest it is likely the case [e.g. 2].

ABSTRACT. Securing positions at higher-ranking institutions is essential for the career success of scientific scholars (Garcia-Sancho et al., 2021; Larivière et al., 2015). Factors influencing such placements include scientific output (Larivière et al., 2015; Siler et al., 2020), educational background (Hengel, 2020; Larivière et al., 2015), funding acquisition (Jackson & Bielby, 2020), and professional social networks (Uzzi & Evans, 2013). However, one often-overlooked factor in the discussion of scientific career success is the role of physical appearance.

ABSTRACT. Prior research has shown that researchers experience pressures to oversell the novelty of their work to gain recognition, but the consequences of such tendency have been understudied. This study investigates whether novelty claims and the gender of the first author affect the likelihood of retractions, corrections, and expressions of concern in high-impact journals, specifically Nature and Science. By joining data from Microsoft Academic Graph and Retraction Watch, I find that papers with more novelty claims face an increased risk of retraction, while papers whose first author is female experience an increased likelihood of having corrections or expressions of concerns about their work published. The results suggest that authors who emphasize the innovative nature of their work, and authors who are female, have their work scrutinized more than their counterparts. My findings push scholars, administrators, and editors to consider how the drive for novelty and gender bias influence scientific publishing, and thereby careers of researchers.

ABSTRACT. Credibility assessment is necessary for science to be self-correcting. Evaluating credibility helps root out flaws and gaps between evidence and explanation, and progress is made by offering new explanations that account for the present evidence and predict future evidence and eliminating explanations that do not. These credibility assessments are pervasive and span the informal to the formal, ranging from conference attendees, interested readers, tenure committees, funders, and other researchers. Despite the diversity, a single method of credibility assessment dominates in the present research culture -- journal peer review. The standard format of peer review is an ad hoc collection of 2 to 5 peer reviewers that critique a paper to help an editor decide its suitability for publication at a specific journal. Much has been written about the limitations of this model.

ABSTRACT. Deep learning-based artificial intelligence (AI) has emerged as a defining technology of the 21st-century. It drives the internet's search and recommendation systems, tumor detectors that outperform pathologists, and generative AI is now writing poetry, solving mathematical theorems, and questioning the future of programming. The speed at which the field of AI research (AIR) has produced these successes is equally impressive. AIR's productivity is sociologically provocative because the field's approach to research differs radically from other scientific fields. Where other fields maintain loose divisions between autonomous, “basic" research and “applied” or “task-driven” research (e.g., immunology v. vaccine development), the interests of academia and industry have long been deeply entangled within AIR. And where other fields assess scientific progress and significance “organically” through mechanisms like peer review, mathematical theory-building, and citation, AIR relies mostly on the formal demonstrations of empirical performance that are common in task-driven sciences (e.g., clinical trials).

ABSTRACT. Understanding the broad impact of science and science funding is critical to ensuring that science investments and policies align with societal needs. Existing research links science funding to the output of scientific publications but largely leaves out the downstream uses of science and the myriad ways in which investing in science may impact human society. As funders seek to allocate scarce funding resources across a complex research landscape, there is an urgent need for informative and transparent tools that allow for comprehensive assessments and visualization of the impact of funding. Here we present Funding the Frontier (FtF), a visual analysis system for funders, policymakers, university leaders, and researchers to analyze the multidimensional impacts of funding and make informed decisions regarding future research investments and opportunities. The system is built on a massive data collection that connects 7M research grants to 140M scientific publications, 160M patents, 10.9M policy documents, 800K clinical trials, and 5.8M newsfeeds, with 1.8B citation linkages among these entities, representing to our knowledge the largest and most comprehensive data aggregation on science funding and its downstream impacts. As such, Funding the Frontier is distinguished by its multifaceted impact analysis framework. The system incorporates diverse impact metrics and a predictive model that forecasts future investment opportunities into an array of coordinated views, allowing for easy exploration of funding and its outcomes. We evaluate the effectiveness and usability of the system using case studies and expert interviews. Feedback suggests that our system not only fulfills the primary analysis needs of its target users, but the rich datasets of the complex science ecosystem and the proposed analysis framework also open new research avenues for both visualization and the science of science communities.

ABSTRACT. We compiled data on the educational and professional backgrounds of 6,751 members across 31 disciplines of the National Academy of Sciences, all elected prior to 2024. Our analysis reveals a pronounced trend of institutional concentration within the natural sciences, social sciences, and medical sciences—indicating a majority of members originate from a handful of elite universities. Conversely, the engineering sector shows a diversification trend, with a broader array of universities represented. Additionally, we highlight the strong interconnections among members, suggesting that such network ties could be a driving force behind the observed institutional concentration patterns.

ABSTRACT. Science of science scholars have studied scientific collaboration, both within individual research teams [6, 5] and across co-authorship networks [7]. However, while science has become increasingly dependent on computational methods [3, 1, 2, 4], researchers have yet to study the relationship between authorship and direct code contribution. We begin by first investigating, who, within individual research teams, creates the analysis scripts which make contemporary research possible. To enable such study, we have compiled an annotated dataset of pairs of author names and their associated GitHub account information (n=3000). We use this dataset to train a logistic regression model capable of matching author names and developer accounts, achieving a binary F1 score of 0.95 (precision: 0.93, recall: 0.96). We then construct a dataset of 37,168 scientific contributors from 4,352 PLoS articles, which provide a link to a source code repository in their ”data availability” notice.

ABSTRACT. Background: The field of Computer science (CS) has rapidly evolved over the past few decades, providing computational tools and methodologies to various fields and forming new interdisciplinary communities. This growth in CS has significantly impacted institutional practices and relevant research communities. Therefore, it is crucial to explore what specific research values, known as basic and fundamental beliefs that guide or motivate research attitudes or actions, CS-related research communities promote. Prior research has manually analyzed research values from a small sample of machine learning papers [1]. None of previous work has studied the automatic detection of research values in CS from large-scale scientific texts across different research subfields. This paper introduces a detailed annotation scheme featuring ten research values that guide CS-related research. Based on the scheme, we build value classifiers to scale up the analysis and present a systematic study over 226,600 paper abstracts from 32 CS-related subfields and 86 popular publishing venues over ten years.

Research Value Scheme: Building on [1], we define that a sentence encodes research value(s) if there is any expression of why something is desirable or undesirable. Through qualitative coding, three authors of the paper manually identified sentences discussing research values from 100 sampled abstracts and arrived at ten popular research values across different CS-related subfields. These values include (1) Performance, (2) Novelty, (3) Efficiency, (4) Generalizability, (5) Openness, Reproducibility, Collaboration, \& Future Work (Openness), (6) Simplicity, (7) Phenomenon Understanding \& Theoretical Grounding (Understanding), (8) Fairness, Bias, Privacy \& Ethics (Integrity), (9) Societal Implications (Society), (10) Usability. Each research value is clearly defined in a codebook with representative examples provided.

Data Sampling \& Annotation: We first curate a list of 86 popular CS-related research venues by consulting csrankings.org and sending questionnaires to 17 doctoral researchers from different communities. We then downloaded 226,600 papers from 2013 to 2022 with the PyS2 library. We randomly sample 12 abstracts from each venue and randomly pick one sentence from each abstract for the annotation task, resulting in 1032 sentences for annotation. Two authors of the paper double-annotate these sentences and hold rounds of discussions to determine what research values each sentence contains. Due to the challenging nature of the value detection task, two authors annotate one value at a time and have finished 7 values out of 10. We split the annotated data into train (40%), validation (30%), and test (30%) sets.

Task Formulation \& Method: We formulate the research value detection task as follows: given a sentence $s$ from an abstract, our target is to identify if a specific value $v_i$ from a set of values $V$ is encoded in the sentence. We build lexicon-based value classifiers and report their performance on the test set. We find that lexicon-based models achieve decent accuracy and F1 scores, with F1 ranging from 0.71 to 0.88 across seven research values. We also experiment with LLM prompting techniques [2] for value detection. Surprisingly, we find that lexicon-based methods outperform LLM-based few-shot prompting methods in 5 out of 7 value detection tasks, with between 0.5\% to 18.5\% gap in F1 scores across these five values. LLM-based methods slightly perform better in Novelty and Simplicity. Therefore, we turn to lexicon-based models for our subsequent analysis and also hope to fine-tune small RoBERTa in the future.

Analysis \& Findings: By running value classifiers on all the abstracts, we are able to characterize individual research communities by their most frequent and most distinctive values. First, we find that researchers in different CS subfields tend to emphasize different research values in abstracts. Traditional fields such as Computer Architecture and Operating Systems are known to give emphasis on Efficiency and Performance. On the contrary, AI-related subfields such as Computer Vision highlight Performance, Generalizability, and Novelty in their abstracts. Furthermore, we've noted that interdisciplinary CS+X fields such as Computational Biology \& Bioinformatics, Ethics, and Human-Computer Interaction (HCI), compared to traditional CS domains, are more likely to mention Openness and less likely to discuss Simplicity, Efficiency, and Novelty. The variation in these values demonstrates the diversity within CS communities in terms of their research practices. Secondly, we have discovered that specific values are experiencing increased emphasis in abstracts over time. Between 2013 and 2022, an uptick in the discussion around all seven research values was observed in the abstracts across AI-related subfields, which include Robotics, Computer Vision, Natural Language Processing, and Machine Learning. This implies that AI-related researchers have adapted their practices to more clearly articulate the research values within their abstracts over time. However, this trend isn't consistent across all CS subfields. Intriguingly, interdisciplinary CS+X subfields have shown an increasing emphasis on values such as Openness, Performance, and Understanding in their abstracts over time. On the other hand, subfields in Computer Theory have demonstrated a decreasing pattern in discussions about Simplicity over time, suggesting that theories may have become more advanced and complex. Interestingly, areas connected with traditional CS Systems have not shown any notable trends with regards to research values over time, showing that there is an established practice of writing in these subfields. Third, we examine the linguistic patterns used to communicate research values in abstracts. We find that these values are occasionally expressed directly with phrases like ``novel solution'', ``simple framework'', or ``generalizable method''. However, more often they are subtly represented through conventional linguistic structures such as ``propose a technique'' or ``create a dataset'' to imply novelty, and ``works in a variety of tasks'' or ``adapts to unseen tasks'' to suggest generalizability. At last, we study the pairwise co-occurrence patterns of values through pointwise mutual information (PMI) and find that Openness and Simplicity tend to co-occur with the other five values. In addition, Novelty and Efficiency tend to co-occur with each other in abstracts.

Conclusion \& Future Directions: Our paper presents a detailed value scheme and conducts a systematic study to uncover research values from scientific abstracts across different CS subfields over time. In the future, we plan to expand the study by (1) finishing annotating the remaining 3 research values out of 10, (2) studying if these values co-occur and are correlated with citations, funding sources, author seniority, and co-authoring patterns, (3) extending the current analysis beyond abstracts such as introduction and even full papers, (4) extending the value scheme to non-CS fields to understand how values are expressed across disciplines.

References [1] A. Birhane, P. Kalluri, D. Card, W. Agnew, R. Dotan, and M. Bao. The values encoded in machine learning research. In ACM FAccT, pages 173–184, 2022. [2] C. Ziems, W. Held, O. Shaikh, J. Chen, Z. Zhang, and D. Yang. Can large language models transform computational social science? Computational Linguistics, pages 1–55, 2024.

ABSTRACT. The transition from graduation to employment among U.S. university professors is far from being a typical job-matching or entry-level hiring problem. Recent evidence suggests that faculty hiring follows steep prestige hierarchies, in which a minority of universities concentrate the supply of faculty labor and rarely demand those trained in less prestigious institutions [2, 9]. Consequently, U.S. faculty hiring networks lead to systemic inequalities that could affect not only the faculty’s career progression but also the overall production and diffusion of scientific knowledge.

In this work, we examine the evolution of faculty hiring of leading U.S. scientists between 1880 and 1990 to answer three main questions: How have faculty supply and prestige hierarchies evolved in the hiring of U.S. preeminent scholars? Are concentration and steep hierarchies persistent in U.S. faculty hiring since the emergence of research universities? Or are they a more recent phenomenon associated with the increasing and dominant role of the U.S. in global science? To answer these questions, we investigate faculty hiring of those who became leading U.S. scientists later in their careers. Specifically, we analyze a century of faculty hiring networks from U.S. universities of education (where scientists obtained their highest degree) to U.S. universities for entry-level faculty hiring (where scientists got their first position as professors). We compute prestige and faculty production rankings for universities using a methodology similar to that described in [2, 9]. Furthermore, we examine the reliance of U.S. universities on foreign-born and foreign-educated scientists over time, as our study encompasses periods of massive migration and high dependence on foreign advanced training.

Our analysis covers the emergence and consolidation of U.S. academic research universities [4, 5]. Although the U.S. had advanced teaching institutions since the 17th century, they were primarily colleges or pre-modern universities with a single general syllabus. The consensus among historians is that U.S. research universities, conducting research in specialized fields, emerged in the late 19th century [4]. With some disagreement about the specific date, it is claimed until the early 20th century, U.S. academic research was still considered nascent and peripheral in global science [4, 3], and the most prestigious institutions were located in Europe. After the mid-20th century, U.S. universities consolidated as leaders in global science, having some of the most prestigious institutions in the world [8].

We rely on biographical data on leading American scientists, extracted from 12 different releases of the series of American Men (and Women) of Science [1, 6]. Our data captures the careers of multiple generations of leading U.S. scientists from 1906 to 2005, and their initial transition from graduation to faculty between 1880 and 1990. Our sample comprises hiring information on 129k leading scientists who held at least one position as professors in U.S. universities. By leading scientists, we consider those who have made significant contributions to their fields, are recognized by their peers, and have conducted a significant portion of their work in the U.S. [6].

Our findings reveal that although the concentration in the supply of U.S. faculty hiring has persisted since the emergence of research universities, it has exacerbated over time. In addition, prestige hierarchies in faculty hiring have become more dominant in recent years, as the role of geographic distance in explaining faculty hiring has decreased over time, while the position in prestige rankings has become more relevant. Until World War I, even universities ranked in the percentile 25-50 were supplying faculty to the top 10% most prestigious universities more than what would be expected if labor were allocated randomly. These differences later disappeared, becoming increasingly unlikely for universities outside the 10% most prestigious institutions to supply faculty for the top 10% universities.

These changes in faculty hiring also coincide with the increasing and dominant role of the U.S. in global science. Until the 1920s, around a quarter of U.S. faculty hires of leading scientists were foreign-educated, mainly in German universities. In recent years, less than 3% of U.S. faculty hires of leading scientists were foreign-educated, although the share of foreign-born U.S.-educated has increased.

Our work has significant implications for research on the long-term dynamics of faculty hiring networks. Until now, research on faculty hiring networks has focused on more recent periods of analysis, mainly after 2010, because of data constraints [2, 9, 7]. Therefore, a long-term analysis allows for identifying the origins and persistent factors behind prestige hierarchies and the concentration of faculty supply. For policymakers, understanding the origins of prestige hierarchies and inequalities in faculty supply can provide empirical foundations for strategies aimed at fostering a more inclusive scientific environment.

ABSTRACT. Women faculty experience academia differently from men in many ways, which can lead them to consider leaving their positions. Using a large-scale survey of 10,071 current and former tenure-track and tenured faculty, representing nearly all U.S. PhD-granting institutions and 29 diverse fields of study, we show that perceived workplace climate is the most gendered aspect of faculty life, compared to stress related to research pressures, work-life balance, and departmental support. Further, analyzing 6,615 free-text responses from the same respondents reveals that devaluation, both in formal evaluations and informal interactions, is the most gendered workplace climate factor. These patterns are especially salient among women of color and tenured women. Women report that devaluation is exacerbated when institutional leadership fails to respond to devaluation, leading many to leave their jobs. Our results highlight that successful remedies must involve organizational change rather than solely individual solutions.

ABSTRACT. In 2019, a math professor called Abigail Thompson published an op-ed in the Wall Street Journal comparing mandatory diversity statements for job applicants to mandatory loyalty oaths that her university required seventy years earlier. The reaction was immediate. Hundreds of colleagues from across the country made or signed statements calling her view “dangerous” and petitioning her employer to investigate whether her behavior was consistent with their “institutional policy”. In 2020, another math professor, Andrea Bertozzi, faced penalties for her work on “predictive policing.” An invited lecture to honor Bertozzi’s “contribution to the mathematical sciences” was cancelled because the organizers “do not believe that mathematicians should be collaborating with police departments” (Castelvecchi, 2020). Social sanctions for both ‘extramural’ speech, like that of Thompson, and ‘academic’ speech, like that of Bertozzi, are increasingly common. Some evidence suggests that these social sanctions may have consequences for academic speech and science more broadly. Frey and Stevens (2023) find that the annual number of attempts to suppress or punish scholars’ speech in the U.S. has increased dramatically over time. A recent survey of university faculty showed that 40% of liberal faculty, 56% of moderate faculty, and 72% of conservative faculty fear losing their jobs or reputations for something they say aloud or post online (Honeycutt et al., 2022). A 2023 survey of academics finds that these same kinds of repercussions affect the kinds of research scholars choose to pursue, potentially altering the direction of scientific inquiry (F. S. Union Tech. Rep., 2023). Despite the increasing prevalence of various sanctions against scholars, there has been little research to understand how these incidents may affect the academic enterprise. In this paper, we investigate whether targeted reputation attacks on scholars reduce the scholar’s recognition in their field and reduce the scholar’s productivity in their area of expertise. Ex ante, it is unclear whether researchers’ professional outcomes would be affected at all, and if they are affected—the effect could plausibly be positive or negative. Affected scholars, for instance, may benefit from additional public attention. Media coverage, a condition for inclusion in our data on incidents, has been positively associated with scholars’ professional success (Philips et al., 1991). Finally, negative reviews can increase sales of products (Berger et al., 2010), which may have parallels to academics receiving criticism, but that draws additional media attention. On the other hand, researchers could face professional penalties and ostracism as the result of an incident. We leverage a new dataset, ‘Scholars Under Fire’, collected by the Foundation for Individual Rights and Expression (FIRE). FIRE is a non-partisan, non-profit organization that defends and promotes free expression in the United States, particularly on university campuses. FIRE gathers information on incidents involving scholars in the U.S. who have received calls for sanction because of their protected speech. The organization collects these records by identifying incidents through news reports. Using a difference-in-difference (DiD) approach, we compare the citations of the work published by the affected scholars before the incidents to a control group of papers published in the same journal in the same year as the affected author’s work. Using this design, we find that the work of affected scholars receives approximately 4 percent fewer citations per year as a result of a “cancellation” incident. Our finding is consistent with the hypothesis that affected scholars are penalized somewhat by the scientific community following their involvement in an incident. We then investigate the effect of incidents on scholars’ future publications and the citations of future work. Compared to a control group of scholars with similar pre-incident characteristics, we find that affected researchers publish 19.5 percent less after an incident and their future work receives 14.5 percent fewer citations. We contribute to the literature in several ways. First, The political-science and legal-rights literatures have long histories of studying free speech in the academy. Researchers studying science and innovation are interested in how speech and controversy affect or distort the scientific enterprise. To date, existing work has largely relied on surveys. We make use of the modern applied economists’ toolkits and look at tangible outcomes on scholars’ careers. Second, we contribute to the theory of ‘Mertonian norms’ (Merton, 1973). Prior work on retractions (Azoulay et al., 2015) and sexual harassment (Widmann et al., 2023) find violations of Mertonian norms in the academy. However, retractions and sexual harassment are both clear examples of professional misconduct. Our empirical setting allows us to test this theory for speech that is usually not an example of professional misconduct. References Azoulay, P., Furman, J.L. and Murray, F., 2015. Retractions. Review of Economics and Statistics, 97(5), pp.1118-1136. “Academic freedom survey” (F. S. Union Tech. Rep., 2023) Berger, J., Sorensen, A.T. and Rasmussen, S.J., 2010. Positive effects of negative publicity: When negative reviews increase sales. Marketing science, 29(5), pp.815-827.Castelvecchi, Davide. 2020. Mathematicians urge colleagues to boycott police work in wake of killings. Nature News, June 19 Frey, K. & Stevens, S.T. (2023). Scholars under fire: Attempts to sanction scholars from 2000 to 2022. The Foundation for Individual Rights and Expression. Merton, Robert K., 1973. The Sociology of Science: Theoretical and Empirical Investigations. Chicago: University of Chicago Press. Phillips, D.P., Kanter, E.J., Bednarczyk, B. and Tastad, P.L., 1991. Importance of the lay press in the transmission of medical knowledge to the scientific community. New England journal of medicine, 325(16), pp.1180-1183. Widmann, R., Rose, M.E. and Chugunova, M., 2022. Allegations of Sexual Misconduct, Accused Scientists, and Their Research. Max Planck Institute for Innovation Competition Research Paper, (22-18)

ABSTRACT. The exponential growth of knowledge presents a paradox: as the collective reservoir expands, knowledge workers increasingly struggle to stay abreast of the latest developments. This predicament, known as the “burden of knowledge”, implies that knowledge workers may disproportionally rely on established concepts over time, particularly as they progress in their careers, which can stifle innovation due to the "knowledge lock-in". Moreover, the reliance on reputation-based signals such as citation counts, reinforced by the Matthew effect, may hinder knowledge workers’ effective engagement with emerging innovation. Recognizing the importance of assimilating new ideas for innovation arises the pivotal question this research seeks to address: How knowledge workers effectively absorb new innovation? Employing a quasi-experimental design, this paper examines how brief, unstructured exposure to new information can drive substantial knowledge acquisition among knowledge workers. This exploration amplifies the power of transient attention in shaping innovation trajectories and provides valuable insights into effective strategies to expedite knowledge absorption.

ABSTRACT. Artificial Intelligence (AI) encompasses many technologies, including deep learning, machine learning, natural language processing, neural networks, and rule-based systems (Lu, 2019). As the evolution of AI accelerates, it plays an increasing role in scientific research and technological innovation (Liu et al., 2020). The global emergence of AI as a relevant technological force has reshaped how policymakers and stakeholders perceive technological advancements (Lauterbach, 2019). In response to AI's global impact, AI policy has garnered international attention and emerged as a critical component of technology policy, given its potential impacts and its range of applications (Agrawal et al., 2019).

This research delves into the intricate evolution of AI policies over the past decade, particularly focusing on the scientific underpinnings that inform these policies. By leveraging the extensive Overton policy documents database, we mapped the contributions from various funders, organizations, countries, and journals to the policy AI discourse, offering a novel insight into how scientific knowledge influences policy formulation in the realm of AI. The study bridges the gap between AI technological advancements and policy development, highlighting the significant yet complex role scientific research plays in policymaking (Bozeman & Youtie, 2017; Gunn & Mintrom, 2021). It underscores the dynamic interplay between the rapid advancements in AI technologies and their reflections in the legislative and governance frameworks across different nations and regions.

To investigate the evolution of AI policies and the scientific literature influencing these policies, our study employed a comprehensive methodological approach leveraging the Overton policy documents database, which contains over 10 million policy documents from a wide array of sources, including governments, intergovernmental organizations, think tanks, and non-profit organizations. Our analysis began with a targeted search within the Overton database for policy documents containing key AI-related terms. Following the initial search, the results were exported to VantagePoint software for advanced processing and analysis. To deepen our analysis of the scientific foundation of AI policies, we extracted the Digital Object Identifiers (DOIs) of scholarly articles cited within these policy documents. A subsequent search for these DOIs was conducted across the Web of Science (WoS) database to extract comprehensive metadata for these articles.

Our findings reveal a marked increase in AI policy documents post-2018, underscoring a global acknowledgment of both the potential and challenges presented by AI technologies. It also indicates that the United States and the European Union are at the forefront of AI policy document production, reflecting their historical and ongoing contributions to AI innovation and governance. Additionally, China's substantial role in the scientific research cited within 3rd International Conference of Science of Science & Innovation, July 1-3, 2024 National Academy of Sciences, Washington, DC, USA

these policy documents highlights its growing influence in the AI domain despite a lower representation in policy documentation compared to its scientific output.

Our examination further identified key funders, organizations, and countries pivotal to the cited scientific research within these policy documents. The United States, for instance, demonstrates a comprehensive funding landscape for AI research through various institutions, highlighting AI's designation as a General Purpose Technology. In contrast, the high funding output from China aligns with its strategic ambitions in AI, underscoring a complex interplay of national priorities and global influence. The analysis also underscored the significant impact of scientific journals in shaping AI policy discourse. Prestigious journals such as Nature, Science, the New England Journal of Medicine, and The Lancet were among the most cited, indicating a preference for high-impact factor research in policy formulation. This preference suggests that policy documents may benefit from incorporating research published in journals with rigorous peer review and broad recognition, though it also prompts consideration of the broader spectrum of scientific work contributing to policy discussions.

Collectively, these findings offer a nuanced understanding of the dynamics at play in the evolution of AI policies. They reveal not only the geographical and institutional sources of influential research but also the thematic focus areas and ethical considerations shaping global AI policy. This discussion contributes to the ongoing dialogue on how best to harness AI's potential while navigating its ethical, societal, and governance challenges.

This research opens avenues for further exploration into the dynamic relationship between science and policy, emphasizing the need for evidence-based policymaking that leverages the full potential of AI technologies while addressing their ethical, societal, and governance challenges. It also lays the groundwork for using the same methodology for other relevant policy topics and understanding their scientific underpinnings. Additionally, we also acknowledge some limitations, including potential biases in the Overton database and the challenges of capturing the full spectrum of influences on policy development.

ABSTRACT. Research has shown that groups who have been historically underrepresented in academic research innovate at higher rates than majority groups members, but their contributions often go unrecognized and unrewarded in their disciplines (Hofstra et al. 2020). In this study, we investigate whether this pattern is reflected in the takeup of intersectionality as a theoretical and analytical framework in sociology. Intersectionality, initially developed by Black feminist scholars, has been described as one of the most significant innovations in sociological scholarship in recent decades (Cooper 2015). As defined by Collins (2015), intersectionality is the critical insight that multiple axes of status and power, including gender, race, sexuality, class, and age, are “reciprocally reconstructing phenomena that in turn shape complex social inequalities.” The concept, theory, and methodology of intersectionality offer important theoretical and empirical contributions to the study of inequality and social stratification, as evidenced by decades of scholarship (Carbado et al. 2013, Cho et al. 2013, Collins 2015). Yet there is reason to wonder whether sociology has fully embraced intersectionality. Early analyses of how far intersectionality has traveled suggested that, even a decade after its introduction to academic literature, the theory remained largely concentrated in specialty journals focused on gender and race (Jones et al. 2013). We expand on these analyses by measuring the extent to which intersectionality, as a key innovation in the study of inequality and stratification, has proliferated through sociology’s academic journals from its entrance into the academic literature up to the present. We track the use of intersectionality in journals, since a main purpose of academic journals is to publish theoretical and empirical innovations. We perform a descriptive analysis of four sociology journals: two of the discipline’s heritage journals, American Journal of Sociology (AJS) and American Sociological Review (ASR), and two leading specialist journals focused on gender and race, Gender & Society (G&S) and the Du Bois Review (DBR). We examine the full set of articles published in these journals from 1990 to 2023, a sample of more than 4000 articles, as well as a subset of these articles that are specifically focused on gender and race. We document the appearance of the term “intersectionality” and its derivatives in articles’ main text, abstracts, and keywords. These sections are likely to reflect differences in the readership’s familiarity with and endorsement of key concepts. Intersectionality in abstracts or keywords signals greater audience familiarity, since authors have limited words to explain concepts in abstracts and since keywords are used to help audiences find articles on topics of interest. In our analyses, we find a consistent pattern: intersectionality has been embraced by specialty journals focused on gender and race, but has largely remained outside of highprestige generalist journals, even in these journals’ gender and race-related articles. As previewed in Figure 1, intersectionality has been most heavily represented over time in the research articles published in G&S. G&S began to publish articles that mentioned intersectionality in the main text in the early 1990s, and in recent years, more than 50 percent of its articles mention intersectionality. In the early 2010s, the percent of DBR articles that mentioned intersectionality jumped, driven in large part by a 2013 special issue focused on 3rd International Conference of Science of Science & Innovation, July 1-3, 2024 National Academy of Sciences, Washington, DC, USA

the topic (Carbado et al. 2013). In comparison, the proportion of gender-or race-related articles mentioning intersectionality in AJS and ASR has consistently lagged behind that of G&S and DBR. These patterns are especially noticeable in analyses of abstracts and keywords. In additional analyses (not shown), we demonstrate that these specialty journal articles not only mention intersectionality earlier than heritage journal articles focused on gender and race; they also engaged intersectionality theory more deeply. We are also currently collecting data that will allow us to document the demographic makeup of intersectionality scholars from 1990-2023.

The failure to adopt an intersectional framework in these high-prestige journals potentially distorts the field’s understandings of gender, race, class, and other forms of stratification and inequality (Choo and Ferree 2010). Additionally, if intersectionality research is shut out of the discipline’s highest status journals, intersectionality scholars, who are disproportionality women and people of color, may face difficulties obtaining career advancing opportunities, including grants, academic positions, tenure, and more. Until the field develops a deeper understanding of and appreciation for this innovation, intersectionality scholars are likely to face persistent publication dilemmas and the scientific community will not fully understand the patterns and production of social inequality.

ABSTRACT. The scope of this study is to provide an overview to retractions in chemistry and related fields. By examining the patterns and causes behind the manuscripts retractions, we aim to highlight the strategies that can encourage adherence to best publication practices moving forward. Moreover, by studying the diagnosed misconduct through retraction notices we aim to cultivate a culture where scholars are better equipped to recognize fraud as well as to acknowledge and correct unintentional errors in their own works. For this study, we analyzed a total of 1,292 research and review articles retracted between 1 June 2001 and 23 December 2021. Retraction data were provided by the Retraction Watch(RW) database on May 24, 2022.1 Manuscripts were selected based on containing chemistry as a subject. However, only 19% (N = 245) of retracted manuscripts were indexed with a single subject. Remaining manuscripts were sorted by subjects into nine subject groups: 1) Physics & Energy – also includes mathematics and cosmology (N = 278); 2) Crystallography – also includes spectroscopy (N = 270); 3) Materials Science (N = 266); 4) Engineering - also includes chemical engineering, computer science and technology (N =244); 5) Biochemistry – also includes forensics and biology (N = 143); 6) Environmental Sciences – also includes climatology and archaeology (N = 96); 7) Nanotechnology (N = 92); 8) Medicine - also includes neuroscience, public health, and occupational health and safety (N = 63); and 9) Social Sciences – also includes business, art, education, and humanities (N = 13).

In addition to subjects, each manuscript in the RW database is indexed with a combination of retraction reasons (>100),2 which we grouped into five reason groups: 1) Misconduct (N = 756) - falsification or fabrication of data, image, results (fraud); plagiarism of article, image, text, data, concerns/issues with referencing/attribution; duplication of article, image, text, data; and copyright claims. 2) Errors & Concerns (N = 338) - error in data, analyses, methods, results, text, image, unreliable/irreproducible data/results, data not provided. 3) Author issues (N = 72) - concerns or issues about authorship, false or forged authorship or affiliation, author unresponsive, miscommunication issues, ethical violations by author, conflict of interest. 4) Publisher issues (N = 64) - fake peer review, duplicate publication, error by journal/publisher, rogue editor. 5) Organization issues (N = 11) - objection, lack of approval, error/ethical violation, concerns/issues about involvement (all by a third party or company/institution).

Based on the abovementioned classification, our analysis of retracted chemistry manuscripts showed the prevalence of misconduct; and most notably, plagiarism (Fig. 1). Of all subjects, plagiarism was more frequently observed in materials science and physics papers. Interestingly, plagiarizing someone else’s work is less common (N = 210) than self-plagiarism (N = 306); the latter is defined as duplication of data, image, text, or article. Article duplications represented 37% of all plagiarism cases in our dataset (173 out of 468). The majority of fraud cases were associated with crystallography; namely, publications in the Acta Crystallogr., Sect. E (N = 135). As compared to interdisciplinary manuscripts, those with a single subject of chemistry have a higher fraction of author issues and errors unrelated to misconduct (Fig. 2).

ABSTRACT. Integrating scientific research into public policy is essential to modern governance, particularly in a world with complex challenges. In healthcare, evidence-based decision-making is paramount (Baicker & Chandra, 2017). It ensures that policies are not only grounded in the best available knowledge but are also attuned to evolving healthcare needs (Gabbay et al., 2020). Brazil's National Commission for the Incorporation of Technologies into the Unified Health System (CONITEC) serves as a prime example of this approach.

By leveraging Clinical Protocols and Therapeutic Guidelines (PCDTs, in its Portuguese acronym), CONITEC standardizes and enhances healthcare practices, embodying a model of health policy formulation that is increasingly recognized as vital for effective and evidencebased healthcare governance (Wang et al., 2020). CONITEC's role in guiding Brazil's healthcare policies through PCDTs mirrors a global trend in health governance, where scientific research directly informs public health measures (Vicente et al., 2021).

In healthcare, utilizing scientific literature within clinical guidelines brings to light important questions about the diversity and relevance of research findings (Maaløe et al., 2021). The geographic and institutional origins of scientific research can greatly influence the nature and suitability of healthcare guidelines (Olayemi et al., 2017). For instance, research from highincome countries might not always address the health challenges of low- or middle-income countries (Orangi et al., 2023).

To contribute to the discussion between scientific evidence and public policies, this study undertakes a comprehensive bibliometric analysis of the scientific literature cited in PCDTs since the CONITEC’s inception. The analysis provides insights into how local and global scientific contributions are incorporated into national health policies. It also underscores the increasing demand for transparency and accountability in how healthcare policies are shaped by scientific research.

Employing a comprehensive data pipeline, this study collects, preprocesses, and analyzes the scientific publications cited in the PCDTs approved by the CONITEC. This methodological framework incorporates techniques such as web scraping, pattern recognition, and bibliometric analysis, enabling a thorough examination of the scientific literature's genesis and its integration into health policy. By extracting and transforming bibliographic data from digital PCDT reports for database querying, and conducting extensive searches in renowned databases like Scopus and Web of Science, the study offers a novel approach to understanding the linkage between scientific evidence and policy-making. The analysis reveals a predominant reliance on research originating from the United States and the United Kingdom, despite notable contributions from Brazilian sources. This indicates a significant global influence on Brazil's health policy, with international research playing a crucial role in shaping national healthcare practices. Moreover, the study identifies major health research institutions and pharmaceutical companies, primarily from outside Brazil, as key funding sources for the referenced scientific literature. These findings underscore the global interconnectedness of health research and its implications for national health policy frameworks.

This research underscores the critical interplay between global scientific research and the formulation of health policy in Brazil, highlighting the substantial impact of international scientific literature on the country's healthcare guidelines. By mapping out the flow of scientific knowledge into health policy, the study not only contributes to a deeper understanding of the policy-making process but also emphasizes the need for a more diversified and inclusive approach to incorporating scientific evidence into health policy. In light of these insights, policymakers are encouraged to consider a broader spectrum of scientific research, including local and regional studies, to enhance the relevance and effectiveness of health policies in Brazil.

ABSTRACT. This study reveals that, following bilateral reduced international visitation and academic exchange, Sino-American scientific collaboration is positioned at a turning point in a declining course. American international students originating from China have declined by nearly 22%, and American students studying in China plummeted to 1.8% of the number in 2018–2019. US-China mutuality in scientific collaboration has also declined remarkably. At the same time, the concentration of influential research collaborated between the United States and China is consistently greater than both nations’ research outputs. Following the discussion of possible substitutes and the roles of American and Chinese researchers in global basic science and emerging issues, I argue that the two nations are so entwined in scientific collaboration that an adversarial rivalry perspective misses much of reality. In the face of rising uncertainties and global disasters, humanity does not have time to waste on nationalistic competitions. It is time for visionary leadership from both countries to promote intellectual exchange and scientific collaboration to address pressing global challenges.

ABSTRACT. Extended Abstract Reproducibility and replicability have gained significant attention in scientific discourse, deeply intertwined with questions about scientific processes, policies, and incentives [1, 2]. Initially centered around the social and behavioral sciences, these concerns now span almost all empirical scientific disciplines [3], including artificial intelligence and machine learning [4]. The open science and science of science communities have responded with innovative initiatives aimed at shoring up the entire research workflow, from conception and study design to data collection and analysis, through to publishing [5]. These efforts have already had important individual and institutional impacts, many of which have been well documented [6]. For example, the Special Interest Group on Computer-Human Interaction (SIGCHI) now recommends providing supplementary materials for ACM publications to enhance replicability [7] and some universities have begun to reward researchers whose work aligns with standards of open science and transparency [8].

Despite these promising advances, however, conversations around reproducibility and replicability have predominantly reflected the voices of researchers in the global North and West [9]. This is concerning for several reasons, most primarily because issues of scientific integrity and scientific process are deeply social and contextual. Our work takes an initial step toward the inclusion of cultural perspectives through a comparative study of researchers in the USA and India. India currently ranks third in research output worldwide, following China and the USA. We conducted a survey-based study involving research faculty from universities in the USA and India. We aimed to gather the perspectives of scientists across different research disciplines. The survey asked participants about their familiarity with the reproducibility crisis, their confidence in work published within their fields, and the factors they believe contribute to this high or low-confidence research. Additionally, we asked participants to share the institutional and practical challenges they faced during their research. We reached out to over 8000 research faculty members and received a total of 430 responses.

Our study offers an in-depth analysis of researchers' views on reproducibility, replicability, and open science practices in the USA and India, casting light on both commonalities and disparities. While Western nations have more proactively tackled the replication crisis, Indian researchers are becoming more conscious of these issues and are making strides toward embracing open science. Yet, our findings indicate that its adoption faces hurdles in both countries and across fields. Respondents across contexts highlight the misalignment of prevailing academic incentives, centered around publications, promotions, and funding, as detrimental to engagement with open science practices. They note a lack of appreciation for replication studies in favor of novelty. We note that new incentives have emerged and should be used as stepping stones for substantial extensions. A notable example is the impact of badges. These simple rewards have a real, measurable impact on engagement with open science practices [10]. The next steps might include the inclusion of badges into reputational metrics and promotional practices. Journals and conferences might consider dedicated opportunities to publish replication studies. Discussions of reproducibility and open science centered in the West have not fully appreciated the challenges faced by researchers working at institutions with fewer resources and less social capital. For example, when attempting to reproduce or replicate a published finding, respondents in India believe that getting responses from the paper's authors in the West is more challenging for them.

Our work confirms shortcomings in researchers' approaches to the assessment of confidence in published work. The author's reputation was noted as a meaningful signal of credibility in both countries and across domains. Current ways for researchers to accumulate good or bad reputations, though, are driven by available existing metrics. The ways in which biases with respect to author reputation may be worsened or mitigated by open science practices, e.g., open peer review, is unclear.

References [1] Maxwell, S. E., Lau, M. Y., & Howard, G. S. (2015). Is psychology suffering from a replication crisis? What does “failure to replicate” really mean?. American Psychologist, 70(6), 487. [2] Schooler, J. W. (2014). Metascience could rescue the ‘replication crisis’. Nature, 515(7525), 9-9. [3] Baker, M. (2016). Reproducibility crisis. Nature, 533(26), 353-66. [4] Willis, C., & Stodden, V. (2020). Trust but verify: How to leverage policies, workflows, and infrastructure to ensure computational reproducibility in publication. [5] Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S., Breckler, S., ... & DeHaven, A. C. (2016). Transparency and openness promotion (TOP) guidelines. [6] Mazarakis, A., & Bräuer, P. (2020). Gamification of an open access quiz with badges and progress bars: An experimental study with scientists. In GamiFIN (pp. 62-71). [7] Echtler, F., & Häußler, M. (2018, April). Open source, open science, and the replication crisis in HCI. In Extended abstracts of the 2018 CHI conference on human factors in computing systems (pp. 1-8). [8] Schönbrodt, F. D., Mellor, D. T., Bergmann, C., Penfold, N., Westwood, S., Lautarescu, A., ... & Montoya, A. (2019). Academic job offers that mentioned open science. Open Science Framework. Retrieved from osf. io/7jbnt [Google Scholar]. [9] Mede, N. G., Schäfer, M. S., Ziegler, R., & Weißkopf, M. (2021). The “replication crisis” in the public eye: Germans’ awareness and perceptions of the (ir) reproducibility of scientific research. Public Understanding of Science, 30(1), 91-102. [10] Rowhani-Farid, A., Aldcroft, A., & Barnett, A. G. (2020). Did awarding badges increase data sharing in BMJ Open? A randomized controlled trial. Royal Society open science, 7(3), 191818.

ABSTRACT. Talent is a scarce resource, and when it comes to innovation, only a few exceptional individuals, like Marie Curie and Isaac Newton, possess the extraordinary potential to revolutionize our way of life ([Akcigit et al., 2020]). For organizations, the talent shortage is a critical problem, as highlighted in the recent Mckinsey Report ([Durth et al., 2023]), which states that ”The talent shortage is a critical problem that is only getting worse. Organizations already face a severe shortage of key talent, and 90 percent say they will have a meaningful skills gap in the coming years.”

Human capital plays a pivotal role in driving a firm’s innovation direction. Dr.Spencer Silver, a chemist at 3M, discovered a unique adhesive with a weak bond but strong sticking properties. Capitalizing on this invention, Silver’s colleague, Art Fry, created the iconic Post-it Notes. Silver’s adhesive innovation ultimately reshaped 3M’s innovation trajectory from purely industrial solutions to accessible and user-friendly products that catered to everyday needs. It paved the way for innovative problem-solving, collaboration, and note-taking, instigating transformative shifts in 3M’s innovation directions ([3M, 2023]).

Motivated by the scarcity of talents in organizations and instances where influential indi- viduals significantly impact a organization’s innovation trajectory, this study aims to uncover how losing talented individuals affect the firm’s innovation direction and explore the underly- ing mechanisms. By leveraging the unexpected deaths of individuals as exogenous events for firms, I utilize a difference-in-difference methodology to examine shifts in innovation direction among firms. This involves comparing firms affected by these events with unaffected coun- terparts before and after the occurrence of the deaths. I aim to address key questions: How does the sudden loss of talent affect the firm’s innovation direction? Do senior executives and inventors drive a firm’s innovation direction? And in what ways do they shape the course of innovation within the firm? This paper constructs a comprehensive micro-level sample dataset by utilizing firm data, employee data, and patent data. Additionally, three different measures are introduced to effec- tively quantify changes in the organization’s innovation direction. Previous studies focus on firm’s innovation direction towards some specific technologies ([Acemoglu and Linn, 2004]; [Acemoglu and Finkelstein, 2008]; [Hanlon, 2015]; [Aghion et al., 2016]). In contrast, this pa- per introduces the Wasserstein distance, total variation distance, and cosine distance to depict the systematic change in technology distribution for firms, based on the CPC codes of patents. The key novelty of these three measures lies in their ability to measure the distance of technol- ogy distributions, systematically capturing how far a firm shifts its innovation technology fields from its previous innovation areas within a given time window.

By merging datasets from Compustat–CRSP, BoardEx, USPTO, patent–firm linkage from [Kogan et al., 2017], and inventor information from [Kaltenberg et al., 2023], I introduce exogenous variations within U.S. public firms using the sudden deaths of 257 upper-tail inventors, 298 lower-tail inventors, and 73 directors. Leveraging a difference-in-difference analysis, I uncover that losing a upper-tail inventor instigates a significant pivot in a firm’s innovation direction. In contrast, the passing of a lower-tail inventor does not influence the innovation trajectory. Similarly, the departure of a directors does not markedly sway a firm’s innovative approach. These findings underscore the heightened impact of human capital at the inventor level on the innovation direction, as opposed to management-level losses. Notably, the impact of losing inventors, especially those with marked achievements ([Azoulay et al., 2010]; [Oettl, 2012]; [Jaravel et al., 2018]), are significant, as evidenced by the substantial changes in a firm’s innovation direction following the loss of upper-tail inventors. These findings suggest that varied talents within a firm have specific functions in innovation. Intuitively, inventors possess fundamental knowledge and skills, positioning them at the forefront of exploring and pioneering innovative ideas. They are the frontline in innovation, driving the development of cutting-edge technologies and solutions. On the other hand, executives and managers at the management level play a vital role in setting up the strategic direction of the organization. They are responsible for charting the course of the firm’s innovation initiatives and overall business strategies. In contrast to the inventor level, a sudden exogenous loss of talent at the management level might not necessarily result in a change in the organization’s innovation direction. This observation aligns with prior studies ([Eisfeldt and Kuhnen, 2013]; [Fee et al., 2013]; [Liu et al., 2023]), suggesting that external factors like death or planned retirements of management executives might not sway a firm’s set strategic trajectory. I further investigate the impact of losing upper-tail inventors on the direction of innovation. Specifically, I find that the death of upper-tail inventor does not notably sway the innovative direction of their non-collaborative colleagues within the firm significantly. These colleagues, engaged before and after the loss, generally maintain their original innovation direction. However, the death of upper-tail inventor has a significant influence on colleagues who have collaborated with the deceased inventor within the firm. Further findings suggest that the change in innovation direction among collaborators does not primarily originate from their own inventions or fixed co-inventions. Instead, it is more likely attributed to the absence of collaborative efforts with upper-tail deceased inventors. These results highlight that the innovation direction of a firm is shaped by both the collective expertise of individual talents and the innovative synergy fostered through collaborative teamwork within a firm. My findings also suggest the loss of patenting from the talent loss and a specific talent match in the team generates specific teamwork’s innovation direction, which cannot be fully replaced by other inventors’ own expertise or other collaborations. I also find that treated firms are less likely to significantly increase their hiring of new inventors in the short term following the occurrence of unexpected deaths, compared to control firm. This observation suggests that, at least in the short term, finding and hiring new inventors may prove to be a challenging task. I also gauge the technological distribution distance different groups of inventors. My findings show that upper-tail deceased inventors and existing inventors have different innovation directions. Additionally, I assess the technology distribution distance between upper-tail inventors and new inventors and the distance between existing inventors and new inventors. My findings highlight the hard-to-replace nature of inventors, where the loss of such upper-tail inventors might result in a diminished inclination for the firm to innovate within the technological fields where these upper-inventors previously excelled. Concurrently, the unique innovation direction of new entrants opens potential opportunities for exploration in untapped technological areas.

ABSTRACT. N/A, the submission is an extended abstract.

ABSTRACT. We examine the change in collaboration patterns among economists before and after COVID. We compare economics working papers in common repositories in from 2010 to 2023. Our analysis suggests that the number of authors per paper has increased, the percentage of papers with four or more authors has increased, yet the frequency of inter-institution collaboration has decreased. Some of these patterns persist beyond the COVID years. Our results suggest that while economics research had become more collaborative in the years leading up to COVID, COVID provided an additional significant boost to the depth and breadth of collaboration.

ABSTRACT. Citation practices are often left up to individual researchers, and they typically vary from discipline to discipline. The citation counts of papers often exhibit imbalances associated with author attributes such as country of affiliation and gender. While recent studies quantified citation imbalances in journal papers in a few disciplines [1, 5], the extent of citation imbalances in computer science remains largely unclear. In computer science, conference publications have a higher status than journal publications [2], and the issue of gender imbalance among researchers has been recognized [3]. Here we investigate citation practices in computer science conference papers in terms of the gender of authors and conference tier.

ABSTRACT. The conference peer review process involves three constituencies: authors desire (fast) acceptance in prestigious venues, conferences aim to accept more high-quality and fewer low-quality papers, and reviewers should be spared overwhelming workloads. These objectives often clash, in large part due to the noise inherent in peer review. In response, conferences have experimented with various policies, including adjusting acceptance thresholds and the number of reviews per submission. We investigate how well various policies work, and more importantly, why they do or do not work. We model the conference-author interactions as a Stackelberg game: a prestigious conference commits to an acceptance policy, and authors respond by (re)submitting or not. Using our model, we explain why about 75% of submissions are rejected despite many of them eventually being accepted at prestigious venues with minor improvements, a phenomenon we term the "Resubmission Paradox". We show that conference prestige, reviewer inaccuracy, and author patience encourage authors to (re)submit and thus increase the review workload for a fixed conference quality. For the robustness of these and other results, we consider variants of our model and conduct simulations based on plausible parameters estimated from real data.

ABSTRACT. In the setting of conference peer review, the conference aims to accept high-quality papers and reject low-quality papers based on noisy review scores. A recent work proposes the isotonic mechanism, which can elicit the ranking of paper qualities from an author with multiple submissions to help improve the conference's decisions. However, the isotonic mechanism relies on the assumption that the author's utility is both an increasing and a convex function with respect to the review score, which is often violated in peer review settings (e.g.~when authors aim to maximize the number of accepted papers). In this paper, we propose a sequential review mechanism that can truthfully elicit the ranking information from authors while only assuming the agent's utility is increasing with respect to the true quality of her accepted papers. The key idea is to review the papers of an author in a sequence based on the provided ranking and conditioning the review of the next paper on the review scores of the previous papers. Advantages of the sequential review mechanism include 1) eliciting truthful ranking information in a more realistic setting than prior work; 2) improving the quality of accepted papers, reducing the reviewing workload and increasing the average quality of papers being reviewed; 3) incentivizing authors to write fewer papers of higher quality.

ABSTRACT. In the competitive realm of scientific research, securing funding is crucial for researchers to propel their work forward and enrich the collective knowledge base. However, concerns regarding the alignment of research project objectives with associated paper outcomes raise questions about scientific integrity. This paper proposes a novel methodology leveraging science of science and natural language processing techniques to measure coherence between project objectives and associated papers. By employing a 5-stage process, including corpus retrieval, word embedding computation, and semantic similarity analysis, we aim to identify potential misconduct patterns. Our method, illustrated with examples, facilitates the detection of discrepancies and empowers stakeholders to uphold scientific integrity. Through this research, we aim to foster transparency and ensure research funding is allocated to projects that genuinely contribute to societal progress.

ABSTRACT. [Extended Abstract attached as a separate file] Short Abstract: The recombination of knowledge is fundamental to science-driven innovation, yet studies show that innovators often struggle to recognize and exploit opportunities for recombination present in their field. In this paper, I show how potent forms of knowledge recombination can nonetheless occur through the work of the specialized and often overlooked technical occupations that steward the materials, instruments, and techniques that innovators have in common. As stewards of core research tools, these ideal-typical “technicians” shepherd many, diverse projects. This positions them to facilitate knowledge recombination across scientific domains. But little is known about how technicians foster recombinant innovation, and how their contributions differ from those of the scientists traditionally considered in theories of recombination. To examine this question, I draw on two years of ethnographic fieldwork at four facilities that make the tools of nanoscale R&D available to researchers in domains as diverse as semiconductor engineering and biotechnology. I draw on ethnographic observations of technicians’ work and sociological theories of craft knowledge to show the patterned ways in which technicians at these facilities recognized and exploited opportunities for recombination among the projects they supported. I thereby uncover an important and underrecognized set of pathways for recombinant innovation: pathways not anchored in the concepts and commitments of any one discipline or domain, but rather resting on pragmatic craft knowledge of materials and tools that are core to multiple innovation domains.

ABSTRACT. This paper investigates the long-term effect of the Space Race, an iconic R&D-related industrial policy, on local commercial development. The identification strategy leverages US-Soviet rivalry in space technology to isolate R&D spending. Using a novel dataset from the National Archives, I found that the Space Race has a long-term effect on local business formation and private investment. The influx of R&D funding, though initially intended for space exploration, spurred innovation that attracted venture capital, subsequently benefiting the commercial sector. Additionally, the Space Race contributed to the training of numerous skilled personnel who later founded firms and transferred space technology into the commercial market. This implies human capital accumulation as a key factor in the sustained influence of the Space Race on local commercial development.

ABSTRACT. Scientific collaboration has become increasingly important to accelerate progress in biomedical research [1]. Collaboration can be a key method to address challenges to progress that arise from increased specialization of fields, siloed institutions, fragmented data ecosystems, rapid generation of large and complex datasets, and the growing expenses of conducting research and clinical trials [2]. In the United States, nationwide efforts such as the White House Cancer Moonshot Initiative [3,4] and the National Cancer Plan [5] bring the goal of fostering greater collaboration to the forefront of cancer research. As the United States’ principal agency for cancer research and training and the world’s largest funder of cancer research, the National Cancer Institute (NCI) supports a substantial extramural workforce of investigators through grants[6]. Given the importance of collaboration in cancer research, we seek to understand and characterize collaboration among the NCI-supported cancer research community. In this talk, we will discuss how we conceptualized and developed a collaboration network of NCI’s extramural workforce. We will provide insight into the network analysis methodology we applied and share our initial findings. We will also discuss challenges with measuring collaboration and how we sought to mitigate them in our work. Lastly, we will discuss how one can use this network and accompanying analysis methodology to inform strategies and policies for fostering collaboration. NCI supports its extramural workforce through a variety of grant mechanisms[7]. These mechanisms are divided into groups representing the types of funding NCI provides, such as Research Grants, Career Development Awards, or Research and Training Fellowships. Because we were interested in measuring scientific collaboration, we chose to focus on Principal Investigators (PIs) who were awarded a research-focused grant to build our network. This includes Research Project Grants such as the R01, R37, R56, and U01 and Research Center Grants such as the P50 or U54. Scientific collaboration may take many forms and some aspects of it may be more tangible than others. Critical to the conceptualization of our network was the concept of measurable collaborations. To determine if PIs had collaborated – that is, worked together in an interdependent fashion toward a scientific outcome – we required evidence of the collaboration in the form of co-funding on an NIH base project or co-authorship on a publication. Not all collaborations will result in co-funding or co-authorship. So, while this does not give us a holistic view of collaboration, it does give us a measurable one. We built our network from a PI-perspective, adding PIs as nodes of the network if they had a qualifying research award in Fiscal Year (FY)* 2017 – FY 2022. Edges were added based on the presence of co-funding or co-authorship between two PIs and therefore represent pairwise collaborative events. To better capture trends in collaboration over time, we included co-funding between PIs that occurred between FY 2012 – FY 2023 and co-authorship between PIs that occurred between 2012 – 2023. Our resulting network consists of over 10,000 PIs representing the nodes of our network. We identified co-funding between these PIs on over 6000 unique NIH base projects and co-authorship on over 200,000 publications. These projects and publications culminate in over one million network edges and over 200,000 unique pairwise collaborations between PIs. Once we had developed our network, we applied a variety of network analysis methodologies to characterize the network. We focused on identifying and applying metrics that could tell us about the strength of the network. In other words, could we see evidence of a robust research community among NCI’s extramural workforce? For example, node degree indicates the number of unique collaborators per PI, while the number of connected components provides insight into how interconnected the network is overall. We also conducted various temporal analyses of the data to understand how collaborations shift over time. We explored whether co-authorship is likely to follow co-funding and investigated the role NCI funding plays in sustaining collaborations and promoting new collaborations to form. We identified several challenges with measuring collaboration that must be considered when interpreting the results of an analysis. First, collaborations can take time to produce measurable events – the absence of co-funding or co-authorship does not imply the absence of a collaboration. Second and related to this point, PIs in the network began collaborating at different points in time and therefore reach collaborative milestones (co-authorship or co-funding) at different points in time as a result. Third, when attempting to measure the impact of an intervention on collaboration, it is important to recognize that most interventions are likely introduced in a setting where collaboration is already occurring. Despite these challenges, a thoughtfully constructed dataset representing measurable collaborations can still be a useful tool to understand and characterize collaborations. Our talk will conclude with insights into how we have used our network of extramural PIs to evaluate initiatives such as the Cancer Moonshot and its goal of fostering greater collaboration and how we plan to use our network going forward.

*The United States Fiscal Year begins on October 1st and ends on September 30th. For example, Fiscal Year 2017 began on October 1st, 2016 and ended on September 30th, 2017.

ABSTRACT. Does academic literature help us make better predictions about the effects of interventions, or are they generating more noise than signals? Despite ongoing scientific endeavors, making effective policy recommendations remains notoriously challenging due to the formidable heterogeneity and contingency we must navigate to predict the outcomes of interventions. This raises the empirical question of which set of academic papers, if any, could actually shift human priors in a direction that enhances predictions under interventions. Motivated by the infeasibility of conducting controlled human-subject experiments of such a nature, we propose a novel method for investigating this fundamental question by using large language models (LLMs) and retrieval-augmented generation (RAG) techniques as proxies for exploring the counterfactual of reading different sets of research papers.

Each version of GPT-4, augmented with a specific set of behavioral research papers, is asked to predict intervention outcomes of cooperation under the high-dimensional, integrative experiment design space of a public goods game (PGG). The design space consists of 211 unique experiment designs with 20 dimensions of varying conditions to map the rich heterogeneity of treatment effects. We demonstrate the LLM’s ability to retrieve academic papers and alter its distribution of predictions for experiment outcomes in directions that are expected based on the documents' contents, going beyond simply retrieving factual information. However, we find no evidence so far that such shifts in distribution contribute to higher accuracy.

Our contributions are threefold. First, we introduce a method for evaluating the potential utility of academic papers based on their contribution in predicting the effects of interventions in highly contingent environments, which can be complementary to other features of papers such as citation, venue, and network metrics. In other words, we study the impact of academic papers through their interactions with intelligent minds, thereby uncovering numerous ways to evaluate the influence of social science on AI, and vice versa. Furthermore, we propose a benchmark for predicting the effects of interventions in highly context-dependent behavioral settings by utilizing an integrative experiment design that reliably maps such heterogeneity. Finally, we ignite several meta-scientific questions. These include the challenges of predicting intervention outcomes under complex context dependencies, the limits of predictability that cannot solely be attributed to model capacity (thus requiring additional experimental data with explicitly constructed design spaces), and the empirical question of how, if at all, human-interpretable explanations are associated with predictive accuracy. To add robustness to our discussions, we are also in the process of eliciting predictions from human participants with varying familiarity with the literature, along with responses from different versions of LLMs. The full results and analyses are expected to be available by the time of the conference presentation.

ABSTRACT. This paper utilizes simulation models and big data to illustrate the emergence of collective intelligence in dynamic cooperative networks within organizational settings. Using an Agent-Based Model, the study examines how members' cognitive diversity influences collective intelligence and investigates how network members' repeated cooperation moderates this relationship. The research reveals two key findings. First, frequent and repeated cooperation among network members positively impacts collective intelligence. Second, an increase in the probability of repeated cooperation positively moderates the influence of cognitive diversity on collective intelligence. To address the simplification of real-world specifics in the simulation, hypotheses are formulated based on the key findings and tested using empirical data from one of China's largest high-tech companies. The fixed-effect panel model confirms the earlier findings. Additionally, further simulations show that high cognitive diversity among employees, along with the probability of repeated cooperation, leads to the emergence of collective intelligence, particularly when they exceed a certain threshold.

ABSTRACT. This study investigates the strategic policy responses to technological innovations across different global regions, focusing on developed and developing countries, as well as the United States and the European Union. Through a comparative analysis of AI policy documents, I identify key thematic priorities and how geopolitical and economic classifications shape policy directions. Network analysis of policy topics reveals a complex web of interconnected issues, highlighting the nuanced landscape of global policy strategies. My findings underscore the importance of contextualized policy-making in navigating the challenges and opportunities presented by technological advancements.

ABSTRACT. This work in progress reports current progress towards WhoFundedIt, an interface to sponsorship data. The project will examine the availability and limitations of data, the differences between freely licensed and subscription data about funding; and provide an interface for visualizing who funds research on a specific topic.

ABSTRACT. AI for medical diagnosis is an increasingly popular venture in patenting behavior. If im- plemented, it could disrupt conventional ideas about status, authority, and expertise within the profession (Jutel 2009; Freidson 2007). In fact, medical diagnosis is a significant area of in- terest within AI development (see figures 12). Given recent developments and improvements of AI, particularly through the fine-tuning large language models (Bommineni et al. 2023), we can expect to see the increased application of AI to medical diagnostics. Does this mean that AI is spurring further innovation in this area? Or, is it narrowing the field into the particular use cases for which AI is presently more developed (Messeri and Crockett 2024)?

ABSTRACT. In this work, we focused on depicting the topology of scientific knowledge and the characteristics of scholarly activities derived from it. Based on the Comparative Toxicogenomics Database(Davis, Wiegers et al. 2023), we quantitatively delineated this field’s scientific knowledge landscape. The topology of the landscape is measured by substantial associations among chemical entities and shaped by activities of publication. Two principal patterns can be targeted: local divergence and global convergence. Furthermore, we approached to characterize the collective attention and individual exploration from the perspective of material relationships among the chemicals they pay attention to. In conclusion, we have shown that sublanguages(Harris 1989) are not only clustered to amplify variations in chemicals’ functions, they also have a significant role in navigating researchers’ explorations. What is the relationship between collective scientific research and the system of knowledge it is working on? Gaining insights from Bourdieu’s theory on scientific fields (1975), we traced the associations between the structure of scientific knowledge and the characteristics of collective scientific activity. We formed the field’s knowledge graph by linking chemicals with their similarities in type and a number of chemical bindings (Morris, Clair et al. 2020) and similarities in related diseases. These two kinds of similarities will be later referred to as structure similarity and function similarity, and will jointly represent the structure knowledge space. At the publication level, we measured the difference among publications in terms of the chemicals they addressed. According to our findings, although each paper addresses chemicals with a wide variety of structures (0.63 average cosine similarity) and functions (0.98), the average distance of chemicals selected by different papers is minimal (0.04). And it's much smaller than the average function distance among papers (0.64). Papers cluster on similar structures and approach various functions. Moreover, the distribution of publications’ structure and function distance displays a double-level structure: local divergence and global convergence. As shown in Fig 1.1, among all the paper pairs, there is a logarithmic relationship between their structural distance and function distance. Function distance increases extremely fast with structure distance among papers addressing chemicals of similar structures, which implies a fierce divergence of function. This divergence does not continue as structure distance keeps increasing. Among papers with considerable structure distance (higher than 0.2), function distance changes slightly, which implies the whole field converges to a limited scope of functions. This suggests the fractal structure in scientific development, where changes in the function per unit lead to exponential structural changes. Scientific publications are organized in a way that functional variations are attained easily at the local level and hard at the global level. Scientific subfields center on the “local zone”. Subfields are operationally recognized by sublanguage (Harris 1989), where papers have close semantic embedding. As shown in Fig 1.2 and Fig 1.3, papers with high semantic similarity display a much higher coefficient between function distance and structure distance. Here, we argue that subfields and their corresponding microparadigms(Rzhetsky, Iossifov et al. 2006) are organized to signal the discontinuous function change derived from continuous structure change. Figure 1. Local Divergence and Global Convergence

This suggestion on the subfield’s epistemological function can be confirmed by mediation analysis of scholars’ exploration activities. Compared with their previous publications, scholars are more likely to attain vast functional distances if they manage to go across the subfield boundaries.

ABSTRACT. See uploaded PDF

ABSTRACT. Rare diseases, individually uncommon yet collectively affecting at least 3.5% of the global population, present a unique challenge to healthcare systems. The Orphan Drug Act of 1983, designed to incentivize the development of treatments for these diseases through market exclusivity and tax incentives, has led to the approval of over 500 orphan drugs. However, this model, predicated on high prices for orphan drugs, becomes financially unsustainable when considering the cumulative impact of over 6,000 known rare diseases on a significant portion of the population and the pharmaceutical companies' misuse of the Orphan Drug Act.

This study evaluates whether the academic reward system is equipped to take on the burden of rare disease R&D, as an alternative to market-driven incentives. Utilizing the PubMed Knowledge Graph, icite, and provenance data in knowledge accumulation in human genetics, we compare the output, impact and careers of researchers on rare versus common diseases.

Our findings indicate that rare disease research significantly contributes to human genetics knowledge,consistently outpacing studies on common disease (Figure.1A). Yet, in recent years, this research is penalized in terms of citations (Figure.1B), particularly in non-US regions without supportive policies like the Rare Disease Act of 2002 (Figure.1B, bottom). Furthermore, researchers focused on rare diseases face higher attrition rates. This citation penalty and attrition rate suggest the current public science incentive system is ill-prepared for the challenges of rare disease R&D.

We conclude by discussing the necessity of reevaluating and adjusting the academic incentive structure to better accommodate and encourage rare disease research. This adjustment is crucial for creating a sustainable model for rare disease R&D that does not solely rely on market incentives, thereby ensuring ongoing innovation and support for affected populations.

ABSTRACT. The theories of scientific evolution often contrast "disruptive'' contributions from "developing'' ones. Funk and Owen-Smith's disruption index has been pivotal in quantifying this disruptive impact of scientific works, revealing insights into the evolutionary process of science through its empirical correlations with various factors in science. However, the discrete and localized nature of the disruption index poses some limitations (see Fig. a). The index can be extremely sensitive to small changes and not able to capture any information that is not encoded in the immediate vicinity of the focal paper. The index also has a low resolution because the varying degrees of linkage between earlier and subsequent works cannot be quantitatively incorporated. Last, it fails to capture disruption events distributed across multiple works, especially if there exist citations between them.

To address this gap, we develop a novel graph embedding algorithm that can effectively quantify the disruptiveness of scientific papers. Our algorithm is based on the observation that disruptive papers typically exhibit a greater conceptual "distance'' between their prior and future works, as future works less rely on the previous works after a disruption. Then, if each paper is represented with two embedding vectors---one depicting its prior works through the citing context and the other representing future works through the cited context---the disparity between them can capture the reliance of future works on the prior works of the focal paper (see Fig. c). To achieve this, we employ an approach that confines the window direction in the skip-gram algorithm (see Fig. b). This method guides one vector to locate closely with future works linked to the focal paper while steering the other vector toward prior works. Then we define the Embedding Disruptiveness Measure (EDM) index of paper $i$ as $\Delta_i =1-\frac{\vec{f_i}\cdot \vec{p_i}}{\| \vec{f_i}\| \|\vec{p_i}\|}$, which is the cosine distance between the past vector (vector representing cited context, $\vec{p_i}$), and the future vector (vector representing citing context, $\vec{f_i}$) of paper $i$.

Utilizing a dataset comprising more than 24 million papers from the Web of Science, we demonstrate that EDM (denoted as $\Delta$) better captures disruptive contributions than the disruption index (denoted as $D$). It shows a higher resolution with less degeneracy and captures a broader spectrum of information beyond a single citation (see Fig. d and e). Examination of Nobel Prize-winning papers reveals that EDM identifies those papers as highly disruptive, whereas the disruption index has a bimodal distribution (see Fig. f). This bimodal distribution was mainly due to the simultaneous disruptions, which the disruption index is unable to capture because of its discrete feature (see Fig. h). Additionally, the $D$ distribution in the random networks, where the number of citations and references, and citation age gaps are preserved, is also located in the top 10\%, suggesting citation and reference counts, and their age gap can be confounding factors. Conversely, the $\Delta$ values for Nobel Prize papers in the randomized network exhibited significant deviations from those in the original network, suggesting that their ability to capture disruptive quality goes beyond mere citation and reference counts. We further explore the capacity of EDM to identify simultaneous discovery pairs and find that our method can accurately figure out paper pairs contributing to the same discovery, even those overlooked for Nobel Prizes, such as the case of the Higgs Mechanism discovery (see Fig. g).

In summary, our study introduces EDM as a refined tool for quantifying disruptive innovation, offering a more granular and global perspective on scientific impact. By overcoming the limitations of the disruption index, EDM will shed light on unrecognized disruptions and provide a useful tool to reveal simultaneous discoveries.

ABSTRACT. The global COVID-19 pandemic changed our daily routines, limiting social interactions and enforcing lockdowns. Extensive research highlights the pivotal role of in-person interaction in fostering the exchange and blending of ideas. For instance, there has been a decline in the weak ties among scholars during the lockdown, which are fundamental for the transmission of novel information. Here, we investigate how the COVID-19 pandemic has affected science, specifically the collaborations between scholars and the impact of the resulting papers.

ABSTRACT. Scientific software has proliferated as researchers increasingly use large amounts of digital data [1]. However, the science of science studies of software are hindered by a lack of high-quality datasets due to the immature software citation ecosystem [5] and even state-of-the-art software named entity recognition (NER) datasets are incomplete and contain anomalies [8]. These limitations negatively impact tracking the flow of scientific ideas in software-enabled science and impede innovation studies of software breakthroughs [4]. A particularly intractable problem in this milieu is disambiguation of software mentions. The lack of standard ways to cite software in papers means disambiguation methods, then, have to rely on the context of the mention. Disambiguation of software mentions in papers would accelerate science of science research by creating structured datasets that provide valuable insights into the use, development, and impact of software in scientific research [3].

In this study, we disambiguated entities used the Software Mentions dataset from the Chan Zuckerberg Initiative (CZI) [6] and an LLM approach. The rationale for using an LLM is to utilize the pretraining of the LLMs as well as the ability to compare between two texts [7]. To develop a labelled test sataset, we selected mentions of the PRISM software from the dataset (n=500) and performed manual disambiguation of the software mentioned using human annotation (among three faculty). This approach was taken because (1) there are multiple software entities using this name; and (2) the majority of PRISM mentions in the CZI dataset were incorrectly disambiguated (i.e., they were attributed to an incorrect software entity). We used the sentences the scientific articles in which the software name was mentioned to conduct the manual disambiguation to be used for the labelled dataset for evaluation of the model.

Next, we posed this problem as a binary classification problem. The features for the classification were obtained by generating a 500 X 500 similarity matrix for the software mentions. We queried a pre-trained large language model (PaLM) [2] to develop this similarity matrix. This was achieved by querying the LLM to output whether a pair of mentions referred to the same software. The LLM was prompted to provide an output of 1 if it identified the two mentions to be the same, -1 if it identified the mentions to be different, and 0 if it was not sure. After generating the similarity matrix, we calculated the sum of each of the rows as the feature for our binary classification model. The model achieved an area under the curve (AUC) of 0.7 for distinguishing between PRISM software and other types of mentions entirely based on a single context sentence of the mention. The ROC curve for the binary classification is shown in figure 1.

The advantage of the LLMs method is that it can be automated to apply across a large dataset of software mentions. Granted, the preliminary results fall short of the manual disambiguation and it remains to be seen how well it performs across a varying set of software. Because the manual disambiguation was done using both the context sentence and the full text of the papers where necessary, results could be improved by incorporating more contextual information. More informative features could also be used in the future to bolster the performance of the model. Overall, this preliminary study suggests that LLMs hold promise as a novel approach to the disambiguation of software mentions for science of science studies.

ABSTRACT. Extended Abstract

In 2019, the Association for Computing Machinery introduced their new post-publication name change policy. Soon after, several large academic journal publishers followed suit. Name change policies facilitate invisible updates to transgender authors’ names in published articles. Before these policies were introduced, trans, non-binary, and gender diverse authors who changed their names were limited to either forfeiting their publication history, or risking being inadvertently outed to their professional networks (Hauck, 2021). Either path could result in damaging consequences to career trajectory. Even with the introduction of more inclusive and privacy centered policies, the process of auditing and updating one’s personal publication history can be a daunting and emotionally exhausting task. To date, a review of the scope and coverage of these policies has not been done. This abstract represents the initial research stages of the author’s dissertation work on the efficacy of author name change policies. Positioned within an ethic of care, this thematic content analysis examines the public facing post-publication name change policies of nine academic journal publishers using open coding techniques. The analysis examines the similarities and differences in both content and format of these policies and reveals how academic publishers use policy to communicate values with authors and balance their needs with the publishing industry’s meticulous ethical and procedural guidelines.

Digitally published policies from nine academic publishers were collected and analyzed. For publisher information, see Figure 1. Publishers were selected based on the availability of a name change policy, their recognizability, and their overall publication output. As is typical in thematic analyses, the primary methodology involved open coding and re-coding for themes in the documents. The iterative coding process resulted in a codebook used for reference and consistency during data collection and as a structural tool for a close reading analysis (Flick, 2018).

Key findings during the coding process help contextualize what publishers understand to be important to their authors and how they use policy to communicate these values. This initial analysis reveals the distribution and frequency of themes and codes in the policies (Figure 1). SAGE has the most comprehensive coverage of relevant topics having at least one occurrence of each code, while Springer Nature and Cambridge University Press have strikingly sparse policies in comparison. Additionally, topics related to the theme of Privacy seem to be highlighted most frequently in policies, while themes of Engagement and Industry Context receive less attention. Finally, the natural discoverability of the name change policy, or the number of clicks from the homepage it took to get to the page containing the policy, on publisher websites varied widely. These findings demonstrate the range and accessibility of information which may be presented to authors who wish to utilize these policies.

[Figure 1. Includes of the publishers in this study, country of headquarters, year policy introduced, and business type. Number in heatmap is the count of the code’s occurrence. Numbers under Page Location capture click count from homepage. ]

Further insights from this data will be gained through deep readings of the documents and compared to the five guiding principles for publishers developed by the Name Change Policy Working Group (2021). Additionally, future work for this dissertation intends to address the efficacy of name change policies, the strategies gender diverse authors use to navigate their identity in academia, and how name changes (irrespective of reason) affect the performance of publications through bibliometric analyses and interviews with key stakeholders.

The context of the larger study is deeply rooted in identity politics and the importance of names and fluid identities over time. The study population straddles two groups facing different but nevertheless interconnected marginalization, underrepresentation, and mistreatment in academic literature (Vincent, 2018). The findings of this analysis illustrate how this marginalization may still occur in how and what publishers choose to communicate through policy, especially as new standards and ethical guidelines are introduced. Through studying academic publisher name change policies, we get a glimpse at how the publishing industry interacts with, learns from, and responds to the intersectional communities it serves. More broadly, we begin to learn about one facet of how science itself is evolving and who gets to have a voice in its future.

References

Flick, U. (2018). An introduction to qualitative research. Sage Publications.

Hauck, K. (2021). Inclusive Author Name Change Policies. Science Editor, 105–107. https://doi.org/10.36591/SE-D-4403-105

Name Change Policy Working Group. Five Guiding Principles and Best Practices. (2021). https://ncpwg.org/resources/principles/

Vincent, B. W. (2018). Studying trans: Recommendations for ethical recruitment and collaboration with transgender participants in academic research. PSYCHOLOGY & SEXUALITY, 9(2), 102–116. https://doi.org/10.1080/19419899.2018.1434558

ABSTRACT. Fighting inequality in the research ecosystem has gained prominence as a topic of discussion among scientists, universities, editorial bodies, and research institutions. The calls for change in the way science is practiced, funded, and promoted largely emanate from women and social minorities who encounter various challenges in entering and establishing themselves in the academic ecosystem, remaining as underrepresented and underfunded groups. While the issue is acknowledged, solutions vary and encounter various challenges.

In this context, attention must be directed to the crucial role played by funding agencies (FAs) in combating inequities in the scientific environment. FAs have ample room to improve their policies and practices and, encouraged by social movements and scientists, have begun to implement policies that consider the many diversities in science in grant proposal processes. The Global Research Council itself, an organization through which national scientific FAs from various countries gather to discuss cooperation, review practices, and promote guidelines for scientific funding, held a meeting in 2023 in favor of racial and ethnic diversity in research, with a greater commitment to promoting structural changes and impact in this regard. Initiatives implemented by FAs addressing elements of Equity, Diversity, and Inclusion (EDI) were identified by Moody and Aldercoote (2020), Hunt et al. (2017), Juk et al. (2023), demonstrating a concern on the part of these institutions to promote a diverse and inclusive environment. However, it is still a challenge to: i) categorize these interventions; ii) determine their effectiveness; and iii) to consider the institutional capacity of an organization to determine why certain initiatives are effective and why some of them fail in actually promoting EDI in the research environment.

While Brazil has a history of affirmative action for the inclusion of minority groups in federal universities through quota policies, there has been a growing concern about expanding activities and strategies that consider other axes of inequalities. For instance, in 2022 the São Paulo Research Foundation, one of the most important FAs in Brazil, created a programm for matters related to EDI, which provides a set of actions to expand the diversity of students and scientists funded and aims to improve internal processes and remove obstacles associated to gender and ethnicity. Although these initiatives are crucial and generally suitable for the Brazilian context, recent episodes demonstrate that the narrative often fails to translate into effective change. The evaluation and discrimination reported by mother researchers who attempted to include their maternity leave information in their curriculum, aiming for it to be considered in academic performance evaluations for productivity scholarships by the Brazilian National Council for Scientific and Technological Development (CNPq), serve as an example. The effectiveness of this initiative did not materialize, as several evaluations ignored the maternity leave period and emphasized the lack of productivity during this period as a performance problem. This episode suggests that while EDI initiatives and policies in research are proposed, their actual implementation and institutional barriers are often overlooked.

Given the presented context, this study aims to map the range of EDI activities related to different contexts and identity characteristics of Brazilian FAs. The study also seeks to investigate the institutional capacity of the analyzed agencies in actually implementing the EDI initiatives. Our guiding research questions are: How are EDI-related initiatives being institutionalized by FAs in Brazil, and which initiatives have proven effective or less effective, and why?

We contacted two national FAs, the Brazilian National Council for Scientific and Technological Development (CNPq) and the Coordination of Superior Level Staff Improvement (Capes); and the National Council of State Foundations for Research Support, which currently brings together 27 FAs in Brazil to sent questionnaires to investigate their commitment to EDI strategies and to measure their institutional capacity in implementing these initiatives. We also relied on secondary data analysis on the topics of EDI and analysis of strategic plans from FAs. To build up the questionnaire we conducted a literature review on EDI activities implemented by FAs around the world.

To measure the institutional capacities of organizations to develop and implement effective EDI strategies and interventions, we investigated the following aspects and classified them into basic, intermediary, and advanced levels: i) The organizations’ understanding of the subject ii) To have dedicated departments, teams, or calls to address EDI-related issues iii) To implement guidelines, promote and institutionalize knowledge on the subject within their human resources (e.g.: through training among evaluators to eliminate EDI-related bias in peer review processes, grant and project evaluations) iv) To identify institutional barriers and lock-ins while implementing such initiatives v) To conduct a systematic evaluation of EDI initiatives and interventions vi) To promote accountability, such as making diversity data available or establishing a communication channel with grantees to consider suggestions and improvements

This is a research in progress, so we present here only preliminary results from the investigation, relying on secondary data from the analyzed FAs. Despite the growing attention to the matter of EDI, discussions still seem to be focusing on narratives, principles, and best practices. There seems to be a focus on gender initiatives, which indicates that the intersectional approach that considers other diversity axes, such as race, ethnicity, and disabilities, is not present in actual interventions.

Investigating how FAs, a core element of the research system, are addressing this issue can be an important starting point to assess if real change is actually taking place.

References

Hunt, L.; Nielsen, M. W.; Schiebinger, L. (2022). A framework for sex, gender, and diversity analysis in research: funding agencies have ample room to improve their policies. Science, 377(6614), 1492-1495.

Juk, Y., Salles-Filho, S., E F Pinto, K., Spatti, A., Coggo Cristofoletti, E., Araujo Tetzner, G. & Campgnolli, E. (2023). Diversity, equity and inclusion: how are funding agencies addressing inequalities in research [preprint]. 27th International Conference on Science, Technology and Innovation Indicators (STI 2023). https://doi.org/10.55835/6442ffc28264b1bf681c48ce

Moody, J.; Aldercotte, A.. Equality, diversity and inclusion in research and innovation: international review. Available at: https://www.ukri.org/wp-content/uploads/2020/10/UKRI-020920-EDI-EvidenceReviewInternational.pdf.

ABSTRACT. Given the ongoing social transformations across various scales and domains, such as climate change, poverty, inequalities, and technological advancements, the significance of understanding the relationship between policy and science has never been greater (Aiello et al., 2021). There is a growing concern among funding agencies, universities, and other stakeholders in the research ecosystem regarding how research can better inform, influence, and impact policy, thereby making this topic increasingly relevant. This also implies discussions about methods to assess the impacts of research (Álvarez-Bornstein & Bordons, 2021).

The literature has explored various methods for identifying, assessing, and measuring the flow of scientific knowledge into policies, encompassing standardization and guidelines (Newson et al., 2018). The utilization of bibliometric and altimetric analyses to assess the nexus between research and policies has seen significant development. This approach often employs keyword searches or text mining techniques to detect references and mentions of research, universities, or researchers within policy documents (Vikings & Grant, 2017). Tools and databases have emerged for this purpose, such as Altmetric.com and Overton (Szomszor & Addie, 2022).

In light of this, the research sought to identify the impact of research funded by the São Paulo Research Foundation (FAPESP), one of the main research funding agencies in Brazil, on policies through the exploration of the Overton tool. In addition, our research also aimed to discuss the potentialities and limitations of this approach to accessing the use and impact of research on policies.

To test Overton, we chose to use the DOIs (articles) from projects and scholarships funded by FAPESP. The data were obtained from the Virtual Library of the Agency. The research team obtained 99,637 DOI records from 52,425 funded projects or scholarships for master's, doctoral, and postdoctoral studies, spanning from 1992 to 2023. The articles are distributed as follows: 0.1% in Linguistics and Arts; 0.5% in Applied Social Sciences; 1% Interdisciplinary; 1.7% in Humanities; 10.5% in Engineering; 11.8% in Agricultural Sciences; 19.7% in Health Sciences; 26.1% in Exact and Earth Sciences; and 28.6% in Biological Sciences. After searching for DOIs in Overton, the data were extracted via Application Programming Interface (API) and integrated into an Excel spreadsheet.

Out of the 99,637 DOIs entered as input, it was identified the mention of 2,994 DOIs in documents indexed in the policy database. These DOIs were referenced in 2,025 policy documents, with 1,132 documents from governmental sources, 387 from International Governmental Organizations (IGOs), 305 from think tanks, and 201 from other sources. It's noteworthy that the majority of these policy documents come from IGOs (387), with the most common originating from the USA (370 policy documents) and the European Union (247). Approximately 80 policy documents from Brazilian sources were identified. Despite the bias inherent in the database (language and indexing bias towards countries such as the USA, UK, European Union, etc.), Overton facilitated the identification of novel data, particularly regarding the use of Brazilian research by foreign government agencies, IGOs, and think tanks. There was a prevalence of themes related to the environment and health in the policy documents. This is exemplified by the IGOs that mention FAPESP articles the most, notably the Food and Agriculture Organization of the United Nations (113 policy documents) and the World Health Organization (55 documents). Another interesting indicator is the identification of 168 policy documents whose source is the Guidelines in PubMed Central, a platform indexing documents related to health research. Despite these interesting and novel results, providing a broader view of research utilization in policies, exploring Overton presents some challenges, notably: analytical effort related to understanding the nature of the various types of policy documents (as the database treats them in a comprehensive manner), the nature of the use and impact of research in these documents and the limitations of coverage, with few results coming from Brazil.

References

Aiello, E., Donovan, C., Duque, E., Fabrizio, S., Flecha, R., Holm, P., ... & Reale, E. (2021). Effective strategies that enhance the social impact of social sciences and humanities research. Evidence & Policy, 17(1), 131-146.

Álvarez-Bornstein, B., & Bordons, M. (2021). Is funding related to higher research impact? Exploring its relationship and the mediating role of collaboration in several disciplines. Journal of Informetrics, 15(1), 101102.

Newson, R., King, L., Rychetnik, L., Milat, A., & Bauman, A.E. (2018). Looking both ways: a review of methods for assessing research impacts on policy and the policy utilisation of research. Health Research Policy and Systems, 16(54). https://doi.org/10.1186/s12961-018-0310-4

Szomszor, M., & Adie, E. (2022). Overton: A bibliometric database of policy document citations. Quantitative Science Studies, 3(3), pp. 624–650. doi: 10.1162/qss_a_00204

Vilkins, S., & Grant, W. J. (2017). Types of evidence cited in Australian Government publications. Scientometrics, 113(3), 1681-1695. https://doi.org/10.1007/s11192-017-2544-2

ABSTRACT. While international collaboration is widely recognized as a key driver of modern scientific growth and has been extensively studied, its internal heterogeneity remains largely underexplored. This oversight is mirrored in the common practice of aggregating heterogeneous ties between countries in most network analyses of international collaboration. Our study addresses this gap by exploring the dynamics of international collaboration through the lens of multilayer networks. This approach allows us to organize countries into distinct layers based on different co-authorship patterns, as determined by the affiliated countries of first and last authors. Focusing on the co-authorship network between Taiwan and its eight New Southbound Policy (NSP) priority partners as a proof of concept, our analysis of layer--layer correlations reveals that collaborations characterized by equal co-authorship and co-affiliation align most closely. In contrast, those led by NSP countries show the greatest similarity to minimal collaborations. This explorative study highlights the importance of distinguishing among co-authorship patterns to fully understand the dynamics of international scientific collaboration.

ABSTRACT. In the evolving landscape of scientific publishing, it is important to understand the drivers of high-impact research, to equip scientists with actionable strategies to enhance the reach of their work, and to understand trends in the use modern scientific publishing tools to inform their further development. Here, based on a large dataset of computer science publications, we study trends in the use of early preprint publications and revisions on ArXiv and the use of X (formerly Twitter) for promotion of such papers in the last 10 years. We find that early submission to ArXiv and promotion on X have soared in recent years. Estimating the effect that the use of each of these modern affordances has on the number of citations of scientific publications, we find that in the first 5 years from an initial publication peer-reviewed conference papers submitted early to ArXiv gain on average 21.1 more citations, revised on ArXiv gain 18.4 more citations, and promoted on X gain 44.4 more citations. Our results show that promoting one's work on ArXiv or X has a large impact on the number of citations, as well as the number of influential citations computed by Semantic Scholar, and thereby on the career of researchers. We discuss the far-reaching implications of these findings for future scientific publishing systems and measures of scientific impact.

ABSTRACT. Science is a risky business by nature. Scientists explore and cultivate uncharted space of knowledge through trials and errors, in which their original ideas are often rejected and expected goals are not fulfilled for various reasons (Franzoni and Stephan, 2023; Veugelers et al., 2022). Such risk and uncertainty tend to be especially high when scientists aim at novel discoveries, which have the potential to open up new avenues and make substantial advancement (Kuhn, 1970). Despite its fundamental role, risk in science has been poorly understood (Franzoni and Stephan, 2023). This study aims to develop a bibliometric approach to quantify the degree of scientific risk in a particular mode of scientific discovery process – recombination. Scientific knowledge is usually generated on the basis of extant knowledge, and combining multiple elements of extant knowledge is an indispensable route to generate new knowledge. Novel discoveries often result from integrating pieces of extant knowledge that used to appear unrelated (Uzzi et al., 2013). Because of such a fundamental role of recombination, it is of scholarly and practical interest to quantify the degree of risk associated in recombination. To quantify risk in the recombination process, we employ machine learning and natural language processing techniques. Drawing on past trajectories of science, we develop a machine learning model that predicts whether a certain pair of knowledge elements will be linked or not in the future. The developed model calculates the probability that the pair of knowledge elements is combined, or put differently, the risk in achieving or failing in recombination. This risk indicator is validated by a questionnaire survey that we carried out, in which scientists self-assessed the anticipated risk of their own projects. Predicting recombination risk. To build a prediction model of recombination risk, we considered a published paper as a knowledge element and that two knowledge elements are combined if a pair of papers are cited together by at least one paper within a certain period. For this exercise we prepared the training and test data with Web of Science (WoS) publication data. We randomly selected 60,000 paper pairs (i.e., successful recombination) that were later co-cited as well as 60,000 paper pairs that were never co-cited in 10 years (i.e., failed recombination). We then converted each paper into a 300-dimensional document vector (for which we developed a word embedding model with WoS text data) and used a pair of document vectors as the input features for predicting whether paper pairs are co-cited or not. We developed link prediction models with several classifiers and assessed the performance of the trained models, finding that the support vector machine (SVM) achieves the highest and satisfactory F1 score of 0.860 among other classifiers. Risk indicator of a paper. Based on this algorithm, we next construct an indicator for the risk of a project. We considered a published paper as a unit of project and its cited references as elements of knowledge that were recombined in this project. Specifically, given a focal paper, we formed all possible combinations of its references (for example, 10 references make 45 pairs). For each reference pair, we computed the risk score based on the above-developed classifier. A focal paper is thus assigned as many risk scores as the nuber of reference pairs, which we aggregated to a single risk indicator of the focal paper. Validation of the risk indicator. To validate this indicator, we carried out a questionnaire survey to ask the corresponding authros of 4,228 randomly selected papers how they had initially perceived the risk of the project, and we collected 378 responses (response rate = 8.9%). The survey inquired into overall risk – "when you started the project, how confident were you that the project would reach the expected finding or any publishable finding at all?" as well as technical risk – "when you started the project, did you anticipate any technical / methodological challenge that could fail your project?" (Fig.1). We then analyzed the correlation between these two survey scores with the bibliometric risk indicators calculated for the respondents' papers. The result suggests that the risk indicator is significantly correlated with overall risk (r = 0.176, p < .001) but not with technical risk (r = .025, p > .1). Risk and novelty. Some previous studies treated risk and novelty interchangeably (Veugelers et al., 2022). While novel research may entail great risk (Franzoni and Stephan, 2021), risk and novelty are not conceptually equivalent. Thus, we examined the relationship between the two concpts using bibliometric indicators and survey measures (Table 1). The result does not show significant correlation between the two concepts, suggesting that risk and novelty are discrete concepts.

ABSTRACT. Data is increasingly considered to be a key component in stimulating innovation. Numerous promising possibilities have been opened up by rapidly emerging, data-intensive technologies, including the Internet of Things and artificial intelligence. The analysis and interpretation of big data are critical in the growth of technology firms in terms of AI training and computing capabilities. Small and medium-sized enterprises, with their limited internal resources, particularly face a serious challenge of implementing innovation, which increasingly requires the effective processing and use of various kinds of data. The smart city provides an important opportunity for creating data-driven innovation. Significant amounts of data are increasingly available from various sources through sophisticated devices and equipment scattered in smart cities. Many smart city projects across the globe provide rich opportunities for SMEs to explore data-driven innovation. China, in particular, has recently been active in collecting and utilizing various kinds of data in smart cities. The availability of and access to data would help private enterprises develop innovative products and services in China, where massive amounts of data resources are collected and maintained by the public sector. There were few empirical studies conducted, however, to examine how data are actually managed and provided in smart cities and how they affect companies’ innovative activities. It remains unclear how public agencies and private enterprises collaborate on data and how that influences the innovation performance of SMEs in China.

This study examines how collaboration between the government and companies in smart cities influenced firms’ innovation performance in China. By focusing on the case of SMEs in the Guangdong province, this study investigates how different types of services offered in government contracts on smart cities have an effect on enterprises’ innovation ability. Here, government contracts on projects related to smart cities are divided into three categories: equipment supply, platform building, and data analysis. Equipment supply includes a broad range of equipment, covering various kinds of electronic devices and machinery components. Platform building mainly refers to helping the government build, upgrade, or operate a platform that stores data related to the smart city. Data analysis is to access and analyze the data collected in the smart city for the government. We examine what impacts are made on the innovation outputs of firms by participating in the different types of projects for smart city development. In particular, we examine the following three hypotheses. H1: The companies that participate in projects related to data analysis will have more improvement in innovation outputs compared with those participating in projects related to platform building. H2: The companies that participate in projects related to platform building will have more improvement in innovation outputs compared with those participating in projects related to equipment supply. H3: Overall, the companies that participated in projects related to smart city development have more improvement in innovation outputs compared with those that did not participate in such projects.

In this study, we adopt propensity score matching (PSM) to create comparable control and treatment groups based on company attributes and then apply them to a generalized difference-in-difference (DID) model separately. It would be possible that those companies that obtained government contracts already possess some internal characteristics that produce innovation performance compared to those that did not obtain contracts. To deal with this issue, PSM is adopted to create control and treatment pairs among SMEs. We used three major indexes to divide control and treatment groups: GDP per capita at the provincial level (National Bureau of Statistics), the industry categorization (Enterprise Information Publicity System), and the registered capital (Tianyancha). Also, we used the outcome variables, namely, software copyright and patents, respectively, to divide the control and treatment groups.

DID is adopted to examine the effects of smart city contracts on the innovation outputs of firms. Regarding contract identification, we classified the government procurement contracts into three types (data analysis, platform building, and equipment supply) based on keyword identification by checking the contents and titles of the procurement documents and contracts. We took the year when the company obtained a government contract for the first time as the treatment year. When measuring the innovation ability of SMEs, we used both patents and software copyrights to evaluate innovation levels. In terms of the independent variable, we identified obtaining a government procurement contract as a treatment. In addition to this, we also added several control variables. GDP per capita at the provincial level is used to represent the level of economic development of the place where the company is located. The better the economic development, the more business opportunities and innovation are expected for the firm. Based on the average national GDP level, the GDP per capita of each province is classified into three large categories: high, low, and medium. We also include the age of the companies as it has been shown that younger firms are more likely to have more innovation activities than older firms. The registered capital is used to represent the size of the firm since the number of employees is not publicly available for many SMEs in China. We also consider the number of associated firms because if a firm belongs to a group of firms, then the parent company often provides human and financial resources to the subsidiaries, which would promote innovation development.

Our analysis showed that overall, the companies that participated in projects related to smart city development have more improvement in innovation outputs compared with those that did not participate in such projects. In particular, the companies that participate in projects related to data analysis have more improvement in innovation outputs compared with those participating in other types of smart city projects. That suggests that contracts on data analysis provide SMEs with access to various kinds of data in smart cities possessed by the government, which contributes to promoting innovative activities by the companies. Government purchases of hardware would promote the use of these products in smart cities. That, however, would not involve any substantive exchange or transfer of data possessed by the government and would not significantly contribute to stimulating innovation at firms. Based on the findings of this study, we would suggest that the government that aims to promote innovation development can focus more on giving more opportunities to SMEs, especially in terms of tasks involving data analysis. These kinds of projects will vastly encourage the enterprises’ innovation in the context of smart city development. Moreover, by improving data sharing and openness, companies will be able to benefit from the data resources available and further improve their innovation performance. In this study, we were not able to use data on R&D at the firm level as a control variable, as in China, many SME firms are not listed and are not required to release company details. Case studies will be helpful in obtaining further details on what kinds of data are available and how companies have access to data in smart city projects, affecting innovation activities and outputs.

ABSTRACT. Creativity is considered the engine of scientific, economic, and artistic advancement. Yet, numerous accounts noted that creativity alone often does not lead to the success of an idea. While the novelty of an idea may initially attract attention, its success depends on how people can understand, relate to, and become comfortable using it. This conundrum—familiar ideas can be more successful than novel ideas—can be better understood considering how idea spaces are multi-dimensional. As the number of dimensions that need to be compared and evaluated increases, the possible combinations of different dimensions increase exponentially, leading to greater cognitive difficulties in evaluation in high-dimensional spaces. Using data from three contexts, utility patents in the US, new entrepreneurial products shared at Product Hunt website, and Korean popular music, we measure the dimensionality of an idea space (technology classes in patents, product markets in Product Hunt, or music genres in K-pop) by estimating the number of principal dimensions from the variations in the data with principal components analysis. Across all three datasets, we find that novelty generally increases the level of success, but only when the idea space is low-dimensional. When the idea space is high-dimensional, we see a decrease in the level of success with novelty. This suggests that familiar ideas are more successful than novel ideas when idea spaces are high dimensional. The implications of our findings can be examined against the backdrop of the rising concern that creativity is dwindling. We suggest that this may be a natural consequence of the challenges in the evaluation of creative ideas, as idea spaces become more high dimensional over time.

ABSTRACT. This study examines how gender and the presentation of novelty in research influence scientific impact across disciplines. Controlling for the intrinsic novelty of the research, I find that male first authors in natural sciences are more likely to assert novelty of their works, and this increases subsequent scientific impact of their research. Interestingly, while there is weak evidence supporting a female citation premium in humanities and social sciences (p-value = 0.057), this gap is not mediated by novelty claims. These findings offer a comprehensive view of the mechanisms behind gender disparities in scientific impact and suggest the need for discipline-specific interventions.

ABSTRACT. This study examines two paradigms for assessing the attribute of scientific articles to evade conventional disciplinary categorizations through a Multiverse Analysis. Nonconformity is more consistently associated to scientific impact than interdisciplinarity.

ABSTRACT. Despite its widespread popularity, limited research has delved into Wikipedia's influence on real-world outcomes. In this study, we investigate Wikipedia's impact on scholarly citations across disciplines amidst the onset of the Covid-19 pandemic. Notably, as research on epidemic evolution surged in early 2020, involving mathematical modeling, including SIR models, non-specialist researchers also engaged in this domain. We propose that Wikipedia served as a key learning source, despite academic guidelines discouraging direct citations. Employing a synthetic control method, we analyze the citation trends of articles referenced in Wikipedia compared to similar non-referenced articles. Our findings reveal a significant positive effect on citations for Wikipedia-referenced articles during 2020, challenging perceptions of its reliability. This study underscores Wikipedia's substantial role in facilitating researchers' acquisition of scientific knowledge, highlighting its nuanced impact beyond conventional citation practices.

ABSTRACT. This study explores the changing landscape of academic leadership in education, focusing on the demographic characteristics and academic expertise of university presidents in the US from 1950 to 2023. The research reveals an increasing trend in female leadership since 1975, and a rise in the average age of presidents since 1980. The academic expertise of presidents is diverse, with a significant proportion from science and technology, and a fluctuating percentage from law & politics and social science. The findings contribute to our understanding of academic leadership and pave the way to study its influence on university research and development to inform more effective science policy.

ABSTRACT. See extended abstract in the paper.

ABSTRACT. When we discuss the social sources of creative destruction (1), there is a common perspective that such innovation typically originates from agile startups rather than from entrenched large corporations. This view is based on the flexibility of startups and their tolerance for risk, enabling them to better explore new technologies. On the other hand, large companies, due to their size and complex internal structures, are generally thought to move more slowly on the path of innovation, and their efforts to protect their existing market share often limit their capacity for disruptive innovation. However, is this view true in reality? How can this view reconcile with situations in the current age of AI that large tech-companies like Google and Meta seem to be playing a bigger role in driving innovations. While previous studies identified the negative effect of team sizes on disruption, few studies have investigated how these findings might scale up to the organizational level, especially considering that many large companies have recently restructured, adopting more flattened and decentralized models for their working units. Therefore, we study millions of company patents from MAG to examine the relationship between organization size and disruptive innovation. Our finding is that disruptive technologies are consistently more likely to occur in large companies across historical periods.

ABSTRACT. How do examiners in the patent review system affect disruptive innovation? Although the patent system was established to protect innovation, previous research has shown that patented innovations are less and less disruptive. We systematically analyzed approximately 2 million USPTO patents from 2004 to 2018. Our Findings show that disruptive innovation is detrimental to a patent granted while examiner workload and examiner work experience both have a positive impact on a patent granted. A puzzling finding is that examiners' high-intensity workloads facilitate the approval of disruptive patents, while low-intensity workloads have the opposite effect. Using control effect regression, we confirm the surprising fact that current examiners are killing disruptive patents.

ABSTRACT. The Library of Congress (LoC), as one of the world's largest libraries, aims to provide the US Congress with top-quality research, analysis, and consultation to support its legislative and oversight duties. To support this critical mission, the library has developed its collection with a strong national interest in mind. This extended abstract presents preliminary findings from our project to investigate the geographical coverage of LoC book collection and how its changing portfolio mirrors shifted national and foreign policies of the United States.

This research also aims to demonstrate the significant research values of the library collection metadata for the community of science of science and research evaluation. The library metadata is created by heavy manual curation by professional librarians and using various controlled vocabularies, which underlie its strong research and cultural values. We argue that science of science research should pay more attention to this data source to cover more diverse research activities represented in published books, which is particularly important for the domain of social sciences and humanities and knowledge created beyond academia.

Utilizing the LoC's 2019 metadata dump, we acquired 7.78 million metadata records of books in the LOC collection published during 1970-2019 that were fully curated by librarians in the Machine-Readable Cataloging (MARC) format. We extracted all geographic terms in the book subject assigned according to the controlled vocabulary of Library of Congress Subject Headings (LCSH). We further mapped all extracted geographic names to the Geonames dataset (https://www.geonames.org/), a comprehensive dataset that covers global historic and contemporary geographic names. Among 4.53 million book records with any geographic terms, 98.1% of them can be mapped to a standardized country or region name, representing a place around the world that the books discuss. Below we discuss our two core findings based on the data.

From Anglo-American to global archives. Our results show that the LoC collection has been increasingly more diverse in its geographic focus over time. In the 1970s, a vast majority of books are focused on three primary regions: North America (31%), Western Europe (16%), and Northern Europe (12%). However, four decades later, only 22% of the books are focused on North America, which is closely followed by East Asia (15%). Regions such as Eastern Europe, Western Asia, Southern Europe, South Asia, and Southeast Asia have also seen their representations increase from the 1970s to the 2010s.

LoC's collection on countries echoes their historical trajectories. We observe a strong alignment between changes in the presentation of some countries and historical events in these countries. For example, there was a significant increase in share of books about Japan during the 1980s, coinciding with Japan's economic and technological boom and its trade frictions with the U.S. Similarly, for Mainland China, marked increases are observed during the reform and opening-up of the country as well as its entry into the World Trade Organization in the new century.

Our preliminary results demonstrate that meaningful patterns about knowledge production and dissemination represented by books can be identified from library collection metadata. In our next step, we will publish our fully processed and augmented raw LOC collection dataset to make it available to the broader community. At the same time, along the research presented in this abstract, we will also delve into the state-level geographic focus of the LoC's collection on the United States to understand how domestic political shifts influence the acquisition of the library collection. This study marks a significant step in analyzing the Library of Congress's role in representing global and local knowledge and its potential as a barometer for U.S. policy and political discourse.

ABSTRACT. National cultures shape the structure of interpersonal relationships across social and economic life. These influence not only the creation and reproduction of social opportunities, but also their consequences for the ideas and artifacts they generate for society. Theorists have argued for a distinction between the formation of Chinese “clan”-like hierarchical structures versus U.S. “club”-like structures, but difficulties in conceptualizing these small-group structural phenomena at scale and assessing their distribution across data in comparable domains has thwarted empirical analysis. Utilizing a fractal conception of micro-social structure embedded within agent models and validated on domains covering science, technology, and culture, here we demonstrate consistent differences between Chinese and U.S. networks that structure the production of innovation. Across innovation domains, Chinese networks manifest shorter path lengths, higher global efficiency, greater average closeness centrality, and lower inequality in degree centrality. U.S. networks, by contrast, manifest greater inequality in degree centrality, less information distribution speed, and more unevenly connected networks. These patterns reflect differences in respect for local authority and have marked implications for collective innovative outcomes. Chinese networks condition more turnover, and more personal mobility across topics, while U.S. networks support greater persistence and a carrying capacity for more ideas.

ABSTRACT. Authors of scientific works use several measures to signal preliminary evidence for suggesting the reproducibility of their work. Inherently reproducible works are perceived differently because of the ease of effort present in recreating the experiments helping researchers to expand on others work. As a research question, the influence of reproducibility on the scholarly impact of a paper as observed through citations is under studied and we present this study to initiate the discussion. Although there exists a rich body of work assessing the impact of various factors while determining the scholarly impact, there aren't studies that take supplementary information, presence of code, data or methods in reproducing the original works as factors of analysis. Some researchers argue, artifacts connecting code or data may not be the end-all be all for reproducibility. Our objective is to explore these sentiments and provide evidence about the impact of a diverse group of features while predicting scholarly impact of papers.

The concept of reproducibility echoes diverse sentiments, and from a taxonomy standpoint, the formal definition of reproducibility has evolved as a term and a concept. We align with the National Academy of Sciences in defining \textit{reproducibility} as the process of obtaining consistent computational results using the same input data, computational steps, methods, code, and conditions of analysis. This definition provides an ideal generalizable standard applicable to large sections of scientific research within the sub-domains of computer science. Consensus on this definition can make it easier to recognize procedures and protocols for validating and verifying scientific claims. The relevance and importance of reproducibility are heightened more than ever, given the current outgrowth of Artificial Intelligence (AI) models into diverse public domains. The modus operandi of scientific workflows in AI has shifted from offering posterior fixes to building a-priori reproducible AI pipelines. Regardless of the complexity, we can observe the push for making models, datasets, and algorithms available in containers, open-source repositories, and webpages. The significance of reproducibility is multifaceted. First, it upholds a standard for sustaining quality in the results and analysis of scholarly works, ensuring that scientific findings are robust, reliable, and unbiased. Second, it enables researchers to innovate and expand on proven findings quickly. Third, in the context of AI, reproducibility addresses essential safety and trust considerations by ensuring accountability within the systems implementing algorithms that make decisions affecting human lives.

We collected scholarly articles from ACM digital library badged as Results Reproduced where the experiments included in a research paper are possible to recreate without the use of author-supplied artifacts, and the primary findings of the publication have been validated and independently verified in a later investigation by a person or group other than the authors. Additionally, we gathered scholarly papers from the same venue that do not have any badge and refer to them as Unbadged articles. Utilizing both these groups of features and collecting the citations of each article as observed on Google scholar from the year 2023 helped us model a scholarly impact prediction problem. We have used a combination of Artifact, Structural, Linguistic, and Miscellaneous to predict the scholarly impact. These features are not conventional to citation prediction but are extremely useful for assessing reproducibility. Our motivation to extend the same attributes was to assess the impact of similar features, for reproducibility assessment on reproducible paper distributions on citations.

The plot from Fig 1.a) represents the feature importance of the best performing regression model. Overall, these findings underscore the importance of transparency, reproducibility, clear communication, and practical contributions in enhancing the scholarly impact of academic papers. They reflect broader trends in the scientific community towards open science and reproducibility, which are key to our interest in addressing the reproducibility crisis in AI.

ABSTRACT. In this paper, we present the results of a study on the impact of the 2022 Russian invasion of Ukraine on the functioning of universities and other research institutions in 9 European countries bordering Ukraine, Russia, or Belarus. We carried out a large representative survey of scientists working in research institutions in Estonia, Finland, Hungary, Latvia, Lithuania, Moldova, Poland, Romania, and Slovakia. The survey questions were developed on the basis of a dozen individual interviews with scholars from the abovementioned countries. This helped us to identify the most important consequences of the invasion and their specific determinants and to develop relevant questions for the questionnaire. Respondents were invited from a random sample of corresponding authors of publications indexed on the Web of Science database. The survey was conducted online and was available in English and 8 other languages used in the 9 countries covered by the study (Estonian, Finnish, Hungarian, Lithuanian, Latvian, Polish, Romanian, Slovak). We collected 3,718 fully completed surveys, with adequate representation of all surveyed countries. About half the respondents answered the open question that requested examples of the impact of the invasion on them and their institutions (the average entry length was 55 words). Survey responses were collected from mid-February 2024 to the second half of March 2024—hence, it is not possible to present detailed quantitative results in this abstract.

The initial interviews and preliminary survey results show the greatest impact on educational functions, with a lesser effect on research and the so-called third role of universities. For a significant number of respondents affiliated with institutions located along a border with Belarus, Russia, or Ukraine, the repercussions of war are complex and multifaceted, with potentially enduring consequences. In brief, the results vary depending on the border, with positive (!) effects in regions contiguous to Ukraine, such as enhanced student interest and increased financial and collaborative opportunities for research. On the other hand, regions adjacent to Russia and Belarus have experienced mostly adverse consequences, such as the suspension of projects and collaborations, a decrease in the number of students, the need to redirect the research interests of particular scholars, and the modification of institutional strategies to cope with these disruptions. The impact of war depends on various factors, including the level of collaboration with Eastern European partners, the dependence of scholars and institutions on such collaborations, the specific scientific disciplines involved, the university’s ability to provide educational opportunities for international students or the strategic orientation of the affected institution.

Several scholars have investigated the impact of the Russian invasion of Ukraine on academic and educational activities. However, most authors have limited the scope of their analysis to the overall effects upon the Ukrainian science sector (Ganguli & Waldinger, 2023), the relocation of universities from occupied territories (Lopatina et al., 2023; Zakharova & Prodanova, 2023), or the prospects for Ukrainian scientists remaining in the country as well as those who emigrate (de Rassenfosse et al., 2023; Fiialka, 2022; Suchikova et al., 2023). However, research on the impact of the Russian invasion of Ukraine on science and higher education beyond Ukraine is scarce. Noteworthy studies relate to aiding scientists from Ukraine (Gharaibeh et al., 2023), the impact of economic sanctions imposed during the war on EU–Russia knowledge flows (Makkonen & Mitze, 2023), the reactions of students outside Ukraine to the conflict, repercussions on academic life (Kapsa et al., 2022), or the universities' responses to the Russian full-scale invasion of Ukraine (Kushnir, 2023; Kushnir et al., 2023).

Our study stands out for its complex research question. We investigate not only the impact of war on the scholarly sector in bordering regions but also its effect on existing inequalities among universities and research institutions in areas adjoining Belarus, Russia, or Ukraine. Many institutions in this region were already facing unfavorable conditions compared to those in more affluent countries in Europe (Benneworth, 2018). Thus, the invasion of Ukraine, in addition to creating new inequalities, may have also, and more importantly, exacerbated those that already existed.

The chief contribution of this study is an empirically based analysis of the impact of armed conflict on the functioning of scientific and higher education institutions located close to hostilities but outside the zone of direct war operations. In a broader sense, this is an analysis of the effects of an external shock on the operation of such institutions. The project results may be valuable for shaping public policy, including research and higher education policy, in response to this specific situation or similar circumstances that might occur in other regions of the world.

References Benneworth, P. (2018). Universities and Regional Economic Development: Engaging with the Periphery. Routledge. de Rassenfosse, G., Murovana, T., & Uhlbach, W.-H. (2023). The effects of war on Ukrainian research. Humanities and Social Sciences Communications, 10(1). Fiialka, S. (2022). "Assessment of war effects on the publishing activity and scientific interests of Ukrainian scholars". Knowledge and Performance Management, 6(1), 27–37. Ganguli, I., & Waldinger, F. (2023). War and Science in Ukraine. In Entrepreneurship and Innovation Policy and the Economy, volume 3. University of Chicago Press. Gharaibeh, F. A., Ahmad, I., & Malkawi, R. (2023). Impact of the Russia-Ukraine War on Education and International Students. 25. Kapsa, I., Lubik-Reczek, N., & Ušiak, J. (2022). War in Ukraine from the Perspective of Polish and Slovak University Students. Athenaeum, 75, 270–287. Kushnir, I. (2023). 'It is more than just education. It's also a peace policy': (Re)imagining the mission of the European Higher Education Area in the context of the Russian invasion of Ukraine. European Educational Research Journal. Kushnir, I., Milani, Z., & Mbah, M. F. (2023). UK higher education sector's response to the war in Ukraine. Journal of Applied Research in Higher Education, Lopatina, H., Tsybuliak, N., Popova, A., Bohdanov, I., & Suchikova, Y. (2023). University without Walls: Experience of Berdyansk State Pedagogical University during the war. Problems and Perspectives in Management, 21(2), 4–14. Makkonen, T., & Mitze, T. (2023). Geopolitical conflicts, sanctions and international knowledge flows: EU–Russia collaboration during the Ukraine crisis. The World Economy, 46(10), 2926–2949. Suchikova, Y., Tsybuliak, N., Lopatina, H., Shevchenko, L., & Popov, A. I. (2023). Science in times of crisis: How does the war affect the efficiency of Ukrainian scientists? Problems and Perspectives in Management, 21(1), 408–424. Zakharova, O., & Prodanova, L. (2023). A university displaced twice: Irreversible and erroneous losses of human capital. Problems and Perspectives in Management, 21(2), 123–132.