previous day
next day
all days

View: session overviewtalk overview

10:30-12:00 Session 2A: Research Careers
Mobility of U.S.-Trained Foreign-Born S&E PhDs – a study of evidence building for science policy research

ABSTRACT. Background and rationale The Chips Act of 2022 highlights topics of critical importance, including the “cyber workforce”, microelectronics workforce, bioeconomy, and artificial intelligence. It is widely known, as the National Science Board has stated, that “foreign-born individuals have long been major contributors to science and engineering (S&E) in the United States”. For example, the Center for Security and Emerging Technology states that “approximately 40% of high-skilled semiconductor workers in the United States were born abroad”. Nationally representative surveys can quantify the share and movement of foreign-born individuals as they move through the U.S. S&E education and training system to the S&E workforce in various fields of interest. For example, most U.S. S&E doctorates who are non-U.S. citizens at graduation expect to remain in the United States after graduation according to the Survey of Earned Doctorates conducted by the National Center for Science and Engineering Statistics within the National Science Foundation; however, when surveyed later, a considerable proportion of foreign-born U.S. S&E PhD recipients have left the United States to pursue future research and employment. This has a significant impact on the S&E ecosystem and empirical evidence can inform various levers employed by government, industry, and academia.

This work aims to quantify the movement of foreign-born U.S. S&E PhD recipients and investigate correlates using nationally representative surveys and bibliometric data in effort to produce high quality evidence related to the participation of foreign-born U.S. S&E research PhD recipients in the U.S. S&E ecosystem.

Initial results The proportion of foreign-born S&E workers in the United States has steadily increased in the past decade, representing 24% of the U.S. S&E workforce in 2019. Among the highly trained U.S. S&E research doctorate holders, the share of foreign-born is larger. It has increased from 25% of the 1971-1975 cohort to 48% of those who graduated in the period of 2016-2020. For the field of semiconductors, foreign-born individuals accounted for more than 50% of S&E research doctorate graduates since the late 1980s, reaching 66% in the latest cohort of graduates from 2016-2020. About 27% of foreign-born U.S. S&E PhD recipients who were on a temporary U.S. visa when graduating resided abroad in 2015, representing a significant amount of talent flow from the United States to other countries.

Methods and further analysis Mobility is studied through the use of nationally representative surveys of science, engineering, and health research PhD recipients. The NCSES Survey of Earned Doctorates, a census of all research PhDs from U.S. institutions, provides respondent-reported demographics and educational history, including birthplace, citizenship status, doctorate field of study, the 2005 Carnegie classification of the doctoral institution, post-graduation employment, sex, race, ethnicity, and type of support for graduate study. It is linked to the NCSES Survey of Doctorate Recipients (SDR), which provides respondent-reported employment outcomes, including occupation, sector of employment, job activities, and location information for the survey reference year. The longitudinal data file of the SDR (2015-19) provides longitudinal data on mobility (geographical and employment) over the period of the surveys. The respondents to the cross-sectional SDR 2015 are linked to the bibliometric database Scopus, providing information on publication history. The author affiliation, co-author networks, and citation topic information contained in the metadata of publications provide unique longitudinal data to explore the mobility of PhD scientists and engineers throughout their publication careers. These datasets will be used to produce analyses of characteristics of U.S. Ph.D. recipients with regards to country of origin, mobility, and various correlates. Multi-variate regression will explore the significance of the correlates between those who remain in the United States after graduation and those who do not.

Significance A fundamental belief of government investment in higher education and training is that U.S. universities play a critical role in training the future workforce in key industries. This data set provides descriptive information about training, publication, and employment as well as self-reported demographics. These rich, detailed data sources allow policymakers to develop evidence for interventions that support critical fields, such as those listed in the CHIPS Act of 2022.

Experiences of Foreign Born/Foreign Trained US STEM Faculty: A Qualitative Metasynthesis of the Literature through the Intersectional Lenses of Gender and Race/Ethnicity

ABSTRACT. Limited research has focused on the experiences of foreign born/foreign trained (FB/FT) faculty compared to the U.S. born/U.S. trained (USB/UST faculty), and even fewer have used the intersectional lens of gender to explore issues faced by FB/FT women faculty compared to either FB/FT men faculty or USB/UST women faculty in STEM departments. This systematic review and meta-synthesis of extant qualitative literature on FB/FT faculty using intersectional lenses should provide valuable information to develop policies and practices tailored to better enhance inclusion, career success and retention of both FB/FT and USB/UST faculty of all genders and race/ethnicities. Since the U.S. academic STEM workforce is highly dependent upon FB/FT faculty, understanding their experiences and how their retention might be enhanced is crucial for innovation and science.

Talking and Walking Interdisciplinarity Across Fields

ABSTRACT. Institutional leaders have long championed interdisciplinary research; however, researchers have paid relatively little attention to the people responding to such calls and their subsequent career outcomes. With the benefit of two large datasets spanning from 1986 through 2016, we show that interdisciplinary dissertations have become consistently more common in recent years as institutional leaders have highlighted the value of boundary-spanning research for solving important and emergent problems. With the benefit of survey data from a near-complete population of all dissertators in the US starting in 2001 through 2016, we observe a consistent upward trend in interdisciplinary dissertations. Unfortunately, we show that these interdisciplinary dissertators have experienced a comparably persistent penalty when considering salaries for their first year after earning the PhD. We also show that among interdisciplinary dissertators, individuals in lower-paying fields tend to earn more when choosing distantly related topic-combinations whereas researchers in higher-paying fields tend to be most rewarded for staying within relatively narrow disciplinary silos.

Quantifying gender and retention patterns among U.S. faculty

ABSTRACT. Background In most academic fields, women remain substantially underrepresented among tenured and tenure-track faculty compared to the U.S. population [1]. Despite broad interest in measuring, explaining, and mitigating gendered retention in faculty careers, evidence for its magnitude, ubiquity, and causes remains controversial. Past work has tended to examine separate aspects of retention dynamics, using either large-scale statistical analyses [2-4] that report within- or cross-disciplinary gendered retention rates but cannot explain them, or small-scale surveys or ethnographies [5, 6] that provide detailed explanations for why women leave academia, but do not provide large-scale evidence or systematic conclusions. Moreover, many studies report conflicting conclusions, depending on discipline, study design, and sample size. A deeper understanding of cross-disciplinary gendered faculty retention patterns would help illuminate the social processes that drive systemic underrepresentation, and would inform policies to improve retention and to mitigate the induced asymmetric loss of talent and concomitant scientific discoveries [7].

Data & Methods We describe the results of a unique large-scale study of women’s retention in academia that combines the generality of a multidisciplinary, census-scale statistical analysis of administrative retention data with the explanatory power of deep survey data on gendered reasons for leaving faculty jobs. We quantify gendered retention using detailed employment data from a census of 282,759 tenure-track faculty who were active in their roles between 2011-2020, spanning all domains of academia, including science, technology, engineering, and mathematics (STEM) fields, the social sciences, the humanities, and medicine, and representing all 391 PhD-granting institutions in the U.S. First, we consider retention across academia as a whole, then analyze large groups of disciplines, then each discipline itself. We then identify detailed explanations for these retention patterns using 10,071 responses to a new survey of former and current tenure-track professors across 17 disciplines, sampled from the larger census dataset. Finally, we complement these findings with a qualitative analysis of nearly 7,000 free-text responses by the survey respondents, which highlight the issues faculty identify as driving gendered retention, and specific policy recommendations to improve retention.

Results I. Longitudinal Analysis We find that across 10 years of observation, at every career age, women faculty are more likely to leave their academic jobs than men—7% more likely as assistant professors, 21% more likely as associate professors, and 29% more likely as full professors—and less likely to be promoted than men—7% less likely as assistant professors and 12% less likely as associate professors (z-test, all p < 0.001). The persistence of gendered attrition and promotion patterns across career age contrasts claims that a lack of gender diversity among senior faculty is primarily due to slow demographic change and long career lengths [8]. The gendered attrition pattern we observe predicts that, for a hypothetical gender-parity cohort of new faculty, women’s representation would fall to 40.6% after 35 years, a loss of nearly 1 in every 5 women faculty. Together, these results show a systematic effect in which women faculty leave academia at significantly higher rates than men, in all years of a faculty career. However, this large-scale statistical analysis says little about the reasons that women and men faculty leave academia.

II. Survey Analysis To understand why women faculty leave academia at higher rates than men, we applied an exploratory factor analysis to responses to our broad survey, based on four categories of faculty stress, grounded in the literature: research pressures related to the job itself (i.e., obtaining funding, scholarly productivity), work-life balance stressors related to juggling work and life (i.e., caring responsibilities, hours worked), workplace culture stressors related to how an academic feels around their colleagues (i.e., dysfunctional departmental culture or leadership, harassment), and departmental support stressors related to the (lack of) external support by their departments (i.e., salary, lack of administrative support).

We find that women leave in response to different stressors than men. Women who left academia selected workplace culture reasons for leaving 1.3 times more often than research pressures, 1.6 times more often than work-life balance, and 1.9 times more often than departmental support. Across nearly all career ages, women identify workplace culture as the most prevalent reason for leaving academia, except in the very early career, when work-life balance briefly dominates. In contrast, men who left academia selected reasons from all four categories with roughly equal frequency throughout their careers. Both current and former women faculty are also more likely to feel pushed out of their jobs than pulled into better jobs, compared to men (t-test, p < 0.001). This was especially true for women of color. While there are some meaningful discipline-level differences, these findings hold true across academia, even in disciplines where the majority of faculty are women.

Via free response, we asked respondents to explain what needed to be different in their former or current jobs in order to reduce the stress they felt in their positions (current), or for them to stay in their positions (former). We characterized the responses, then we estimated the themes and their relative frequencies. This qualitative analysis further emphasized the differential impact of workplace culture on women’s retention, and also allowed us to derive additional insights about the causes of stress and leaving that are not fully captured by the survey.

Significance Individual faculty experience academia differently depending on their gender, race, and career stage. Our results broadly quantify the complex and structural nature of retention, and show that gendered retention patterns are ubiquitous, significant, and driven by differences in how different faculty experience academia and their disciplines. Specifically, while past studies focus on work-life balance reasons for women's differential attrition, our work shows that issues with workplace culture are the most common reasons, especially for older women. To achieve equity in academia, departments and institutions will need to address the underlying reasons, across all career stages, that drive these unequal outcomes.

References [1] Wapman K.H. et al. “Quantifying hierarchy and dynamics in U.S. faculty hiring and retention.” Nature, 610, 120-127, 2022. [2] Kaminski, D., & Geisler, C. “Survival analysis of faculty retention in science and engineering by gender.” Science, 335(6070), 864-866, 2012. [3] Gumpertz, M. et al. “Retention and promotion of women and underrepresented minority faculty in science and engineering at four large land grant institutions.” PloS One, 12(11), e0187285, 2017. [4] Ceci, S.J. et al. “Women in academic science: A changing landscape.” Psychological Science in the Public Interest, 15(3), 75-141, 2014. [5] Martinez, L.R. et al. “Fleeing the ivory tower: gender differences in the turnover experiences of women faculty.” Journal of Women's Health, 26(5), 580-586, 2017. [6] Gardner, S.K. “‘I couldn’t wait to leave the toxic environment’: A mixed methods study of women faculty satisfaction and departure from one research institution.” NASPA Journal About Women in Higher Education, 5(1), 71-95, 2012. [7] Kozlowski, D. et al. “Intersectional inequalities in science.” PNAS, 119(2), e2113067119, 2022. [8] Hargens, L.L. & Long, J.S. “Demographic inertia and women’s representation among faculty in higher education.” The Journal of Higher Education, 73(4), 494-517, 2002.

10:30-12:00 Session 2B: Measuring and Governing Global Science
The latent structure of global scientific development

ABSTRACT. Science is essential to innovation and economic prosperity. Although studies have shown that national scientific development is affected by geographic, historic, and economic factors, it remains unclear whether there are universal structures and trajectories of national scientific development that can inform forecasting and policymaking. Here, by examining countries’ scientific ‘exports’—publications that are indexed in international databases—we reveal a three-cluster structure in the relatedness network of disciplines that underpin national scientific development and the organization of global science. Tracing the evolution of national research portfolios reveals that while nations are proceeding to more diverse research profiles individually, scientific production is increasingly specialized in global science over the past decades. By uncovering the underlying structure of scientific development and connecting it with economic development, our results may offer a new perspective on the evolution of global science.

Exploring Alternatives for Measuring National Scientific Capacity

ABSTRACT. Rationale No standard measures exist comparing science capacity across nations. Many proxies attempt to capture the underlying construct or dynamic of scientific capacity. The Organization for Cooperation and Development (OECD) compares subsets of wealthy nations . The National Science Board’s Science and Engineering Indicators Report compares a limited number of nations based on investment in R&D, performance indicators (output), an intensity indicator for eight nations (Table RD-7), and a comparison of country R&D performance for 17 countries (Table RD-8). UNESCO reports on national statistics and indicators in its Science Report . Data without a workable framework or theoretical construct inhibits comparisons for policy purposes, a gap which we seek to address with this study. Across the literature, we find authors choosing many different, distinct variables or clusters of variables to compare nations. These analyses are customized for the purpose of a single article, and they do not provide consistent measures. A descriptive theory of scientific capacity and a standard set of measures would help to compare nations. However, no research program has adequately examined nations conducting science to produce a useful set of indicators or a national science capacity index. This paper reviews the literature to identify and assess the variables used to measure national scientific capacity. A complex-systems heuristic is presented to hierarchically categorize indicators and provide a multi-level theoretical framework for the categorization of numerous indicators. Next, we present data regarding indicators of national scientific capacity and conduct an exploratory analysis to assess measurement coverage, clustering, and convergent /discriminant validity in relation to other well-established variables. Lastly, we seek to ultimately move towards a composite index of indicators which may be of use in both theory testing and policy decision making.

Literature Support This paper draws inspiration from earlier efforts to compare nations on science capacity (alternately termed ‘impact’ (King, 2004), ‘wealth’ (May, 1997), or ‘productivity’’ (Cole & Phelan, 1999). Each of these measures has merit in themselves, but they do not disambiguate the individual national measures to an internationally comparable set--which we acknowledge is a challenging task. We approach this task by, first, defining the construct of interest as one of ‘capacity’ rather than productivity or impact—since those latter features will include contributions from international collaborations, which confound measures of national strength. We view capacity as the explicit and latent resources, capabilities, and competencies available to a nation to absorb and exploit new knowledge from wherever in the world it emanates (Porter, 2011). Capacity will not always be evident in outputs such as number of papers published or number of citations, or as inputs such as government or private sector funding. Thus, additional measures related to scientific and technical human capital (Bozeman, Dietz, & Gaughan, 2001)., regarding number and quality of researchers in a nation are important. So too needed are measures that capture the position of the nation within the international network of scientists (Whetsell, 2022), as nations may leverage relationships and network position within the system to deploy domestic capacity. Measures of national infrastructure, rule of law, political climate/academic freedom, and regulatory quality also need exploration (Coppedge et al. 2011; Grimm and Saliba, 2017). By proposing a broader approach than taken by King or May, we combine measures to propose a composite index. The index will distinguish among inputs, capacity, and outputs, with a complex systems view of hierarchical but loosely coupled interdependence between inputs, capacity, and outputs (Simon, 1996). Further, complex systems theory suggests that measures across nations often manifest empirically in highly skewed distributions that suggest power law dynamics where we can test efficiency of national systems (Katz, 2006; Wagner, 2009).

Methods Science capacity can be defined as the resources, capabilities, and competencies needed to create and support knowledge creation, conduct research and development, and disseminate science. Nations invest in science capacity in accordance with policy to the extent that resources are available. In the 21st century, in imitation of the wealthy nations, many more nations have adopted a science policy that targets science capacity building, and with some success. Data are now available that were lacking just a few years ago. We suggest that a greater variety of indicators can be used to assess national capacity to increase the “likelihood of converging to an accurate understanding of knowledge produced by research” (Kostoff, 1995, p.8) where a collection of indicators is more likely to give a better evaluation of research strength than one indicator alone (Martin, 1996). We begin by exploring the usefulness and complementarity of more than 12 indicators, grouped into three theoretical domains of their tiered contribution to science capacity: 1) facilities and infrastructure are needed to support and build capacity, represented by national wealth, and an educated populous who contribute to and create demand for knowledge economy; and social order and political environment; 2) means by which science is conducted, including numbers of trained scientists and engineers; R&D spending; institutions and equipment, and openness to international engagement; and 3) output of scientific and technological knowledge and its diffusion to the larger world through coauthorships; scholarly articles; citations and their impact; teaching; and patents as a proxy for embedded knowledge and demand. Frame (2005), in attempting a similar project, noted that “quantitative indicators of scientific and technological activity are often of questionable validity and reliability. This is particularly true in lesser developed countries, where the lack of data gathering skills may frequently result in the development of misleading indicators….” Frame did, however, find correlations among a similar set of indicators to those used in the current study (science and technology manpower data, expenditure figures, student enrollment in higher education, and number of degrees conferred), suggesting that the approach can have some validity. Data have improved. The Frame approach has new validity.

Efficiency of Investment A composite index of scientific facilities, means, and output could provide a useful measure of national science capacity, which we explore. We note that, at the time when variables were collected, capacity may be underused, used efficiently, or a nation may be outproducing what would be expected based on inputs. Thus, capacity should not be viewed as a fixed state but rather a continuous dynamic state that has here been measured at just one point in time, so dynamics must be considered in future research. As with any complex system, change and interrelationships are non-linear and there are multiple sites of agency with intricate levels of causality that cannot be fully captured in an index. To go beyond our variables and give a sense of science efficiency among nations, we will consider a Triple Helix (Leydesdorff & Etzkowitz, 1996) context—and explore how to contributions of these characteristics can aid in efficient use of resources.

Conclusion In this paper we propose an approach to measuring national science capacity based on a set of indicators, tied to previous research, presented within a complex-systems theoretical framework. We attempt to apply these concepts to compare the scientific facilities, means, and output of nations. We hope that the approach suggests pathways forward for the community to create useful tools for assessing current science capacity and for supporting international collaboration, and we look forward to comments.


Bozeman, B., Dietz, J. S., & Gaughan, M. (2001). Scientific and technical human capital: an alternative model for research evaluation. International Journal of technology management, 22(7-8), 716-740. Cole, S., & Phelan, T. J. (1999). The scientific productivity of nations. Minerva 37, 1-23. Coppedge, M., Gerring, J., Altman, D., Bernhard, M., Fish, S., Hicken, A., ... & Teorell, J. (2011). Conceptualizing and measuring democracy: A new approach. Perspectives on Politics, 9(2), 247-267. Frame, J.D. (2005). Measuring scientific activity in lesser developed countries. Scientometrics, 2, 133-145. Katz, J. S. (2006). Indicators for complex innovation systems. Research policy, 35(7), 893-909. King, D. A. (2004). The scientific impact of nations. Nature, 430(6997), 311-316. Kostoff, R. (1995). The Handbook of Research Impact Assessment. US Government Printing Office, DTIC Report Number AOA296021. Leydesdorff, L., & Etzkowitz, H. (1996). Emergence of a Triple Helix of university—industry—government relations. Science and public policy, 23(5), 279-286. Martin, B.R. (1996). The use of multiple indicators in the assessment of basic research, Scientometrics 36 3, 343-362. May, R. M. (1997). The scientific wealth of nations. Science, 275(5301), 793-796. Porter, M. E. (2011). Competitive advantage of nations: creating and sustaining superior performance. Simon and Schuster. National Science Board (2019). Publications Output: U.S. Trends and International Comparisons. Science and Engineering Indicators. OECD (2021). Main Science and Technology Indicators Database. OECD (2015). Frascati Manual 2015: Guidelines for Collecting and Reporting Data on Research and Experimental Development, The Measurement of Scientific, Technological and Innovation Activities, OECD Publishing, Paris. doi: Simon, H.A. 1996. The Sciences of the Artificial, 3rd ed. MIT Press UNESCO (2020). The UNESCO Science Report: Towards 2030. Online access ISBN 978-92-3-100129-1. Wagner, C. S. (2009). The New Invisible College: Science for Development. Brookings Institution Press. Whetsell, T. A. (2022). Democratic Governance and International Research Collaboration: A Longitudinal Analysis of the Global Science Network. arXiv preprint arXiv:2203.01827.

Changing landscape of global science funding

ABSTRACT. Financial investments are instrumental to national scientific competitiveness. Cutting-edge scientific research is resource-intensive—e.g., facilities, equipment, materials, and labor—and investments in science are key drivers of scientific activity. However, scientific investments are not distributed equally across the world. Three regions—North America, East and Southeast Asia, and the EU—account for more than 85% of global Research and Development (R&D) expenditures. Meanwhile, national scientific investment has been primarily investigated through R&D expenditures which include a huge range of institutions and activities that go beyond basic scientific research. In this study, we investigate how countries fund national and international research by tracking research grants disclosed in funding acknowledgement sections of scholarly publications. Our results suggest that, although China is rising as the dominant scientific funder to global science, it has limited influence on other countries from a research investment point of view. The U.S. is the leader in international funded publications that, on average, countries experience the largest lost if the U.S. stops co-funding activity. Furthermore, science in African countries are heavily underfunded and scientific system of developing countries are more fragile in the global funding perturbation.

Democratic Governance and International Research Collaboration: A Longitudinal Analysis of the Global Science Network

ABSTRACT. The democracy-science relationship has traditionally been examined through philosophical conjecture and single country case studies. There remains limited global scale empirical research on the topic. This study explores country level factors related to the dynamics of the global scientific research collaboration network, focusing on structural associations between democratic governance and the formation, persistence, and strength of international research collaboration ties. This study combines longitudinal data between 2008 and 2017 from the Varieties of Democracy Institute, World Bank Indicators, Scopus, and Web of Science bibliometric data. Methods of analysis include temporal and weighted exponential random graph models (ERGM). The results suggest positive significant effects of both democratic governance on international research collaboration and homophily between countries with similar levels of democratic governance. The results also show the importance of exogenous economic, population, and geo-political factors, as well as endogenous network factors including preferential attachment and transitivity.

10:30-12:00 Session 2C: Innovation Capabilities for Transformative Change
Panel: Innovation capabilities for transformative change: opening a critical knowledge dialogue - Title: Institutional capacities and capabilities for Transformative Innovation. Learnings from diverse experimental policy engagements

ABSTRACT. The Transformative Innovation Policy (TIP) is an emerging framework that advocates for an innovation policy that focuses on environmental and social challenges beyond economic growth, that opens up transformation pathways to a diverse set of solutions to complex challenges, is highly inclusive and embraces conflict by acknowledging power struggles, has learning as a fundamental trigger to change practices, norms and institutions, and fosters systemic change (Schot and Steinmueller, 2018). TIP has been gaining more track in the innovation policy narrative in the last decade, gaining theoretical and practical depth through experimental policy engagements (EPE) between researchers, policymakers and practitioners that are part of the Transformative Innovation Policy Consortium (TIPC – more information in for the last five years. Being part of different engagements in Sweden, South Africa, Colombia, and Catalonia, we reflect on the required capabilities for advancing transformative innovation from different levels (individual, collective) and groups (researchers, policymakers, practitioners, and social groups). The discussion will have two blocks. The first looks into innovation capacities and capabilities based on a paper in progress related to the (EPE) developed with Vinnova, the Swedish Innovation Agency, related to adding a formative evaluation layer to their mission-oriented innovation policies methodology. The second block takes the capacities and capabilities framework and discusses both sets from the process developed by the core groups part of EPEs in Colombia, South Africa and Catalonia. The mission-oriented innovation policy (MOIP) approach is based on the establishment of clearly defined goals – i.e. missions – within a specific timeframe to direct cross-sector innovation (Janssen et al. 2021; Mazzucato 2018b, 2018a). Transformative innovation policy (TIP) (Diercks et al. 2019; Schot and Steinmueller 2018; Weber and Rohracher 2012) moves away from clearly defined cross-sectorial missions to focus on emergent, open-ended transformations in ‘socio-technical systems’ (Geels 2004), as conceptualized in the research field of sustainability transitions (Köhler et al. 2019). In contrast to the emphasis on measurable, pre-defined outcomes and state-led governance processes of the MOIP framework, the TIP framework emphasizes the need for policy to engage with ongoing transformations, through the participation of new (‘niche’) agents and civil society at large, to open multiple directionalities for innovation. The Vinnova paper argues that the two approaches can be treated as complementary by advancing a capacity/capability-based Transformative Mission-Oriented Innovation Policy (TMOIP) framework. The paper aims to identify the types of institutional capacities and capabilities needed to design and implement transformative missions. In line with Diercks et al.’s (2019) typology of challenge-led policies, we argue that TMOIP can address a particular type of societal ‘missions’: those with multiple (open-ended) goals that cannot be unequivocally anticipated or defined. We first provide a synthesis of the MOIP and TIP approaches to outline the differential characteristics of TMOIPs in terms of the capacities and capabilities needed to define and implement them. We then review how the mission-oriented innovation policy literature discusses the importance of institutional capacities and capabilities using concepts drawn from business administration, public administration and policy, and human development to identify different types of capacities and capabilities needed for transformative missions, which are formalised into our analytical framework. The resulting TMOIP framework is then applied to an explorative case study of the Swedish innovation agency Vinnova, a frontrunner organization currently experimenting with methodologies to design mission-oriented programmes following transformative innovation policy principles. Complementing the theoretical and practical findings from the Vinnova case, in the second block we discuss the type of capabilities required by the core groups in the EPE, which are usually composed of researchers and policymakers. We extrapolate the capabilities to group dynamics and the specific skills facilitators require to build and nurture transdisciplinary spaces with a base of epistemic capabilities coming from the human development capabilities approach (Velasco and Boni, 2020; Fricker, 2015; Medina, 2017). We discuss how a human development view of transitions, in general, and of innovation capabilities, in particular, can strengthen the concept and practice of just transitions (Velasco et al., 2021). We base our arguments on EPEs developed with the Ministry of Science, Technology and Innovation of Colombia, particularly with the team leading a social appropriation of science national programme called “A Ciencia Cierta”; the “Living Catchment Project” EPE developed as part of the South African National Water Roadmap strategy, funded and sponsored by the Department of Science and Innovation and the Water Research Commission of South Africa and implemented by the South African National Biodiversity Institute; and the process developed with the Shared Agendas in Catalonia, as an expression of TIP in practice. We expect to provoke a discussion with other panel members and session assistants to further develop the meaning of value from Sen’s definition of capabilities as the real possibilities and opportunities of leading a life that a person has reasons to value (Sen, 1999) when discussed from the notion of innovation capabilities. References Diercks, Gijs, Larsen, Henrik, and Steward, Fred (2019), 'Transformative innovation policy: Addressing variety in an emerging policy paradigm', Research Policy, 48 (4), 880-94. Fricker, M. (2015) Epistemic contribution as a central human capability. In G. Hull (ed.), The Equal Society (pp. 73–90). Cape Town: UCT Press. Geels, Frank W. (2004), 'From sectoral systems of innovation to socio-technical systems - Insights about dynamics and change from sociology and institutional theory', Research Policy, 33 (6-7), 897-920. Janssen, Matthijs J, et al. (2021), 'The promises and premises of mission-oriented innovation policy—A reflection and ways forward', Science and Public Policy, 48 (3), 438-44. Köhler, Jonathan, et al. (2019), 'An agenda for sustainability transitions research: State of the art and future directions', Environmental Innovation and Societal Transitions, 31, 1-32. Mazzucato, Mariana (2018a), 'Mission-Oriented Research & Innovation in the European Union: A Problem-Solving Approach to Fuel Innovation-Led Growth', Mission-Oriented Research & Innovation in the European Union. --- (2018b), 'Mission-oriented innovation policies: Challenges and opportunities', Industrial and Corporate Change, 27 (5), 803-15. Medina J (2017) Varieties of Hermeneutical Injust. In Kidd I, Medina J and Pohlhaus G Jr. (eds.) The Routledge Handbook of Epistemic Injustice. Routledge: 41–52. Schot, J., & Steinmueller, W. E. (2018). Three frames for innovation policy: R&D, systems of innovation and transformative change. Research Policy, 47(9), 1554–1567. Sen, A. (1999) Development as freedom. Oxford: Oxford University Press. Velasco, D., & Boni, A. (2020). Expanding epistemic capability in participatory decision-making processes: The Universidad de Ibagué capabilities list. In Participatory Research, Capabilities and Epistemic Justice (pp. 27–57). Palgrave Macmillan. Velasco, D., Boni, A., Delgado, C., & Rojas-Forero, G. D. (2021). Exploring the Role of a Colombian University to Promote Just Transitions. An Analysis from the Human Development and the Regional Transition Pathways to Sustainability. Sustainability, 13(11), 6014. Weber, K. Matthias and Rohracher, Harald (2012), 'Legitimizing research, technology and innovation policies for transformative change: Combining insights from innovation systems and multi-level perspective in a comprehensive ‘failures’ framework', Research Policy, 41 (6), 1037-47.

Innovation capabilities for transformative change: opening a critical knowledge dialogue

ABSTRACT. After five years of operating the Transformative Innovation Policy Consortium (TIPC), a global research network involving researchers, policymakers and funders from 12 countries, a second phase of TIPC is about to be launched in 2023. The second phase will draw on the research, experimentation, policy engagements and lessons learned over the past few years and the research agenda developed. A key lesson from the first phase of TIPC is the need to better understand what capabilities are needed to design, implement and evaluate Transformative Innovation Policies (TIP). In response to this lesson, several of the TIP research activities involved the development of new tools, such as transformative theories of change guided by a framemwork of transformative outcomes and set within a formative evaluation framework for TIP projects and programmes. The insights reveal that researchers and policymakers working on TIP need certain sets of capabilities (we refer to these as “TIP capabilities”, for clarity in this context). Although some capabilities may be shared, others may be specific to researchers (who are active facilitators of TIP policy experiments), or to policymakers or funders (who connect experiments to other political initiatives). Therefore, it is not entirely clear, yet, what the full range of these capabilities entails, hence the importance of this panel session to help unpack the issue with inputs from a wider audience. To this end, the session will feature researchers who have been involved in TIP experiments that cover Africa, Europe and Latin America, to reflect on their experience of working with policymakers and other researchers in TIP projects. Reviews of the relevant literature reveal a gap in the conceptualization and empirical research on the issue of the types of innovation capabilities required for transformative change and the specific nature of what could be understood as transformative innovation capabilities. Each of the speakers has taken up this issue specifically in their work, and through dialogue around the following questions, we will bring these ideas together in a synergistic way.

Making Urban Sustainability Transitions happen: Transformative Innovation Policy in Six European Cities

ABSTRACT. Background and research questions Against the background of a steadily rising rate of rapid urbanization and the mounting pressures imposed by climate change, pollution, austerity, aging populations, limited resources and inequality, cities are facing unprecedented challenges along with potential disruptive events and external shocks (Fastenrath et al., 2019). Cities are not only particularly vulnerable to these challenges, but also well-positioned to address and navigate these changes through the tools of urban innovation policies in conjunction with other local policies (Bulkeley et al., 2019). In fact, with the shift of political agendas to matters of sustainable developments, the center of gravitation in innovation policy is shifting from national to urban and international levels. Cities are knowledge centers in their national economies; multiple actors with resources and capabilities can be mobilized in local milieus to experiment, co-shape and co-design solutions in order to build resilient and sustainable systems (Bulkeley et al., 2016; Fuenfschilling et al., 2019; Geels, 2005;).

Our research thus starts from the following assumptions: - Pro-active innovation policies are emerging at the local level, accompanied by the novel tools and activities available at this level too. Formulating, experimenting with and implementing novel policies and solutions in cities differs significantly from other levels of government, but is not yet well understood. - Cities are democratic entities where futures are imagined and alternative directions of change are contested through much more direct citizen feedback than at higher policy levels. Cities formulate overarching visions and policies, yet initiatives tend to be placed within sectoral departments, usually characterised by strong fragmentation, which complicates the horizontal cooperation needed for experiments as well as collaboration with internal and external actors and stakeholders. - The most daunting step in transition policies is to move from individual experiments to broader and generalised transformations. In order to enable sustainable transitions, local contexts will need to address all stages of the policy cycle from envisioning and design to implementation and learning, in order to generalize innovative solutions and thereby contribute to a wider transformation process. Moreover, local policies are also required to practically address emerging constraints and mobilise the opportunities offered by national and transnational/European policies in order to accelerate transitions. Generalisation of innovations is driven by yet another important dimension in the context of urban innovation policies, namely the mutual learning processes across cities. Current European policy initiatives try to foster cross-city replication and adaptation processes.

The objectives of this paper are i) to propose a conceptual framework for Transformative Innovation Policy in Cities that takes into account the aforementioned changing features of urban policies for sustainable transitions, ii) identify common patterns, differences and their determinants across a range of cities in Europe, iii) point to some lessons learned from these experiences for urban innovation policy in general.

Approach and methods Underpinned by literature research, a series of internal workshops and a preliminary study on the case of Amsterdam (Bundgaard et al. 2022), we have developed as part of an abductive research process, a five-layer conceptual framework. The Framework seeks to better understand the processes of shaping and implementing both incremental and transformative innovation policies in the urban context. This framework covers the guiding visions behind transformations, the mental models underpinning innovation policy, the organisational and institutional structures and processes for – mainly incremental – innovation, the spaces and conditions for experimenting with radical novel solutions, and the processes of transformative generalisation to turn these experiments into widespread practices. Based on this framework, six case studies have been conducted in three European countries(Sweden: Lund and Malmö, in France: Pau and La Rochelle, in Austria: Linz and Klagenfurt). By assessing and comparing the preliminary results of these cases we aim to deepen our understanding of advanced practice in their pursuit. The empirical investigation is based on secondary sources on strategies and policies of the six selected cities and a programme of interviews with policymakers, partner organizations, CSO’s and other stakeholders, to complement and refine our theoretically founded framework and allow us to further explore the instruments and activities relied upon for the purpose of transformative change further. The framework is planned to be operationalized and made accessible to policymakers through a series of workshops over the course of 2023.

Expected results Through this work, we expect to identify what policies, instruments, activities and actors are central to the emergence of sustainability-oriented innovation and transformation policies in cities. Moreover, we want to develop more differentiated hypotheses on the influence of the respective national contexts on the scope and practices of urban innovation policy, by taking into account both the differences between the institutional conditions in the three countries, and how they determine what kinds of policies are actually promising and feasible. Building on this knowledge we expect to provide insights into “good practice” in terms of what can proactively be done by municipalities and other stakeholders to facilitate and steer cities towards sustainable futures. Through the advancement of our conceptual framework, we hope to devise novel inroads for how Transformative Innovation Policies in Cities could be better understood, designed, assessed and ultimately improved.

Significance of research approach and findings There is a growing demand for novel sources of inspiration and knowledge to help city authorities develop and implement truly transformation-oriented policies and thus to better cope with the challenges of sustainability in an increasingly turbulent global and societal environment. Solutions for complex challenges call for inter- and transdisciplinary approaches. Current policy and governance challenges are characterized by high uncertainties in transformation processes, multi-actor settings and often face a shortage of organizational capacities and domain/system knowledge to design, implement and manage complex change processes. Consequently, the framework developed is fed by contributions of scientists from different backgrounds and disciplines (such as sociology, urban planning, economics, geography, political science, innovation studies etc.), who apply the respective context-specific lenses on the case studies in the selected cities. Moreover, it is inspired by dialogues with practitioners from local authorities to European policy and various stakeholder organisations. With these ingredients, we hope to mobilise our insights for a more transformative governance of cities.

References Bulkeley, H., Coenen, L., Frantzeskaki, N., Hartmann, C., Kronsell, A., Mai, L., ... & Palgan, Y. V. (2016):Urban living labs: governing urban sustainability transitions, Current Opinion in Environmental Sustainability, 22, 13-17 Bulkeley, H., Marvin, S., Palgan, Y. V., McCormick, K., Breitfuss-Loidl, M., Mai, L., … & Frantzeskaki, N. (2019): Urban living laboratories: Conducting the experimental city?, European urban and regional studies, 26(4), 317-335 Bundgaard, L., et al. 2022): Transformative Innovation Policy in Cities: Illustrating a Five-layer Framework using the case of Amsterdam, Paper for the EU-SPRI 2022 Conference, Utrecht, 1-3 June 2022 (submitted for publication in Technological Forecasting and Social Change) Fastenrath, S., Coenen, L., & Davidson, K. (2019): Urban resilience in action: The Resilient Melbourne Strategy as transformative urban innovation policy?, Sustainability, 11(3), 693 Fuenfschilling, L., Frantzeskaki, N., & Coenen, L. (2019): Urban experimentation & sustainability transitions, European Planning Studies, 27(2), 219-228 Geels, F. W. (2005): Technological transitions and system innovations: a co-evolutionary and socio-technical analysis. Edward Elgar Publishing.

Transformative innovation capabilities in practice: insights from the Living Catchments policy experiment in South Africa

ABSTRACT. The emergence of the transformative innovation policy (TIP) approach is disrupting the systems of innovation for development landscape of academic discourse and engagement with policy practice. It draws on multi-level systems theory to advance a theoretical and methodological framework, a ‘transformative’ theory of change (TToC) that hinges on the achievement of ‘transformative outcomes’. We address a gap in understanding relate to the capabilities – that is the ‘transformative innovation capabilities’ – required to design, implement, and evaluate TToC through experimental interactive practice. Although, there is no clear conceptualization of “transformative innovation capabilities” as such, in the TIP related literature (personal communication with Johan Schot, 2022, Carolina et al, 2022), we build off the work of Penna, Schot, Velasco and Molas-Gallart (2022) on institutional capacities and capabilities for ‘transformative mission-oriented’ policies. The object of our analysis is the emergence and evolution of the “transformative innovation capabilities” of a coalition of actors in South Africa, aimed at co-creating and strengthening transformative innovation policy initiatives. We aim to understand what transformative innovation capabilities are and how these are being strengthened. What do the coalitions of actors need to know and have the power to do to achieve transformative innovation outcomes? What are the principal structural, systemic, landscape and regime level forces that are directly enabling or conditioning the expression of their innovation capabilities in practice?

10:30-12:00 Session 2D: Assessing Use of Research Evidence in Legislatures
[Panel: Akerlof et al., Quantitative Methods for Assessing the Use of Research Evidence in Legislatures] Detecting Evidence Citation and Quotation in the U.S. Congress: A Methodological Case Study

ABSTRACT. Background and rationale As policy issues have increasingly become scientific and technological in nature, research information pervades the work of legislatures. Yet, since the U.S. Congress has no requirement to cite its sources, it remains unclear where the evidence comes from, who is using it, how, and to what end. These represent core questions about how science impacts society on which new techniques in data analytics and machine learning may be able to shed light. We investigate the extent to which digital records from the U.S. Congress can reveal patterns in evidence citation and quotation through a case study of an organization mandated to provide government advice, the National Academies of Sciences, Engineering, and Medicine (NASEM). As a private, non-profit organization congressionally chartered during the Civil War, the National Academies is directed to “investigate, examine, experiment, and report on any subject of science or art” for any department of government. This presentation will review methodological challenges and findings from three types of analyses of congressional and NASEM data: 1) which entities in Congress cite the National Academies most frequently and in what contexts; 2) co-occurence of National Academies references with named entities; and 3) the development of techniques for identifying National Academies quotations in Congressional documents.

Data All available text from U.S. Congressional data—hearings, bill text, committee prints, and the Congressional Record and reports—were downloaded between June-October 2022 from govinfo (API), the Sunlight Foundation and Congress github site (bulk) and Propublica (bulk) sources. National Academies consensus and workshop reports were downloaded from the NASEM site in HTML form in May 2022.

Approach overview Meta-data and processed information was deposited in a mongodb document store. The document data was analyzed using a python-based data mining approach. Our ETL (Extract-Transform-Load) process used minimal steps during the transformation phase. Data was loaded locally as it was available. Minimal metadata was associated with files to enable matching across various sources, i.e., subsequent versions of the same bill have different abbreviations in the file names. Only the latest version of a bill/document has been used. The content itself of the document was converted to text from original html or pdf format. For data analysis we used Python and its libraries sbert, sklearn, and faiss; mongodb for meta data storage and full text index; and Pdf2jpg and easyocr for image-based PDF-to-text conversion.

Analyses For the first two sets of analyses, we identified named entities and their co-ocurrences within all sentences in the congressional document texts, limiting the results to those related to the National Academies. In order to detect Congressional quotation of NASEM documents, we conducted a three-step analysis. First, all sentences from the NASEM reports were indexed using faiss with their vector embeddings using the sbert sentence transformers library and its pre-trained paraphrase-multilingual-MiniLM-L12-v2 model. All sentences from congressional documents were also embedded with the same sbert model. For each sentence within the congressional documents, the top 10 most similar sentences from the NASEM reports were selected for more detailed analysis in steps two and three. In the second step, common legal verbiage, such as the titles of acts and statutes, were removed using a SVC (support vector classifier) trained on sbert embeddings of selected human coded (labeled) sentences. Finally, unlike in the first step, which measured semantic similarity based on vectors, exact phrase-based quotations were identified, based on matching of k consecutive tokens. We experimented with k and (no)stemming.

Results Initial results illustrate the frequency with which the National Academies (sciences, engineering, and medicine, and the National Research Council) have been cited within congressional hearing and bill text between the 103rd and 117th sessions of Congress. Within all available bills and ~95% of downloaded hearings at the time of the analysis, 332 bills and 188 hearings referenced the Academies, which represents ~0.2% of total texts. Most of the citations are for the National Academies (216), National Academy of Sciences (127), or National Research Council (94), with fewer references to the National Academy of Medicine (26) and Engineering (6). Bill meta-data indicated that the following topics were areas in which the Academies were most likely to be cited: health, armed forces and national security, and science, technology, and communications. The committee hearings most likely to reference the Academies in bill texts include: House Committee on Science, Space, and Technology, House Committee on Appropriations, and House Committee on Veterans' Affairs. The committees associated with the bills that cite NASEM most often include House Energy and Commerce, House Transportation and Infrastructure, House Science, Space, and Technology, and House Armed Services.

Anticipated results A more comprehensive named entity recognition (NER) and co-occurrence analysis is still being processed and will be available with faceted analysis in relation to various government document properties, like the few mentioned above. In the third analysis we distinguish between potential directionality of the quotation based on the sequence of publication dates, i.e., whether the text was published in congressional or NASEM documentation first. We note that many ‘quotations’ actually refer to previous congressional documents, further, many of the semantically similar sentences refer to different concepts, documents, or events. To address the noise in the findings, the sentences have been stemmed and the analysis conducted on a longer sequence of k consecutive tokens, starting with 8 tokens. We will show examples of successful quotation detection as well as false positives.

[Panel Proposal] Quantitative Methods for Assessing the Use of Research Evidence in Legislatures

ABSTRACT. Research on the conditions under which evidence use is most likely to occur in legislatures has historically been hindered by limitations in available data and relevant metrics. Without formal systems for the citation of evidence, tracing the policy impact of engagement between the scientific and legislative communities can be challenging. However, new methodologies may offer the opportunity to address these deficits and, in turn, build scholarship on evidence-based policymaking processes and reveal ways to bolster institutional capacity for decision-making on complex scientific and technological issues.

Four interdisciplinary studies from the United States, United Kingdom, Germany, and international datasets illustrate the development of potentially transformative quantitative methodologies for the study of evidence use: machine learning, natural language processing (NLP), and behavioral modeling. K. L. Akerlof and colleagues employ data analytics and machine learning to explore institutional patterns in citation and quotation from the U.S. Congress and National Academies of Sciences, Engineering, and Medicine (NASEM). Similarly, Afagh Mulazadeh’s study identifies and analyses the explicit sources of scientific knowledge on antimicrobial resistance used in UK parliamentary scrutiny by applying NLP across two corpuses she is building using open data and Elsevier’s International Center for the Study of Research (ICSR) lab database. Combining ICSR and Overton databases, Basil Mahfouz analyzed more than 15,000 education policies published by governments throughout the COVID-19 pandemic to evaluate if decision makers used the ‘best’ available scholarly knowledge. In contrast, Henriette Ruhrmann from the Technical University Berlin applied quantitative behavioral modelling based on psychometrically validated survey data from 1,115 researchers and 162 legislators in Germany to predict science-policy engagement.

These techniques from the computational and quantitative social sciences offer the potential to shed new light on how legislatures use evidence, what types of evidence, and what drives researcher engagement and its use by legislators. Focusing on legislatures is a highly relevant extension of an existing body of research on science use for policy that predominantly focuses on the executive branch of government. Further, the methods enable researchers to evaluate the effectiveness of legislatures at using scholarly research, at scale and with the potential to replicate the studies in different geographic contexts. The panelists will also describe current methodological challenges.

Panelists & Presentation Titles:

• Akerlof, K. L. Detecting Evidence Citation and Quotation in the U.S. Congress: A Methodological Case Study • Ruhrmann, H. Quantitative Modelling of Science-Policy Engagement – Leveraging Behavioral Science to Strengthen Legislative Science Advice • Mulazadeh, A. Identifying and Analysing the Explicit Sources of Scientific Knowledge Used in UK Parliamentary Scrutiny: A Case Study on Antimicrobial Resistance (AMR) • Mahfouz, B. School Closures: Did Policy-makers Use the Best Science in Designing Education Policy During the COVID-19 Pandemic?

Did governments leverage the ‘best’ research during COVID-19? An education policy case study.

ABSTRACT. Introduction

In response to the Covid-19 pandemic, governments worldwide enacted school closures, disrupting learning for more than 1.2 billion students globally. When schools reopened, students also faced challenges from other measures, such as mask mandates, social distancing, and hybrid learning.

The measures were implemented or commented on through more than 42,000 policy documents published by governments, think tanks, non-governmental organisations, and international agencies. This case study analyses policy documents published by governments using a range of scientometric methods to answer: To what extent did governments use the best research?


The study leverages the databases of Elsevier’s International Centre for the Study of Research (ICSR) Lab and Overton. ICSR Lab provides access to the metadata of research articles indexed by SCOPUS, while Overton’s platform comprises a repository of policy documents and their citations.

First, we identify governmental entities that were responsible for education policy during the Covid-19 pandemic by searching for government policies whose subject area is classified as Education on Overton(n=604,000). The results are further filtered to include only policies published after February 2020, which is when the World Health Organisation formally declared the COVID-19 outbreak a “public health emergency of international concern” (n= 466,000). We then run a query identifying policy documents that include at least one COVID-19 related keyword (n= 42,000). The corpus was then filtered to include only policy documents that include at least one formal citation to a scholarly article (n=15,000). The research team then selects the parliamentary entities with the most publications for the analysis based on the availability of data, language barriers, and other technical considerations. The shortlist includes the United Kingdom Parliament Select Committee on Education, United States House of Representatives Committee on Education and Labour, the European Union Directorate General for Education and Culture, among others.

From these policy documents, we extract all their citations and match their bibliometric records in the ICSR Lab database using unique digital object identifiers (DOI). Scholarly papers are then clustered based on their SciVal topics, a natural language processing mechanism for identifying the topic of papers. For each of these topic clusters, we extract all papers with the same topics indexed in Scopus and published after 2020. For each topic, we run a logistic regression where our response variable is whether a paper was cited in our policy corpus, while our independent variables are paper citation counts, the publishing journal’s CiteScore, the author’s h-index, and the authors’ institutional Times Higher Education 2021 score.

Initial and Expected Results

The current methodology is an enhanced version of a previous attempt at the same analysis. In the last run, instead of starting with relevant policies and tracing their citations, we first built a corpus of research papers focused on education during COVID-19, and then traced their policy citations. It became apparent that the previous model had some limitations, namely that policies citing our scholarly corpus were not all relevant to education policy during COVID-19. We also found that a significant portion of COVID-19 education policies did not cite our scholarly corpus, and so were excluded from our earlier analysis. Further, the regression on the previous model treated the whole corpus as a single unit, with no room for variations based on field or topic.

In this section, we base our expected results on the initial results we achieved in the first run and explain how our new approach might lead to better results. For example, in the initial analysis, we uncovered certain biases in how governments sourced their data. Only 0.62% of scholarly papers were cited by all government entities. Even when comparing citations bilaterally, the highest rate of evidence shared by two entities was slightly over 10% (measured as ratio of one entity’s total citations also cited by another entity).

Second, although over 40% of the research on education during Covid-19 was authored in developing countries, the papers represented only 16% of policy citations. This may be due, however, to a data bias, as Overton is more effective at indexing English language policy documents from western governments. This leads to the third aspect; a clear preference for policy makers to cite local research, which makes sense because education is highly localised, meaning policy makers needed more contextualised evidence.

Fourth, when we applied a key word co-occurrence algorithm using the VOSviewer software, we identified topical differences between cited research that was published after 2020 versus research that was published before the pandemic. Governments were much better at citing recent medical research about COVID-19, often referencing papers within a few months of their publication, than they were at citing new education research, mostly referencing education papers published in 2016-2020.

Fifth, we found that scholarly papers cited in policy tended to also have more academic citations, be published in journals with higher CiteScores, and from authors with higher h-indexes. This is further affirmed in our regression, which calculated a positive relationship between the independent variables and the probability of being cited in policy. Of the four variables, citation count was the only one with a p-value less than 0.05. Citation count was also over 10 times greater at influencing our response variable than the other three. However, with a coefficient of 0.35, citation count remains a poor predictor, while the model overall had a poor fit, with a recall rate of less than 33%.

We expect our enhanced, topic-based methodology will lead to more accurate results and a better fitting regression model. By starting with the policy documents and then tracing their scholarly citations, our analysis will be based on a more robust, relevant corpus. Further, by using natural language processing to model topics, we can account for field-based variations in the data by comparing papers that were cited to papers that were not within the same topic space. Finally, the new methodology unlocks multiple opportunities for comparative analysis, for example, comparing the effectiveness of the use of evidence across research fields and governments.


Quantitative research, especially when utilising machine learning approaches, suffers from risks based on wrong assumptions and data bias. Our corpus includes only peer-reviewed papers indexed by SCOPUS, which involves a quality control process, and Overton which more reliably indexes policy documents from North America and Europe. Further, due to time sensitivities, scientists during the pandemic opted to share results in alternative, faster forms to conventional peer-reviewed journals. Many of these sources are not indexed by SCOPUS and are not included in our analysis

There is also a fundamental issue with the construction of the regression. Our independent variables are correlated amongst themselves, requiring significant normalisation. Further, we assume a linear relationship between policy impact and academic excellence. To mitigate, we will explore non-linear regression models as well as adding new, other non-academic variables, such as media coverage, on policy influence to improve the model’s fit and accuracy.

Quantitative Methods for Assessing the Use of Research Evidence in Legislatures

ABSTRACT. Research on the conditions under which evidence use is most likely to occur in legislatures has historically been hindered by limitations in available data and relevant metrics. Without formal systems for the citation of evidence within legislatures, tracing the policy impact of engagement between the scientific and legislative communities can be challenging. However, new methodologies may offer the opportunity to address these deficits and, in turn, build scholarship on evidence-based policymaking processes and reveal ways to bolster institutional capacity for decision-making on complex scientific and technological issues. Four interdisciplinary studies from the United States, United Kingdom, Germany, and an international dataset illustrate the development of potentially transformative quantitative methodologies for the study of evidence use: machine learning, natural language processing (NLP), and behavioral modeling. K. L. Akerlof and colleagues employ data analytics and machine learning to explore institutional patterns in citation and quotation from the U.S. Congress and National Academies of Sciences, Engineering, and Medicine (NASEM). Similarly, Afagh Mulazadeh’s study identifies and analyses the explicit sources of scientific knowledge on AMR used in UK parliamentary scrutiny by applying NLP across two corpuses she is building using open UK parliamentary data and Elsevier’s International Center for the Study of Research (ICSR) lab database. Combining Elsevier’s ICSR Lab and Overton databases, Basil Mahfouz analysed over 15,000 education policies published by governments throughout the COVID-19 pandemic to evaluate if decision makers used the ‘best’ available scholarly knowledge. In contrast, Henriette Ruhrmann from the Technical University Berlin applied quantitative behavioural modelling based on psychometrically validated survey data from 1,115 researchers and 162 legislators in Germany to predict science-policy engagement. These techniques from the computational and quantitative social sciences offer the potential to shed new light on how legislatures use evidence, what types of evidence, and what drives researcher engagement and its use by legislators. Focusing on legislatures is a highly relevant extension of the existing body of research that predominantly focuses on the executive branch of governments. Further, the methods enable researchers to evaluate the effectiveness of legislatures at using scholarly research, at scale and with the potential to replicate the studies in different geographic contexts. The panelists will also describe current challenges in using bulk data compiled by legislatures, and the creation of new datasets, such as the Elsevier ICSR Lab database, Overton, and a large-scale psychometrically validated survey of German legislators and researchers.

Quantitative Modeling of Science-Policy Engagement – Leveraging Behavioral Science to Strengthen Legislative Science Advice

ABSTRACT. Background and Relevance Techno-scientific innovation needs democratic guidance. For democracies to revisit societal values in light of technological transformation, science must rethink how to engage with legislators. Legislatures represent diverse constituent interests, act as a forum of debate on controversial issues, and scrutinize government action. Therefore, legislators are responsible for aggregating, articulating, and representing societal interests and values in light of scientific progress. Researchers, conversely, should contribute to the co-development of a shared knowledge base to inform parliamentary action and the direction of future research.

Knowledge Gap Previous behavioral science-based research addresses the behavioral determinants of researcher engagement with industry and the public, but evidence on science-policy interaction remains scarce. On the policy side, quantitative modeling remains focused on the executive. Therefore, evidence on the behavioral determinants of legislator research use is urgently needed to promote engagement effectively and efficiently.

Research Question Focusing on the individual engagement behavior of researchers and legislators, which personal and institutional factors drive engagement activities and research use? How can quantitative modeling inform change in the science system? The core hypothesis motivating the study is that the institutional support research organizations offer (e.g., incentives, resources, training, or recognition) promotes policy engagement.

Theoretical Framework The study adopts an interdisciplinary approach linking science-for-policy studies and behavioral science. The study leverages behavioral science theory to explain and model researcher-policymaker/legislator engagement behavior. The COM-B model, developed by Michie et al. (2011), structures the inquiry. The COM-B model posits that (C)apability, (O)pportunity, and (M)otivation determine (B)ehavior. Researchers can apply the COM-B model by operationalizing the domains for any target behavior. The unique advantage of the COM-B model among behavioral science models is that it systematically links the framework domains (C, O, M) to specific types of behavior change interventions. This linkage allows me to translate the research findings into actionable practitioner guidance for research organizations seeking to support engagement.

Data and Methods Situated in Germany, my empirical research design combines two methodological approaches: First, I conducted 22 qualitative interviews with highly-engaged researchers and legislators in Germany around a data-rich technology policy case study in early 2022 (A). Based on these semi-structured 30-60-minute interviews, I conducted a two-step qualitative content analysis. In the first step, I assigned primary deductive codes based on the COM-B model, and in the second step, I derived inductive sub-codes following Mayring (2000) and Hsieh and Shannon (2005).

Second, I developed and conducted two quantitative, psychometrically validated survey studies based on the COM-B model for knowledge ex-change behavior with 1,115 researchers and 162 legislators from federal and state parliaments in Germany in mid-2022 (B). The goal of the survey studies was to quantitatively test the external validity of hypotheses developed in the interview study (A) for larger samples of researchers and legislators using multiple regression analysis. The survey design operationalizes the COM-B model based on insights from the interview study and literature on science policy engagement. I leveraged or adapted validated item batteries wherever possible. To operationalize researcher policy engagement, the dependent variable, I adapted Tartari et al.'s (2014) Academic Engagement Index for policy engagement activities. I validated the selection of engagement activities in focus groups with 16 experts in the field of knowledge exchange.

Results Based on the interview study (A), I aggregated the core motivations of researchers for policy engagement, their institutional support, and engagement activities in a Sankey chart. The qualitative data supports the hypothesis that institutional support leads to more in-depth policy engagement. For legislators, the interview study findings demonstrate that they adapt their engagement strategies to their limited time, often prohibiting engagement with researchers whose communication does not meet their needs. Moreover, legislators tend to favor the field of their academic training. In general, they demand more researcher engagement and, in particular, more consideration for their needs as knowledge users.

The survey studies (B) compare the relative effects of variables in the domains of capability, opportunity, and motivation in driving science-policy engagement. I identified that for researchers, mission-driven motivation to solve societal problems and apply research findings is the strongest predictor of researcher policy engagement, followed by communication skills and institutional support. For the researcher survey, the multiple regression model explains 32.7% of the variation in policy engagement behavior (adjusted R2). For legislators, preliminary results suggest that motivation and capability (specifically, research literacy) are the strongest drivers of research use. In the area of opportunity, social influences (specifically, injunctive norms) promote evidence use. Different regression model specifications explained 40.9% to 44.4% of the variation in legislator research use.

Understanding the interplay of these three determinants (capability, opportunity, and motivation) can inform the design of evidence-based behavioral change interventions; for example, institutional support mechanisms for researchers engaging with policymakers/legislators. Ideally, such support structures should target the most critical determinants of engagement behavior to optimize their effectiveness.

Limitations Modeling human behavior is inherently challenging. Even methodologically rigorous survey research remains limited. This study has three limitations: First, survey respondents (and interviewees) self-selected to participate in the study creating the risk of non-response bias. Second, respondent errors or necessary parsimony may negatively affect their precision with which survey item scales capture the underlying constructs. Lastly, though the data may carry implications for science policy systems globally, I encourage caution in extending conclusions outside the study’s German context.

Conclusions and Policy Issues As research organizations acknowledge their new responsibility as interlocutors in societal transformation processes, they are obligated to build institutional capacity. Specifically, they should create new organizational structures to support their researchers in integrating increasing demands for policy engagement with expectations for excellence in research and teaching.

Based on the behavioral model, I propose an evidence-based approach for research organizations to support policy engagement in practice- novel to Germany. My interactive practitioner resource catalog presents support mechanisms currently in use and highlights how they compare in terms of efficiency. By translating the study findings into actionable guidance for research organizations, I hope to contribute to more strategic, evidence-based capacity-building of organizational structures to promote engagement between research and policymaking.

10:30-12:00 Session 2E: Government Funding
Research misconduct investigations in China’s science funding system

ABSTRACT. As stewards of public money, government funding agencies have the obligation and responsibility to uphold the integrity of funded research. Despite an increasing number of empirical research deals with misconducts, a majority of them focus on retracted publications. How agencies spot funding-relevant wrongdoing and what happens next to those responsible remains unexplored, particularly for emerging science powers. Following a chronology of China’s anti-misconduct policies, we retrieved and analyzed all publicized investigation results from China’s largest basic research funding agency over the period from 2005–2021. Our findings reveal that both the “police patrol” and “fire alarm” approaches are adopted in identifying misconduct and deterring fraud. The principal reasons for investigations are journal article retractions, whistleblowing, and plagiarism detection software. Among six funding-related misconduct types punished and publicized, the top three are fraudulent paper, information fabrication and falsification in the research proposal, and proposal plagiarism. The most frequent administrative sanctions are debarment and recouping of grants. This article argues that more systematic research and cooperation among stakeholders is needed to cultivate research integrity. Specific training and education should be provided for young scientists and researchers in less-developed regions, both of whom make up a large share of funding-relevant research offences.

Is this grant scarce and desirable? The perception of competition in public research funding

ABSTRACT. Background and rationale

Since the 1980s, competition has become a widespread mechanism for the allocation of public research funding in most advanced countries (Geuna 2001; Musselin 2018). Competition has been implemented at the organizational level through the introduction of performance-based funding (Hicks 2012; Krücken 2019; Teixeira, Biscaia and Rocha 2021), as well as at the individual level through various types of competitive grants managed by research funding organizations (Lepori and Reale 2019). This implementation has occurred in public sector contexts within which new policy rationales have supported the adoption of private sector practices, such as quasi-markets and efficiency-focused management models (Teixeira et al. 2004; Lepori 2011).

Whereas we know much about how competition is implemented in research systems, we know little about how it subsequently unfolds. The sociological literature suggests that competition should be understood as a socially constructed phenomenon (Arora-Jonsson, Brunsson and Hasse 2020), which is enacted in (strategic action) fields (Fligstein and McAdam 2011) and is largely dependent on social hierarchies (Krücken 2019) and networks (White 2001) in these fields. This means competition in research systems will unfold differently depending on how it is organized, as well as on how it relates to specific historical contingencies (Arora-Jonsson, Brunsson and Edlund 2023). The ways in which these processes unfold are likely to affect core dimensions in competition for research funding: self-selection among applicants (Viner, Powell and Green 2004), narratives to request resources (Velarde 2018), and strategies deployed by scientists to acquire funding (Laudel 2006). Such dimensions are, ultimately, likely to have an important impact on the outcomes of competition, including the novelty of funded applications (Boudreau, et al 2016), and, thus, the ability to support research with the potential to engender scientific breakthroughs (Laudel and Gläser 2014).

Set against this backdrop, the aim of our paper is to generate new knowledge about two elements that are central to constructing competition for funding: the extent to which grants are perceived as scarce and desirable (Arora-Jonsson, Brunsson and Hasse 2020). From previous research funding literature, we know that perceptions of scarcity and desirability vary depending on the type of grants and the position of scientists (Laudel 2006). There has, however, not been any systematic investigation of how grants are perceived by the actors involved in the competition, and, specifically, on the ways in which this may be affected by rules and discourses disseminated by funding agencies.


To address this aim, we build on theory from the sociology of quality markets literature (White 2001; Krücken 2019), as well as from the commodity and consumer behavior literature (Lynn 1991; Wu, et al 2012), which provides us with extensive insights into how scarcity perceptions are generated, how they affect perceptions of quality and desirability, and how they, in turn, impact the behavior of actors (Aggarwal, Jun and Huh 2011; Nichols 2012).

Empirically, we provide a case study consisting of two schemes housed under the European Union’s Framework Programs (EU-FP) for research and innovation: the European Research Council’s (ERC; Laudel and Gläser 2014) bottom-up individual grants and the European Commission’s (EC) top-down collaborative actions (Ulnicane 2015). These two schemes represent contrastive cases in terms of their goals, narratives, and implementation approaches.

Using a mixed methods approach involving official documents, a survey, and a series of interviews, we thus contrast perceptions of scarcity and desirability among applicants to ERC grants and collaborative projects, seeking to grasp how these perceptions are affected by the goals, narratives, and implementation approaches from public authorities and funding agencies. To avoid disciplinary bias, we focus on potential applicants in the area of immunology.

Scarcity and desirability of ERC and collaborative grants

The ERC’s budgets are, as with any other economic resource, limited, and this, by default, creates certain scarcity. Scarcity is, however, intertwined with desirability in constructing competition for the ERC’s project grants, because resources that are scarce and difficult to access typically become desirable and attractive to access as well (Zuckerman, 1977). The desirability of ERC grants has been reinforced through bottom-up applications – implying that scientists do not require any nominations from universities or research councils – and beneficial funding conditions – including large monetary amounts that scientists have at their autonomous disposal for lengthy duration periods at European host organizations (Schreck, 2007; Wolinsky, 2010). The desirability of these grants has, finally, been further constructed through Europe-level allocations, which are adjudicated by prominent panellists that are seen as elites and experts in their respective disciplines. This serves to envelop the ERC’s grants in a meritocratic aura that is reminiscent of Merton’s (1942) scientist norms.

Scarcity in collaborative calls varies according to the budget allocated and the broadness of the calls. For example, we can expect lower success rates in broad calls supporting the application of digital technologies in the health area than in calls targeting the development of a specific treatment for a particular disease. We can denote overall increasing levels of desirability and thus competition throughout the years, notably due to the shift from an EU-FP relying on the juste retour principle to today’s predominant excellence narrative. The juste retour principle implied that EU Member States could expect to receive EU-FP funding in proportion to their financial contributions to the EU budget. From the 2000s and onwards, notably with the Lisbon Strategy targets and the inception of the European Research Area, the policy narrative underlying EU-FPs saw increasing importance of New Public Management elements, with past results as a key criterion for the allocation of funding, and of the concept of excellence that encompasses all EU-FP funding instruments (Young, 2015; Hoenig, 2018). Further, collaborative schemes in EU-FPs provide resources to strengthen and expand collaboration and thus stimulate knowledge exchange. They provide networking opportunities and generate new collaborations that can result in co-publications, patents and new projects.

The foreseen survey and interviews can help us explore more in-depth how researchers perceive both grant schemes, notably in terms of career opportunities, but also identify mechanisms of self-selection or the potential influence of institutional settings on the desirability of such grants. At the moment of writing, we drafted a questionnaire that will be sent to hundreds of potential applicants to EU-FP grants in November/December 2022. We expect to be able to share advanced results by the time of the conference.


Our paper offers contributions to the research policy literature and the sociology of markets literature, by advancing new theoretical and empirical understandings of scarcity and desire as core elements in the construction of competition in public research funding.

From a policy perspective, and as suggested by the sociology of law literature (Edelman, Uggen and Erlanger 1999; Lascoumes and Le Galès 2007), we highlight how, beyond setting rules for competition, the design of grant schemes also sets norms and values that might deeply affect the perceptions of applicants concerning the desirability and scarcity of grants. And the perceptions of applicants can, in turn, impact their behavior because, as we have known for some time, “if (wo)men define situations as real, they are real in their consequences” (Thomas and Thomas 1928: 571-572). Taking this into account may help tailor policy measures to better connect the outcomes of competition to the achievement of policy goals.

Cross-disease spillovers: Evidence from HIV research targeting

ABSTRACT. Background rationale: The arrow of funds and research

In 1984, the US health secretary announced the discovery of HIV (human immunodeficiency virus): “The arrow of funds, medical personnel, research and experimentation [we] aimed and fired at the disease AIDS, has hit the target. First, the probable cause of AIDS has been found… Second a new process has been developed to mass produce this virus… Finally, we hope to have a vaccine ready for testing in two years”.

Since then, the HIV/AIDS ‘target’ has grown to be a major locus of social, economic and political activity. For more than 20 years, over half a per cent of the entire US federal budget has been allocated to HIV/AIDS. Globally, over half a trillion USD has been spent on HIV/AIDS since 2000. A new UNAIDS programme within the United Nations, an International AIDS Vaccine Initiative, and a plethora of other HIV/AIDS organisations emerged. The largest ever global health program focused on a single disease target, PEPFAR, was launched.

HIV/AIDS has been the focus for one of the biggest research efforts of the last 40 years. The US National Institutes of Health (NIH) alone has spent over $70bn on HIV research. The effort included a new Office of AIDS Research within the NIH, Congress mandating that ten per cent of NIH research funds be set aside specifically for HIV, and the Presidential appointment of an AIDS co-ordinator. Such investments ushered in antiviral drugs, helping to reverse a rapidly growing HIV/AIDS pandemic.

Notably, some of the outputs of that research effort could have ended up being different to what was targeted. Long before HIV/AIDS emerged, Vannevar Bush’s centrepiece of post-war science policy claimed that spillovers were not only “certain”, but also likely to occur “often”. Over half a century later, as resources were being mobilised for HIV/AIDS, policymakers highlighted that some of the outputs of the research effort could end up going beyond what was targeted. Senator Kennedy (D., Massachusetts) argued that “This knowledge enhances our ability to fight not just AIDS, but all diseases.” Such claims were becoming a staple feature in policy debates, as advocates lobbied for research to target specific diseases.

In response to concerns about the allocation of research funds across diseases, NIH Director Harold Varmus argued in his testimony to Congress: “There are legitimate limits to our ability to plan science. Because science attempts to discover what is unknown, it is inherently unpredictable… Research aimed in one direction frequently provides benefits in an unexpected direction.” This raises the question, how frequently?

Methodological approach: Frequency, magnitude and unexpectedness of HIV spillovers

Here, we explore the frequency and magnitude of spillover and non-spillover outputs for a given disease. We develop a methodological approach for offering some empirical estimates, and undertake extensive manual review. Our findings suggest that spillovers are far from rare. Lastly, we discuss implications for theory and policy.

The frequency of research spilling out across fields would inform both priority setting and calculations of returns from research investments. It is conceivable that some spillover areas will be overlapping with HIV and only slightly off-target, whilst others will be exclusive of HIV and distinctly more off-target. This will further affect how spillovers are perceived and valued. In short, the aggregate frequency of HIV spillovers might belie variation in the way HIV spillovers are distributed. This raises the need to be able to examine not just their frequency, but also the composition and magnitude of the spillovers themselves.

Since diseases are not mutually exclusive, research outputs can relate to more than one disease. One way to understand the magnitude of spillovers is to locate them on a network that reflects disease relationships. Fortunately, there have been studies offering ‘global maps of science’ on which to locate spillovers, and which can help us draw inferences about their distance and direction from HIV. These networks offer a view of underlying cognitive and social structures in research, reflecting how various pathogens and symptoms are understood to be related to each other, as co-infections, co-morbidities, secondary complications and so forth. Spillovers can be viewed, not only in terms of high-frequency or low-frequency in a given category but also in terms of their location across disease-space.

We analysed 118,493 publications, and found that 62% of these were spillovers. We used Medical Subject Headings assigned to publications to explore the content of these spillovers, as well as to corroborate non-spillovers. We located spillovers on a network of MeSH co-occurrence, drawn from the broader universe of medical publications, for comparison. We further assigned spillovers a ‘distance’ from HIV (using cosine similarity) and reviewed 1,000 grant-publication pairs from a local sample, and 1,000 pairs from a remote sample. Coding reliability was assessed in 200 pairs, selected at random. Intercoder reliability was sufficient for our purposes, with Kappa coefficients of at least 0.89 for each category (at 95% confidence level range of ±0.05).

Anticipated results: HIV spillovers are frequent but not random

Overall, our results show that HIV research targeting delivered outputs not only in relation to HIV/AIDS but also to other disease areas beyond HIV/AIDS. Such cross-disease spillovers were far from rare, with spillover outputs occurring at least as often as – if not more often than – on-target outputs.

We have known for a long time that research in general has very high rates of return. HIV research seems consistent with this and, moreover, it seems that spillovers form a large part of why we find such high returns to research in terms of aggregate social welfare gains. However, whilst spillover frequency is high, we also see that HIV spillover traffic is not random.

Firstly, spillovers were unevenly distributed across disease areas. Most of the HIV spillover traffic went to areas that overlap with HIV/AIDS or neighbouring disease areas. We were able to visualize these differences across disease space on a network of MeSH categories. This also showed some spillovers occasionally went further afield, with their categories more distinct from HIV.

Secondly, reviewing research projects at their outset allowed us to distinguish spillovers that were expected, from spillovers that seem to be unexpected. When separating spillovers based on their proximity to HIV, we found remote spillovers were more likely to seem unexpected than local spillovers.

Thirdly, there is notable variation across NIH institutes. For some institutes, more than half of their outputs were spillovers whereas for others it was less than a third. This offers a strong indication that research organization plays an influential role in the direction of research outputs.

Significance: implications for research targeting more generally

It seems that aggregating the returns to research, without explicit attention to spillovers and their variety, masks some potentially important trade-offs in research policy. Such a finding would relate to one of the central questions of the Atlanta 2023 conference “Should policies stoke innovation fires as the engines of growth or direct them to solve environmental challenges?”

Joining the dots between government, funders and academia: are Areas of Research Interest the missing cog in the system?

ABSTRACT. Background: With the aim of making it easier for researchers to produce policy-relevant research, the UK Government now requires all departments and arms-length bodies to publish annually-updated statements of their evidence needs, called ‘Areas of Research Interest’ (ARIs). We describe how ARIs are produced, and how they are used to support this aim.

Aims and objectives: In this paper we offer a description of ARIs and their development by UK governmental departments, and an assessment of how different stakeholders have responded to or otherwise used the ARIs.In the UK, the Government Office for Science (GOS) supports the Government Chief Scientific Adviser (GCSA), and holds a cross-government remit to support science capability across departments. It relies on 'soft power' networking and influencing rather than mandating. It also holds the policy for Areas of Research Interest. As part of this remit, Giulia Cuccato led a team of civil servants in the Government Office for Science (GOS) to develop guidance for departments in developing their ARIs, and supporting and tracking their ARI-related activities. Alongside this work, Kathryn Oliver and Annette Boaz have been embedded in GOS to explore and support better production of ARIs, and more effective engagement with them by funders, researchers and intermediaries. We have been involved in supporting the development of ARIs across the UK government, in helping departments use ARIs to access relevant evidence and expertise, and in researching how ARIs could be optimised to support the research-policy system.

Methods: We draw on 25 interviews, approx 50 hours of observations of meetings and roundtables, and our 3 years' embedded researcher experience to describe the ARIs, their different functions across settings, institutions and audiences. We describe how ARIs are produced, what ARI-related engagement occurs, and how they function as a systems level intervention.

Findings: ARIs were intended to identify strategic research priorities for departments, but in practice we have found that they have a much broader set of uses. We found that departments use them to improve internal working and relationships, to implement the agenda of the Chief Scientist, to support other governmental processes such as spending reviews, the Integrated Review, and the Science Capability review. For some they are a reflection of their policy priorities; for others an articulation of the activities and structures of their internal science system; a statement of likely research commissioning priorities; and/or a statement of research areas around which they would welcome collaboration or input. However, the end products appeared to be mostly appropriate for the departments in question. By and large, they were seen as useful internal tools to negotiate and communicate with policy colleagues around budgets and priorities, and useful external tools to solicit help.

Universities and academics find them useful to plan engagement activities such as workshops and fellowships, but often tend to view them as poorly-written research questions. ARIs can help the research community to understand what government departments want from them. This happens most effectively when there are opportunities for dialogue or a clear narrative about the policy history behind each ARI.

Identifying relevant expertise and research is a real challenge for government departments, particularly where resources are limited. Framing problems is an important step for departments, because it dictates what research and which experts are considered relevant and appropriate. We found that officials in government departments were committed to addressing the challenge of diversity and inclusion in academic-policy engagement, but we unsure how best to go about improving practice in this area.

Strengths of ARIs: ARIs work well as an external articulation of research and evidence needs for departments. They offer funders, intermediaries and researchers insights into what departmental research agendas. Universities and intermediaries in particular have used ARIs to develop their own strategic engagement plans (see, e.g Heckels, 2020). Most departmental ARI documents now contain contact details as well as ‘asks’ and ‘offers’ for each ARI. This makes it easier for funders, intermediaries and researchers to know how to respond (e.g. by getting in touch for a conversation, arranging a research collaboration or responding to a research tender).

ARIs as a systems intervention: The ARIs were proposed to encourage the production of more policy-relevant research. This has been describe as a ‘deficit model’, suggesting that if decision-makers had better evidence, their decisions would improve. This model has been widely criticised as being based on some fundamentally flawed assumptions about how decision-making works (Jones and Crow, 2017) and on how evidence informs that process (Locke, 2002). The ARIs may have been planned to address this illusory ‘deficit’, but in practice perform a far greater range of functions which help to connect the policy research system in complex ways.

The true value of ARIs may be in illuminating the ways in which the research-policy system is connected, and how we can intervene most effectively to support this system.

Weaknesses of the ARIs: systems challenges

Not everything can be or is articulated as an ARI: ARIs are not able to articulate the totality of departmental research needs. For some departments, this is due to political or operational sensitivity, and for others, they prefer to only publish ARIs on topics where they are currently seeking external input. It would be a mistake, therefore, to think of ARIs as a complete and exhaustive list of the topics on which government is seeking input.

ARIs are not research questions: Academics frequently describe ARIs as poorly written research questions. An alternative, more useful phrase might be “research needs”, which helps to give the impression that there is a process attached to them, that they are valued, and broader than research questions. They are more usefully thought of as topics for conversation.

ARIs are difficult to access and analyse: By 2018, most departments had published at least one version of their ARIs, which then sat on the government website in pdf or html formats. There is as yet no easy way to search for ARIs by topic, department or year, which makes it difficult for researchers to identify relevant topics or potential collaborators This also means that departments are not easily able to identify shared cross departmental areas of interest.

Finding relevant evidence and expertise takes time and work: While some departments had resources dedicated to engagement around the ARIs, others did not. While relevant research often exists (as bodies of primary research, in research and practice communities, or in ongoing funding investments), this knowledge is often inaccessible, being behind paywalls or requiring time and skill to find and absorb.

Key conclusions: The ARIs have great potential to enable funders, government, research organisations, researchers, and intermediaries to work together in a more effective way. We have observed that merely producing ARIs is not a sufficient intervention; instead, it requires skilled mobilisation work by people within all these organisations to be able to optimise their production and use.ARIs are a mechanism for organisations to share their research interests with external audiences in the form of a published document,. They also have a much broader set of uses, including connecting departments with each other and helping intermediaries shape engagement plans. All groups would benefit from more robust evidence to choose effective engagement mechanisms, and more can be done to make the ARIs discoverable and useable. Overall, the ARIs are a useful tool to illuminate, and begin to connect different parts of the research-policy system.

10:30-12:00 Session 2F: Climate Sustainability and Environmental Justice
Policy perspectives regarding benefits and challenges of connecting with citizen science initiatives. A case study on environmental justice

ABSTRACT. The rationale

Citizen science (CS) is a research approach with potential to build policy-society bridges. On the one hand, citizens can draw the attention of scientific research to understudied topics that could then been addressed by policy making. On the other hand, CS opens opportunities to democratise the policy processes (Cohen & Doubleday, 2021) which makes it particularly promising in the case of environmental policy.

The Rio Declaration developed in the context of 1992 United Nations "Conference on Environment and Development" established the basic principles for environmental democracy, which included access to information, public participation and access to justice. Similarly, the Escazú Regional Agreement (2018) on Access to Information, Public Participation and Justice in Environmental Matters in Latin America and the Caribbean, seeks to strengthen citizen rights to participate in environmental decision-making through specific forms of policy governance.

Although the potential exists, CS literature has highlighted several challenges faced by CS initiatives when engaging to the policy-making sphere. They range from issues regarding data quality to power imbalances and institutional resistance (e.g. Nascimento et. al. 2018; Haklay, 2021). The EU has created an inventory of selected practices from CS projects that attempted to link with environmental policy-making sphere (EC 2018). But there is no empirical research analysing the perspectives of policy makers regarding their connection to CS projects. This paper aims to fill this gap in the context of environmental justice in the Matanza-Riachuelo basin.

The context

The basin is a heavily polluted area in Buenos Aires, Argentina. There are almost 14,000 industries, many of them dumping their waste into the river, while 18% of the population is not connected to the drinking water network. The area is ruled by different government jurisdictions (Municipalities, Province, and Nation) normally governed by different political parties, which has made it very difficult to advance with comprehensive solutions.

We are involved since 2020 in a CS project, which co-designed along with 59 community actors from the basin, a digital platform for citizens to share experiences on key areas for environmental justice. These data open several opportunities for sanitation policy making (e.g. cost effective monitoring of water quality, democratic approaches to natural areas conservations, etc.), including the possibility of broadening participation, as it is legally requested. The early stage of the project (2020) through interviews and workshops we created a knowledge coalition of researchers (12), policy actors (13) and community actors (47). As a result of this process, we understood that policy makers believe in the potential of CS related to a better identification of community priorities and in trust-building for policy actions, but there are several challenges and reservations that need to be addressed (e.g. on data quality, the politics of participation, etc.).

Thus, in 2022 during the last stage of the project we aimed systematising the perspectives of policy actors regarding the potential uptake of CS in sanitation policy. These are the entry point to identify policy options that can contribute to strengthen the CS-policy link, because they shed light over different dimensions where CS initiatives could contribute to decision-making processes, from their standing points. This paper describes the result of this exercise. The method

We used Q-method to identify policy makers’ perspectives regarding the uptake of CS knowledge in policy formulation. It is a research technique to identify underlying patterns of meaning and discerning people’s perceptions of their world (Webler et. al., 2009, McKeown & Thomas, 2013). People participate in interviews where they are provide with a set of statements expressing ideas on a given topic to order them in a normally distributed grid. Participants are invited to explain the reasons justifying their way of organising statements in the grid. The statistical analysis then proceeds by correlating participants’ sorting of statements, which take the place of variables in factor analysis (McKeown and Thomas, 2013). Factors are interpreted with inputs from the interviews.

We presented 52 statements to 14 policy actors. Statements were developed through an iterative process by both co-authors using, primarily, inputs from the review of 36 international papers reporting evidence of CS projects that intended to establish some type of relationship with environmental policy spheres. In addition, some few statements were built using transcriptions of the public events in which policy actors in Argentina made affirmations regarding CS potential. We codified statements in three dimensions

• Linking mechanism: three forms of citizen intervention through CS initiatives that may connect to public policy: i) citizen-driven-data, ii) citizen participation in agenda-setting and iii) citizen participation in the governance of common goods or public services. • Policy-making phases: three phases of the policymaking processes to link with CS initiatives: i) problem identification, ii) policy implementation, iii) policy-change (e.g. creation of new programs). • Valorisation of CS-policy link: positive or negative.

Two factors were extracted using principal component analysis and rotated using varimax methods.


We found two non-confrontational perspectives regarding the potential contribution of CS in policy-making processes. We confirmed our findings of the knowledge coalition building process that there is a generally optimistic opinion regarding the potential contribution of CS to socio-environmental issues. Stakeholders believe that CS promotes empowerment, inclusion, and more responsive policies since they benefit from situated knowledge. The highest valued mechanism explaining optimism in the CS-policy link is the participatory nature of CS -rather than the capacity to produce and make available citizen-driven data. This is not surprising in the context under analysis that has participation as a formal mandate for sanitation policy.

Although they have some common background in the way they value participation, the two perspectives differ regarding what specific contribution to the policy making process CS could enhance, or, in other words, in what aspects of policy making there are more prospective links with a CS approach. One of them (Perspective A) finds the potential mostly relate to the initial phase -problem identification- and in creating opportunities to change policy, while the other (Perspective B) believes that the highest contribution is in the implementation phase. Remarkably, policy actors more directly involved on environmental issues load more heavily on perspective B.

By combining this quantitative analysis with qualitative material from interview transcripts, we spell out the perspective as following

Perspective A: "CS produces ideas to create new policies or policy programs. These initiatives are particularly useful in the context where there is a lack of policy actions, to fill a vacant area, thus complementing the available policy knowledge. Citizen participation makes visible concrete socio-environmental problems".

Perspective B: “Synergies can be created between public policy agencies and CS initiatives, contributing to the improvement of existing policymaking processes. Participation increases trust in policy actions, which is necessary to enhance policy effectiveness”.


We contribute to the scarce empirical literature addressing the potential of linking CS with policy making using evidence from a highly complex context of Argentina.

Our approach is different to empirical papers providing information on the CS-policy link because while they share insights obtained from the experience of CS, we explore the nuances of the other side of the CS-policy link analysing the perspective of policy actors.

We confirmed that the potential contribution exists also in the minds of policy actors. However, depending on the specific policy context, the contribution could be more complementary or more synergetic with existing policy actions

STI policy and climate change: insights from Uruguay and South Africa

ABSTRACT. Science, technology and innovation (STI) are called to play a fundamental role in the quest for overcoming the current global challenges, and deep tensions around the models of economic growth, and the social and planetary unbalances and the aggravated and reinforced dynamics among them (UNDP 2020). We are faced with an increasing inequality in wealth, global unsustainability, child labor, new ways of slavery, political polarization and challenges in human rights (Giuliani 2018). Thus, the call is for urgently rethinking the current development paradigm, to prioritize sustainability and social inclusion, and radical new relationships between humans and nature within the planetary boundaries (Rockström, Steffen et al. 2009).

This very complex scenario leaves open and uncertain questions that require new approaches, methodologies, and frameworks in the search for new practices, institutions, ways of life, guiding principles and ethical standards to define new ways of life within safe planetary boundaries which leave behind a growth paradigm based on the depletion of natural resources and the environment. Science, technology and innovation must be placed at the forefront of this process, as instruments of sustainable and inclusive development. In a different vein, for some decades STI policy have gained an explicit space within the policy concert, particularly in the developing world where this process is lagging behind. In Latin America, countries have defined STI policies and, in some cases, strategic plans, although with diverse emphases, scope, objectives, approaches and guiding principles (Dutrenit, Aguirre-Bastos et al. 2021). In spite of this well established and acknowledged role, its legitimacy is clear at the discursive level, but its intertwinement and embeddedness into crucial development aspects is still missing (Dutrenit and Sutz 2013, Bianchi, Bortagaray et al. 2021, Bortagaray and Aguirre-Bastos 2021). Furthermore, it is not evident what is best governance scheme to better advance in this direction, to foster the necessary changes across the policy spectrum placing STI as a fundamental driver of change. What type of policy space should STI have within the overall policy spectrum? Should it be organized as a discrete policy domain, or should it be designed in a way that cuts across others like agriculture, health, industry, social development? What scope, governance models and systems better serve such transformation, and with what specific policies, institutional arrays, and the extent to which they need to be changed or created? (Sachs, Schmidt-Traub et al. 2019). Current societal challenges are profoundly complex and novel, calling for flexibility, adaptation, experimentation and policy learning. STIP key (new) role is part of a “normative turn” (Daimer, Hufnagl et al. 2012), a paradigm shift in STI policy with growing importance of directionality and normativity in STI, and an instrumental role of STIP for solving societal challenges, and advancing sustainable human development.

This work attempts to analyze such changing scenario, its scope and scale in the context of two very different countries, Uruguay and South Africa, particularly taking into account how STI policy connects with climate change strategies. The work is in its initial stages, and draws on an ongoing collaborative effort in this regard.

Empirically it studies both the STI and the climate change policy spaces on their own, to then analyze (i) the extent to which STI permeates climate change strategy, (ii) how does climate change appears in the STI policy domain, (iii) and their linkages and intersections. It also investigates their connections to other policy domains, such as agriculture, industry, health. The explored hypothesis is that STI policy has evolved in relative isolation of other development arenas and in the particular climate change policy in the case of Uruguay. In South Africa, this trajectory has been different, and STI policy has been more tuned to development problems including climate change. Part of the explanation lays in the different ST policy governance systems these two countries have.

References Bianchi, C., I. Bortagaray, F. Liurner and E. Magallán (2021). Aportes de las políticas de ciencia, tecnología e innovación al desarrollo sustentable en el Uruguay del siglo XXI. Principales desafíos para una agenda de transformación. Serie Ideas para agendas emergentes. UNDP. Montevideo. Bortagaray, I. and C. Aguirre-Bastos (2021). Innovation Policies for Inclusive and Sustainable Development: Insights from the Central American Region. Policy and Governance of Science, Technology, and Innovation. Social Inclusion and Sustainable Development in Latin America. G. Ordóñez-Matamoros, L. A. Orozco, J. H. Sierra-González, I. Bortagaray and J. García-Estévez, Palgrave Macmillan Cham. Daimer, S., M. Hufnagl and P. Warnke (2012). Challege-oriented policy-making and innovation systems theory: reconsidering systemic instruments. . Innovation system revisited - Experiences from 40 years of Fraunhofer ISI research. F. ISI. Stuttgart: Fraunhofer Verlag: 217-234. Dutrenit, G., C. Aguirre-Bastos, M. Puchet and M. Salazar (2021). Latin America. UNESCO World Science Report 2020. Paris, UNESCO. . Dutrenit, G. and J. Sutz (2013). Introducción. Sistemas de innovación para un desarrollo inclusivo. La experiencia Latinoamericana. G. Dutrenit and J. Sutz. México D.F., Foro Consultivo Científico y Tecnológico, AC - LALICS. Rockström, J., W. Steffen, K. Noone, Å. Persson, F. S. l. Chapin, E. Lambin, T. M. Lenton, M. Scheffer, C. Folke, H. J. Schellnhuber, B. Nykvist, C. A. De Wit, T. Hughes, S. van der Leeuw, H. Rodhe, S. Sörlin, P. K. Snyder, R. Costanza, U. Svedin, M. Falkenmark, L. Karlberg, R. W. Corell, V. J. Fabry, J. Hansen, B. Walker, D. Liverman, K. Richardson, P. Crutzen and J. Foley (2009). "Planetary Boundaries: Exploring the Safe Operating Space for Humanity." Ecology and Society 14(2). Sachs, J. D., G. Schmidt-Traub, M. Mazzucato, D. Messner, N. Nakicenovic and J. Rockström (2019). "Six transformations to achieve the Sustainable Development Goals." Nature Sustainability 2: 805-814. UNDP (2020). The next frontier. Human development and the Anthropocene. Nueva York, UNDP.

Equitable Energy Transition Issue Analysis: A Comprehensive Success Factor Analysis (CSFA) Approach

ABSTRACT. Background and Rationale: Issue analyses are essential in framing sociotechnical systems challenges to reduce unknown-unknowns, identify barriers, mitigate risks, and achieve high-impact outcomes,. For large scale sociotechnical systems challenges like developing an equitable energy transition plan, issue analysis becomes more complex, as capturing subjectivity of all stakeholders becomes a daunting task. Current approaches to framing energy transitions embrace either a reductionist or holistic lens. Both framing approaches have their limitations towards achieving an equitable plan. A reductionist approach fails to capture the complexity within the system, and the subjectivity needed to attain an equitable renewable energy-based economy. Holistic approaches on the other hand do not adequately capture cross-scale, cross level interactions that occur at different abstraction levels within the Spatial (patches, landscapes, region, globe), Temporal/Rates/Duration (Slow, daily, seasonal, annual, fast, short, long), Jurisdictional (local, provincial, national, intergovernmental), Institutional (operating rules, laws, constitution), Management (tasks, projects, strategies), Network (family, kin, society), and Knowledge (specific, contextual, general, universal) scales. To address these limitations, this work employs a holistic problem analysis framework designed to highlight challenges spanning levels of abstraction, plurality, context, and scope in complex problem contexts to carry out issue analysis towards an equitable energy transition plan. Method: We employ comprehensive success factor analysis (CSFA), a method developed for grand challenge problem analysis. CSFA is a holistic method that identifies a broad array of success factors that are necessary to achieve a desired system level vision. CSFA reduces unknown-unknowns by identifying questions that needs to be answered for a system to yield desired outcomes. CSFA incorporates pespectives from 9 overarching disciplinary lenses - psychology, physiology, politics, operations, education, environment, economy, technology, and sociology (P3OE3TS). To carry out the analysis we utilize a toolkit developed at the Institute for Innovation Science at Purdue University. Our analysis involves pattern-based exploration of web-derived information on equitable energy system transitions. The system is analyzed at a national level. Solar, wind, bioenergy, hydro-electric, hydrogen and geothermal are considered as renewable energy for this research work. To identify success factors needed for a low carbon, renewable energy transition, we reviewed nearly 400 documents and drew out perspectives on 16 categories of success factors that compose CSFA, namely: infrastructure, equipment and supplies, workforce and talent, capital/finance, practices and mechanisms, awareness and acknowledgement of needs, motivation, enabling strategies, adoption and habit conversion, mechanism for evaluation, mechanisms for sustainability, resilience, security and safety, policies, government/ leadership support, and stakeholder interaction. Result: Several hundred success factors were identified that are vital to achieving an equitable energy transition. These factors provide a conceptual framework that can guide energy transition policy deliberations towards achieving equity by ensuring that 1.) individual and collective experiences, identities, backgrounds, and environments, are taken into consideration during policy discourse. and 2.) the dynamics of cross scale and cross level interactions within the sociotechnical system are accounted for. Collectively, these factors highlight that transition interventions must differ by level of abstraction, and are influenced by cross-scale (e.g., jurisdictional, management and, network scales) interactions, calling attention to the need for a portfolio of solutions to achieve widespread transition success Significance This research identifies success factors that need to be addressed by policy makers and stakeholders to achieve equitable energy transitions. Our findings, which are location agnostic, provide a user-friendly means to support energy transition planning and discourse that helps reduce unknown unknowns and reduce expert bias (identified in discourse theory) during problem framing, helping to foster more equitable outcomes

“Walking the green line”: government sponsored R&amp;D and clean technologies in the US

ABSTRACT. We examine whether government sponsored R&D affects the development of clean technologies with a higher impact on subsequent technological development. The empirical analysis uses information on USPTO patents granted during the 2005-2015. Starting from patents acknowledged in government funding, we build a control group by matching firm level information on Patstat. We combine linear regression and propensity score methods to control for an eventual sorting of riskier projects into public funded projects by firms and for non-random (public) treatment at technology level. We also assess the distributional impact of government supported R&D in green technologies. Results show that green patents benefiting from public funding have a significantly larger impact that the other patents developed by the firms in the sample, and that this impact becomes larger along the distribution of patent citations. Our results complement the literature on market-pull policies to reduce greenhouse gas emissions, showing that technology-push policies represent a relevant option to determine the speed and direction of technical change in the field of clean technologies.

10:30-12:00 Session 2G: Responsible Research and Innovation
Inclusion as Science and Innovation Policy Objective: Comparing Responsible Research and Innovation and Broader Impacts Frameworks

ABSTRACT. Background and rationale

In this paper, we analyze inclusion objective in the science, technology and innovation (STI) policy context. Inclusion is increasingly gaining importance as a STI policy objective aiming e.g., to promote closer interaction between science and society. Hence, it is valuable to scrutinize its use and symbolic role in the broader STI policy language. We begin the analysis by reviewing the extant (STI) policy literature. Secondly, we carry out a comparative analysis of the US Broader Impact Criterion (BIC) and the European Responsible Research and Innovation (RRI) frameworks. The objectives of the research are twofold. Our STI literature analysis aims at synthesizing the scattered knowledge and multiple approaches to inclusion as a policy objective. Secondly, we analyze how inclusion as a science and innovation policy objective is both defined and operationalized in the BIC and RRI evaluation frameworks.

The rationale for the comparative approach arises from the alternative ways by which inclusion is conceptualized, understood, pursued and evaluated in different policy contexts. In other words, we are interested in the different contextual and systemic realities and value-bases in which inclusion is promoted as a science and innovation policy objective. We hypothesize that these realities have an effect not only on the development of the societal impact itself, but also the actual evaluation practices as focusing devices in understanding the impacts.


Our empirical research is based on international comparative policy analysis that focuses on the contextual differences in policy formulation and implementation. In addition, it focuses on the policy instruments, the broader innovation system and its elements that influence innovation processes and outcomes, and sheds light on the conceptual ambiguity characterizing innovation concept and inclusive policies in general. The co-existence of multiple definitions in academic literature and among practitioners hampers the understanding of these terms and their further uptake in both the private and public sectors.

The comparison analyzes the concept of inclusion in the context of two different frameworks. Our theoretical literature based conceptual framework synthesizes the key dimensions of the broad science and innovation policy. This helps to compare the cases in terms of their approaches to inclusion. In practice, we analyze the extant research on BIC and RRI. Additionally, we analyze the key policy documents related to the frameworks and the evaluation criteria operationalizing objectives related to inclusion and evaluation practices. The document analysis is based on the principles of theory-driven content analysis, utilizing the conceptual framework built in the theoretical section of the paper.


The findings present different approaches to conceptualize and operationalize inclusion as a science and innovation policy objective. RRI and BIC are importantly different criteria in terms of their approaches to inclusion, the former taking a more process-oriented and the latter more outcome-oriented approach to promoting inclusiveness. While RRI focuses on collaboration, partnerships and interactive processes as a part of the research, BIC aims primarily at benefiting e.g. science, technology, engineering, and mathematics by means of concrete outcomes of the research. In the RRI framework, inclusion refers primarily to public engagement and “inclusive participatory approaches”. Instead in the BIC, inclusion is approached primarily as the participation of different marginalized groups in research and education activities, disseminating research results to wider audiences, and collaboration and networking activity with external actors.


Comparative analysis of the different ways to understand and operationalize inclusion can increase the societal impact of science and innovation policy. Such analysis can also contribute to the ongoing academic debates on societal impact as well as its evaluation amidst increasing societal complexity. It sheds light on the embeddedness of policies in different systemic contexts and value-bases that affect not only the creation and use of the policy language but also the concrete evaluation practices attempting to capture the impacts of policymaking.

The analytical distinction between inclusion as means and inclusion as an objective is useful especially in light of STI policies emphasizing the societal impact of research and innovation. Such inclusion concept is emphasized in the processual view on inclusion that is clearly different from output related inclusion. These different meanings given for the inclusion are clearly adding to conceptual ambiguity. Hence, identifying the primary goal is essential. A central question in this context is whether inclusion is primarily seen as a goal or as a means of policies promoting societal impact. Furthermore, the design of inclusive policies needs to consider agency in impact creation.

Altogether, extant research has identified diverse disconnections not only between the policy and practice of increasingly complex innovation space but also between the agency and structures that direct attention e.g. to the lacking acknowledgement of the role of users and consumers (including minorities and disadvantaged groups) in transformative innovation policy discussions. These disconnections, combined with the growing complexity of policy concepts, are decreasing the legitimacy of STI policies. Inclusion can be a powerful policy concept in promoting societal impact of research, and research on its conceptualization and operationalization can increase its value as an integrative policy concept. However, in addition to the identified need for transformative policies to be more inclusive, future research needs to account for the growing complexity and ambiguity of policy language regarding inclusive policies.

Framing and operationalizing ethical and responsible innovation in AI-driven manufacturing innovation

ABSTRACT. Background and rationale

The growing application of artificial intelligence (AI) to manufacturing, particularly when integrated with smart factory robotics, automation, and logistics, promises to transform the sector. AI will generate impacts across all aspects of industrial systems and industrial innovation, including product development and design, manufacturing production, supply-chain management, sustainability, competitiveness, user interaction and workforce training. Impacts will also be generated for employment and job tasks, affecting both shop floor and white-collar occupations. The potential positive impacts of AI in manufacturing have been recognized by industrial and innovation policymakers, with strategies and programs initiated in the US and other leading economies to accelerate the development and diffusion of advanced manufacturing and AI.

Alongside business and policy expectations that AI-driven manufacturing will drive industrial performance, generate innovation, and revitalize national and regional industrial economies, there are also many worries, generating debates about responsible innovation in AI manufacturing. Among these concerns are apprehensions about the displacement of human agency and labor, the rise of new mundane (and less-well paid) tasks that support AI, and the ethical implications associated with bias and privacy, as well as issues related to data security and consequences for small and mid-size enterprises.

While the scale, scope, and directions of impacts associated with AI in manufacturing will vary across sectors, companies, and occupations, strategic choices made early about how AI-driven manufacturing will be designed, implemented, and governed could lead to material differences in avoiding or mitigating adverse effects and risks and ensuring a more equitable balance of benefits and costs. This is a key premise underwriting initiatives to embed attention to ethics and responsible innovation in AI manufacturing. This paper offers a formative assessment of this proposition by drawing on early experiences with a large-scale project to develop and promote AI manufacturing. We reflect on whether and how consideration to ethics and responsible innovation can be framed and operationalized, anticipate future developments, and derive insights for innovation management, university-industry-community partnerships, and innovation policy.


The case study focus is the Georgia Artificial Intelligence Manufacturing Technology Corridor (GA-AIM) which aims to accelerate research, scale-up, and transfer of AI and related digital technologies in industry in Georgia, USA. Through a university-industry-community partnership, led by the Georgia Institute of Technology, GA-AIM seeks to integrate the deployment of AI technologies in industry with related job training and outreach to underserved communities. The project has received federal funding ($65 million) under the US Economic Development Administration’s Build Back Better Regional Challenge.

GA-AIM has an embedded ethics and responsible innovation team (which includes the authors of this paper) which is focused on two key program elements: an AI-manufacturing test-bed facility and AI-manufacturing community engagement. The team is using action-research methods, including targeted studies, and engagement with designers, users, and other stakeholders to facilitate GA-AIM in addressing ethics and responsibility and to foster learning and adjustment that advances the responsible and ethical use of AI-manufacturing technologies.

Results and significance

GA-AIM started in 2022. For the paper, we anticipate findings from targeted initial studies on the use of augmented reality (AR) systems for training and upskilling manufacturing workers and AI technologies for monitoring the health of manufacturing systems. We will highlight key ethical issues that are implicated in the design, development, and use of these systems and how designers are addressing them. These include ethical issues that are commonly discussed in AI ethics – including bias and privacy – and ones that are less commonly discussed, including AI-human interaction. AI-human interaction is a particularly important issue in manufacturing contexts, although it is an issue that is relatively neglected in AI ethics codes and policies. Additional ethical issues we will consider are trust and trustworthiness. Many AI researchers place a lot of emphasis on trust – on the importance of developing AI systems that users trust. Less attention has been given to the question of the criteria that should be met for a system to be worthy of the trust of users, including users of different backgrounds and levels, as is likely in manufacturing environments. We will explore what is changed through attention to ethics and responsible innovation in the design and deployment of these systems, what are the impacts of those changes.

There is some prior work on the involvement of social scientists in responsible research and innovation (RRI) in emerging technology research centers and projects, including in nanotechnology and synthetic biology. There is a critique that RRI to date has focused more on responsible research than responsible innovation. The GA-AIM project not only targets the use of AI innovations but also integration with workforce training and community economic development. This is significant in that it adds new dimensions and challenges related to the framing and operationalization of ethical and responsible innovation for emerging technologies. In addressing this topic, and drawing on action-research findings, we aim to contribute to scholarly discussion about how responsibility and ethical approaches in AI can be framed, adding considerations related to manufacturing environments and community development which, to date, have been less explored. We will examine important (and often overlooked) issues of the operationalization of attention to ethics and responsible innovation, with the aim of deriving exploratory insights for innovation management, university partnerships, and innovation policy.

Role of Scientific Institutions in the Development of Molecular Diagnostics in India: Need for Responsible Innovation System Approach

ABSTRACT. 1. Background & rationale of the Study The main purpose of the present study is to examine the significance of Scientific Institutions in building the Molecular Diagnostics (MDs) innovation system in India. In the recent episode of COVID-19 pandemic, MDs is considered as the gold standard test for the confirmation of the presence of SARS-COV-2 virus in the patients. It is emerging as one of the most dynamic and transformative areas of diagnostics technology, that has led to advances in research which is revolutionizing healthcare across a wide range of diseases and health conditions. The role of scientific institutions is crucial for the production of the knowledge that serves as a direct source of ideas for any technological development in the society. The knowledge base developed within the scientific community can effectively enable the strategies of applied research for the development and diffusion of the technology within the system. The scientific institutions are mainly constituted by the State-supported universities and research institutes which leads to technological discovery and facilitates new knowledge production that serve the foundation for any technological innovation. The new knowledge creation for the development of technology and especially concerning for human health is a complex task that requires dedication and demands time. In such a scenario the role of the government to support scientific institutions and their system building activities becomes important to (i) facilitate funding for the development of human resource & new idea generation, and (ii) maintaining the continuity of financial support till the desired outcome is received. At present, the MDs innovation system in India is 70-80 percent import dependent which is characterized by the two major context-specific issues i) Unaffordability, as imported technologies are highly-priced, ii) inaccessibility due to unsuitability of imported technologies for resource-poor healthcare settings. Concerning the context-specific issues and present episode of the covid-19 pandemic, the expectations for the development of MDs innovation system in India are enormous i.e., on one hand, it has to deal with the detection challenges associated with the increasing incidence of corona virus itself and already existing highly burdened infectious and lifestyle diseases like TB, Malaria, cancer, diabetes etc. On the other hand, the technology has to deal with the issues of availability, affordability and accessibility. In such a scenario, the argument of the present study drives towards the development of need-based technological innovation system for MDs development. 2. Objectives Considering the fact that the role of scientific institutions is significant for the development of MDs innovation system and continuous support from the Government is critical in facilitation of the system building activities of scientific institutions, the central objectives of the study are (i) to analyze the role and conducts of Scientific Institutions in the development of MDs innovation system (ii) to analyze the role of the Indian government in mobilizing the Scientific Institutions, and, (iii) to analyze the factors that are hindering the pathways of development.

3. Methodology

Since the development and diffusion of MDs innovation system has to deal with the context specific challenges, the study uses the Socially Responsible Innovation System Approach (SRISA) framework. The SRISA framework would be driven by combining the two frameworks of the innovation studies:

First, the Technological Innovation System (TIS) Framework (Carlsson and Stankiewicz,1991; Bergek et al., 2008) that will help in identifying the innovation actors and institutions and their functions involved in the development of MDs innovation system. Second, the System Failure Framework (SFF) (Klein-Woolthuis et al.,2005; Van Mierlo et al., 2010) to identify the constraints and challenges faced by the innovation actors and institutions in building for the development of MDs innovation system.

The study uses mixed method that is quantitative as well as qualitative. Following are the steps involved (i) Identification of the potential actors involved in MDs innovation system development (ii) Mapping of the system building activities of these innovation actors using secondary literature like published research articles, national reports, government health policy documents/proceedings, and databases like SCOPUS, Web of Science, PROWESS, and UN Comtrade. (iii) Development of questionnaires and interviews with the innovation actors that helped in development in gathering information on challenges encountered in the development and diffusion/adoption of new medical diagnostic technologies. 4. Results Study is an attempt to examine the System building activities of Indian Scientific Institutions in the development of the Molecular Diagnostic (MDs) Innovation System. Scientific Institutions are the precursor of any technological development with their capabilities in the generation of new ideas. MDs is an advanced and accurate diagnostic technology that has considerable scope to serve the diagnostic needs and requirements of the healthcare system. System framework helped in analysing the System Building activities of scientific institutions involved in the constitution of the science base for the development of MDs technology. The system performance was evaluated in terms of the Technological Innovation System (TIS) functions and the systematic challenges are assessed through the System failure framework. Based on the secondary and primary survey of major science base actors, the study finds that the State is playing an important role through mobilization of resources in the facilitation of knowledge production and contributes significantly to the development of MDs technology. Knowledge production has got significant momentum in the last decade with the development of specialized human resources and the establishment of dedicated research institutes. However, the innovative capabilities in terms of attaining need-based TIS are found to be sub-optimal pertaining to the specific diagnostic needs of highly burdened diseases in India. The system analysis reveals that the TIS functions are underperforming because of the absence of a well-defined funding mechanism and goal-oriented targeted policy regime of the State. Since MDs have a transformative effect on the present healthcare diagnostic system, State has to address the system-based challenges and issues for developing a need-based technological innovation system for MDs in the country. 5. Significance of the Study

Recent pandemic of COVID-19 has alarmed the Government, scientific community and policy makers in the world to develop the technological capabilities in order to attain the stage of self-reliant to deal with the severe consequences of the disease. The accuracy of the treatment is majorly depending on the diagnosis of the disease, the availability of diagnostic test is the major concern for the countries in order to identify the infected population and to start the treatment. In such a scenario the study would be a significant contribution for the creation of knowledge base in the development of MDs innovation system. The study will help in the identification of both strengths and weaknesses in the existing innovation system for MDs innovation system and will help in recommendation for the development of policy in building a context specific National Innovation System for MDs in India.

Engaging learned societies in promotion of open science and responsible research

ABSTRACT. Background and rationale

Since the 17th century, learned societies - voluntary non-profit organizations involving academics - have existed for the advancement of scholarship, research, disciplines, publishing and public understanding of science. Publishing scientific journals and books has traditionally been an important part of the activities of learned societies but they also take other various activities including arranging conferences, supporting research, and popularizing knowledge.

The most well-known learned societies are the national science academies, in which membership is typically based on invitation and merit. However, there also exists a much broader archipelago of local, national and international societies, whose membership is open to all academics, and often also to interested professionals and citizens. In English-language literature, a learned society can also be referred to as a learned/scholarly/scientific association or scholarly society. Occasionally, the term professional society is also used. We have estimated in a working-paper that there are around 9000 learned societies in Europe (

In historical perspective, learned societies constitute, along with universities and other kinds of research performing organizations, the foundation of contemporary academia. International Survey on Research Integrity (IRIS: provides the broadest available indication of the continued relevance of learned societies. Over 82% of 60,885 active researchers from Europe, United States and other countries, who in 2021 answered the IRIS survey question concerning professional societies, identified at least “a little bit” with the societies, and 62% identified “moderately”, “a lot” or “a great deal”.

Yet, surprisingly little information and research exists on their current number, forms of organizations and operation, or contributions to scientific and societal impact of research in specific countries, regions or globally. While more attention has been paid to the obstacles and challenges for promoting open access in the landscape of learned society journals, next to nothing is known or said about the (potential) role of learned societies in the area of responsible research, including research integrity and research assessment. However, we argue that these societies create networks of scholars and professionals and have discipline specific expertise on scholarly activities that should be exploited more.

In this paper we address the following questions: 1. What role does open science and responsible research play in societies’ activities? 2. Do the members expect open science and responsible research to be part of the societies’ activities?

To answer these questions, we would need a global review of learned societies, their activities and membership. The scope of our present study is mainly focused on Finland, from which we have a comprehensive report in 2019 ( Besides Finland, one survey study has covered learned societies in Portugal, and another one social science societies in the UK.

Methods and data

We provide data from Finland and from Finnish societies. Currently, almost 300 societies are members of the Federation of Finnish Learned Societies (TSV), and they have more than 250,000 individual members (Finland has a population of 5,5 million). Recently TSV collected two broad datasets from its member societies and the individual members of these societies.The first dataset (N=114 unique societies) concerns the actions related with responsible research of learned societies. The second dataset (N=1540 individual members of learned societies) concern the membership of the societies. By integrating these two datasets we are able to provide insight about the open science and responsible research activities from the viewpoint of the societies and their membership.


Over 94% of 114 learned societies in Finland, who answered the survey in 2021, indicated that the promotion of scientific activities is a major part of their activities, followed by the promotion of general understanding of science and societal impact. Generally speaking, the promotion of responsible research and open science has not yet become a key part of learned societies’ operations. Yet around 70 per cent of the societies participate in developing open science, 60 per cent publish immediately openly available publications. However, activities linked to research integrity, open data, open education and citizen science are rarer.

Nevertheless, the majority of societies were interested in developing their activities in one or more areas of responsible research: identifying and proposing experts to working groups, committees or as evaluators (65 % of societies), promoting open science and organizing events (more than 50 %), evaluating the quality of research (more than 40 %), assessment of social impact, science education and research ethics (approx. 30 %).

The membership of the learned societies regards traditional forms of operation, such as networking, publications, events and the popularization of science as the most important parts of the societies’ operations. Over 90 per cent of the respondents agreed at least partially with the statement that the promotion of the openness of research-based knowledge is important, and more than 80 per cent agreed that this should be free of charge to the readers. Almost 90 percent of the respondents agreed at least partially with the statement that the promotion of research integrity should be an important task of societies, and almost 80% agreed also on the importance of responsible assessment of researchers and research quality. Almost 60% agreed on the promotion of citizen science, while 45% agreed that producing open learning materials is an important task of societies.


Our case study in Finland shows that societies are active in the most traditional forms of open science, such as open access publishing. Publication activities are indeed one of the learned societies’ key forms of operation, and a significant proportion of learned societies’ publications already complies with the principles of open science. The societies have been important enablers and promoters of open, non-profit and scientific publication activities without author fees (so called Diamond OA) in Finland. Other forms of open science and responsible research are not yet established but highly supported by the members of the societies. The perceptions from the membership should guide the future development of the societies.

Data on society memberships shows that learned societies represent the science community on a broad scale in terms of age, field of education, work organization and place of residence. The networks created by the societies are cross-organizational and often international and provide a unique resource for identifying experts of different domains and creating collaboration between the domains. Thus, societies have a great potential in implementing and developing field-specific open science practices through engagement of researchers at the grass-root level towards a more responsible direction. Policy makers, research performing and funding organizations could develop more strategic collaboration with learned societies as a valued intermediary between science and society.

13:15-14:45 Session 3A: Research Careers and Institutional Types
Institutional reproduction of intersectional inequalities in science

ABSTRACT. Introduction Universities play a key role in shaping the social structure in which scientific activities are performed. Wapman et al. (2022) showed that graduates from the 20% most U.S. prestigious universities occupied 80% of the faculty positions in the country. This is due to the fact that authors from top universities benefit from an environment that gives them higher productivity and recognition (Way et al. 2019). Such inequalities at the individual level influence the research priorities of the country, as it allows those top institutions to have a fundamental role in setting the research priorities within the scientific community (Clauset et al. 2015). In tandem, research has shown a strong alignment between an author’s sociodemographic characteristics and the topics they study (Kozlowski et al. 2022a). However, little research has studied the intersection of these intersectional inequalities and how they might be mediated through institutional prestige. In this project, we focus on how prestige relates with the socio-demographic representation of authors, of research topics and research impact. Also, we will analyze the role of mission-driven institutions, such as Historically Black Colleges and Universities (HBCU) and Women’s Colleges (WC). Two main research questions drive the analysis. First, we want to assess whether those types of institutions have a specific topical profile, and how it varies as a function of authors’ intersectional identities. Second, while it has been widely studied that institutional prestige drives impact (Way et al. 2019), it is important to acknowledge how this relates with race and gender inequality. How, if at all, does the citation bias against marginalized groups—women, Black and Latinx authors— vary by the prestige of the institutions in which they work?

Methods To answer these questions, we use more than 4.5 million articles from more than 800 US universities published between 2008 and 2020 and indexed in the Web of Science (WOS). Following the method developed by Kozlowski, Murray et al. (2022b), authors of the selected papers were assigned a probability over each racial group based on the association between their family names and racial categories found in the 2010 US Census (USBC 2016). Gender was inferred using authors' given names, based on Larivière et al. (2013). We consider a researcher's identity as the combination of four racial categories —Black, Latinx, Asian, White—and gender, considered in a binary way. Given the limitations of the data and inference algorithms, it is not possible to automatically classify authors into Census categories “Native Americans” and “Two or more races”, and gender outside the women-men binary, which further highlights the importance of studied based on surveys and self-identification of authors as complements to large-scale secondary analytical studies. We used historical WOS data to compute the average number of citations of US universities between 1980 and 2019. We use three different classifications as complementary indicators of prestige, as proxies for perceived prestige (US News & World Report), research prestige (average citations) and selectivity prestige (Carnegie Selectivity Index). Each of these three rankings is divided into three groups: ‘high,’ ‘middle’ and ‘low’. We also consider mission-driven classifications: HBCU and WC. We used topic modeling (Blei et al. 2003) to infer the research topics of articles based on their titles, abstracts and keywords. We define the topical profile of a group as the proportion of papers this group contributes on each topic with respect to the total number of publications in the topic. Topical profile can be applied both to groups of researchers' identities and institutional categories. To compare two topical profiles, we use the Spearman rank correlation, as the relation between topical profiles is not linear. We focus on the correlation between the topical profiles of institutional categories and authors' identities. If the correlation between those groups is high, it means that they tend to publish on similar topics. We also build a linear model to predict the effect of author's identities on impact (citations and Impact Factor). By splitting articles by their institutional prestige groups and running the linear model for each, we show the differential effect of race and gender by institutional prestige.

Results and discussion There is an increasing debate around the role of elite institutions in the reproduction of systemic inequalities in science (Clauset et al. 2015; Wapman et al. 2022). These institutions do not only have a higher accumulated economic capital—as exemplified by their endowment—but also of symbolic capital. A handful of institutions produce a large proportion of research articles, and those attract an even larger impact (Way et al. 2019). This study provided an intersectional perspective on the role of institutions. It provided evidence that institutions are key to the stratification process of science and that, across different institutional tiers, the experiences of racialized and women authors is qualitatively different from that of White men. Aligned with their missions, HBCUs and WCs occupy a topical space that is closely related with that of Black and Latinx, and White women (respectively). In previous works we found that thesetopical profiles relate with the lived experiences of these populations, and have a larger focus on issues such as racial discrimination, migration, and gender-based violence (Kozlowski et al. 2022a). However, as we climb the ladder of institutional prestige, we observe a sharp decline of participation in those topics. Far from being a simple consequence of the composition of their faculty, this reflects on a more nuanced phenomenon: authors from marginalized groups in top institutions have a topical profile that differs from the one of their institutions, but also differs from that of their own identity in other institutions. Top institutions are failing to leverage topics that are relevant to marginalized populations. We have also found that the impact gap by race and gender is affected by institutions. Articles from top institutions gather more citations and are published in higher impact journals. But within institutions the gap persists, and the highest impact gap is found on top institutions, even after controlling by topic and other relevant covariables. These results demonstrate the strong role that institutions play in shaping the research profile of a nation. Initiatives that serve to diversify hiring may serve to expand the topical profile of an institution, but only if other mechanisms are in place to ensure that scholars are free to pursue the full landscape of research questions. Our work suggests a penalty for this deviation that is borne particularly by minoritized populations.

Cited References

Blei DM, Ng AY, Jordan MI. 2003. Latent dirichlet allocation. J. Mach. Learn. Res. 3(null):993–1022 Clauset A, Arbesman S, Larremore DB. 2015. Systematic inequality and hierarchy in faculty hiring networks. Sci. Adv. 1(1):e1400005 Kozlowski D, Larivière V, Sugimoto CR, Monroe-White T. 2022a. Intersectional inequalities in science. Proc. Natl. Acad. Sci. 119(2):e2113067119 Kozlowski D, Murray DS, Bell A, Hulsey W, Larivière V, et al. 2022b. Avoiding bias when inferring race using name-based approaches. PLOS ONE. 17(3):e0264270 Larivière V, Ni C, Gingras Y, Cronin B, Sugimoto CR. 2013. Bibliometrics: Global gender disparities in science. Nature. 504(7479):211–13 USBC UCB. 2016. Frequently Occurring Surnames from the 2010 Census. Wapman KH, Zhang S, Clauset A, Larremore DB. 2022. Quantifying hierarchy and dynamics in US faculty hiring and retention. Nature. 610(7930):120–27 Way SF, Morgan AC, Larremore DB, Clauset A. 2019. Productivity, prominence, and the effects of academic environment. Proc. Natl. Acad. Sci. 116(22):10729–33

Labor advantages drive the greater productivity of faculty at elite universities

ABSTRACT. Researchers at elite universities tend to dominate scientific discourse, with higher productivity and far greater prominence, and thus shapes the direction and pace of scientific discovery as a whole. Such dramatic disparities could be due to simple meritocratic sorting, reflecting genuine differences in individual scientific ability, but they could also reflect non-meritocratic factors like luck and structural advantages. Understanding which mechanisms drive this prestige-productivity pattern would shed critical new light on efforts to accelerate scientific discovery, and inform policies aimed at broadening participation.

Past work has shown that among early-career faculty researchers, placement at a more prestigious institution can cause greater productivity and prominence. That is, where you work is the greater determinant of your impact than where you trained. However, the particular mechanism through which prestige underlies this pattern remains unknown. Through multiple lines of evidence, we show that differences in available funded non-faculty scientific labor drive substantial prestige-productivity inequalities, and the scientific dominance of elite universities can be explained by their substantial labor advantage over researchers at less prestigious institutions, primarily in disciplines where faculty lead and collaborate with a research group. Our analysis leverages cross-disciplinary, longitudinal data on the education, employment, and publications of 78,802 tenured or tenure-track faculty spanning 4,492 departments across 25 disciplines in science, engineering, and the social sciences at 292 PhD-granting U.S.-based universities, over the period 2011--2017, which we combine with researcher-level productivity data encompassing 1.6 million publications from the Web of Science. We complement these data with institution-discipline-level counts of graduate and postgraduate (non-faculty) researchers, institutional covariates, and discipline-specific measures of prestige.

First, we show that faculty's annual productivity, measured crudely as their mean publications per year, increases substantially with environmental prestige, with elite researchers being roughly twice as productive as researchers at the least prestigious institutions. We isolate the component of total productivity that could be driven by differences in labor by partitioning each faculty's ``total'' productivity into two sources: (i)~group productivity (publications coauthored with non-faculty research group members), and (ii)~individual productivity (all other publications). In disciplines with group collaboration norms, a larger group will tend to drive greater group productivity, but not in disciplines without such norms. We show that in such disciplines, group productivity is both substantial and grows systematically with prestige, even as the individual group members are no more or less productive. Finally, we show that research labor is highly concentrated within the most prestigious environments, indicating that elite researchers tend to have larger research groups.

We then test this ``labor advantage'' hypothesis using a series of predictive models, showing that funded labor consistently plays a significant role in predicting productivity and group sizes in disciplines with research group collaboration norms, but not in disciplines that lack these norms. Next, using a matching experiment on mid-career moves, we show that faculty who move to an environment with more available funded labor tend to have groups that are significantly larger after the move than those who go to environments with less labor. Finally, we quantify a systematic relationship between larger faculty group sizes and greater group productivity that is independent of prestige.

Taken together, these results identify the environmental mechanism by which prestige drives greater scientific productivity, and show that it is the profound labor advantage of elite working environments that allows their scientists to dominate scientific discourse. More broadly, our findings suggest relatively simple interventions for both increasing scientific productivity and increasing the diversity of scientific advances. Accounting for the non-meritocratic effects of research environments will be an important component in developing predictive theories of knowledge production.

Faculty Research Productivity in Striving Research Universities

ABSTRACT. Background and Rationale Over the last forty years, many institutions in the U.S. higher education ecosystem have changed their institutional strategy and/or identity to place a greater emphasis on research and graduate education (O’Meara, 2007). This widespread phenomenon, often termed “striving,” has occurred for many reasons, most notably via strategic efforts by academic leaders to enhance institutional prestige and ensure long-term stability and prosperity (Jaquette, 2013). Although striving behavior can occur at any institution, this paper looks specifically at the case of institutions seeking to become or advance their competitiveness within the ranks of doctoral research universities – the most prestigious and research-oriented of American universities. Importantly, while the striving phenomenon has been examined by a variety of scholars, often overlooked are the experiences of tenure-track faculty, the very people who are expected to increase the research profile of their institution (Gonzales et al., 2014; Jackson et al., 2017; O’Meara & Bloomgarden, 2011). This study examines the differential experiences of faculty at striving research universities to better understand the patterns of workload and academic research productivity that emerge in these transitioning institutions. Faculty at striving institutions are typically expected to enhance their research programmes in furtherance of institutional goals. However, the degree of institutional support and relief from other job duties (teaching and service) they receive in furtherance of these goals varies considerably, sometimes creating intractable time conflicts (Sabagh et al., 2022). Striving institutions also broadly experience a transitional phase in their internal norms, expectations, and logics (Morphew et al., 2018; Morphew & Huisman, 2002), which can lead to ambiguity in evaluation (Jackson et al., 2017; Véliz & Gardner, 2019). Critically, support and reward systems for research are weaker at striving research universities than at established R1 research universities (Griffith & Altinay, 2020; O’Connor et al., 2011). This results in a complex system of institutional policies that both hinder and support faculty research productivity at striving research universities – one which is likely to have disparate effects on faculty at different career stages and from underrepresented groups. In the broader context of studies of academic research productivity in individual faculty, examining striving research universities helps address an important theoretical gap. Most or all of the major inputs into research productivity have been identified by previous studies (Bland et al., 2005, 2007; Toutkoushian & Bellas, 1999). These include individual characteristics (e.g. intrinsic motivation and research skill), characteristics of the employing intuition of the researcher (e.g. rewards structures, clear goals and guidance, sufficient time allocation, lack of red tape, doctoral students, and support structures), and social characteristics (e.g. welcoming climate, peer support, peer expectations, and collaboration opportunities). Despite these many factors being well known, their interrelation is complex and remains ambiguous (Gläser et al., 2014). We do not know, for instance, how varying degrees of institutional support for research may disparately affect individuals. Another complication is that understanding the relationship of the factors which support research productivity is particularly challenging because they tend to largely co-occur, leading to multicollinearity, particularly for institutional factors (Bland et al., 2007). Academic work environments typically have either most or few of the characteristics supporting research productivity, with little in-between. In other words, different categories of institutions (e.g., liberal arts colleges) tend to have internally consistent bundles of research supporting and hindering policies, such as teaching loads, rewards structures, and graduate programs. Researchers have thus struggled to determine which institutional factors are most causally important for research productivity. This greatly complicates the interpretation of studies that compare colleges across intuitional categories, models, and niches. That is, studies with doctoral research, comprehensive, and primarily undergraduate institutions in their sample. Such studies, unsurprisingly, find that the average productivity of faculty varies across institutional type, due to the strong correlation of institutional type and the presence research-supporting characteristics. It has been known for some time that the strongest predictive factor of a random academic researcher’s productivity is the type of institution at which they work (Pellino et al., 1981). In other words, the research productivity of individual faculty tends towards the average research productivity of their employing institution. This, however, does little to enhance our theoretical understanding of research productivity. In particular, cross-sectional comparison creates a risk of conflating the distinctive characteristics of these institutional categories that are cause variation in research productivity with those that are unrelated. That is the core motivation to study research productivity in the striving university setting. Because these universities are undergoing change, we can observe variation in institutional support for research within a broadly similar category of research institutions, particularly in teaching loads. Striving institutions have unusual combinations of characteristics that are both supportive and deleterious for research productivity, due their transitional nature. This will hopefully lead to a greater understanding of which factors are most important for supporting research productivity. To examine these complex issues, we ask two questions: 1.) In a striving research university environment, how do individual characteristics and institutional factors interact to enhance and inhibit faculty research productivity? 2.) In a striving research university environment, what is the precise relationship between faculty time allocation and research productivity?

Methods This study uses a subset of approximately 800 faculty working at 117 striving research universities from a 2011 national survey of tenure-track faculty in the U.S. The dataset provides a reasonably large sample size along with detailed individual characteristics, work histories, job duties and time allocations, work psychology measurements, and self-reported measures of research productivity. Striving research universities were identified as those that entered into the Research 1 or Research 2 Carnegie Classifications between 2010 and 2021, after having previously been in a lower classification. The survey data were analyzed using descriptive statistics and a combination of regression techniques, including OLS and count regression.

Anticipated Results Although there are many interrelated factors, preliminary findings suggest: First, faculty interest in research is varied and an important factor in research productivity. Next, teaching duties have a more negative impact on research productivity than comparable service loads. Additionally, there is variation in research productivity by race and sex not explained by other factors in the model. Finally, there are diminishing marginal returns to time allocation to research, particularly when research comprises the majority of a tenure-track faculty member’s work. More findings are likely to emerge as this study is refined further.

Significance This research helps advance our understanding of the relationship of the many factors that influence research productivity and may inform the decisions of academic leaders and policymakers. Better understanding of the connections between the mechanisms by which research productivity operates could help improve the effectiveness and equity of striving institutions’ strategic initiatives. This is particularly relevant as long-term downward trends in both domestic and international enrollment at U.S. colleges suggest that the financial pressures which have helped spur striving behavior are likely to continue. In addition, these insights could help identify ways to assist faculty build research capacity more broadly.

The value of competitive hires within institutions under audit: A global comparison of university departments in the UK, Italy, Australia, New Zealand, Norway and Hong Kong

ABSTRACT. The realities of periodic research audit in operation in many research-intensive countries means that HEIs are incentivized to implement policies and mobilize their research workforce towards achieving a high return within these evaluations. This preliminary research uses a global comparative approach using HEI case studies in the United Kingdom, Australia, Norway, New Zealand, Hong Kong and Italy to investigate the productivity and visibility benefit of a competitive hires to university departments. Competitive hires are identified as a mobility event into a department two years prior to an evaluation event (T1), and their value to the department (the extent that the competitive hire was successful) is determined by comparing their productivity and visibility value three years post hire (T2) when the evaluation event is complete. Particular attention was paid to the value-added by international, versus locally based hires, especially in countries with a research workforce that is internationally diverse (UK, Australia & New Zealand). Results show an interesting interplay and conceptualization of the potential benefit of hiring individuals to a department that extend to a consideration of their wider networks. These results provide alternative considerations for research recruitment policies, as well as extend current thinking of the value of an individual vis-à-vis their wider research capacity and existing network.

13:15-14:45 Session 3B: Institutions and Science
Explaining growth and decline of academic field at public universities.

ABSTRACT. This paper investigates structural change in the field of public universities since the mid-1990s with a focus on the potential influence of factors related to the growth and decline of academic fields. Studies of change in disciplinary structures at universities that both span decades and consist of sufficiently large samples have so far mainly been conducted in the United States, where in the period between 1971 and 2006 colleges and universities implemented cuts to subjects in the humanities and natural sciences, while the applied and professionally oriented subjects grew strongly.

To explain the dynamics in the development of academic fields at universities (and colleges), a number of different theoretical approaches have been applied in the literature, including the “world polity” perspective of sociological neo-institutionalism, “population ecology” and the “theory of institutional stratification”. We refer to these three approaches in our analysis and gauge to what extent they have explanatory power, using the field of public universities in Germany (1995-2018) as an empirical example.

Studies are often based on comparisons of the shares of disciplines in a sample of universities. Growing (or declining) shares of a discipline are interpreted as relative growth (or dismantling) of that subject field. By contrast, our analysis not only considers shares, but draws on the Relative Specialization Index (RESP), which enables the disciplinary profiles of individual universities to be characterized. We compute the RESP for the following variables: academic staff (professors and non-professorial scientific staff), basic public funding, and grant funding. Our analysis differentiates between technical universities (TUs) and non-technical universities (NTUs).

Results at the level of all German universities show a more dynamic development of NTUs compared with TUs. Inversely, the proportion of shrinking universities is higher for TUs than for NTUs. We confirm that the humanities are somewhat in decline, and we find a shrinking of the natural sciences at NTUs as well as growth in the social sciences in all universities.

Based on these descriptive findings, we investigate whether a university‘s growth depends on its particular disciplinary profile, size, age or prestige. Using statistical tests, our results suggest that smaller and younger universities are more likely to exhibit an academic profile comprising growing subject fields, while larger and older universities more often embrace declining fields. Furthermore, we show that NTUs funded by Germany‘s Excellence Initiative (2005-2018) tend to have an above-average proportion of subject fields in the humanities and natural sciences, while the universities that were not funded by this program are more likely to have above-average proportions in engineering and the social sciences.

Our empirical results suggest that the decline in the humanities and the natural sciences as postulated by neo-institutional theory can only be partially confirmed: in the case of NTUs, the decline in the humanities is seen mainly in the numbers of professors, but not conclusively in terms of other variables. By contrast, decline in the natural sciences is observed for NTUs, but not for TUs. Strongest empirical support for neo-institutional theory comes for the social sciences, which exhibit growth over the entire observation period and for all variables.

By comparison, the population ecology approach receives consistent empirical support, since it is primarily the larger and older universities – in contrast to the smaller and younger ones – that appear to have the carrying capacities for shrinking academic fields. Furthermore, the younger and smaller universities follow the developments in their environment more closely, most visibly in the social sciences, which are generally expanding.

Our results also provide some support for the theory of institutional stratification, as the natural sciences – which are generally considered “high-status” basic subject areas – have clearly benefited from the Excellence Initiative. Conversely, universities with profiles in the social sciences – the “low-status” disciplines – were less successful in the Excellence Initiative. In other words: universities with “high status” disciplinary profiles had much better chance of receiving Excellence funding than those with more “low status” disciplines.

The paper concludes with a discussion on growth and decline of subject fields more generally, and therein a brief comparison between higher education institutions in Europe and the United States.

Scholarly publishing at US federally funded research & development laboratories: influences on public-private science

ABSTRACT. Background Government investment in basic research produces benefits that counteract the forces that lead the private sector to underinvest in such research (Rosenberg, 2009). Governments differ in how they organize publicly funded scientific research. In general, this spending is divided between funding distributed to performers outside of the government, such as universities and private firms (i.e., extramural R&D), and funding to government research facilities (intramural R&D). Intramural research facilities, often termed “public research institutes” or PRIs, are assumed to undertake research directly relevant to governmental needs that might not be pursued by extramural performers. However, few studies of PRIs exist in the published literature. The OECD launched a study in 1989 on government research laboratories, with a follow-up study conducted in 2011 (Sanz-Menéndez et al., 2011). In the United States, Bozeman and Crow (1998) published a series of examinations of federal government research laboratories. Cruz-Castro et al. (2020) and Zacharewicz et al. (2017) produced a series of scholarly works looking at organizational and managerial issues in public research organizations, and Hallonsten (2017) recently conducted a review of what he terms the “third sector of R&D” (encompassing government research laboratories but including non-governmental research institutes).

This study focuses on a specific form of PRI in the United States: nationally owned research laboratories operated as federally funded research and development centers (FFRDCs). FFRDCs evolved from their origin in World War II to constitute a significant part of the U.S. public R&D enterprise, accounting for approximately 36 percent of the research performed at federal facilities in 2018. The FFRDC is an organizational form different from most PRIs in other nations. These institutions are typically formed and owned by the U.S. federal government, but operated by academic, industrial, and special-purpose entities under bespoke agreements with specific agencies (known as the FFRDC’s “sponsor”). As a result, these PRIs straddle the boundary between intramural and extramural government laboratories. They retain the status of a governmental entity, but with many of the attributes of a non-governmental organization. The current study is limited to a specific subset of FFRDCs, the Research and Development Laboratories (RDLs). Compared to other types of FFRDCs (Study and Analysis Centers and Systems Engineering and Integration Centers), RDLs are major producers of multi-disciplinary scientific knowledge (Hruby et al., 2011).

  Key Research Questions Our paper presents descriptive statistics about scientific output from 1981 through 2020. We analyze the publications for all Research and Development Laboratories (RDLs) and for specific subsets of the RDLs (such as laboratories functioning primarily as scientific user facilities) by field of science. We characterize the scientific impact of that output using measures such as the average of relative citations. We also analyze patterns of collaborative research at RDLs (based on co-authorship networks across all scientific papers). Specifically, we examine the following sets of research questions:

RQ1 How has the production of scientific knowledge at RDLs (measured as the number of scholarly articles) varied relative to overall federal basic research funding to FFRDCs? What differences can we observe in scientific productivity and impact at specific RDLs, especially between user facilities and multidisciplinary research centers?

RQ2 As US federal entities, RDLs are expected to focus on national priorities, and their R&D activities are expected to benefit the US science and technology enterprise. We can characterize the “consumption” of RDL input by analyzing the institutional affiliations of authors who cite RDL publications. How does the citation of RDL publications vary across the nations and types of institutions citing those works?

RQ3 US federal R&D funding is concentrated in mission-oriented agencies, such as the Department of Defense, Department of Health and Human Services, and NASA. How has the disciplinary focus of RDLs shifted in relation to the R&D budget priorities of their sponsoring agencies?

Data and Methods This research-in-progress analyzes the place of FFRDCs in the US basic research landscape, primarily through bibliometric analysis of the publication output of the 29 RDLs as designated by the National Center for Science and Engineering Statistics (NCSES). Data for this paper was obtained from the Web of Science (WOS) database, covering the Science Citation Index Expanded, the Social Sciences Citation Index, and the Arts and Humanities Citation Index, over the 1973-2021 period. The name variations of each FFRDC analyzed were retrieved from the institution and department fields of the WOS. Each paper was assigned the subfield of the journal in which it is published according to the NSF field and subfield classification of journals (Hamilton, 2003), and this classification scheme was used for the field-normalization of their number of citations (Waltman et al., 2011). Full counting of papers was used—each FFRDC contributing to a paper obtained a whole unit. Networks were created with the UCINET and Netdraw softwares (Borgatti, 2002; Borgatti, Everett & Freeman, 2002).

We also obtained public-use data files and other outputs from two NCSES surveys: the annual FFRDC R&D Survey (the FFRDC Survey), and the Survey of Federal Funds for Research and Development (the Federal Funds Survey). The FFRDC Survey captures measures of R&D spending, and some data on the sources of funding for those R&D expenditures. The Federal Funds survey provides data on overall R&D obligations by specific federal agencies by field of science and engineering, and also R&D obligations to individual FFRDCs.

We use the FFRDC and Federal Funds Surveys to provide information on the resources (funding obligations) and activities (expenditures) for R&D at each RDL, and for their primarily federal government research sponsors. We then set up various models to derive findings on how the FFRDCs’ scientific output changes over time in response to shifts in federal agency R&D priorities and in the nature of research conducted at these facilities.

Significance Link and Scott (2021) conducted an analysis of RDL publications for eight laboratories, and used their data to estimate the elasticity in the production of scientific output for those laboratories (the number of additional publications generated from incremental budget increases). Our work goes beyond this foundation by providing a more comprehensive dataset of publications over a longer period of time, so we can detect long-term trends. We also are able to investigate more thoroughly how publication volume and patterns vary based on the nature of the laboratory and the distribution of R&D funding received by agency and field. We will present our findings showing that the RDLs, in general, produce publications with greater impact than overall U.S.-authored publications (using citation measures). We also show that RDLs are heavily engaged in international collaboration, and that their knowledge outputs are also consumed significantly by nations other than the United States. We also estimate how RDLs, consistent with the literature on PRIs, change their scientific output in response to changes in the R&D priorities of their sponsors and of the U.S. national government.

University scientific coproduction becomes more social in crisis

ABSTRACT. See file attached.

Imprinting or Learning? Charting the Organizational Evolution of Scientific Laboratories

ABSTRACT. 1. Question

How do scientific laboratories vary in how they operate, and do they change over time? A legacy of scholarship in the study of scientific productivity has established the demographic, career, and resource factors that explain scientific production. A notable pattern in this work is a preeminent focus on individual investigators.

Whereas some scholars underscore that this focus on individual investigators is an artifact of the availability of data, an emphasis on individuals is especially problematic in the context of scientific productivity because labs are known to exhibit considerable heterogeneity. Science and technology studies demonstrates that how labs assemble people, technologies, and resources varies substantially and meaningfully in ways that are consequential to productivity and innovation. Furthermore, organizational studies further attests to the fact that organizational heterogeneity is significant for explaining differences in performance, innovation, and turnover. Organizations embrace ideologies, logics, “blueprints,” or models for organizing which is instantiated in how they operate and can guide subsequent organizational evolution.

Therefore, we ask, how do lab models vary among scientific labs across organizations and over time?

2. Theoretical Framework

From the literature on organizational emergence, persistence, and change, we establish two tenets on how organizations evolve. First, organizations are reflections of their time and are imprinted with the characteristics at the time of their founding. In this view, the features adopted by organizations during founding are highly stable and persist over long periods of time. Second, organizations can learn, adapting to new experiences or developing new forms of knowledge through sensemaking and the enactment of new patterns of cognitive associations. In this view, organizations may change with the experience of success and failure, changes in networks, the incorporation of new information, ambiguity, or by random chance.

Drawing on and comparing the literatures on organizational imprinting and organizational learning, we compare two competing sets of propositions, notably:

1. Imprinting: Scientific labs express an organizational model established at time of founding that persists over time. 2.Learning: Scientific labs express an organizational model that changes and adapts over time that increasingly differs from its time of founding.

We evaluate these competing propositions in the field of academic scientific labs across over 30 American universities using a combination of statistical models and machine learning.

3. Data

The data we use to examine the organization of scientific labs comes from the Institute for Research on Innovation & Science (IRIS) UMETRICS (Universities: Measuring the Impacts of Research on Innovation, Competitiveness and Science) research dataset. The 2020 UMETRICS release consists of data provided by 33 universities over a time frame of 1 to 19 years. UMETRICS captures expenditure data, including data on awards, wage payments to individuals, and vendor transactions resolved at the month level.

To identify labs, we associate each faculty member to his/her/their set of awards. For each faculty member, we generate nonoverlapping, 1-year windows that represent “snapshots” derived from monthly expenditures. For each faculty-window, we construct manifest variables derived from expenditure activity. These measures provide a window into the variety of ways labs search for support, spend their money, and assemble personnel. As such, we use these measures to identify lab models.

4. Methods

To characterize the heterogeneity of lab models, we exploit two independent clustering methods: a model-based method, latent class analysis (LCA), and an unsupervised machine learning method, k-medoid clustering based on Gower’s distance. A fundamental analytical decision involves specifying the appropriate number of clusters to represent the data. Thus, we utilize both clustering strategies to assess the extent to which clusters derived from both methods overlap. To assess the extent to which lab models persist or change over time, we sort lab windows into clusters and track the trajectory of a lab across its windows over time.

5. Results

5-1. Lab Models

Our interpretation of the latent classes and a k-medoid clustering analysis of a sample of the UMETRICS data suggest a 6-class solution represents the most parsimonious solution.

Most labs employ what we term a small venture lab model: these labs are unexceptional when it comes to expenditures, number of federally funded projects, and personnel, yet nonetheless make use of competitive federal funds to conduct research.

In contrast, a subset of labs employs models that are characterized by relatively big spending and the lion’s share of federal awards. Two of these have a strong association with medical schools and include the Medical Research Facility and the Specialized Medical Boutique: the former is notable for its employment of postgraduate research staff, while the latter is notable for being helmed by an English PI. The Productive Powerhouse, while associated with a medical school, holds multiple administrative affiliations across the university. It employs relatively more graduate and postgraduate employees, and these employees are constituted by relatively more women, ethnically English individuals, and individuals on multiple awards (i.e., their funding is split across different award accounts).

The two remaining lab models are unexceptional when it comes to expenditures and awards yet pull their weight in the employment of graduate students. The Engineering Factory is a leaner version of the Productive Powerhouse and is strongly associated with a school of engineering. The Doctoral Factory is associated with a greater number of graduate employees and graduate employees with split funding.

5-2. Lab Models Over Time

To characterize the persistence and change of lab models over time, we estimate the posterior probability that a particular faculty-year “belongs” to a particular latent class and assign each to one and only one class. Next, we assemble each faculty-year for each faculty member (N = 64,733) to construct a chronological account of each faculty’s lab model.

We find that a slim majority of faculty employ the same lab model throughout the course of their career on record (27,565) compared to those who make at least one switch (25,114). However, the number of switches a faculty member incurs appears to correlate with the length of their recorded career. Furthermore, faculty who begin their careers with certain lab models are less likely to make a switch later in their career.

Thus, we find strong evidence for organizational imprinting, i.e., that scientific labs express a model established at time of founding that persists over time. However, we also find evidence for organizational learning, i.e., that scientific labs express a model that changes and adapts over time that increasingly differs from its time of founding.

6. Significance

Our analysis advances our understanding of scientific production and the organization of scientific laboratories in three ways. First, our study addresses the persistent call to study the evolution of organizational populations beyond the rarified set of Fortune 500 companies. Second, our study offers an operationalization for the organization of scientific labs that enhances the fit between our understanding of science as a social system with our actual measurement. While data platforms like UMETRICS improve our capacity to analyze the value of American research and development, our conceptual tools for utilizing such platforms in service of theorizing scientific production require attention. Finally, we shift the spotlight away from individual investigators to laboratories that propel the frontier of scientific knowledge, participate across multiple networks of activity, and produce graduates who go on to populate the scientific and engineering workforce.

13:15-14:45 Session 3C: Regions II
The Geographic Content of Research: What are the places that get studied?

ABSTRACT. Background and rationale

An increasing amount of works have studied the geographic dimension of research using publication data. Most of them traditionally rely on metadata associated with authors' affiliations, as they usually assign publications to the geographic location of their authors' institutions. Although this approach is useful to identify the places where scientific knowledge is produced and the disciplines places specialize in, it tells little about the location of the places that get studied. The latter is crucial, as there is a potential mismatch between the places authors work at and the places they write about; the places that fund research and the places that get direct benefits from it; and, more generally, between the social demand for research where it is more needed and researchers' supply where it gets higher individual rewards.

Multiple situations can lead to these mismatches. For example, an author located in a country, say Mexico, can focus her research on other countries (e.g., Sweden) and write no papers about Mexico because of data availability issues; simultaneously, researchers at the Nordic Africa Institute, a research center located in Uppsala (Sweden), are meant to write about Africa; research funded in China can translate into papers about the US if those papers are aimed at accessing top American journals; while scientists from all around the world would increasingly study Wuhan (China), the epicenter of COVID-19, as the pandemic became an issue of global interest; or authors may write about the places they like, where they have cultural, family or personal ties, even though they could work elsewhere. To understand all these cases, it seems necessary to look at the geographic content embodied in research works from a more systematic and general approach to go beyond publication and author metadata restrictions. This work tries to contribute to this first but necessary step: setting a general analytical framework and providing some first estimations of geographic content and geographic attention. Its main goal is to describe the places that get studied in absolute and relative terms.


This work proposes a new methodology to extract the geographic content of research, defined as the places that get studied in a document. First, using Natural Language Processing (NLP) models for Name Entity Recognition (NER) in 13 different languages (i.e., English, Japanese, German, Spanish, French, Chinese, Portuguese, Korean, Russian, Italian, Polish, Dutch, and Swedish), I extract the words that refer to geographic locations from titles and abstracts . As opposed to existing approaches that rely on lists of places or established administrative boundaries, an NLP-NER approach allows identifying words that refer to geographic places based on their context and their role in a sentence without imposing predefined classifications. Second, I match all these places with the GeoNames database to assign coordinate points and look closely into the characteristics of these places (e.g., type of place, population, among others).

I apply the proposed methodology to the whole universe of documents in Microsoft Academic Graph (MAG). In particular, to all the titles and abstracts, for all the publication types and disciplines, in 13 different languages. As a result, I created a database that links documents to places at multiple levels of analysis, which can be shared without license restrictions. I also provide statistics on how the relative geographic attention places receive distributes worldwide, disaggregating data by discipline, author's location, and language.


Disciplines have different propensities to refer to geographic places in their content. Although in the overall sample, the share of documents with geographic content is 15.46\%, thirteen of the nineteen considered academic fields have shares larger than 16\%. Not surprisingly, geographic content is highly present in geography, where it accounts for 63.44\% of the total documents in the field. The following three academic fields, by their prevalence of geographic content (history, political science, and economics), span from 49.52\% to 43.08\%. In contrast, for the three fields with less predominance of geographic content (chemistry, computer science, and materials science), geographic locations are only found in less than 5\% of their total documents.

In absolute terms, geographic attention is concentrated in the US and China across all academic fields, as measured by the number of papers about these countries. Regarding their relative share in the world's population, the US, Australia, and European countries receive disproportionate attention. Relative to their share in the global GDP, the US and Nordic countries still received higher-than-average attention, although results vary according to the studied academic field. In all cases, African countries seem to be the most neglected areas. Overall, geographic attention seems to align with a country's income more than its population's share. Hence, dollars of income seem to deserve more attention than individuals.

When combining geographic content with the authors' location or the documents' languages of publication, the above patterns still hold. Additionally, authors from a country tend to devote more attention to their neighboring countries and to the countries that share geopolitical, commercial, and historical ties. For example, authors located in France devote more-than-average attention to their former colonies than to studying other European countries, as measured by the relative distribution of documents.


The study of the geographic dimension of research has been reduced to the places where knowledge is produced. Understanding the places that get studied will open the door to new questions on the geographic distribution of research attention and the incentives to research a place. As there is an increasing awareness of the geographic concentration of research attention in a few countries in the Global North, geographic content measures can contribute to exploring other dimensions of this phenomenon.

Current efforts to identify the places studied in a document have important methodological issues and do not belong to a unified framework. Instead, they usually rely on small samples of documents written in English, focusing primarily on social and environmental sciences. Moreover, estimating geographic content measures requires significant computational resources that may not be accessible to many researchers. This work solves some of these technical challenges and provides a more comprehensive dataset, which could facilitate a more extensive exploration of geographic content.

For policymakers, the analysis of geographic content could reveal how research efforts translate into direct knowledge for the places they govern. Geographic content can complement current measures of research impact, as it can help monitor how much people care about particular geographies. Although increasing the number of documents about a place should not become the main objective of science policy, geographic content can contribute to identifying how actors perceive specific geographies. Moreover, it can help measure which places benefit from the research produced (and funded) by others.

Comparing states’ AI capabilities in governance and performances

ABSTRACT. As the economic value and technological importance of artificial intelligence (AI) have grown, measuring countries’ technological capabilities in AI becomes one of the most popular issues in the research field. However, while there have been some elaborating indices and measurements that can cover the wide range of the technology, most of these assessment systems heavily depend on traditional technology development measurements: scientific publications and patents. These two traditional measurements are beneficial to quantify technological development and capabilities, however, focusing too much on these measures can miss another critical part of AI development, AI governance. Given the impact of the technology, consideration of AI governance including ethical issues and accountability of the technology should constitute a state’s AI capabilities. This study aims to compare states’ AI governance preparedness and their rankings in performance-based assessment. To measure states’ AI governance, algorithmic impact assessment (AIA) tools and states’ AI-related legislation documents will be used. As “a tool for assessing possible societal impacts of an AI system before the system is in use,” AIA provides governance components of the technology. AI governance preparedness will be measured by applying AIA to a state’s AI-relation legislation documents. Because there is no one-size-fits-all AIA tool yet, we will combine the currently used AIA tools (Canadian algorithmic impact assessment model, IEEE’s AI Standards, UN Guiding Principles on Business and Human Rights, and European Commission’s High-level Expert Group on AI’s assessment list for trustworthy AI) using their shared components. After finalizing the critical part of AIAs, states’ AI-related legislation documents will be reviewed based on these components. For the performance-based ranking, we will use AI Index by Stanford Institute for Human-Centered Artificial Intelligence (HAI) which is one of the widely accepted states’ AI capability assessment reports. Based on their performance ranking and the existence of AI-related legislation, the following 10 countries will be the cases in this research: US, China, India, Canada, UK, South Korea, Germany, Italy, Spain, and France.

Rationality of export promotion policies in Costa Rica: is this a mix of GVC, innovation system and middle income trap approaches?

ABSTRACT. The Costa Rican economy has long been regarded as one of the region’s most stable and socially developed economies. In this paper we made an analysis of the evolution of Costa Rica’s strategy of economic diversification and capabilities building traying to analyse the rationality of the policies in different stages.

Most of the literature on the phenomena known as the “middle income trap” (MIT) suggests the relevant roles of capabilities, structural change and technological progress as the key factors able to better positioning a middle income country moving to better conditions (Vivarelli, 2014). A general argument, as suggested by Perez-Sebastian, 2007; Agenor and Canuto, 2012, is that once some countries reach the middle-income level, the pool of unemployed and underemployed rural workers drain out, wages start to rise, but an significant problem is that benefits from imitation and importing foreign mature technologies decrease in importance. In this condition, changing the structure of the economy (diversification from low productivity sectors into high-productivity ones) and on the types of product exported are the most important drivers in the strategies to overcome MIT (Gill and Bhattasali, 2007). It is important to consider that capability building and catch-up by domestic firms depend greatly upon the nature and features of innovation systems and it is necessary to consider the heterogeneous nature of the knowledge base, the specificity of the national, sectoral, and regional contexts, and the role of institutions in which innovative activities occur (Malerba and Lee, 2021). In the innovation system approach, the set of interrelated components is understood as a system working towards a common objective. Therefore, the set of parts and aspects of the economic structure and institutional set-up that affect learning and research are understood as the Innovation System (Lundvall, 1992). Chaminade and Edquist (2010) understand innovation policies as those public actions that drive innovation processes, both for development and dissemination processes. The rationality of innovation policies is based on trying to solve system problems. There are some failures in the system that hinder the possibility to reach these objectives. System problems to meet those objectives can usually be understood as problems of: infrastructure provision and investment problems; of transition; of "lock-in"; institutional; network problems; skills and learning; exploration-exploitation imbalance; complementarity problems (Chaminade and Edquist, 2010). Borrás and Edquist (2013), suggest a possible way to group policies and instruments to foster innovation, considering: regulations (intellectual property, regulations), economic transfers (competitive funds, exemptions), soft instruments (alliances, agreements). Orozco (2017) argues that to increase the efficiency of such policies and instruments, it is important to consider the geographical, sectoral or firm size specificities. The rationality of policies in GVC approaches is aimed mainly to facilitate trade. According to OECD (2012) and Catteneo et al (2013), in the approach of GVC there are two main objectives for policies: suppressing/reducing obstacles to trade at the border, including trade facilitation and increasing the accessibility and connectivity of the domestic market, and the security, predictability, reliability and efficiency of transports/logistics, telecommunications and ICT. More specific to innovation and building capacity, the approach recognizes that GVCs facilitate capacity constraints, since a country does not need to develop a fully integrated industry to participate in international trade. But capacities and productivity remain key factors for foreign investors and lead firms. In the new context, lead firms have to define strategies where innovation centres are decentralised. Lead firms need to innovate in developing countries, and solve their specific needs and eventually, they can bring the results back home, thereby contributing to boosting the developing countries’ exports (Govindarajan and Trimble, 2012). This requires the host developing country to develop innovation capacities, based on education and skills. There is a strong connection between the policy approaches of innovation systems and GVC. It is clearer for the approach of sectoral systems of innovation. But it is also clear that for more geographical approaches of innovation systems, it is also necessary to consider both the challenges and the opportunities from the conditions of international trade. Most of the challenges for economic and social upgrading in GVC require strong innovation systems. Considering the rationality of both approaches, it is clear that Costa Rica's export promotion and foreigner investment attraction policies have been mainly based on the GVC approach. Indeed, the strategy has focused on lowering barriers to international trade and consolidating an institutional framework to attract investment and promote new export products. From the point of view of the innovation system approach, there are still many gaps in the country's policies, as will be discussed in the paper. Our argument is that these gaps have contributed to the MIT in the country. We develop an historical analysis of the policies in the country and present some data to show the main changes in the structure of the economy and the exports.

Signaling innovation activities within agri-food ecosystems

ABSTRACT. Innovation studies traditionally employ bibliometrics and patent analyses to model ecosystems – methods that are less suited to measuring innovation in industry sectors where patents and publications are not used frequently by innovators. Some such sectors may be quite innovative despite a lack of patents or publications, for example the agriculture and agri-food sector, which is currently undergoing deep transformations relating to digitalization, climate changes and consumer preferences. Studying innovation in these sectors can be challenging due to a lack of accessible data. Official statistical sources often do not provide adequate coverage, and traditional surveying techniques such as personal interviews, focus groups, paper-and-pen or electronic questionnaires are increasingly problematic due to declining response rates or insufficient quality of data. It is crucial to explore alternate data sources and methods in sectors such as agri-food (Gök et al. 2015). Experimenting with webscraping techniques provides a non-intrusive method of collecting the public expressions of organizations with regards to their innovation activities. Previous studies have shown the potential of using text analysis in innovation studies (Antons et al. 2020; Youtie et al. 2021) and some underline the potential of content analysis of organizations’ public communication (Daas et van der Doef 2020). In this context, this study investigates whether innovation activities in sectors such as agri-food can be identified and captured; and what aspects of innovation activities are disclosed using public communications. The case we explore is the Canadian food and beverage processing sector and value chain due to its characteristics (Finco 2018; Vlachopoulou et al. 2021) and we develop a method to detect innovation-related themes conveyed by the sector’s organizations. In brief, the model is able to recognize paragraphs on websites that pertain to "innovation" and classify those, producing a coding scheme that can be used to analyze the agri-food sector. Data We base the analysis on a list of 2,506 organizations belonging to the Canadian agri-food ecosystem for which websites (URLs) have been identified. This agri-food ecosystem contains food and beverage processors, private sector suppliers of services and goods, private and public research and development institutions, government and non-for-profit organizations, and higher education institutions. The list was developed using manual and automated keyword web searches (food and beverage manufacturers by type such as products produced, etc.), and augmented by membership lists from associations of types of food and beverage processors, directories of organizations, and other public sources. Using a proprietary web analyzer tool created for this research, the data employed in this study were retrieved in May 2022. For this project, 47,531 English-only web pages are analyzed. Method We develop the method in four (4) main steps: 1) pre-processing of text data; 2) selection of paragraphs containing words related to innovation as a topic and vectorization; 3) topic analysis of the contexts in which innovation is used; 4) supervised classification process of select paragraphs related to the topic of innovation in new websites. This fourth step comprises two (2) sub-steps: 4.a) expanding the training corpus with a positive and unlabeled (PU) data classification; and 4.b) testing the model with a test data set. In step 1) pre-processing of text data, we apply classical Natural Language Processing (NLP) steps to prepare unstructured textual data for further analysis. Specifically, we execute morphological analysis and a lemmatization procedure for each word, then filter out stopwords so that only tokens corresponding to nouns, adjectives, and verbs that respect a document frequency threshold of 30 are kept. Next, we identify the most frequent bigrams of words to create a bag-of-words model. In step 2, we select text segments containing at least one of the words from a curated list representing innovation concepts. This results in a Document-Term Matrix with 69,826 segments and 3,879 unique unigrams and bigrams. In the third step, we use the Latent Dirichlet Allocation (LDA) approach to detect key topics across all segments containing on the innovation-related words. Next, a series of parameter tests occur that result in a model with eight (8) topics selected with a topic coherence score of 0.54. The LDA output consists of two matrices that allow us to analyze and interpret each topic and to create latent features for documents that feed the fourth step of our method. Step 4) consists of expanding the training set using supervised classification. The expanding method is based on the contextual modulation phenomenon involving conceptual expressions. This phenomenon describes the semantic process of the text segments where a concept is evoked, but without any standard lexical anchorage. Specifically, we formalized this task as a positive and unlabeled data (PU) classification problem. This type of method aims to expand the set of positive data from unlabeled data. Thus, we label those segments containing innovation-related words as the P dataset (positive dataset), and then we mark the unlabeled dataset U. Then we retrieve an equal number of labels that do not contain the identified words. Next, we apply the previously developed LDA model to retrieve feature vectors for the U dataset. Finally, we train models and select the best model using the F1-measure. This final step leads using the supervised training method on unseen websites to select organizations having more intensive innovation branding. With the help of agri-food experts, we then evaluate the results. Expected results We have implemented all but the final step in the method. We developed a significant topic model that encompasses the main subjects of our corpus well. In our first experiment, the model obtained an F1 score over 0.8. We anticipate that further iterations of the model will lead to improvement and that the model, programming, or methodology will be able to classify unseen websites to identify those organizations communicating about the identified topics, as well as those with higher intensity of such communications. Conclusions This study experiments with machine-learning and using unobtrusive methods to expand knowledge of innovation ecosystems. It provides a model to identify firms with higher innovation intensity in their communications. As next step, research into validation of the technique, for example conducting collaborative research with industry associations representing the main stakeholders of the Canadian agri-food ecosystem would be recommended.

Bibliography Antons, D., Grünwald, E., Cichy, P. et Salge, T. O. (2020). The application of text mining methods in innovation research: current state, evolution patterns, and development priorities. R&D Management, 50(3), 329‑351. Daas, P. J. H. et van der Doef, S. (2020). Detecting innovative companies via their website. Statistical Journal of the IAOS, 36(4), 1239‑1251. Gök, A., Waterworth, A. et Shapira, P. (2015). Use of web mining in studying innovation. Scientometrics, 102(1), 653‑671. Finco, A. (2018). Lessons of Innovation in the Agrifood Sector : Drivers of Innovativeness Performances. Lessons of Innovation in the Agrifood Sector : Drivers of Innovativeness Performances, 181‑192. Vlachopoulou, M., Ziakis, C., Vergidis, K. et Madas, M. (2021). Analyzing AgriFood-Tech e-Business Models. Sustainability, 13(10), 5516. Youtie, J., Ward, R.*, Shapira, P., Schillo, R. S., Earl, E.L. (2021). Exploring New approaches to understanding innovation ecosystems. Technology Analysis & Strategic Management, 7(Sept):1-5.

13:15-14:45 Session 3D: Governance
Research Governance and the Dynamics of Science: A Comparative Analysis of Governance Effects in Organisational Context

ABSTRACT. Background This paper aims to contribute to, and advance, the understanding and empirical study of the effects of research governance on scientific fields by exploring the governance effects on a scientific field in the context of two different universities. That is achieved by building on a recent framework for the study of governance effects on research fields (Nedeva et al., 2022) and extending it, conceptually and empirically, by conducting a comparative analysis of the reported behaviour of members of the same research field, within the same governance regime and two rather different universities. This paper is the third within a research line on studying governance effects on scientific fields (Nedeva et al. 2022, Morales-Tirado, in progress).

Rationale Debates regarding the study of research governance effects on global scientific fields in the literature unfold along several lines. Some studies, quantitative as well as qualitative, investigate governance effects on context-specific research organisations, namely universities and research institutes (Lorenz 2012; Luukkonen and Thomas 2016; Vinkenburg 2017; Glaser 2019; Luo, Ordóñez-Matamoros & Kuhlmann, 2019; Thomas et al. 2020; Strinzel et al. 2021; Kozlowksi et al. 2022; Ramos-Vielba, Thomas and Aagaard 2022; Feenstra, & López-Cózar, 2022,). Other, more nuanced studies extend their research interest to include studying the effects of governance on the epistemic choices of members of local knowledge communities (Glaser, 2019). Methodologically studies of governance effects usually aim to measure change using opinion-based survey techniques, case study/interview approaches, bibliometrics or lines of investigation seeking to unpack the (soft) causality mechanisms that may or may not affect organisational, personal and group selections.

While contributing to the understanding of governance effects on science, these approaches share a significant shortcoming in that they generally fail to extend beyond the local conditions for knowledge creation and hence fail to capture the aggregate governance effects at the level of transnational, global research fields.

This was conceptually addressed in a recent paper (Nedeva et al., 2022) by proposing a novel heuristic for linking the characteristics of performance-based evaluation arrangements (PREAs) and the properties of research fields. Next, a comparative analysis of the governance effects of different governance arrangements of the same research field extended this framework empirically. This paper explores the (potential) difference in responses of the local members of the same research field, within the same governance arrangements and two different universities.

Methods We used a novel framework to study governance effects on scientific fields, one that recognises three contexts where different effects may occur, e.g., the research space context where performance based evaluation arrangements (PREAs) are embedded, the research field context where knowledge claims are assessed to award reputation (publications, grant capture etc.), and the context of research organisations where individual and collective performance are evaluated for organisational career purposes (Thomas et al., 2020).

We kept the PREA and research field contexts constant and allowed variance in terms of the organisational context. Hence, we interviewed members of research groups in two universities. To use a terminology from Paradeise and Thoenig (2015) one of the universities is a ‘top-of-the-pile’ and the other one is a ‘wannabe’.

Our questions were designed to capture the interactions between university leaders (administrators) and the members of the local knowledge network, or research group. We conducted a total of twenty interviews (13 for a top-of-the-pile university and 7 for a wannabe university). We explored interactions (and power play) in the context of nine selection points (Nedeva et al., 2022).

Preliminary results We analyzed the interviews around selection dimensions concerning organizational authority, namely (1) organizational career, (2) knowledge production, and (3) knowledge dissemination. Results indicate that university governance arrangements matter but do not change the actions of this type of field members.

We contrast the responses of both universities around the selection of new group members, promotion and probation, and we find that for the wannabe university members, recognition of the university authorities is more important than for those from a top-of-a-pile organization. Yet, field considerations (e.g., field recognition) override organizational pressures. In regards to selection for access to research infrastructure, methods and skills, behaviour is unaffected by local influences in both instances. Similarly, our results suggest that selections for access to knowledge networks, decisions over publication outlets and submissions for PREA assessments are dominated by F-type notions.

Significance This paper is an important empirical test of the framework for the study of research governance on scientific fields. We also believe that it contributes to the understanding and methodology for tracing governance effect on global scientific fields.

References Feenstra, R., López-Cózar, E.D. (2022) The footprint of a metrics-based research evaluation system on Spain's philosophical scholarship: An analysis of researchers' perceptions, Research Evaluation, 1-15. Glaser, J. (2019) ‘How Can Governance Change Research Content? Linking Science Policy Studies to the Sociology of Science’, in Simon D., Kuhlmann S., Stamm J. and Canzler W. (eds) Handbook on Science and Public Policy, 419–47. Cheltenham, UK/Northampton, MA, USA: Edward Elgar. Kozlowksi, D., Larivie're, V., Sugimoto, C. R., and Monroe-White, T. (2022) 'Intersectional Inequalities in Science', Proceedings of the National Academy of Sciences of the United States of America, 119, e2113067119 Lorenz, C. (2012) ‘If You’re So Smart, Why Are You under Surveillance? Universities, Neoliberalism, and New Public Management’, Critical Inquiry, 38, 599–629. Luo, J., Ordóñez-Matamoros, G., Kuhlmann, S. (2019) The balancing role of evaluation mechanisms in organisational governance—The case of publicly funded research institutions, Research Evaluation, 28 (4), 344–354, Luukkonen, T., and Thomas, D. A. (2016) ‘The ‘Negotiated Space’ of University Researchers’ Pursuit of a Research Agenda’, Minerva, 54, 99–127. Nedeva, M., Tirado, M.M., and Thomas, D.A (2022) Research governance and the dynamics of science: A framework for the study of governance effects on research fields, Research Evaluation, 1-12. Paradeise, C., and Thoenig, J.-C. (2015) In Search of Academic Quality. Houndmills, Basingstoke: Palgrave Macmillan. Ramos-Vielba, I., Thomas, D. A., and Aagaard, K. (2022) Societal Targeting in Researcher Funding: An Exploratory Approach, Research Evaluation, 31, 202–13. Strinzel, M., Brown, J., Kaltenbrunner, W., de Rijcke, S., and Hill, M. (2021) Ten Ways to Improve Academic CVs for Fairer Research Assessment, Humanities and Social Science Communications, 8, 251. 1057/s41599-021-00929-0. Thomas, D. A., Nedeva, M., Tirado, M. M., and Jacob, M. (2020) Changing Research on Research Evaluation: A Critical Literature Review to Revisit the Agenda, Research Evaluation, 29, 275–88. Vinkenburg, C. J. (2017) Engaging Gatekeepers, Optimizing Decision Making, and Mitigating Bias: Design Specifications for Systemic Diversity Interventions, The Journal of Applied Behavioral Science, 53, 212–34.

The Technological Convergence of Emerging Intelligent Technology Ecosystem

ABSTRACT. This paper is submitted to the panel entitled "The Dynamics of Science and Technology Governance in Emerging Sectors" (submission 6247)

Abstract This paper aims to study the emergence of emerging innovation ecosystem with focus on three selective intelligent technologies, including artificial intelligence (AI), Internet of things (IOT), and intelligent manufacture. The key research question this talk try to answer is how to form and develop the innovation ecosystem of emerging technology industries? How does the institutional factor influence the evolution of innovation ecosystem during the development process of science and technology? As emerging technology is constantly developing rapidly, its definition, scope, application, and technology governance approaches align with the emergence of emerging technologies. This paper considers the current development status of various industries and the recent policy content of emerging technology industries mostly aims to promote smart technologies such as artificial intelligence, the Internet of Things and smart manufacturing. Though, establishing innovation policy framework to enhance the governance of the emerging technology are important, but the empirical studies focusing on the convergence of various emerging sectors and social impacts on the technology emergence process are still less clear. This paper will therefore explore the formation of the innovation ecosystem base on the multilevel prospective, taking academic research, technological development, commercialization process, and social impact on the three emerging intelligent technologies. In addition, we will focus on the interplay between multilevel prospective on emerging sectors. Finally, the main focus of this panel will move onto macro level with focus on the interactions between emerging sectors and the institutions. Combining social network analysis and multilevel analysis, this research will analyze the innovation ecosystems of various emerging sectors. Ultimately, this project will propose tailor-made policy recommendations to further promote the developments of emerging sectors.

Knowledge Intermediaries and Evidence Use in State Policymaking: Topic Modeling and Analysis of Two Consequential Policy Areas

ABSTRACT. Evidence-based or evidence-informed decision-making is the paradigm by which public decision-makers leverage knowledge from various sources to craft policies and programs that are fair, efficient, and effective. Evidence can come in a multitude of packages and be leveraged with different calculuses about what matters most. In this work, we build on Head’s three lenses of evidence-based policy where he articulates that evidence can come from science (what we traditionally think of as “evidence”), from practice (the tacit and earned knowledge of front-line public servants), and from the political sphere (knowledge about political dynamics, authority, and feasibility) by examining policy documents from two consequential public policy domains in the United States: autonomous vehicles and state-level opioid strategic plans.

The focus is on how state governments, often in conjunction with knowledge intermediaries attempt to monitor and understand innovation trends and develop strategies to adapt to changes in science and technology. Policymakers and public administrators frequently draw from research in science and technology to address problems confronting society. Relatively little is known regarding how state public managers scan for, access, and employ consensus science. This panel explores the variety of sources of information that can be accessed, the development of intermediary organizations to help structure and access information at the state and local levels, and the methods local governments use to communicate that information to their constituencies.

Policy reports and plans can be used as artifacts of policy processes that capture the decisions and dynamics of complex policy activities, rather than as an end of the policy process (Freeman and Maybin 2011). As such, detailing the people and sources used to create the policy document can reveal important aspects of how the policy evolves in a location. We use policy documents from the two domains mentioned above to identify important aspects of how the policy deliberations occurred and the impact they had on the outcome (report construction). We created unique datasets for this investigation.

Data and methods. All reports from the 50 states and Washington DC relative to the two substantive areas were scraped from publicly available sources on the internet. All reports from 2013 until 2020, when the COVID-19 pandemic started) were considered for this investigation. Reports were then reviewed for their focus on program implementation and governance, rather than a technical focus. There were 68 reports on the AV side and 72 in the opioid case.

Reports were then analyzed on a number of dimensions to understand how they used evidence in their work. Our methods were centered on bibliometric techniques which catalog and enumerate the contents of scientific documents. We assessed images and figures, references, word/page count, topics covered using NLP, authors, and committee/task force membership. We also ran clustering models to find distinct patterns of reports.

We used topic modeling to characterize the contents of the state reports, state laws, and executive orders. The core text of each document was extracted for modeling. Wordstat 9.0.8 was used for text processing and topic modeling using the nonnegative matrix factorization (NMF or NNMF) method. The text was preprocessed using snowball stemming. Words that occurred at least 30 times in the corpus were retained, or about 2,000 words in total. Other settings remained Wordstat defaults. Examining coherence statistics showed a peak at about 30-35 topics; 35 topics were specified in the final model.

Findings. We found three distinct clusters of reports within each of the two substantive areas. While the specifics of the clusters across the two areas are slightly different, they are remarkable in that they have the hallmarks of Head’s three lenses of evidence for policymaking. On the AV side, we see three clusters that are expert-engineering (practice evidence), expert-academic (scientific evidence), and convening, legislative focus (political evidence). For the opioid reports, we see that reports that have a lot of private organizations (rather than state agencies) emphasize practice-based knowledge, while those reports that are oriented by task forces emphasize political knowledge, and finally, those reports authored by groups with heavy representation from public health departments emphasize scientific knowledge.

Building resilient technology policy through public participation: The case of the Chilean National AI Strategy

ABSTRACT. Background The emergence of artificial intelligence (AI) as new general-purpose technology is influencing how societies shape their development models for the next decade (Bresnahan & Trajtenberg, 1995; Klinger et al., 2018; Trajtenberg, 2018). Even though the concept of AI was conceived in 1956, it was not until the last decade that AI’s adoption scale, speed, and risks became a central policy challenge for governments (Taeihagh, 2021). Since the first national AI strategies publication in 2017, there has been an explosive increase to over 700 policy initiatives in more than 60 countries (OECD, 2021).

Recent studies and international discussions on AI governance have emphasized the need to build broad societal consensus around ethical principles and institutions (e.g., Calo, 2017; Gasser & Almeida, 2017; OECD, 2019; UNESCO, 2021). However, members of society with different imaginaries and expectations have different opportunities to frame what is relevant, urgent, possible, or inevitable in technology policy (Sand, 2019; Konrad & Boyle, 2019). Moreover, most AI policies have been developed by “domain experts,” leaving “lay people” on the margin, failing to incorporate multiple visions. The latter is problematic because multi-stakeholder processes are becoming ubiquitous amid crises of trust and social unrest that have proliferated during the last few years.

Countries face difficulties in implementing and ensuring the continuity of AI strategies due to changes in governing coalitions and competing social priorities. For example, Mexico and Argentina published their strategies in 2018 and 2019, in the last year of their governments, and were not implemented in depth nor continued by the next governing coalition (Gómez Mont et al., 2020). This manuscript discusses how a participatory approach to AI policymaking can enable policies to achieve higher consensus and resilience. To do so, we conducted an in-depth analysis of the building process of the Chilean National AI policy, in which approximately 10,000 people participated through different mechanisms and stages. This is an interesting case to study as, unlike other AI policies in the region, the Chilean AI policy achieved a high degree of consensus–as measured during a public consultation before the policy was enacted–and has been able to withstand changes in government coalitions to date.

Methods We undertook a longitudinal case study research approach, accompanying and participating in the development of the AI policy between 2019 and 2022. Utilizing interview data, official documents, and transcripts of grassroots discussions, we mapped the AI policy development utilizing process tracing techniques (Garud, Berends, & Tuertscher, 2018; Langley, 1999, 2007; Van de Ven & Huber, 1990; Van de Ven & Poole, 2005). We chose this design because case studies have been used for modeling and assessing complex causal relationships (George & Bennet, 2005) and have been found helpful in illuminating decisions (Yin, 2017), both of which enable an understanding of how technology and innovation policy is developed. Longitudinal case studies also enable us to explore AI's technological and social contexts, and the collection of qualitative evidence allows us to identify key process variables.

We divided the policy development process into four stages using an engineering systems architecture approach (Selva, Cameron & Crawley, 2015). In the “Conceive” stage, stakeholders address the challenges and opportunities of AI systems and think of possible solutions to them (e.g., regulations and strategies). In the “Design” stage, stakeholders design a roadmap and build the solution chosen during the Conceive stage. In the “Implement” stage, stakeholders start executing the actions developed in the Design stage. Finally, in the “Operate” stage, the solution owner monitors results, identifies future opportunities and challenges, and can decide to initiate a new policy process. We acknowledge that this framework can simplify uncertain and messy real-world politics. However, practitioners can use this framework to guide AI governance discussions.

Results and discussion Chile’s AI strategy was initially conceived as a top-down, expert-driven process. However, two external crises (Chile’s social riots of 2019 and the COVID-19 pandemic) lowered the barriers to adopting a bottom-up approach as there were mounting pressures from different stakeholders demanding involvement and requesting specific policy outcomes. The government responded by adapting the process and its governance to address the bottom-up pressure and navigate the conflicting demands of heterogeneous stakeholders. Furthermore, authorities agreed to foster self-convocated roundtables and organize regional discussions to gather information for the draft, leaving the expert committee only as a consultative group.

The constant relation between demands and responses shaped the AI development process and defined the level of overlap between the four stages (i.e., Conceive, Design, Implement, and Operate). To convince heterogeneous groups to participate, the government purposely developed tools to generate trust and lowered barriers to participation. For example, in the Design phase, there was an open deliberation period in which everyone could discuss and contribute content for the policy. Public officials developed the first draft with that deliberation input, which was later presented for consultation to the public. The option to review and comment on the draft was open to the public. This two-staged model gave the public more accountability and fostered trust in the AI strategy. Another trust-building example was when the government actively argued against the expert/non-expert dichotomy, responding to domain expert groups complaining because of the involvement of “lay people” in the process.

The process generates insights into how the intertwined nature of technology and development in emerging countries shapes public deliberation and moves processes beyond the expert/non-expert dichotomy. Deliberation during the process was usually framed based on Chile’s singularities, deficits (Pfotenhauer et al., 2019), and opportunities to address social goals, all of which directed the AI’s development (Schot & Steinmueller, 2018). Key initiatives in the policy’s action plan are a US $5 million grant for economic reactivation through AI, a public-private enterprise to foster AI and data science using Chile’s unique astronomical potential (see Guridi et al., 2020), and the prioritization of three industrial sectors for an AI Sandbox (i.e., healthcare, logistics, and fintech). Thus, the discussion focused on how AI contributed to the country's overall development, which contrasts with other policy narratives based on developing science and technology for its own sake.

The resulting policy achieved a high level of consensus and acceptance and has survived for more than a year since its publication. Nearly 90% of the people who participated in the consultation highly agreed or agreed with the proposed objective, and more than 80% with the topics and objectives proposed. Furthermore, the policy survived through a change in the country’s governing coalition, and to date, the new administration has continued implementing it and engaging in international outreach activities.

We contribute to technology and innovation policy literature by providing insights on how to enable participatory processes to build more resilient technology policies. Governments should acknowledge citizens’ reflexive agency to build democratic legitimacy in technology discussions (Biale & Liveriero, 2017). Furthermore, public discussion allows the creation of technology policy with a focus on anticipation, experimentation, participation, and directionality, following Schot & Steinmueller's (2018) transformative change framework. Finally, we show how crises and social unrest can be leveraged to foster participation and innovation when constructing technology policies.

13:15-14:45 Session 3E: Measuring Innovation
Understanding the use of innovation-related concepts in enterprises’ websites

ABSTRACT. This study explores the use of keyword frequency from corporate websites to build innovation indicators of firms. We used the online data from 2,413 companies that participated in 29 questionnaire-based investigations between 2010 to 2016. We extracted the content of the corresponding websites via snapshots hosted on The Wayback machine and analyzed them based on keywords related to innovation concepts (innovation, collaboration, open innovation, R&D and IP). We built a nominal scale from the questionnaire-based data that categorize firms based on the level of evidence of innovation and on their level of time to market. Then, we produced multinomial logit regressions with the nominal scale indicator as dependent variable to test the contribution of our Web-based indicators to increase the relative risk ratios of being in a group of firms that shares similar evidence of innovation and time to market pattern versus other groups. Our preliminary results show that when one of the web-based indicators is high, our control group (no evidence of innovation while being in the bottom 20% in terms of time to market), does not see any significant positive increase in relative risk ratios when we compare it to any other groups. Other results are consistent with what one would expect from groups with greater evidence of innovation, which show a significant positive increase in relative risk ratios when we compare them to the other lower groups. However, some other significant results seem at first glance to be inconsistent with what would be expected. Indeed, this raises the question of whether we are capturing patterns of market signaling from the mid-lower tier while observing the absence of the need to perform any form of market signaling from the companies in the upper tier. After all, actions still speak louder than words it seems.

Examining Firms’ Engagement with Different Forms of Knowledge Disclosure: Website, Publication and Patent data

ABSTRACT. Background and rationale Firms engage with a variety of practices to disclose knowledge resulting from their R&D efforts. Although these disclosures can lead to unintended knowledge spillovers and hinder a firm from fully capturing the benefits of R&D, they also enable the firm to accrue a range of benefits – e.g. reputation building, gaining access to external knowledge and financial resources, establishing intellectual property rights – that contribute to the firm’s innovative performance. Scholars have extensively examined patenting and publishing as forms of knowledge disclosure and their impact on innovation (e.g. Alexy et al., 2013; Arora et al., 2018; Hicks, 1995; Rotolo et al., 2022), and more recently, the extent to which these forms of disclosures are substitutes (Blind et al., 2022). Yet, our understanding of how firms disclose knowledge through their websites and how this channel of disclosure relate to other channels of disclosure remains scant.

In line with recent efforts aimed at expanding our understanding of the innovation process on the basis of novel text-based indicators (Bellstam e tal., 2021; Gatchev et al., 2022), in the paper, we examine firms’ behaviour in disclosing and signalling their R&D activities and strategies on their websites, and compare these disclosures with disclosures firms make in patents and publications. To do so, we build a large-scale textual dataset derived from the website pages of a sample of firms. We employ Natural Language Processing (NLP) and topic modelling to map firms’ knowledge activities in term of topics as reported on their websites.

Methods We piloted the study on a dataset that combines publication and patents data related to a sample of 9000 UK firms with textual data extracted from the websites of these firms in the year 2020. We employed a text mining-based approach that uses firms’ textual data at scale to delineate topics from website data and to match these with the well-established Microsoft Academic Graph (MAG) Field of Study (FOS) model. The MAG is the largest publicly available dataset of scholarly publications and the largest dataset of open citation data (Shen, Ma, & Wang, 2018). MAG data models scholarly communication activities which consist of six types of entities – publications, authors, institutions (affiliations), venues (journals and conferences), fields of study and events (specific conference instances) ¬– and the relations between these entities – e.g. citations, co-authorship. The relations between the entities are described in more detail in (Sinha et al., 2015). The FOS classification can be conceived as one of the broadest classification systems for knowledge transparently available for use. FOS are the results of a hierarchical topic model run on the entire MAG data corpus. More precisely, the hierarchical topic model produces unique FOS identifiers (ids) by creating a hierarchy of five levels (about 700,000 topics).

Extant research has shown the potential of website data in creating valuable information on innovative activity at a firm-level (Bellstam e tal., 2021; Gatchev et al., 2022). For example, website data can be used to better understand innovation outcomes, strategies, and relationships (Gök et al., 2015). Running the analysis at scale also allows drawing industry-wide structures from the data. Ashouri et al. (2021) used website data from 96,921 medium-high and high-technology firms to create a model of industry structure Our study leverages the framework presented in Ashouri et al. (2021) to devise a transparent approach to creating a classification: we infer a classification to firms’ website data using NLP and the hierarchical topic model-based MAG FOS categories.

Utilizing the firms’ URLs with an automated website scrapping system, the textual content of firms’ websites was retrieved – the approach is described in detail in Ashouri et al. (2021). The data platform uses the capacities built in BIGPROD1 project []. For retrieving and hosting the raw data, a ‘hybrid’ design was adopted, with part of the infrastructure being located on-site premises and the other part in the cloud (MS Azure Cloud Platform). Raw text collected from firms’ websites, web scrapping task and MAG publications’ text required cleaning and harmonization. Therefore, pre-processing steps involved cleaning procedures (e.g. removal of stop words and non-alphanumeric characters, stemming and lowercase transformation) applied to harmonize and increase the consistency of the text. For NLP to work, the natural language (text) needs to be transformed into a numerical vector form. Text vectorization techniques, namely tf-idf, Bag of Words and vectorization, are very popular choices for machine learning algorithms, can help convert text to numeric feature vectors. Therefore, to quantify and convert text into numerical representation in documents, we compute a weight to each phrase that signifies the importance of the phrase in the document and corpus. The tf-idf approach is a transformation applied to texts to get vector representation of vocabularies. Then it is possible to obtain the similarity of any pair of vectors to a quantified measurement.

Assigning FOS is now possible to text representations coming from firms’ websites. The delineation of these categories on firms’ website data and the descriptions of firms’ products is very granular given the large range of FOS categories, the values are standardized descriptions of what a company is doing.  Because these FOS categories are hierarchically linked, it is possible to easily assess how similar any pair of FOS categories are to each other. This can be used to go from low-level specific descriptors to high-level categories. In this study, we used the 100 most popular FOS ids. For each of the firms in the sample, their website was scraped and indexed using a pre-determined vocabulary. A weighted vector of vocabulary codes was then used to assign a vector of FOS ids to each of the websites. This process resulted in a weighted vector for each firm in the sample: a vector represents a firm’s website content using FOS ids and their associated weights.

Expected results and implications We expect that the study will shed light on the extent to which firms engage in knowledge disclosures on their websites as revealed by the emergence of topics from website textual data. This will enable us to delineate different types of firms on the basis of their disclosure behaviour as well as to increase our understanding of how website disclosures may relate to other forms of disclosures such as patenting and publishing in terms content (e.g. what topics are disclosed on websites that are/are not disclosed in patents and publications). In this regard, by comparing the R&D topical signals received from firms’ website content with the ones collected from patent and publication data, we can isolate firms’ behaviour in communicating their activities to their stakeholders and examine the dynamics associated with this phenomenon (e.g. what topics are disclosed on websites before being disclosed in patents and publications). We expect the study to contribute to two main streams of research in innovation studies literature, i.e. research examining firms’ knowledge disclosures and the role that these have in the innovation process, and research developing novel measures and indicators of innovation on the basis of the increasing availability of text data.

Quantifying biomedical firms’ basic research

ABSTRACT. Background and rationale Basic research is the catalyst for promoting the growth of high-tech firms that are the main driving force of technological innovation. As society progresses, the link between science and technology gets closer. The development of high-tech industries has an increasing demand for original knowledge and innovationse, making basic research even more important nowadays. Therefore, understanding how basic research influences the development of firms becomes a significant issue. Identifying and measuring basic research accurately is fundamental to investigating its influencing mechanism. However, it seems that a standard measurement method has not yet been determined, because of the abstraction of the concept basic research. For now, there still remains limitations in existing methods in current empirical research. For example, most methods are either indirect to basic research or too coarse-grained to measure the basic research intensity. Such measures include overall R&D expenses and the number of scientific papers, which would lead to the inaccuracy and even incorrectness of findings and policy suggestions. High-tech firms, as one of the main bodies in research investment, assume important responsibilities in R&D activities. Encouraging firms to invest in basic research is a crucial means for a country to enhance the competitiveness of science and technology. Therefore, it is particularly important to study firm’s basic research, especially whne the empirical research on this topic is still lacking. Biomedicine is a research-intensive industry in both China and abroad. Its progress and development is closely related to basic research, especially the high-level basic research. In this study, we measure the basic research in biomedical firms by applying an indicator, Level Score, that can quantitatively reflect the basicness of an individual paper.

Method “Level Score”, a quantitative index that Ke (2019) proposed, can well measure the basic level of a research paper. By establishing the occurrence network of MeSH terms and using the LINE network representation learning algorithm, we can obtain the translational axis in a vector space, pointing from the center of basic type MeSH terms to the center of applied type MeSH terms. Then we can calculate the appliedness score of each MeSH term, which is the cosine similarity between the translational axis and the MeSH term. The Level Score of a research paper is the average appliedness score of all its MeSH terms, ranging from -1 to 1. The smaller the score is, the more basic the research paper is. Based on this indicator, we then quantify the basic research of biomedical firms in the biomedicine industry. We derive a list of all biomedical firms from 2015 to 2017 according to the Standard Industrial Classification (SIC) from Compustat database. Using the method of keyword matching, more than 20,000 firm papers published from 2015 to 2017 are successfully identified from MEDLINE database. Their reference papers are obtained from Microsoft Academic Graph (MAG) using doi as an intermediary. We then calculate the Level Score of all firm papers and their reference papers. We observe the distribution of the basic level of firm papers and their references, investigate the dependence between firm’s basic input and output, and come to some valuable conclusions.

Results The preliminary results of our study show that: (1) In the biomedical industry, there is an obvious binary distribution of "basic research-applied research" in firm papers. That is, firms clearly publish scientific papers of both basic research and applied research. In the meanwhile, papers’ basic level increases along the path of “MEDLINE papers - firm papers - firm reference papers”, revealing a hierarchical supporting mechanism of basic research. (2) Biomedical firms adopt diversified R&D strategies. There is also a binary division of "basic-applied" in firm’s research preference. But this reference deflects more to basic research, as basic type papers accounts for 60%-80% in most biomedical firms. Further, the R&D strategy adopted by biomedical firms is related with firm size. Our results show that, small and medium-sized firms are more likely to choose only one of the research type, a relatively extreme strategy. Yet large firms appear to be more conservative, carrying out both basic and applied research at the same time. (3) Basic research is important for biomedical firms in terms of the relationship between research input and output. We find that biomedical firms cite more papers of basic research more than those of applied attributes, and that some firms even only cite basic papers. Regarding references as a kind of knowledge input and papers as an output, we find that the basic type knowledge input can produce both basic and applied output. But it is more difficult to obtain basic type output from applied input than to get applied output from basic input.

Significance First, by applying the Level Score index in biomedicine industry, we further verify its feasibility in measuring basic research in this study. According to the results and the conclusions, we believe that this index has the potential to be further promoted in both theoretical and empirical research of science and technology assessment. In this way, our study can provide new ideas for the relevant empirical research in the future in terms of measuring basic research. Second, we find some interesting preliminary results regarding the status and rules of basic research in biomedical firms, through measuring the basic level of more than 20,000 firm papers and more than 1 million reference papers. The results of our study reflect the important role that basic research plays in the development of biomedical firms. The conclusions will help improve the measurement methods of basic research and provide more empirical evidence for the field of biomedicine. Third, based on the results of our study, we tentatively propose some suggestions for firm basic research. Theoretical support will be provided for government to formulate relevant policies. An implications will also be provided for firms to strengthen the efforts in basic research to improve R&D and innovation capabilities, which is of great significance to achieve the long-term development of firms.

Are Federal Contractors Less Innovative?

ABSTRACT. One of the roles of federally-funded R&D, beyond its primary goal of advancing technologies critical to US security, is to fuel economic growth and US competitiveness. Thus, a recent cause for alarm (e.g., Arora, Belenzon and Patacconi 2018, Gruber and Johnson 2019) is the 67% decline in federal R&D. Not surprisingly, this decline corresponds to declines in US R&D productivity and nominal GDP growth. One policy response is the Chips and Science Act of 2022, which authorizes approximately $174 billion through FY 2027 to support the nation’s science and technology base, 75% of which is for Research (R). Thus the Act responds to a common misperception that the decline in federal R&D has come from R. In fact, federally-funded R has maintained a constant share of GDP. Essentially the entire 67% decline in federal R&D has come from development (D). Because R goes principally to universities, while D goes principally to firms, universities’ share of federal R&D has grown sevenfold (from 5% to 35%), while firms’ share has declined 70% (from 71% to 22%). This shift is curious, since at least one study found that federal R&D contracted to industry stimulated more private R&D investment than other federal R&D (Levy and Terleckyj 1983). Moreover, the same study found that federal R&D contracted to industry increased labor productivity, while other federal R&D decreased labor productivity. Such dramatic shifts and their coincidence with declining R&D productivity suggest the former may be responsible for the latter. If so, a likely explanation for declining R&D productivity is that federal R&D enhances firm R&D productivity. Such a finding might come as a surprise, since a common conception of federal contractors is that they are less innovative than commercial firms (Gansler 2013, Srinivasta 2019, Fischetti 2020). One reason federal contractors might be less innovative is that they evolve to thrive in the highly idiosyncratic and bureaucratic federal procurement system (e.g., Josephson et al. 2019). A second reason is that they may substitute lobbying and influence expertise (necessary to secure government contracts), for innovative expertise (necessary to succeed in the market). While it is obvious that government contracting is cumbersome, it is less obvious the impact this has on innovation. As a shining counter-example, Operation Warp Speed (OWS) represents federally-funded innovation at its best. In nine short months, beginning late March 2020, the government put out a call for vaccines, evaluated 100 responses, funded development and manufacturing expansion for six candidates, and administered the first vaccine December 13. Nor is OWS an isolated example. It is well known, for example, that the Internet originated with ARPAnet, but federal R&D also led to other important general-purpose technologies such as lasers, MRI and GPS (Singer 2014). Accordingly, the net impact of federal R&D contracting on firm innovation is an empirical question. While prior research has examined this question, and concluded that the productivity of federal R&D was a small fraction of that for firms own internal R&D (Griliches 1986), the data was cross-sectional, and pre-dated the decline in federal R&D funding. Accordingly, it may have captured a period when the federal government was overinvesting in development. We re-examine the question of federal R&D contracts on innovation by comparing federal contractors with commercial firms. Our measure of innovation is firms’ Research Quotient (RQ), defined as the firm-specific output elasticity of R&D. Accordingly, RQ captures the percentage increase in firm revenues associated with a 1% increase in R&D. In aggregate, RQ captures the contribution of industrial R&D to economic growth (one of the goals of federal R&D). To conduct our test, we first characterize RQ separately for firms with and without federal R&D contracts. While the RQs of both type firms have been decreasing, those with federal R&D contracts have higher RQ throughout. This is true after adjusting firms’ financials to treat R&D contract dollars as an R&D input, rather than as revenues. We next examine whether the higher RQ of federal R&D contractors reflects a selection effect or a treatment effect through a difference in differences (DiD) estimation relative to the time of first federal R&D contract. Our results reveal that the higher RQ of federal R&D contractors reflects both treatment and selection. Federal R&D contractors have significantly higher RQ prior to first contract, and RQ increases significantly following the first contract. The question then turns to why federal R&D contracts increase firms’ RQ. One plausible explanation is that the government issues R&D contracts to firms from whom it expects to procure the ultimate good/service. If so, then RQ is mechanically higher for these firms—increases in R&D precede planned increases in revenue from procurement. We test this by exploiting a shock to federal funding—the Budget Control Act of 2011 (BCA), also known as sequestration, which dramatically decreased federal spending. Using dynamic DiD estimation, we find that while the RQ of federal R&D contractors decreases following sequestration, the decrease is not-significant. Thus, the higher RQ of federal contractors is not merely an artifact of the R&D being an early stage of a longer-run contract. We therefore explore whether federal contracts change the character of firms’ R&D in ways that enhance its productivity. We look at two dimensions of R&D character, its expansiveness and its impact. Both measures are based on patents, thus we can’t test the 50% of firms who choose not to patent their innovations. Nevertheless, for the subset of firms who do file patents, we find that the R&D of federal contractors is more expansive and impactful than that of commercial firms. Taken together our results indicate that federal R&D contractors are at least as innovative as their commercial counterparts. Thus, the benefits of federal R&D contracting appear to outweigh the bureaucratic costs they impose. These benefits reflect both selection and treatment effects: government contractors have higher RQ prior to their first federal R&D contract, but in addition their RQ increases after receiving these contracts. The higher RQ of federal contractors cannot be explained by the fact these contracts are pre-cursors to subsequent procurement contracts. Rather it appears that federal R&D contracts enhance the character of firms’ R&D—making it more expansive and impactful. One reason this may be true is that the funding agencies provide a conduit from research at universities and government labs to firms who commercialize those innovations. In doing so, these contracts help solve the “valley of death” problem that few university and lab inventions are commercialized. Our results have implications for policy. First, we find no evidence that greater use of commercial firms would increase the impact of federal R&D. In fact, we find that federal R&D contractors are at least as innovative as their commercial counterparts. More importantly, our results suggest the decline in federally-funded R&D to companies likely contributed to the decline in US R&D productivity. By extension, the rise in funding to universities appears not to have generated any benefit. Perhaps this is because the shift in from D at firms to R at universities has produced an imbalance, such that research generated by universities now exceeds industrial capacity to commercialize it.

13:15-14:45 Session 3F: Energy Technology
The impact of innovation cluster policies on Energy-transition: Learning from leading energy clusters in Germany.
PRESENTER: Mahendra Singh

ABSTRACT. Introduction:

Innovation clusters are often known as innovation ecosystems and have a significant impact on the German economy and regional development. Different federal states already have placed various cluster policies to support the cluster initiative. These policies are designed to congregate diverse actors (e.g., SMEs, large companies, research institutes, and regional institutions) with similar interests in particular demographic proximity [1]. Moreover, clusters with excellence in energy topics also lead territorial energy-transition and support multiple actors to develop rapid and scalable innovations. Seen in this light it became a natural interest for policymakers, investors, and firms to analyze the ongoing activities of leading energy clusters. Such analysis could give a broader overview of current trends and the mutual interest of a diverse set of stakeholders. In this context, the present work considers analyzing energy clusters with different perspectives such as geographical scope, members companies, and focus topics. A total of 44 energy clusters along with 4524 members are taken into account in this study.

Background and Context:

In the literature, the theory of innovation clusters has been widely studied by various scholars. There are different approaches have been developed to analyze an innovation cluster. These methods include namely qualitative analysis, interviews, patents, etc. Nevertheless, the majority of approaches rely on either controlled input (e.g., cluster manager interview) or a lack of updated activities (e.g., patent analysis) [2]. Along these dimensions, previous studies were unable to provide ground activities of energy clusters and lacked providing the real impact of various energy cluster policies. Following the discussion above the present work has tried to answer the following research questions.

Research questions:

1-Who are the major actors involved in setting up an energy innovation cluster and how these clusters are organized to meet the goal of various participants? 2-How unstructured data from the cluster and their member's websites can be used to gain insight into recent innovative activities in the energy sector?

From the theoretical point of the view, the work has provided a comprehensive overview of major cluster policies in different federal states of Germany.


Over time website data is thriving and is considered as an essential source to measure emerging innovative actions. In this context, the energy-cluster website contains the most updated about cluster activities such as ongoing activities, events, projects, and publications. Therefore, the proposed methodology primarily banks on the website data. As an initial step, a list of energy clusters is obtained from the Cluster platform Germany [3]. A total of 44 energy clusters along with 4524 members are taken into account in this study. Furthermore, an internally developed web-scraping tool is used to crawl the cluster and member's data. The tool performs systematic and guided web-scraping for searching a keyword presence on the particular web-page. Two distinct sources namely European Structural and Investment Funds (ESIF) and Scival platform are used to mine energy-innovation related keywords. Different categories and sub-categories are assigned to cover a wide range of topics and actions related to major energy fields. Moreover, a commercially available company database is used for matching firms in the member's list. Python based natural language toolkit (NLTK) is used to clean the data and different analytics such as co-relation analysis of topics, decile ranking, and frequency analysis is done by utilizing text data.

Main outcomes: The main result highlights are summarized below:

• The work has analyzed the major actors driving cleantech innovation clusters in Germany. To this end, the composition of different clusters has been analyzed. Similar to innovation clusters active in other fields, the key members in an energy cluster are firms (Large, Medium, Small, and Startups), research institutes, universities, public organizations, and financing actors. However, cluster composition in the energy sector is also populated with several other actors (e.g., Training centers, Innovation matchmakers, co-working facilities, innovation consulting, etc.) and individuals.

• The overall results have indicated that the majority of energy clusters are very specialized in certain topics such as hydrogen, carbon, and bioenergy and are getting notable attention from various stakeholders. In addition, various cross-sectoral topics (e.g., bio-hydrogen, Organic solar cells) are also emerging due to the growing interaction between different sectors.

• Energy clusters and their member count is unevenly distributed in Germany. Therefore, new policy measures are required to create a supportive framework for the regions with relatively lower participation.

References: 1- G. M. Z. Koecker, Clusters in Germany - an empirical based insight view on emergence, financing, management and competitiveness of the most innovative clusters in Germany, Report (04 2008). 2- M. Tvaronaviciene, K. Razminiene˙, L. Piccinetti, Aproaches towards cluster analysis, ECONOMICS & SOCIOLOGY 8 (2015) 19–27. doi:10.14254/2071-789X.2015/8-1/2. 3-Cluster platform Germany. (accessed in November, 2021).

Using Granular Start-up and Project Data to Analyze Global Scaling of Novel Climate and Energy Technologies

ABSTRACT. Background and Rationale Accelerating technological innovation is essential for meeting long-term decarbonization goals and for supporting economic development and employment by expanding emerging green industries. Currently mature technologies will meet only a quarter of long-term sustainable energy goals according to the International Energy Agency. Addressing this challenge therefore requires ensuring that new climate technologies scale up rapidly and can be deployed cost effectively. Energy innovation systems research suggests that various actors—especially governments, start-ups, large firms, and private investors—and their interactions will be involved in scaling up innovation to a commercially ready product. However, currently, there is no systematic approach for connecting empirical data on commercially viable, early-stage technologies and the various actors involved, with systems models that investigate large-scale deployment and inform energy and climate policy decisions.

Standard methods of developing learning curves require a track record of deployment that new technologies lack, and expert elicitations are resource-intensive and may be less amenable to analyzing commercial breakthroughs. Linking realistic, bottom-up empirical evidence on technologies such as start-up investment data, project databases, and historical technology analogues with macro-level energy systems models can help overcome these difficulties to inform scaling trajectories for emerging technologies. Here, we propose a framework for integrating such granular real-world data into an open-source integrated assessment model, i.e., the Global Change Assessment Model (GCAM), and use direct air capture and hydrogen as examples of how this framework can support analysis of technology development and just energy transitions.

Methods The technologies selected are based on the following criteria: (a) technologies in early development stages that will be important for deep decarbonization pathways (b) technologies developed by start-ups with substantial investment records, particularly from private investors (especially strategically and financially motivated corporations), (c) technologies with new model developments in GCAM (often related to previously understudied areas, indicating a strong potential for high-impact results).

We examine two case studies. Our assessment of direct air capture (DAC) technologies draws on publicly available investment data on commercial purchase agreements for carbon dioxide removal, as reported by Stripe and, and plant capture rates from IEA project data and announcements from company websites. Our examination of hydrogen utilizes bottom-up project data and company data on global investments to investigate regional scale-up trajectories and resulting equity implications in future energy systems.

We use these two cases to estimate variations in technology diffusion through projected changes in the S-curve. An S-curve has three degrees of freedom: the inflection point, the growth rate, and the maximum value. The inflection point and growth rate are derived from historical technology analogues by fitting an S-curve to historical capacity data for those technologies. The maximum value is calculated using investment data and publicly available purchase agreements or plant operation announcements.

We implement the changes in S-curves in GCAM. GCAM is a technology rich model with detailed regional representation of energy production, transformation, distribution, and end use. GCAM projects systems outcomes through 2100 at five-year intervals, including energy supply and demand by technology and vintage, regional prices for goods and services (e.g., electricity, agricultural commodities, etc.), emissions of greenhouse gasses and short-lived species, and climate outcomes. For each five-year time step, GCAM solves for a set of market prices such that supply equals demand for all markets and sectors and across all regions (32 geopolitical regions, 384 land-water regions).

In the DAC case study, we use purchase agreements and plant announcements for 16 DAC start-ups which report the amount of carbon a company plans to capture in the near future. Assuming a start year of 2010, the first year of operation for a DAC plant according to IEA, we project the next ninety years of DAC assuming a growth rate and inflection year identical to different historical analogues. We specify the exact capacity of DAC for which GCAM should solve at every five-year time step, using the S-curve parametrization calculated with historical technology analogues and near-term DAC projections.

For hydrogen, we separate start-up and IEA project-level data by region and technology type (e.g., green hydrogen, blue hydrogen, etc.) to examine how different scale-up trajectories may evolve in different geographical areas. Because hydrogen is a manufactured product rather than an extracted resource, future production will be heavily dependent on current investments in innovation and infrastructure rather than solely on place-dependent natural resources. We compare the granular data to expected need as seen in national and regional hydrogen strategies, and in a net-zero pathway in GCAM. This allows us to see the future geopolitical implications of current funding inequities between the Global North and Global South and determine where hydrogen technologies are likely to be underdeveloped compared to expected needs.

Anticipated Results Preliminary results suggest that, if DAC adoption follows historical analogues, the timing, growth, and market saturation of DAC technologies differ significantly from what is typically observed in models that currently inform global, national, and sub-national climate policy. The median logistic growth of technologies in our dataset (measured in cumulative capacity) is 17%, but with a wide range of growth (2% - 200%). This result suggests that technologies grow at vastly different rates, impacting the cumulative capacity of the technology as well as the time it takes for the technology to scale up to its eventual steady state.

Preliminary analysis of regional hydrogen development shows differing patterns of deployment between developed and developing countries. Developed countries have many small-scale hydrogen projects and significant innovation support for start-ups that span a wide range of technologies. Developing countries have fewer start-ups with less innovation support and tend to build a small number of high-capacity projects. Several countries that expert elicitation suggests will be key players in future hydrogen markets currently fall short of their national targets (including Chile and the United States), while others are on track to meet or exceed their targets (China and South Africa).

Significance To reach a net-zero world, rapid development and deployment of pre-commercial technologies is essential. In order for researchers and policymakers to accurately assess the potential contributions of these technologies to decarbonization pathways, new methods of analysis are needed to determine realistic scaling pathways. Here, we demonstrate two case studies of how such analysis can be developed, and the resulting implications for ambition for carbon dioxide removal, negative emissions, and global equity in future hydrogen systems.

Leveraging technology spillovers to accelerate clean energy innovation

ABSTRACT. *Background and Rationale* Spillovers of knowledge across technology domains are one of the key drivers of technological innovation, as knowledge initially developed in one technology area can be applied in another area to enable discovery and invention, reduce technology costs, and improve technology performance. Contributions of technology spillovers to clean energy innovation are expected to play a major role in the global push towards cost-effective decarbonization of the energy system. Despite the importance of technology spillovers, we have limited understanding of the micro-level processes of how knowledge spillovers across technology domains occur, what factors enable or affect spillovers, or how spillovers can be most effectively spurred and leveraged by public policies targeting the decarbonization of the energy sector.

*Methods* We have undertaken a multi-year mixed-methods investigation of technology spillovers in three important clean energy technologies: solar photovoltaics (PV), lithium-ion batteries (LIB), and solid-state lighting (SSL) (specifically, white light-emitting diodes (LED)). We use process tracing to combine and integrate quantitative and qualitative empirical evidence from a broad survey of the scholarly literature and primary documents, expert interviews, analysis of citations in patents and scientific publications, and a machine learning-based method of patent text analytics to understand the mechanisms and enablers of technology spillovers. Using these methods, we reconstruct the historical contributions of spillovers to individual innovations in the three technology areas, and then develop an inductive generalized typology of technology spillover mechanisms and enabling factors.

We find evidence of crucial knowledge contributions made by technology spillovers to 15 innovations in solar PV, 12 innovations in LIB, and 9 innovations in white LED-based SSL. In all three domains, identified spillovers were particularly important at the early stages of innovation, as they enabled key components and manufacturing processes that eventually became integral parts of the first commercial LIB and white LED products and the solar cell design that dominated the PV market for several decades. However, the impact of spillovers is not limited only to the stages of early research and development (R&D). We also find evidence of spillovers occurring at later stages of technology demonstration and market formation. Across the three areas, spillovers contributed to innovations in technology components, materials, system architecture, and manufacturing processes. In addition, spillovers drove a majority of improvements in consumer experience characteristics of white LED-based lighting products.

*Results* Based on the analysis of identified spillover processes in three technologies, we identify four types of “spillover mechanisms,” different ways by which a spillover can occur: (1) learning and researching, (2) communication and collaboration, (3) human mobility (both physical and across disciplines or fields), (4) exchange of physical objects that embed knowledge, such as manufacturing equipment. Importantly, these four mechanisms are not mutually exclusive, as a spillover can operate through several simultaneously occurring and supporting mechanisms. Observed spillovers also differ in how intentional they were. In many cases, spillovers occurred as a result of a targeted search for external knowledge needed to solve a local problem or targeted application of available knowledge in a new area or for a new purpose. However, a few notable spillover cases also occurred serendipitously in the process of undirected “blue-sky” research.

We identify many broad categories of enabling factors for technology spillovers, including, but not limited to, external sectoral and market shocks; booms of R&D in external but related technology domains; freedom of search in the laboratory setting, both in academia and industry; multidisciplinary education and training; cross-disciplinary hiring and team composition; multi-sectoral firms; and academic-government-industry partnerships and knowledge exchange events.

Finally, we highlight four categories of public policies that enable spillovers: (1) public R&D funding that stimulates cross-disciplinary knowledge search and collaboration; (2) public funding for technology demonstration projects; (3) deployment policies that create incentives for strategic entrepreneurial knowledge search by incumbent firms in other industries; and (4) cross-cutting policies that support cross-sectoral and systemic coordination, such as government-industry round tables and roadmapping exercises. We also find that “stop-go” funding cycles negatively affect spillovers into clean energy technologies, potentially delaying resulting innovation and technology deployment.

*Significance for Policy* Based on our findings, we propose a set of five principles that can guide the design of energy and innovation policies and management practices that leverage technology spillovers to accelerate clean energy innovation.

First, for any organization, firm, or funding agency, there is a need to recognize and acknowledge the trade-offs that exist in supporting innovation activities in a focal knowledge domain against supporting a broader multi-disciplinary knowledge base involving multiple knowledge domains. Access to a broad pool of knowledge can accelerate innovation in the focal domain through technology spillovers, but it may not work on its own without a deep understanding of the focal domain and the ways by which external spillover knowledge can be applied in it. These trade-offs suggest that there is a balance to be found between the depth and breadth of knowledge search in an organization, and that this balance can be proactively pursued and managed.

Second, policies that aim to leverage technology spillovers for innovation should be flexible, both in the choice of particular policy and funding instruments and in policy design, to allow for knowledge search in unexpected directions. Some of the most notable spillovers in our study occurred when researchers were able to pursue “blue-sky” research or worked in public- or industry-funded mission-oriented R&D while still being allowed a certain freedom of search.

Third, we find that continuous knowledge exchange between science, engineering and manufacturing is important for the generation of spillovers, which often requires deliberate management of boundaries within organizations and disciplines to alternatively nurture development of knowledge within an organization or discipline and cross-pollinate with external knowledge across the stages of innovation.

Fourth, cross-disciplinary, cross-sectoral exchange should be supported at all organizational levels: from individuals (e.g., through temporary or extended placements of public sector and university researchers in industry) and events (e.g., conferences mixing scientists, engineers, and industry representatives) to teams (e.g., multidisciplinary team composition), organizations (e.g., through academia-industry collaborations and alliances) and platforms for broad cross-sectoral and cross-disciplinary collaboration (e.g., industry roundtables, roadmapping, and visioning or foresight visioning exercises).

Finally, all instruments and activities supporting technology spillovers should not be restricted only to R&D policy. They should also be included in the policy mix for the support of innovation at all stages, including technology demonstration and market deployment. This principle is particularly relevant for innovation in clean energy technologies, in which deployment policies are known to have played a crucial role in stimulating dramatic cost reductions and technology deployment over time and are expected to continue playing this role in the future.

Mission-oriented policy learning for an agile energy research program – The case of Germany

ABSTRACT. Background and rationale

For Germany, the transformation of the energy system has become more urgent than ever. The necessary acceleration of climate protection and the gas supply shortage since the Russian war in Ukraine have initiated a reorientation of the German energy transition as a mission-oriented innovation policy strategy. Its objectives consist of a triangular between environmental objectives such as emission reduction and renewable energy diffusion, affordability and security of supply. However, recent developments have increased the importance of technological sovereignty and strategic industrial policy as additional policy objectives. In this situation, the German Ministry of Economic Affairs and Climate Protection aims at reconceptualizing its Energy Research Program until 2023. In a research project funded by the ministry, our team receives the opportunity to accompany this reconceptualization. The ministry's objective is to make use of a better operationalization of the overall socio-technical objectives of the mission-oriented energy transition. Moreover, the Energy Research Program shall be better coordinated with other stakeholders in the innovation policy system and other policies. These interfaces to different stakeholders and policy instruments need to be identified and explicitly addressed in a new governance concept of the program. For example, missing interfaces need to be discussed from both a theoretical and a practicability viewpoint for better coordination of how successful projects from the program can receive further support via regulatory or demand-sided instruments. In that way, the new Energy Research Program shall be embedded more narrowly to the mission - and guarantee better coordination with other existing innovation policy and energy policy instruments.

Research questions and objectives

The recently initiated research project has the opportunity to accompany innovation policy design processes in real-time to dismantle the processes of designing and coordinating policy instruments. The insights into the reorientation process help explore innovation policy from a practitioner's perspective, which needs to be included in conceptualizing policy recommendations. The project pursues two objectives. First, it aims to translate the existing knowledge from the literature and our previous research on an agile, mission-oriented science and innovation policy (Weber et al., 2021) to the German Energy Research Program's case. Thus, suitable and practicable policy recommendations for a new governance concept and possible instruments shall be developed and simultaneously discussed with the relevant policy actors. Second, the project aims to accompany the debate and the implementation process for the new Energy Research Program to explore what challenges practitioners need to deal with during the policy design and implementation process. Therefore, the project addresses the following research questions: • Which policy instruments and governance processes are part of the German Energy Research Program? • How can new research on an agile, mission-oriented innovation policy be conceptualized into practicable policy recommendations for the reconceptualization of the Energy Research Program? • What organizational capabilities are required for the concerned policy actors (ministries or project operators) to support the design and implementation processes and learning cycles within the Energy Research Program? The first research question uses previous research on the analysis of innovation policy instruments and their design principles to structure existing activities within the Energy Research Program. Such a structured view is missing so far. Based on the literature, the mapping of policy instruments and governance processes helps to find starting points for proposing policy recommendations. With the term governance processes, our project refers to the processes how instruments such as calls for projects address specific socio-technical needs of the energy transition and how they are designed in detail. Governance processes also include strategic adjustments of instruments along specific socio-technical needs. Moreover, these processes target the capability to select successful projects for further support or other policy instruments aiming at increasing the production or diffusion of the successful technology. Governance processes include the monitoring of key performance indicators and evaluation activities to adjust instruments and governance processes. The second question focuses on the translation of existing literature to the case of the Energy Research Program. The third question examines the implementation challenges for policy actors' mission-oriented instruments and governance processes.


The research project is based on a qualitative case study design using different data sources. It consists of two phases. First, the mapping of instruments and the governance processes is conducted based on a structured review of public documents, the existing research literature and participating observations during meetings and workshops in the ministry. Simultaneously, a structured literature review on an agile, mission-oriented and transformative innovation policy, with a particular focus on the energy transition, is conducted. As a result of the mapping and the literature review, a proposal of possible mission-oriented governance processes will be elaborated and discussed in workshops with the relevant actors and stakeholders in the ministry. This includes the challenge to actively address the interfaces of the Energy Research Program with other ministries and instruments within the energy transition innovation policy mix. In the second phase, we accompany the implementation of the new Energy Research Program via semi-structured expert interviews in the relevant departments of the ministry, the project operators and other relevant stakeholders. Thus, we may refine the conceptualization of mission orientation in its daily policy implementation processes. The interviews, the participating observations and the public documents on the Energy Research Program represent the empirical base of the qualitative case study, which is analyzed in an iterative coding process using MAXQDA. As the second phase is supposed to begin in March 2023, we will present the proposal for an agile, mission-oriented energy research program and preliminary empirical findings on its implementation.

Preliminary results and significance

Our previous research on agile innovation policies suggests that mission-serving programs based on several instruments and governance processes require the connection between strategic objectives and the operative selection and implementation of policy instruments and their design features (Weber et al., 2021). The proposed project explores the connection between the strategic and operative levels of policy-making by considering how strategic objectives are operationalized in the Energy Research Program, for example, in the form of calls for projects. In addition, the identified interfaces to the concerned policy actors and to other instruments and programs of the German energy transition policy mix help to understand how coordination at the strategic and operative level of innovation policy-making looks like. Moreover, previous research has highlighted the problem that often new instruments are implemented without considering if existing instruments can cope with new challenges. As a result, programs and policy mixes become far more complicated than necessary (Flanagan et al., 2011). By explicitly identifying the instruments and design features of the program, we can analyze unnecessary policy path dependencies and discuss the option to replace or abolish instruments. Furthermore, implementing mission-oriented policy concepts on former purely supply-sided instruments such as direct project funding programs represents a new field because mission-oriented policy research often focuses on demand-sided measures for accelerating the diffusion of innovations. The proposed project fills this gap by using the long experience of this supply-sided program and by detecting the interfaces to other instruments within the mission-oriented innovation policy mix.


Flanagan et al. (2011): Reconceptualising the 'policy mix' for innovation. Research Policy 40(5), 702-713. Weber et al. (2021): Agilität in der F&I-Politik. Konzept, Definition, Operationalisierung. Study on the German Innovation System 8-21 for the German Expert Commission for Research and Innovation.

13:15-14:45 Session 3G: Open Science and Accessibility
The Economic Logic of Open Science in Fusion-Energy Research: A Systematic Approach to Policymaking

ABSTRACT. 1. Background and Rationale

Climate change is a grand challenge of our era. From an economic point of view, government intervention is justified to address two market failures: one environmental and one technological. The former results from an overprovision of electricity from fossil fuels, while the latter from an underprovision of research to develop greener technologies. Environmental technological policy must jointly address both market failures due to their interconnections. On the one hand, the standard approach to climate policy has been to limit the emission of greenhouse gases (GHG). On the other hand, the adoption of greener technologies can reduce or eliminate GHG emissions.

This paper focuses on the development of one of such greener technologies: nuclear fusion, which is the reaction that occurs within the stars, including the Sun, to generate their own energy. A technological breakthrough in nuclear fusion energy can help to decarbonize the power system because it can provide clean, safe, and affordable energy without the amount of nuclear waste resulting from fission reactions, which currently occur in fission power plants around the world. However, fusion-energy research is still on the edge between science and technology because it still requires large investment in basic research before developing technological applications.

Despite the infantry stage of fusion-energy research, this industry is experiencing a transition from publicly to privately funded research. The influx of private funds is changing the objectives of fusion-energy research from an approach based on open science, where the scientific results are shared within the global fusion community, to an approach based on proprietary technologies, where each company protects its technological applications with Intellectual Property Rights (IPRs).

Whereas the enforcement of IPRs can create economic incentives for private companies to generate even more technological applications in the field of nuclear fusion and will eventually lead to the commercialization of fusion power plants, stricter IPRs are often associated with high social costs that result from restricting the access to the knowledge that was previously shared within the global fusion community and thus limit the effectiveness of nuclear fusion in climate-change mitigation policies.

2. Methodology

Building on the new Economics of Science, this paper develops a systematic approach to science and innovation policy in fusion-energy research as it transitions from public to private funding. The overall objective of this systematic approach to policymaking is to strike a productive balance between the publicly funded community of scientists, who favor full disclosure of their results, and the community of privately funded technologists, who favor disclosure only after a patent. This productive balance between the two communities can be achieved when science policies fund open science and check against excessive incursions of claims to Intellectual Property Rights. To achieve this objective, the methodology employed by this paper is to interview key players in both communities to gather information on how they view open science in fusion-energy research. Interviews allow to gather rich information on this topic and they are a qualitative research method that can be used as a foundation for future quantitative research on this topic. To the best of our knowledge, no other authors have outlined a systematic approach to policymaking in this industry. Despite its limitations, the qualitative method employed in this paper has the ability to assist policymakers who seek a regulatory framework to guide the future of this industry.

3. Anticipated Results

Some of these interviews have already been collected by Michel Claessens, who was the spokesperson from 2011 to 2016 for the International Thermonuclear Experimental Reactor (ITER) project, which is currently the largest publicly funded project in fusion-energy research. These interviews provide preliminary evidence that scientists who are working in the ITER project follow the Mertonian norms of Communalism, Universalism, Disinterestedness, Originality, and Skepticism (CUDOS).

These preliminary results support a systematic approach to policymaking in fusion-energy research that stresses the benefits of openness in science because all the intellectual property developed in the project is shared equally by all the global members of ITER.

Claessens also interviewed some of the private companies that are supplying equipment to ITER. These managers also confirmed that openness in science allows the communal, universal, and original approach required to supply hi-tech components that meet the stringent specifications of a first-of-a-kind project, such as ITER, whose consortia of suppliers span across the world.

4. Significance

Climate change is a complex problem that requires multiple solutions from multiple disciplines. Science and innovation policy in greener technologies can help to mitigate this complex problem by fostering the development and adoption of technologies that can reduce GHG emissions. Although not a panacea, nuclear fusion can provide clean, safe, and affordable energy in the future.

However, fusion-energy research is undergoing two paradigm shifts that will jointly shape its future. First, it is transitioning from science to technology as more and more technological applications are providing different approaches to the future commercialization of fusion-generated electricity. Second, it is transitioning from public to private funding as more and more private companies are entering the newborn fusion industry. This large influx of private funding is shifting the focus of fusion-energy research from full disclosure of scientific results within the fusion community to stricter enforcement of intellectual property rights that could have a negative impact on research.

Now more than ever fusion-energy research needs a regulatory framework to guide this transition. This paper plans to build a systematic approach to policymaking in fusion-energy research to show that science and innovation policies that favor openness in science can provide greater economic and social utility to both the publicly and privately funded research communities in fusion energy. The main policy recommendation derived from this paper urges the governments that are currently sponsoring private endeavors in fusion energy to keep science open in exchange of public funding.

The systematic approach to policymaking derived from this paper can provide a policy toolkit for managing complex and dynamic technologies in several other sectors, such as aerospace. Moreover, the implications for sustainability derived from this paper can assist the policymaking system with a multicriteria approach to assist the formulation and implementation of climate-change mitigation policies in other technologies of public interest, such as transportation.

How to make research data available and ensure compliance with the OSTP Memo

ABSTRACT. In August 2022, the Whitehouse Office of Science and Technology Policy announced new requirements from federal agencies “Ensuring Free, Immediate, and Equitable Access to Federally Funded Research”. Namely, this requires executive agencies and departments to:

Update their public access policies as soon as possible, and no later than December 31st, 2025, to make publications and their supporting data resulting from federally funded research publicly accessible without an embargo on their free and public release; Establish transparent procedures that ensure scientific and research integrity is maintained in public access policies; and, Coordinate with OSTP to ensure equitable delivery of federally funded research results and data.

In 2019, we saw the number of open access academic papers published globally, passing 50% of all papers published. There are also many new business models and routes to support open access as the push has steadily gained moment since it was first conceived in the 1980s. Authors and funders have a plethora of green and gold open access publication routes, although managing the cost implications of the latter is an ongoing discussion.

Open academic data is largely a new concept to researchers and funders alike. Whilst several fields have been benefiting from making data available for some time, for example, genomics and astrophysics - many communities are struggling with the concept of where and how to make their research data accessible. Several are struggling even with the concept of what ‘data’ means to their field.

The State of Open Data report from Digital Science surveys researchers on their attitudes and concerns with regards to open academic data. Since 2016, we have monitored levels of data sharing and usage. Over the years, we have had 25,000 responses from researchers worldwide providing unparalleled insight into their motivations, challenges, perceptions, and behaviours toward open data. The State of Open Data is a critical piece of information that enables us to identify the barriers to open data from a researcher perspective, laying the foundation for future action. This year’s report also includes guest articles from open data experts at the National Institutes of Health (NIH), the White House Office of Science and Technology Policy (OSTP), the Computer Network Information Center, Chinese Academy of Sciences (CNIC, CAS), publishers and universities.

A separate Digital Science report led by Ripeta is titled The State of Trust & Integrity in Research has insights into how researchers are actually acting in practice when it comes to open research data. It demonstrates that improved data sharing policies will boost public trust in research

In addition to its detailed analysis of the five major funders, Ripeta found that with regard to 62 key funders worldwide: 71% of funders required data management plans 68% of funders covered expenses of data management and sharing 66% specifically mentioned data sharing repositories as a mechanism of making data publicly accessible 26% of policies mentioned how long data should be retained.

Locating funding agency policies proved to be particularly difficult, with many not surfacing through web searches. There was also significant policy variability between funding agencies, with many policies having differing requirements for implementation. Utilizing Ripeta’s services, funders and government agencies have the ability to analyze and monitor compliance within established open science or data management and sharing policies, while also identifying areas of good institutional research practice.

Exclusive analysis by Ripeta in one section of the report compares the policies and practices of five major world funders: the Bill and Melinda Gates Foundation, the European Commission (EC), the National Institutes of Health (NIH) in the United States, the National Natural Science Foundation of China (NSFC), and the German Federal Ministry of Education and Research (BMBF). The analysis shows that although each of the funders have their own policies in place to support data sharing, there are discrepancies between those policies and how they translate into practice. For example, the number of research papers published in open access (OA) journals varied greatly depending on the funder. The NIH (95%) and the Bill & Melinda Gates Foundation (93%) far outstripped the representation of OA publications of other funders, especially the NSFC (33%).

This session will discuss what we know about how to encourage researchers to make their data available and how we can ensure compliance with the OSTP memo going forward.

Open Access policies, their effectiveness and unexpected consequences

ABSTRACT. Introduction: in the last part of the previous century, the world witnessed tre upraise of a movement aiming at organizing the access towards scientific knowledge in a more egalitarian manner. In particular the university librarians in the US plead for a more evenly distributed access to global scientific literature for scholars in the Global South, and hence declared war on the power of the scientific publishing industry. So, the initial aims of the Open Science movement, as it was coined, focused on equal access to scholarly literature, as subscription rates were sky-high, which meant that reading of the most important scholarly journals was impossible for many in low- and middle-income countries. Only when three major statements were made, in gatherings in Budapest, Bethesda, and Berlin, with increasing attention, and known as the B-B-B declarations on open science, science policy and research funding got involved, leading to the development of national open science mandates, institutional open science policies, and open science requirements for research grants. Note that in these initial initiatives, it centered mostly around measures on open access to scholarly publishing. In this study we will aim at a few of these policies, and the way these have worked out. In doing that, we will analyze the full broadness of results, including any un-wished for outcomes, in line with Stone’s book “The Policy paradox”, in which she claims that undertaken policies often have paradoxal consequences, for which one did not aim at the time the policies were developed and undertaken.

Policies: As stated above, many countries issued national policies and mandates on open science, in particular on open access for scholarly publishing. Policies focused on various types of open access publishing, like a preference for Gold Open Access in the UK and the Netherlands, and a focus on Green Open Access in for example Denmark. This national perspective often resulted in likewise policies on the institutional level, although these often also covered other aspects of open science, such as open data, open source code, open logbooks, etc. In 2018, a consortium of international funders, under the name of cOAlition S, launched Plan S, an outline on how to publish in open access format. Plan S prioritized Gold open Access over other types of open access, in particular over Hybrid Open Access. The difference between Gold and Hybrid Open Access is in the fact that Hybrid still relates to the subscription journal, so publishers still cash in on both subscription incomes, as well as the individual payments for individual single papers submitted to their journals, while the Gold Open Access type means that the whole journal is open access, and accessible for everybody who is interested, without any restrictions whatsoever. A crucial role is played in the debate around Plan S by these payments for open access publishing, the so-called Article Processing Charges (APCs). Plan S proposed to use a cap, a maximum of money one single open access publication might cost. In the hybrid situation, it is still up to the publisher, often owners of highly prolific internationally oriented journals, to determine the value of the APCs.

Studies: In this section we will discuss the outcomes of two studies, both related to open science policies and both evolving around the phenomenon of APCs. In the first study we will show that the national open science policy of the Netherlands, started in 2013/2014, and aiming at Gold Open Access as the standard format for open access publishing, is clashing with the development initiated by the Dutch universities. Ibn the Netherlands, universities do have a high degree of autonomy, and are well organized in contexts such as university library settings, evaluation cycles, and overall governance of the institutions. After the national government issued a national open access policy, prioritizing Gold as the default open access format, the universities, organized together with the royal library in the so-called UKB, negotiated with the publishing industry specific deals that allowed Dutch scholars to publish in journals within the subscriptions with these publishers, so in Hybrid open access format. This development started in 2016/2017, and we will show the effects of these two contradictory initiatives and the way scholars from various scholarly domains have reacted to this. The second study focuses on the Gold Open Access publishing on a global scale. While the original motivation for the open science movement was a better access for the scholars of low- and middle-income countries to reading scholarly literature form the Northern hemisphere, this analysis shows that the current development of Gold Open Access publishing is driving into the direct opposite direction. The study will show that scholarly publishing in Gold Open Access journals via the payments of APCs is becoming more and more expensive. A direct translation of numbers of publications times APC-rates is showing that publishing becomes more costly, but when normalization for national welfare situation is conducted, by applying the OECD based PPP-rates (Purchasing Power Parity), the situation even worsens. So while initially, scholars form the low- and middle-income countries could publish, but had a hard time reading scholarly literature, we are now moving into a situation in which these scholars can read the global literature, but do have little to no access to the publishing dimension anymore.

Conclusions: What we witnessed in the two studies we conducted is a complex power structure, with a variety of actors, both supra-nationally and nationally, conflicting interests within the national context, funding agencies, and a variety of motivations (academic, commercial, individual). A clear issue in the debate around Plan S was the relationship of the consequences of Plan S with the existing reward & recognition systems, as well as career perspectives of early career researchers in an international context. This is also reflected in the study on the Dutch system, whereby academic freedom to choose the journals that suit you best is conflicting with prescribed ways of publishing in both the national mandate as well as in Plan S. Finally, the study on global Gold Open Access publishing reflects a development towards a more unequal access to scholarly publishing, along the line of available financial resources. All these outcomes clearly show unexpected consequences of the policies undertaken.

Stone, D., “The policy paradox”, New York: W.W. Norton & Company, 2002

Information accessibility and knowledge creation: the impact of Google’s withdrawal on Chinese scientific publications

ABSTRACT. Since Google entered mainland China in 2006, its share of the total search engine market of mainland China rapidly increased to 40.08% by the end of 2009. Together with the Chinese firm Baidu, which offers a similar service portfolio and held a market share of 58.47%, Google effectively became part of a duopoly (Kong et al., 2022). Google was, hence, a main source of information in China, especially of information from foreign countries (Kong et al., 2022; Wang et al. 2020). Like any search engine provider operating in China, Google was obliged to follow the strict censorship guidelines imposed by the Chinese government, but, in January 2010, Google decided to discontinue the censoring of search results on its China search page. This decision rapidly escalated in a sudden and unannounced withdrawal of all Google services from China, leaving millions of users without access to the world’s top search engine overnight. From the 30th of March 2010 onwards, users in China could not access Google services anymore (The Guardian, 2012; The Official Google Search Blog, 2012; Bloomberg, 2014; Xu et al., 2021; Kong et al., 2022). In this paper, we investigate the effect of Google’s sudden exit from China on the scientific research output of Chinese scholars. Access to information in the form of books and research material has been shown to be crucial for the generation of new knowledge (Furman and Stern, 2011; McCabe and Snyder, 2015; Waldinger, 2016; Berkes and Nencka, 2019; Mueller-Langer et al., 2020; Furman et al., 2012; Biasi and Moser, 2021; European Commission, 2012). A lack of access or high accessibility costs can, hence, be a key barrier to new discoveries and knowledge creation. Not surprisingly, information and communication technologies have been shown to enhance science production by increasing the availability of information and, hence, reducing search costs (Agrawal and Goldfarb, 2008; Ding et al., 2010; Winkler et al., 2010; Kim et al., 2009). Google’s sudden exit from China, therefore, bears the risk that Chinese researchers lose touch with the research frontier and persistently lag behind their foreign peers. Using Google’s exit from China to assess the effect of barriers to information accessibility has several advantages which address common challenges for causal estimation. First, Google’s exit was exogenous to science production and unexpected as it was the result of a rapid escalation of political tensions between the Chinese leadership and Google (Zheng and Wang, 2020; Xu et al., 2021; Kong et al., 2022). Second, Google was, at the time of the sudden withdrawal of its services, one of the main sources of knowledge for China (Kong et al., 2022; Wang et al. 2020) and its scientists (Qiu, 2010). Our empirical analysis focuses on the field of economics following prior studies such as Kim et al. (2009), McCabe and Snyder (2015), Liang et al. (2022), and Piracha et al. (2022). Economics is a research field with a simple knowledge production function as it does not rely on material and expensive equipment (Stephan and Levin, 1992). New insights are published almost exclusively in scientific journals rather than in books and conference proceedings which are often not well covered in bibliometric databases (e.g. Michels and Fu, 2014). Hence, an estimated effect of the sudden decrease of information accessibility on scientific output is less likely to be confounded by other effects resulting from the knowledge-generating process or publication strategy of the field. To derive causal results, we use a Difference-in-Difference (DiD) approach employing a control group of researchers located in Taiwan and Hong Kong following Zheng and Wang (2020) who argue for a control group that is culturally, economically, and geographically closely related to China. Our results show that researchers affiliated with Chinese institutions experience a significant decline in both their research output quantity and impact as measured by citations received by the future literature. The magnitude is about 28% for co-author-weighted publications and 30% for co-author-weighted citations. We explore the proposed underlying mechanism of information accessibility further and show that the productivity and impact of those Chinese scholars that work with foreign co-authors are less affected by Google’s exit. These scholars can use their interpersonal networks as a channel for knowledge access (Singh, 2005; Mohnen, 2022). The publication output and impact of these scholars decreases by smaller shares of 20% and 22%, respectively, supporting that the mechanism of knowledge accessibility is responsible for the decline in publication output after Google’s withdrawal. In further analysis, we find that the effect in terms of quantity and impact is stronger for those scholars with the highest impact as measured by their citation stock over publication stock before Google’s exit. The publication output and impact of the top 25% scholars decrease by 39.5% and 37.5%, respectively, while the publication output of the scholars at the bottom of the impact distribution decreases by 20%. There is no significant effect for the scholars at the bottom of the impact distribution in terms of impact. The large effects on the top scholars raise concerns about the ability of China to stay in touch with the research frontier in the medium and long run with potentially harmful implications for economic growth (Griliches, 1992; Jaffe, 1989). We make several contributions to the literature. First, our work adds to our understanding of the determinants of knowledge creation (Stephan and Levin, 1992; Stephan, 1996, for an overview) and more specifically of the role of information and communication technology in knowledge creation (Agrawal and Goldfarb, 2008; Ding et al., 2010). Prior studies have shown that access to network technology (Agrawal and Goldfarb, 2008; Ding et al., 2010, for the case of BITNET) eases information accessibility and facilitates the knowledge production of scientists. Here, we focus on Google as a general search engine and complement prior findings for different technologies. Second, we contribute to recent literature that focuses on positive information shocks such as the availability of access to libraries (Berkes and Nencka, 2019; Furman et al., 2012; Biasi and Moser, 2021), of research resources (Furman and Stern, 2011) and of online access to scientific journals (McCabe and Snyder, 2015; Mueller-Langer et al., 2020) and their impact on knowledge creation. We differ from these studies in two ways. First, these studies focus on the access to prior scientific knowledge available in form of books, journals, and research resources while we focus on the access to a search engine that covers a much broader scope of information. Second, we explore a negative shock of information availability to assess the effects on science production while prior studies focus on positive shocks of information availability. Third, we add to the developing literature that focuses on the implications of Google’s China exit. These include a higher stock crash risk for firms (Xu et al., 2021) and a decrease in corporate innovation (Kong et al., 2022; Zheng and Wang, 2020). Other than these prior studies, our focus is on the scientific rather than on the corporate sector.

15:15-16:45 Session 4A: Decisions and Scientific Productivity
Does Double-blind Peer-review Effectively Correct for Demographic Disparities in Research Funding?

ABSTRACT. Background

We examine the ‘Villum Experiment’ (VE), a funding program instigated by the Villum Foundation, a Danish private funder who supports blue-sky research in the natural and technical sciences in Denmark. The VE instrument is supposed to support unorthodox ideas in their early phase. More specifically, the instrument was created for those research projects out of the ordinary that challenge the norm and have the potential to change fundamentally the way we approach important topics. The VE should therefore promote ‘risky ideas’ and ‘new paths’ in the natural and technical sciences. Interestingly, to mitigate well-known biases in peer review, applicants are kept anonymous to the reviewers. Assessments are solely based on short structured applications where the focus is upon the ‘idea’, no CVs or names are disclosed. Non-blinded peer reviews are seen as conservative and supposedly incentivise ‘safe applications’. Likewise, exposure of names, affiliations and past performances through CVs is assumed to causes biases resulting in for example, Matthew effects, and gender and ethnicity disparities. A double-blind mechanism is assumed to alleviate such challenges. The questions is to what extent?


We use all application data (e.g. names, ids, full text) and review scores from five rounds of applications (2017 to 2021) - corresponding to 2000+ applications with around 250 funded projects. We examine different aspects of the VE programme, such as its mechanisms and potential impacts. Here, we focus on potential gender bias and disparities in who is funded. The current overall application and success rates, somewhat surprisingly, given the double-blinded mechanism, suggest a consistent gender difference where men are seemingly more successful than expected given their share of applications. Despite a double-blinded review process, women are 4.3 percentage points less likely to have their proposals funded than men (9.18 \% vs. 13.5 \%). To make sense of this persistent gender disparity, we use a series of bayesian multilevel models and post-stratification to investigate potential explanations.


Deeper analyses reveal two important self-selection/sorting mechanisms and potential gender disparities related to the demographics of the overall researcher population. First, we show that gender differences in success rates are partly explained by women self-selecting into one evaluation panel ('Life Science') to a much higher degree. Where men are more evenly distributed across the four evaluation panels ('Earth and Space Science', 'Life Science', 'Physical Sciences and Mathematics', 'IT and Engineering'), 46 \% of women applicants apply within life science. Because success rates differ across panels, and the 'Life Science' panel has consistently lower overall funding rates, women tend to self-select into stronger competition. The overall success rate differences are then a case of Simpson's Paradox, where results almost reverse when dis-aggregated across evaluation panels and application years.

Second, more experienced applicants tend to be more successful in having their proposals funded, with professors more successful than associate professors, and associate professors more successful than assistant professors and postdocs. However, women comprise a much smaller fraction of the more experienced applicants. Gender differences in funding rates are therefore also a function of the demographic disparity in academic ranks more broadly.

Lastly, women comprise around 25 \% of professorships in Denmark and 15 \% within the natural sciences, However, among the applicants with the rank of professor, only 11 \% are women. Using a Multilevel Post-Stratification (MrP) model, we calculate weights and simulate success rates for each academic rank and gender combination among applicants as if they were proportional to the entire population of Danish researchers.The model shows that the remaining gender disparity in funding rates are likely influenced by a lower self-selection into the program by the highly experienced women, and differences would largely disappear if more women applied.

In summary, gender disparity can still arise even when doubled-blinded review standards are employed. Larger structural imbalances in field composition and career advancement in the science system contribute to differential success in funding competitions.


The results shed light on two important discussions within science policy. Echoing recent calls for attention \parencite{traag_causal_2022}, we exemplify the importance of distinguishing between gender bias (discrimination) or gender disparity (differences) when assessing fairness in the allocation of merit in science. Gender differences in funding rates can still emerge when bias (conscious or not) in review processes are extremely unlikely, due to indirect causal effects of imbalances in the scientific system. Assessing the mechanisms contributing to gender differences in funding rates are important as changes to peer-review systems cannot eliminate such disparity by itself. Unequal access to research funds are also a product of inbuilt differences in career advancement (i.e. a leaky pipeline), and less representation of women in some fields of science. Solutions to such disparities necessitates a better understanding of the driving factors, because gender bias and structural disparities are not easily solved using the same policies.

Parody, Joke, or Insanity? Retracted Publications Continue to Garner Attention

ABSTRACT. “I think the article is a parody thing, or a joke, or maybe just [obscenity] insane. Fair warning, you will not be any smarter or better informed after reading it, but you might get a good belly laugh or two [retracted publication].” -Twitter User

Background: Scholarly literature can be retracted for multiple reasons, including faulty analysis, plagiarism, or falsification. Retracted scholarly literature raises questions about the validity and rigor of academic work. In some cases, retracted articles can hinder an academic field’s growth by inserting misleading or wrong information into a topic’s body of knowledge.

Especially in cases where articles are retracted due to flawed reporting or methods, it is useful to examine the nature of retracted papers. Conversely, it is also important to understand retracted papers because it is arguably worse for a flawed paper (i.e., one with faulty analysis, not reproducible, or another reason that is related to its results) to not be retracted, or take extended periods of time to do so.

Methods: Basic analysis The publicly available RetractionWatch (RW) dataset comprises our base set of data. We only include articles with a DOI in our analysis - this restriction appears to mainly exclude articles affiliated with Russian institutions. To inform our basic analysis, we perform summary statistics on the original dataset, including assessing retractions by year, country, reason for retraction, and subject matter. We also examine these measures on the subject of papers that are technology-related - we measure this by the RW subject (either “computer science” or “technology”), or if the paper is assigned an AI-related label using a SciBERT classifier.

Citations analysis Using a corpus of scholarly literature (containing merged publication data from Web of Science, Digital Science, arXiv, Microsoft Academic Graph, and the Chinese National Knowledge Infrastructure) we assess the number of times each retracted paper was cited, comparing the retracted papers’ number of citations to a stratified sample of our corpus. To create the stratified sample, we controlled for paper publication year, publication country, and a binary variable indicating whether or not the paper is technology-related based on the criteria described earlier. For this comparison, we additionally employ “citation percentiles,” which indicate the percentage of papers published in the same year and field as a given scholarly article that have fewer citations (e.g., a biology paper published in 2010 in the 95th citation percentile is cited more than 95% of 2010 biology papers). Finally, to examine the practice of citing retracted papers, we measure retracted papers’ citations before and after their retraction years, again subsetting technology-related articles against all articles.

Social media impact analysis To gauge social media impact of retracted papers, we connected our papers to Dimensions’ Altmetrics database. Altmetrics provides data on tweets, public Facebook posts, and other social sharing data on papers in the Dimensions database. It does not cover the full Retraction Watch database, but does cover most English-language publications. We compare social media metrics on retracted articles to the same stratified sample of scholarly literature that we use for our citation analysis.

Pilot results/expected results: Basic analysis: China has triple the number of total retracted articles than the second-highest country with retracted articles, the United States, with India, the United Kingdom, and Japan in 3rd, 4th, and 5th places respectively. Iran has the 6th highest global retractions despite not placing in the top 20 countries for scholarly paper publication globally. Regarding technology papers, China has ten times as many retracted papers as the United States, with the United States, India, Iran, and Malaysia following with slim margins respectively. Biology and medical papers were by far the most common subject in the general corpus, with triple the number of retractions as the second most common field, technology-related papers (over 12000 versus 4000), followed by business and engineering respectively. More papers over the past decade have been retracted than in previous decades, as other analyses have reported (insert source). On average, it took two calendar years for a technology-related paper to be retracted; in contrast, it took four calendar years for a non-technology related paper to be retracted. This discrepancy drops to 2 years to 3 years when we only consider papers originally published after 2009.

Citations analysis: Retracted papers continue to be cited after retraction. On average, 53% of citations for retracted papers occurred after the retraction year for non-technology-related papers, while 57% of citations occurred after the retraction years for technology-related papers.

The Kolmogorov-Smirnov test shows that retracted and non-retracted papers have different citation-percentile distributions for both technology-related and general research papers. Specifically, retracted publications follow a uniform distribution (i.e., the median percentile is 0.5) more closely, and the sample papers have higher percentiles.

Altmetrics social media analysis: To understand how the public discusses and interacts with retracted research, we used the Altmetrics API to retrieve tweet IDs that mentioned retracted technology-related papers. We retrieved the tweet texts and applied the Valence aware dictionary for sentiment reasoning (VADER) sentiment analyzer on the text. Repeating this process with our sample of non-retracted technology-related papers, we compare the difference in tweet activity of retracted and non-retracted publications.

VADER detects the polarity of text and provides a sentiment score on scale of -1 (negative) to 1 (positive). Retracted papers have a low average sentiment score of 0.03, compared to the non-retracted papers with an average sentiment score of 0.16. For retracted papers, positive tweets reflected sarcasm and negative tweets reflected frustration and disappointment.

For example:

Positive sentiment: “I'm no chemist, but pretty sure this is the greatest paper Big-E has published.“

Negative sentiment: “Sadly, misinformation is not only plaguing social media but the scientific community. Here are two bizarre examples of poor peer-review standards and the flourishing of low-standard “scientific” journals.”

For non-retracted papers, positive tweets reflected praise of and genuine interest in the research mentioned and negative tweets generally received a low score due to the research topic (e.g., depression or cancer).

For example:

Positive sentiment: “Oooh! Shiny! This was certainly worth waiting for. Also great to see coverage of leading edge research in high profile non-academic press”

Negative sentiment: “New research on breast cancer risk assessment”

Significance/discussion: China, the United States, and India lead the world in the number of retracted papers, both for technology and non-technology-related papers. The top 10 countries with the highest number of retracted papers differ when disaggregated by topic, and differ from the top 10 countries ranked by general scholarly output. Russia, Iran, and other countries with less sophisticated scholarly research infrastructures are overrepresented in the top countries list; this phenomenon may be due to less rigorous research methods, peer review, or research standards on the front-end.

Papers are both more likely to be retracted over the past decade and papers retracted in more recent years have a shorter retraction period (as to be expected, if they exist in this dataset). The higher retraction rate may be due to more rigorous review of papers post-publications and reduced stigma regarding research retraction. Still, papers tend to be cited more often post-retraction than before, which has significant implications for advancing scholarly research in all fields, but particularly nascent ones like technology-related fields.

Gender bias in funding evaluation: A RCT field experiment

ABSTRACT. Evaluation of proposals for allocation of research funding is done mainly through peer review. Some research concludes that there is gender bias in research funding; others sustain that gender differences exist but bias evidence is elusive and findings are contradictory. Despite some conceptual ambiguities, the debate revolves around to what extent the (potential) existence of gender bias in research assessment is causing gender differences in outcomes.

Evidence in observational approaches is either based on outcome distributions (e.g. success rates by applicants’ gender) or at best on modelling bias as the residual, once all relevant variables are considered. In observational research causal claims are usually mixed with simple statistical associations.

The standard problems of selection bias, confounding, lurking or unobserved heterogeneity have not always been properly addressed in previous observational research on funding research and peer review. Research addressing the links between research funding and gender bias has often faced problems of validity, both external validity (as it usually referred to a single country, funding agency or funding instrument) and internal validity, more related to the research design and the causality approach.

The funding organizations usually define the criteria for quality or merit assessment and their weighting. If gender bias exists in research funding, it would be the outcome of the reviewers’ assessment, under the evaluative framework set by the funding agency.

Trying to overcome validity shortcomings, in this paper we use an experimental design in which, instead of trying to identify the causes of observed effects (observational approaches strategies), we aim to measure the effects of a cause, the treatment effect of the gender of the principal investigator (PI) in a research funding proposal score.

We embedded a hypothetical research proposal description in a field experiment and with the research design we addressed some of the most important challenges of the “experimental approaches” that help to avoid bias estimates: 1) authenticity of the treatments: whether the treatment used in the study resembles the intervention of interest in the real word; 2) the realism of participants: whether the participants in the experiment resemble the actors that usually participate in this type of process; 3) the genuineness of the context: whether the context within which subjects are receiving the treatment resembles the context of interest, and 4) the truth of the outcome measures: whether the outcome measures resemble the actual outcome of theoretical or practical interest.

Our subjects were the reviewers selected by a funding agency, and the scoring task was done using the same criteria, framing and weighting of the call, additionally it was implemented simultaneously to the agency peer review assessment.

We manipulated the item in the proposal, which described the gender of the PI, with two designations: female PI and male PI. Treatment was randomly allocated with block assignment and response rate was 100% of the population, avoiding problems of biased estimates in pooled data.

Our results show that proposals led by female PIs received a lower mean score than the ones from male PIs, but the differences were not statistically significant. The origin of the small difference was that female reviewers assessed female PI proposals less favorably than male PI proposals.

Contrary to previous research, we find no evidence that male or female PIs received significantly different scores, nor did we find evidence of same-gender preferences of reviewers regarding the applicants’ gender. Thus, we cannot reject the null hypothesis that PI gender has no significant effects on scoring, nor do we find support for the matching hypothesis between applicant and reviewer gender.

Large-scale assessment of editors' impact on publishing in the social sciences

ABSTRACT. The scientific elite wield considerable power in shaping the evolution of scientific research (Azoulay et al., 2019; Chu & Evans, 2021) and reap sizeable rewards for their contributions (Allison et al., 1982; Merton, 1968; Xie, 2014). Opportunities to enter the scientific elite, however, are limited and are distributed with clear disparities across any number of social cleavages. This presentation reports on research that tackles the issue of publishing in elite journals, perhaps the most salient mechanism for entering the scientific elite (Heckman & Moktan, 2020), and in particular, the effect that journal editors have on that process.

Research shows that editors and reviewers are more likely to support papers that are closer in topic to their own research areas (Krieger et al., 2021), and which are written by academics who are nearby in the collaboration network (Dondio et al., 2019; Ductor & Visser, 2022; Teplitskiy et al., 2018), who have won notable awards (Huber et al., 2022), and are members of elite professional networks (Crane, 1967; Laband & Piette, 1994). Current estimates of editorial gatekeeping’s effect size are small, but it is difficult to know how much confidence to place in these results, since much of this work is focused on contexts where an editor is unlikely to be an active gatekeeper on the one hand, or an effective gatekeeper on the other. We have a growing number of studies set in non-elite and in multidisciplinary journals (Dondio et al., 2019; Teplitskiy et al., 2018), while those studies that address elite journals have virtually all been in economics (e.g. Colussi, 2018; Ductor & Visser, 2022; Laband & Piette, 1994).

Our project will provide the most substantial evidence to date relating to an editor’s effect on the publication process. We have collected the most comprehensive longitudinal dataset of journal editors in the social sciences, with over 3000 editors at over 1000 journals. This allows us to make examine the level of editorial influence in elite journals, specialist journals, and the broader mass of journals in each social science field. The breadth of the data mean that we can also provide some of the first looks at editorial gatekeeping outside economics in the social sciences.

To make full use of this data we are leveraging the Web of Science with state-of-the-art author disambiguation to assemble multiple measures of the distance between authors and editors. The Web of Science itself is used to assemble the collaboration network in science, the canonical approach to judging editorial gatekeeping. We also assemble the institutional affiliation network to provide an alternative view on the same principle of relational closeness. To further control for the topic similarity between authors’ research and an editor’s research we link papers to their SPECTRE document embeddings, high-dimensional text embeddings provided by Semantic Scholar. All of these measures of similarity are allowed to vary over time, not only as editors change at journals but also as researchers’ (and editors’) research interests evolve over time.

Using a series of field-specific relational event models (de Nooy, 2011; Quintane et al., 2014; Schecter & Quintane, 2021) we will report on the likelihood that someone gets published in elite journals, conditional on their distance from the current editor(s). We expect effect sizes to be small, but also for them to vary substantially across fields and across the journal hierarchy.

The most significant limitation to our research is that we do not have access to reviewers, nor can we pair them to the papers they evaluated. Editors of course can and do still wield power over the publication process, but it is typically less direct than that of reviewers. Another area where our design falls short of what would be ideal is that we cannot disentangle self-selection by authors themselves from editorial decision-making. Researchers at least in part—though this almost certainly varies dramatically across (sub)fields—make their choice of which journals to submit to on the basis of the identity of the current editor. This means that while it is entirely reasonable to speak of the effect that editors have on a person’s likelihood of publishing in a given journal, it is insufficient to prove that owes to editorial gatekeeping, specifically.

Still, we are optimistic about our paper’s potential. There are three main contributions we expect to make with this research. It will (1) provide a rare look at elite publishing practices in social sciences other than economics; (2) highlight variation in how editors affect elite publishing across several social science fields; and (3) document variation in terms of how editors affect publication across the hierarchy of journals within a field.

Exploring Transparency and Openness Through TOP Factors and Citation Indicators

ABSTRACT. Background The Open Science movement continues to grow across the globe with support from the European Commission (EC), the OECD (OECD) , the United States National Academies (National Academies of Sciences, 2018), amongst others. The Center for Open Science (COS), a non-profit organization, was founded in 2013 to “increase the openness, integrity, and reproducibility of scientific research” (CoS). In 2015, CoS, with several universities, funders, and publishers, published the Transparency and Openness Promotion (TOP) Guidelines, eight standards scholarly journals can implement to further openness and transparency (Nosek et al., 2015). In 2020, CoS launched the TOP Factor to assess the degree to which journal policies are promoting transparency and reproducibility. The TOP Factor is based on the TOP Guidelines, e.g. transparency of data/code/materials/research design and pre-registration. The TOP Factor is calculated by summing the level of implementation of the 8 original standards plus 2 additional standards (CoS).

The Web of Science Core Collection is a global, multidisciplinary citation index widely used for scientometric analysis. The list of journals in Core Collection is available on the Web of Science Master Journal List (Clarivate). The Core Collection includes 21,879 active journals as of January 2022, covering sciences, social sciences, and arts & humanities. These journals pass a rigorous editorial selection process (Clarivate).

In 2020, Clarivate collaborated with the Center for Open Science to incorporate TOP Factors on the Master Journal List (Clarivate). As of February 2022, 1,232 of the Core Collection’s journals (5.6%) have a TOP Factor. Previous studies have analyzed TOP Factor for journals in specific disciplines, including sleep research and chronobiology (Spitschan et al., 2020), herpetology (Marshall & Strine, 2021), and sport science (Hansford et al.), (Hansford et al., 2022), (Hansford et al., 2021). Other studies have found a positive correlation between open data sharing, one aspect of open science, and citation impact (Piwowar et al., 2007), (Piwowar & Vision, 2013). This study explores the relationship between TOP Factor and journal impact measures across all research disciplines.

Methods The TOP Factors were drawn from the Web of Science Master Journal List in February 2022. To calculate TOP Factors, Center for Open Science uses 10 standards scored individually on a scale of 0-3 (disclose=1, require=2, verify=3). These 10 scores are summed to create the overall TOP Factor with a maximum of 30 (CoS). The citation metrics are from InCites Benchmarking & Analytics, a research analytics database based on Web of Science Core Collection data. Institutional Profiles journal categories were used. This category scheme uses 6 broad academic disciplines.

The citation indicator used is Journal Citation Indicator (JCI). The JCI is the mean category normalized citation impact (CNCI) for the journal. CNCIs are calculated at the document level for articles and reviews from the previous three years, counting citations for four years. The JCI is normalized for document type, publication year, and category. The average JCI for any category is 1. A JCI of 2 indicates that a journal is receiving twice the expected number of citations than the average journal in the category.

Results The majority of the 1,232 Web of Science Core Collection journals with a TOP Factor are in social sciences (786), and the fewest are in physical sciences (69). Social sciences also had the highest percentage (12%) of its titles having a TOP Factor, and physical sciences had the lowest (2%). The median TOP Factor was 1. Very few approached the maximum TOP Factor of 30. Physical sciences has the highest average TOP Factor (4.7) despite having the lowest number of journals and lowest percentage of its category having a TOP Factor. The 2020 JCI for these journals ranges from 0.11 to 11.25 with a median of 1.26. We found no clear correlation between the journals’ JCI and TOP Factor, despite previous studies that found a citation advantage for papers with open data. Since open data is only one of the attributes that TOP Factor reflects, further research is needed to explore the relationship between openness and impact.

Significance This is one of the first studies to analyze TOP Factors across a multidisciplinary set of journals. There are some disciplinary trends, with social sciences having the highest number and percentage of titles with a TOP Factor, and physical sciences having the lowest. Overall, only 5.6% of the Web of Science Core Collection journals have a TOP Factor, and the median TOP Factor is 1 out of a potential 30. This suggests that adoption of the TOP guidelines is not widespread in the publishing community, and those who have pledged to adopt them are not fully embracing them. We found no correlation between the TOP Factor of a journal and its normalized citation impact despite the perception that greater openness leads to greater impact. Further research could explore disciplinary trends, as well as changes over time.

15:15-16:45 Session 4B: Barriers, Bottlenecks and Disruptions in Innovation
Team collaboration network, team structure and evolution of knowledge network: An empirical examination of solar energy industry

ABSTRACT. Background According to Schumpeter’s theory of innovation, innovation is the reorganization and recombination of knowledge elements. Knowledge restructuring is the scientific combination of various types of knowledge to build a rational knowledge structure, which makes knowledge elements are connected into knowledge network to affect innovation performance. Studies have cognized knowledge combination is an aggregation of knowledge elements used by organization for inventive activities. With the advent of knowledge economy era, the ability of knowledge creation and technological innovation are the core competence in the organization. They are the reorganization of existing knowledge elements. While researchers have paid more attention to the results of knowledge combination, such as invention and innovation. Little attention is paid to the process of innovation performance generation. However, invention and innovation are realized in the process of knowledge reorganization. The reorganization of different knowledge elements leads to the formation of innovation networks, and the evolution of knowledge networks in a given domain represents a constant and continuous combination of knowledge. The characteristics of the knowledge elements and the structure of the connected knowledge network directly influence invention and innovation. Therefore, knowing about the evolution of knowledge network allows companies to use their resources more efficiently so that they can create more inventions to promote technological innovation as well as increase productivity. In recent years, a number of studies have been conducted to explore the impact of collaboration on innovation performance from the perspective of social networks or social capital, and these studies suggest that collaborative networks influence the reorganizational combination of knowledge elements. However, a lot of studies have analyzed from the individual or organizational level, lacking the intermediate level of the team. This paper considered the necessity to analyze the impact on innovation network from the perspective of team level with the following points. First of all, teams are the most common form of organization for a particular research innovation. Members within a team often have a certain degree of similarity and connected knowledge, and the innovation strategy of this knowledge can be influenced by the team structure. For example, a centralized team structure may prefer exploitative innovation, while a decentralized team structure may prefer exploratory innovation, thus affecting the development of this knowledge in the knowledge network. In addition, for a collaboration network possessing different team structure, the difficulty of communication, the average degree of the network and so forth will result in various knowledge combination. Secondly, collaboration occurs not only within teams but also between teams in a field. Cooperation between teams play a crucial role in the distant combination of knowledge. The knowledge bases and reserves are highly heterogeneous among teams. Through cross-team collaboration, highly heterogeneous teams and knowledge may be combined together. Involving collaborators specialized in a topic to another topic may bring fresh ideas and unexpected solutions. Therefore, the purpose of this paper is to explore the impact of cross-team collaboration and team structure in networks on the evolution of knowledge network. In this paper, based on social network and organization theory, the following assumptions from two dimension are hypothesized. On the one hand, it’s analyzed from the perspective of social network. Social network theory emphasizes the significant differences of the members’ ability in network different positions to access information. Two dimensions, centrality and structural hole, have often been analyzed in many researches. Centrality represents the resources and information that the team can access and control, and structural hole represents the heterogeneous knowledge that the team can acquire. Therefore, this paper hypothesized that the knowledge of teams occupying more structural holes and a higher degree of centrality will have more opportunities for combination. One the other hand is based on the organization theory. Organizational theory suggests that different team structures exhibit different decisions and behavior patterns, among these centralized and flattened being the two most common organizational structures. This paper argued that teams with a centralized structure will be detrimental to the introduction of external knowledge, while a flat team structure will be more receptive to new knowledge. Therefore, this paper assumed that knowledge with centralized teams will get less opportunity to combine, while knowledge with flat teams will get more opportunity to combine.

Data and Method The sample consists of invention patents of solar energy industry whose applicants were organizations granted from 2012 to 2022. All the patents data were selected from PatSnap. There are 175,864 sampled patents, yielding 145,422 inventors. Patent data includes information such as inventors and IPC (International Patent Classification number). Inventors can be used for building collaborative network and IPC for knowledge network. Inventors will cluster with a threshold of 10 using the method of clustering for obtaining team. The team structure was measured with the software python and gephi. For method, ERGM is a model to analyze the effect of one network formation of another network. Therefore, this paper used the method of ERGM to build the network of inventor-inventor, team-team, IPC-IPC, inventor-IPC and team-IPC, which measured the influence of team network on knowledge network. It’s tested the impact of collaboration five years ago on knowledge innovation during the following five years. And using the patent data of the first five years as the pre-test data was aimed to calculate the indicators and build the collaboration network, while using the data of the second five years was to generate the knowledge network as the observation network.

Results This paper empirically obtained the impacts of different team structures on knowledge network evolution. It was found the knowledge followed by what kind of team will develop better, and it will be used for more innovation in the future. Firstly, the knowledge associated with the team owning greater centrality in the team-team network will have more opportunities to be combined. Secondly, knowledge associated with teams possessing greater structural cavity in the team-team network will have more opportunities to be combined. In addition, the knowledge of teams with flat structure will have more opportunities to be combined.

How do firms overcome barriers to innovations?: the mediating roles of external and internal funding of Korean Green firms

ABSTRACT. Background: Over the last several years, growing international interest in environmental protection has led to the promotion of low-carbon industries and pursuit of sustainable development on a global scale (Mensah et al., 2019). For research and development (R&D), which is essential for strengthening a firm’s innovation capability, the most important element is funding. This also applies to green firms, for whom financial investment into R&D is crucial (Noailly & Smeets, 2021; Yuan et al., 2020). Researchers using a resource-based view of the firm have four categories: physical, financial, organizational, and human capital (Barney, 1997). We analyze R&D funding based on resource-based view theory. Arrow (1962) argued that if R&D is funded from the outside, a moral hazard and adverse selection due to asymmetric information are possible. Gertler (1988) explained that external funding may be difficult and more expensive than internal funding if elements of market imperfection are present, such as transaction cost, agency cost, and asymmetric information. Some scholars have argued that firms tend to invest their internal R&D funds because of this risk. In other hands, Noailly and Smeets (2021) argued that the green firms, like clean energy technology sector, are capital-intensive and large investments are a prerequisite. We address the following research question: Does the innovation barrier of green firms affect innovation performance? and what are the effects of external and internal funding-based innovation strategies, respectively, on this relationship?

Purpose: This study examines whether green firms can overcome barriers to innovation and improve innovation performance through research and development(R&D) funding. We analyzed innovation performance with internal and external R&D funding for firms facing barriers to conducting green technology innovation activities.

Data: This research utilized the data on the Korean Green industry from the 2015 Green Industry Innovation Survey (GIIS), which was conducted by the Green Technology Center, Korea. As the data contains overall innovation, including the objectives of green innovation, resources used for green innovation (input of innovation), and green innovation performance, GIIS data is believed to be appropriate for this research.

Result: The analysis results indicated that internal R&D funding was more influential than external R&D funding. The results showed significant innovation performance for firms using internal, rather than external, R&D funding to overcome barriers; internal R&D funding mediated the relationship between innovation barrier and performance.

Contribution: First, This study demonstrates the need for green firms to recognize barriers they face and conduct R&D despite these barriers. Further, we add to the academic discussion on the importance of R&D investment in technological innovation when overcoming barriers to innovation. Second, the findings emphasize the importance of green firms using internal resources to fund R&D to improve innovation performance, rather than injecting external R&D resource, despite the recognized barriers. This study indicates that, despite recognized barriers, entrepreneurs need to invest in R&D for innovation, and it is more important to invest internal resources than to take on the many risks associated with external funding.

Limitation and Further study: First, it did not take time lags into account. This study was conducted by analyzing only the 2015 data. Innovation performance includes sales growth rate and patents, which may have time lags. Second, knowledge exploration and external cooperation can be used as strategies, in addition to R&D funding, to overcome barriers. Third, barriers to innovation may vary depending on a firm’s characteristics and industrial environment.

A cluster analysis of innovation barriers in agricultural sector: A k-modes machine learning algorithm approach

ABSTRACT. Background and context

In the last few years, innovation profiling and segmentation has become an essential method in academic and policy circles for analysing innovation behaviour of firms. When firms innovate, they are often faced with a wide variety of challenges or impediments. These are more pronounced in the agricultural sector, especially, since businesses in the agricultural sector have experienced a variety of issues that have hampered their innovation efforts. These obstacles include, among other factors, resource (financial and human), institutional and regulatory, as well as, environmental factors (climate change, droughts, floods, etc.).

Meanwhile, in an effort to address some of the challenges to innovation faced by agricultural businesses, policy-makers have often applied a blanket approach to design policies which, in some circumstances, have failed to meet the needs and requirements of agricultural businesses. Some of these policies have not produced the desired effects due to a lack of understanding of the characteristics and profiles of innovation in agricultural businesses and, in particular, the barriers to innovation in these firms. This is reflected in weak innovation performance in agricultural businesses, which has remained much lower when compared to other sectors of the economy, especially those in developing nations.

Innovation remains nonetheless a critical vehicle for addressing the sector's challenges, notably those of poor productivity, provided that a conducive framework for innovation is developed which addresses the impediments to innovation in the sector. This can help ensure the agricultural sector’s long-term viability and sustainability (Ulvenblad et al., 2018).

Unfortunately, simple aggregate indices of the number of innovation-active enterprises that consider a specific barrier to innovation as very important provide little relevant information for policymakers. To gain a better understanding of the structure and complexity of barriers to innovation in the agricultural sector, it is necessary to profile businesses based on the barriers they face, given that alleviating or at least minimising barriers to innovation may stimulate more innovations in the sector, thereby increasing the number of businesses that are actively engaged in innovation.

Objectives Although existing literature on innovation barriers has, to a limited extent, explored the different types of barriers to innovation in other sectors of the economy, the primary goal of this study is to investigate clusters of agricultural businesses based on barriers to innovation they experience.


This study used data from the South African baseline Agricultural Business Innovation Survey covering the period 2016-2018 (Agri-BIS 2016–2018). The Agri-BIS 2016-2018 was based on the guidelines of the Organisation for Economic Co-operation and Development’s (OECD) Oslo Manual (OECD/Eurostat, 2005). The survey used the methodological recommendations for the Community Innovation Survey (CIS) of the European Union (EU) countries, as provided by Eurostat, the Statistical Office of the European Commission. The survey focused on ascertaining how agricultural businesses innovate. The core questions asked about the businesses’ product, process, organisational and marketing innovations. The survey also asked questions about the different innovation activities and outcomes.

To determine the barriers to innovation, the survey incorporated additional questions on the factors that impede agricultural innovation. These questions asked businesses about the different factors that they considered highly important during the reference period 2016–2018.

This study applied a K-modes machine learning clustering algorithm to analyse the clusters of agricultural businesses that have experienced a wide range of innovation barriers. Python programming language was used to implement the k-modes machine learning algorithm on the dataset using the Scikit-Learn and pandas library.

Significance and contribution

This study makes an important contribution and bridges a key gap in literature in that, it is one of the first of its kind to apply a machine learning algorithm to cluster agricultural businesses in terms of the barriers that affect them. Secondly, this study allows for a more detailed examination of the implications in terms of prospective innovation policy instruments targeted at alleviating impediments to innovations on weak innovation performance in agricultural businesses as a result of the different types of barriers. The results may be used to inform policy by better targeting instruments and tailored policies to that address barriers to firm innovation in specific clusters in the agricultural sector.

Voluntary Human Capital Mobility, Involuntary Mobility, and Innovation; Evidence from the Collapse of Nortel Networks

ABSTRACT. Researchers on innovation and human capital mobility have considered mobility as a voluntary event where employees choose to enter the job market. However, mobility can be involuntary (forced), and it is often the case. By exploiting the fall of Nortel Networks as a natural experiment, we distinguish voluntary mobility from forced mobility and examine the consequences on innovation performance at the individual level. We find that the two aspects of mobility have different consequences on employees' innovation performance when studied separately from the perspective of the source and destination firms. We contribute to the literature on human capital mobility as a driver for knowledge creation and diffusion, particularly forced mobility which has so far received very little attention. Also, we underline a better understanding of the mobility mechanism on knowledge creation and diffusion patterns by showing how forced mobility compares to voluntary and no mobility.

15:15-16:45 Session 4C: Digital and Data Led Innovation
The Smart City as a Field of Innovation: Effects of Public-Private Data Collaboration on Innovation Performance of Small- and Medium-sized Enterprises in China

ABSTRACT. Data is increasingly considered to be a key component in stimulating innovation. Numerous promising possibilities have been opened up by rapidly emerging, data-intensive technologies, including the Internet of Things (IoT) and artificial intelligence (AI). The analysis and interpretation of big data are critical in the growth of technology firms in terms of AI training and computing capabilities. Small and medium-sized enterprises (SMEs), with their limited resources internally, particularly face a serious challenge of implementing innovation, which increasingly requires the effective processing and use of various kinds of data. The smart city provides an important opportunity for creating data-driven innovation. Significant amounts of data are increasingly available from various sources through sophisticated devices and equipment scattered in smart cities. Many smart city projects across the globe provide rich opportunities for SMEs to explore data-driven innovation. China, in particular, has recently been active in collecting and utilizing various kinds of data in smart cities. The availability of and access to data help to improve the software development of private enterprises in China, where massive amounts of data resources are collected and maintained by the public sector. In the process of smart city development, there are also many tasks that are complementary to each other, including connecting databases, building online platforms that connect different data coming from various data sources, operating and maintaining these online platforms, and providing products and services to citizens. These diverse kinds of tasks involved in smart city projects initiated by local governments have brought about new business opportunities for innovative SMEs in China. To implement the policies of encouraging the development of SMEs by the central government, municipal governments have introduced policies that give priority to SMEs in participating in smart city projects. Those companies that have access to the data collected in smart cities and held by government agencies are expected to benefit from utilizing the rich data for creating innovative products and services. There were few empirical studies conducted, however, to examine how data are actually managed and provided in smart cities and how they affect companies’ innovative activities. It remains unclear how public agencies and private enterprises collaborate on data and how that influences the innovation performance of SMEs in China. In smart cities, different types of public-private collaboration are involved, including hardware purchase, platform building, platform operation, and data analysis. It is not yet well understood how these different types of collaboration influence the availability and accessibility of data and consequently the innovative performance of SMEs. In mainland China, smart city development has been promoted actively to tackle severe urban issues, including air pollution, traffic congestion, and public safety. The Ministry of Industry and Information Technology (MIIT) established the China Smart City Industry Alliance in 2013 to implement smart city projects, and a policy paper, Guidance on Promoting the Healthy Development of Smart City, was published in 2014 by eight government departments. In 2015, the development of smart cities was promoted by the Prime Minister as the future development direction for cities in China. Encouraged by these policy measures, nearly 300 smart city projects have been approved so far. Smart city development in China has particularly emphasized the linkage and synergy with big data through advanced information and communication technologies, including IoT and cloud computing. Very few empirical studies have been conducted so far to investigate what kinds of data are collected in smart cities, how these data are available, who has access to the data, how these data are managed, what incentives are provided to encourage data sharing, and what impacts are made on stimulating innovation. In this study, we examine how data are managed through collaboration between the government and companies in smart cities and how the mode of data collaboration influences firms’ performance on innovation in China. By focusing on the case of SMEs in the Guangdong province, this research aims to shed light on what kinds of data are available and used in smart cities and how the government and enterprises collaborate on data to facilitate innovation. The analysis of this study utilizes data on more than eight million contracts extracted from the Government Procurement Database managed by the Ministry of Finance. The database contains rich information on government procurements, including the goods and services procured, the date of the contracts, and the monetary size of the contracts. Data on companies are assembled with regard to the registered capital, industry, software products, and patents in 1990-2021 from the database of Tianyancha. The applications of patents are also analyzed by using the patent database maintained by the State Intellectual Property Office (SIPO) of China. The SIPO patent database provides complete information on all patent applications and grants in China, including the application and publication number of the patent, the application and grant years, the classification number, the type of patent, and the assignee of the patent. The knowledge and technological domains are identified by analyzing the patent classification numbers and the content of the descriptions of the inventions. By using these data sources, panel data is established with key characteristics of SMEs, software and patents outputs, and their record on government contracts. The contracts are divided into three categories, namely, data analysis, platform building, and hardware purchase, based on keyword identifications. To deal with the unbalance between the treatment group (the companies that obtained government contracts) and the control group, we use propensity score matching (one-to-one nearest neighbor matching) to narrow down the sample size of the control group to that of the treatment group. Then we apply the event study methodology to examine whether there are significant differences in innovative outputs of software products and patents before and after the companies receive government contracts. We also compare how the innovation performance of companies differs based on the types of contracts these companies obtain. That makes it possible to identify what kinds of data collaboration would be effective in improving the innovative performance of SMEs. Our preliminary analysis suggests that many of the government contracts obtained by SMEs that have innovation outputs would be concerning platform building. After receiving government contracts, firms that conduct data analysis for smart city projects would tend to have more patent outputs compared with companies with similar characteristics. On the other hand, firms that implement platform building and platform operation for smart cities would be likely to have more software products. That difference in innovative performance would be considered to be influenced by the quantity and quality of data available and accessible to companies. While government contracts for buying hardware devices and equipment for smart city development would encourage innovation in technological products and services in SMEs, they would not have a significant impact on the innovative performance of these firms. Government purchases of hardware that contains data-intensive technologies would promote the use of these products in smart cities. That, however, would not involve any substantive exchange or transfer of data possessed by the government and would not significantly contribute to stimulating innovation at firms. Policy implications will be explored for various modes of data governance, including government-led, industry-led, and public-private partnership approaches, with their impacts on facilitating innovation and addressing societal concerns about data security and privacy.

Are Digital Innovation Policies Effective in Promoting the Development of Digital Economy in China?

ABSTRACT. Title: Are Digital Innovation Policies Effective in Promoting the Development of Digital Economy in China?

How latecomers of emerging countries catch up with the incumbents on technological innovation is always an important topic (Guo et al., 2021). China is also facing challenges in upgrading its manufacturing value-added in the global market and transferring to a more sustainable economic growth model. Digital technology has become the driving force of Chinese economic growth and transition (Li, Hsu, Mao, & Zhang, 2022). In 2013, the size of the digital economy was about 9.5 trillion, accounting for 20.3% of China's GDP. In 2020, the size already increased to around 39.2 trillion, taking a share of 38.6% of GDP. China is also a leading digital economy in the global market. According to CGTN ( Empowering digital transformation for china's economy. 2022), the size of China's digital economy is ranking second in the world in 2020.

Three elements shape the development of China's digital innovation: a large consumer market, a well-developed supply network of digital products, and a unique societal and regulatory structure (Li et al., 2022). On the one hand, in the past ten years, the rise of the middle classes has stimulated consumer demand for high-end products. It has brought a large group of users to digital innovation enterprises in China. On the other hand, the "deep supplier networks, a large skilled workforce, and a developed logistics infrastructure" have enabled China to develop a completed digital innovation ecosystem. As the Chinese government plays a predominant role in economic development, China's institutional system has significantly affected demand and digital supply-side innovation. China has pointed out to develop a national strategy for the digital economy in its Medium to Long-term Planning for Scientific and Technological Development plans. For example, in the Thirteenth Five-Year Plan Outline of National Economic and Social Development of the People's Republic of China, the Chinese government proposed implementing a national big data strategy and promoting the open sharing of data resources (Xinhuanet, 2016). Furthermore, in Fourteenth Five-Year Plan Outline, the Chinese government has proposed accelerating digital development and building a Digital China (Xinhuanet, 2022). In response to the government's national strategy, Chinese governments at all levels have introduced policies for digital economy development and digital transformation. The data from the Chinese policy database shows that in 2013, the Chinese government introduced 342 digital innovation policies, and in 2021, the Chinese government introduced 1242 digital innovation policies. The figures have tripled in the past eight years. Although the Chinese government has implemented a series of innovation policies to stimulate digital innovation in different regions of China, existing digital innovation studies may not systematically evaluate and compare the effectiveness of these policies in fostering digitalization in China. The government may waste many valuable resources on policies that are not very effective. Therefore, it is incredibly intriguing and helpful to find out how the Chinese government uses policy instrument mix to promote the development of the digital economy and comprehensively evaluate the policy effectiveness of digital innovation. Thus, the research questions of this paper are as follows:

Research Question 1: "What kind of policy instruments are frequently used by the Chinese government to promote the digital economy?"

Research Question 2: "Is every type of policy instrument effective in boosting the digital economy?"

Research Question 3: "Among all applied policy instruments, which policy instruments are more effective in facilitating the digital economy?"

The unit of analysis is a province (excluding Hong Kong, Macaw, and Taiwan). This study will collect two kinds of data. The first is the 2013-2021 digital innovation policy document at the province level, collected from Bailu Zhiku ( by a web's crawler. The second is the 2020 digital economy index by provinces, collected from the database. The approach of Natural Language Processing will be used to clean and process the collected policy documents. In this process, some Chinese keywords (“税收减免Tax deduction,” “孵化器 Incubators,” “政府采购 Procurement” and “补贴 Subsidy” etc.) will be used to identify the policy instruments in the policy text. Based on the innovation policy instrument framework proposed by Elder et al. (2016), we define thirteen types of instruments in this study, including "R&D tax credits," "Direct R&D support," "Training and skills," "Entrepreneurship," "Technical services and advice," "Cluster," "Collaboration," "Innovation network," "Procurement," "Innovation prizes," "Standard," "Regulations," "Technology foresight." This study will also construct a multivariate regression model to test if a certain innovation policy instrument significantly impacts the digital economy development of provinces in China and to compare the effectiveness of different types of innovation policies.

The preliminary results show a huge regional heterogeneity of digital innovation development in China. Coastal regions like "Guangdong" (N=302), "Zhejiang" (N=193), "Jiangsu" (N=188), "Fujian" (N=165), "Shandong" (N=128), "Beijing" (N=100), "Tianjin" (N=85) and "Shanghai" (N=73) have launched more digital innovation policies. Southern provinces pay more attention to digital innovation than northern provinces (Guangdong, Zhejiang, Jiangsu, and Fujian compares to Shandong, Beijing, and Tianjin). Besides, Chinese governments frequently use direct support for R&D and innovation, training and skills, and cluster policy when developing a digital economy. Lastly, from the multivariate regression results, we observe that both demand (=12.57, p<0.05) and supply policies (=2.62, p<0.05) can significantly improve the development of the digital economy development and the demand-side policy is more effective than the supply-side policy.

  References Elder, J., Cunningham, P., Gok, A., & Shapira, P. (2016). Handbook of innovation policy impact. Beaverton: Ringgold, Inc. CGTN (Producer), & . (2022). Empowering digital transformation for china's economy. [Video/DVD] Retrieved from Guo, B., Ding, P., Greidanus, F. J., & Li, W. H. (2021). What makes a successful industry-level catch-up? general framework and case study of china’s LED industry. Frontiers of Engineering Management, 8(2), 284-309. Li, L., Hsu, C., Mao, J., & Zhang, W. (2022). Contextualising digital innovation in today's china: Local practices and global contributions. Information Systems Journal (Oxford, England), 32(3), 623-629. doi:10.1111/isj.12379 Xinhuanet. (2016). 中华人民共和国国民经济和社会发展第十三个五年规划纲要. Retrieved from xinhuanet. (2022). 中华人民共和国国民经济和发展第十四个五年规划和2035年远景目标纲要. Retrieved from

Assessing Organizational Capabilities and Effectiveness of Digital Transformation Implementation Agencies: A Case of the National IT Industry Promotion Agency (NIPA) of South Korea

ABSTRACT. _Background and Rationale_: Although studies have examined the benefits of digital transformation and the effectiveness of certain policy instruments for digitization, the policy implementation capabilities of public agencies have received little attention despite their critical importance in the effective delivery of support programs. Using a new framework developed by the World Bank for assessing the organizational effectiveness of innovation agencies, this research aims to address this knowledge gap with a case study of a Korean implementation agency tasked with supporting firms' adoption of digital technologies. This study analyzes the effectiveness of the Korean agency in four key dimensions–delivery of goals, stakeholder satisfaction, processes and functionality, and resources–of organizational effectiveness and draws lessons in each of the dimensions, as well as from a systemic point of view. The findings show that phased efforts that start with a focus on building critical mass across the digital economy and that are based on existing capabilities are key to improving the effectiveness of public agencies supporting firms' digitization.

_Methods_: From the experience of assessing the policy mix and functionality of science, technology and innovation (STI) policies in developing and middle-income countries, it became clear that there was a need to develop a methodology focused specifically on the features of the agencies that played a key role in various phases of these policies, ranging from design through implementation and governance. Especially important was the need to gauge their effectiveness in fulfilling their mission and goals. With this a novel analytical framework was developed inspired in the organizational effectiveness approach from Quinn and Rohrbaugh (1981) adapted to the specifics of STI policy agencies. It takes into account recent reports on observed features of these agencies by NESTA (Glennie & Bound 2016) and the World Bank (Aridi and Kapil 2019). The analytical framework develops the framework of organizational effectiveness into a model with specific categories relevant to STI agencies that guide an empirical strategy for assessing their effectiveness.

_Data and Results_: The project gathered data from available agency reports and interviews with relevant agency personnel. Reports were used dating back to 2104 regarding budgets, mission transitions, project catalogs, statements of objectives and goals and evaluations of beneficiary satisfaction. Several rounds of interviews were conducted and followed up with written questions which received written responses and supplementary information on the available reports. Two sorts of results were of interest in this project: First, an empirically grounded assessment of the effectiveness of NIPA as a digitization innovation agency in Korea was sought. Second, the project aimed to establish whether the proposed analytical framework is useful for determining the profile and effectiveness of innovation agencies generally in other countries.

On the first result, the project found that NIPA has a very specific profile that is not found in other innovation agencies generally. It is an implementation and services organization that maintains close connection with the beneficiaries of the programs it implements and manages. It takes significant feedback directly from beneficiaries to adapt its approaches to better serve them and caters to their perception of what is most important and useful. Understanding the profile of the organization in this way, in its tightly constrained policy context, it can be said that it is very effective in its role. However, it is a limited role lacking many of the functions and capabilities that would be required for an organization that would be expected to have greater responsibility in broader technology policy strategies. The broader role of innovation policies is distributed across policy domains under the purview of several ministries giving implementation agencies a very narrowly focused role closely tied to their beneficiaries’ satisfaction.

The second conclusion is that method is useful to determine the profile and effectiveness of innovation agencies generally. Even though there were some limitations in the availability of information, we were still able to determine the profile of the agency and get a sense of its effectiveness and how it may be considered as a model in other contexts. We were able to conduct interviews but were not able to run surveys or other more intrusive approaches to information gathering. Future use of the method would require fuller access to take advantage of its potential. Interestingly, the approach seems sufficiently general and, at the same time, detailed enough, that it did not require much ex-ante knowledge of the organization to arrive at the key findings. It seems the method is flexible enough for use in almost any government context if enough information is made available.

References: Aridi, Answer and Natasha Kapil. (2019). Innovation Agencies: Cases from Developing Countries. World Bank Group. Glennie, Alex and Kirsten Bound. (2016). How Innovation Agencies Work: International Lessons to Inspire and Inform National Strategies. NESTA, UK. Quinn, Robert E. and John Rohrbaugh. (1981). “A Symposium on the Competing Values Approach to Organizational Effectiveness.” Public Productivity Review 5, 2: 122-140

Data-driven innovations in Germany: Drivers and determinants of adoption by SMEs

ABSTRACT. Studying the dynamics of data-driven innovations (DDI) is becoming increasingly important to map the drivers for developing data-intensive solutions by SMEs. Such an analysis provides substance to an analysis of barriers to the adoption of DDI as well as the impact of policies and legislative acts, like the Data Act, and public-funded data-sharing platforms, on fostering innovation and promoting growth and social well-being. To that end, a combined collection of structured and unstructured data will be used to analyze the adoption and diffusion of DDI in the German industry.

15:15-16:45 Session 4D: Networks
Capturing Research Field Dynamics through Multiplex Network Structures

ABSTRACT. Background

Studies of science and science policy often rely upon analyzing the ‘structure’ of research fields. This is used to explore how scientific fields evolve, whether they are affected by policy and funding, and performance comparisons across universities and research systems (Porter & Rafols, 2009; Braam & van den Besselaar, 2014; Langfeldt et al. 2020). Traditionally, field structure has been investigated via citation-based relationships, touching upon fields’ knowledge pools (Van Raan & Tijssen 1993; Creswell 2009; Porter & Rafols 2009; Boyack & Klavans 2014). Elsewhere, as in classical sociology, structure is explored through a mix of social organisation and intellectual aspects, including the norms, intellectual conventions, governance rules and authority relations of ‘scientific communities’ (Merton 1968, Crane 1971, Whitley 2011).

In this paper, we instead explore co-existing ‘structures’ of a research field, comprising an ‘interlaced’, ‘multilayered’ or multiplex network phenomenon (Heimeriks et al. 2003, Teurtscher et al. 2014).


Traditional approaches to investigating and mapping the structures of research fields, we posit, while in part useful to the study of science dynamics, have some unfortunate shortcomings. First, by assuming a unitary structure of science/scientific fields these approaches limit the investigation of complex, inherent dynamics. And second, by often inferring structures and relationships rather than mapping these directly, the explanatory power of traditional approaches is somewhat limited.

Still, unpacking the sources of inherent science dynamics, or the dynamics of scientific fields, is vital for understanding the evolution of science, the ways in which exogenous factors interlace with inherent sources to shape the workings of the sciences.


Building on an understanding of structure originating with ‘new institutionalism’ (Powell & DiMaggio, 1991) we assume that relationships can be empirically accessed through (social) exchange or distribution.

Using co-nomination to trace three different kinds of exchange occurring within scientific networks, we mapped three different structures in a physics field – intellectual, collaboration and technical. (For more on the use of co-nomination to map research fields please see Karaulova et al. 2020)

Social network analysis techniques were used to investigate further the links between the different networks/structures. The network maps were generated using VantagePoint and Gephi.


To test this ‘structures’ perspective, we used multiple rounds of co-nomination analysis, a reputation-based approach combining snowball sampling and social network analysis (Karaulova et al. 2020). We mapped three exchange networks for a particle physics field.

We then highlight actors simultaneously active in one, two or three networks. This shows researchers performing multiple roles, and engaging multiple exchange relationships, perhaps not captured by citation-based approaches or study of single networks. It foregrounds the importance of other roles beyond intellectual influence, such as functions that technicians and equipment developers undertake, and social organizer roles that may help crystallize the field – a task more often associated with women, according to our in-progress analysis.


This ‘structures’ approach adds multidimensional character to whether scientists appear central or peripheral in a field. It has potential implications for how researcher performance is evaluated by universities, for organized mitigation of intersectional biases in science, and for how funders tackle coordination dysfunctions in fields developing over time (Tuertscher et al. 2014; Kozlowski et al. 2022).

Overall, it promises to transform how we study research fields and science dynamics, by shifting our perspective from one-dimensional ‘structure’ to key multiplex ‘structures’. Using it to study additional cases could lead to better understanding of how research governance can support fields. It could also drive policy and funding to become tailored to bespoke dynamics of key field ‘types’, as revealed by their multiplex structures.


Braam, R., van den Besselaar, P. (2014) Indicators for the dynamics of research organizations: a biomedical case study. Scientometrics 99, 949–971. Boyack, K. W., & Klavans, R. (2014) Creation of a highly detailed, dynamic, global model and map of science. Journal of the Association for Information Science and Technology 65: 670–685. Scopus. Crane, D. (1971). Transnational networks in basic science. International Organization,25, 585–601. Creswell, J. W. (2009). Editorial: Mapping the field of mixed methods research. Journal of Mixed Methods Research,3, 95–108. Heimeriks, G., Hörlesberger, M., & Van Den, P. Besselaar (2003). Mapping communication and collaboration in heterogeneous research networks. Scientometrics, 58, 391–413. Karaulova, M., Nedeva, M. & Thomas, D.A. (2020) Mapping research fields using co-nomination: the case of hyper-authorship heavy flavour physics. Scientometrics 124, 2229–2249. Kozlowski D, Lariviere V, Sugimoto CR, Monroe-White T.(2022) Intersectional inequalities in science. PNAS 119, 1-8. Langfeldt, L., Nedeva, M., Sorlin, S., and Thomas, D. A. (2020) ‘Co-Existing Notions of Research Quality: A Framework to Study Context-Specific Understandings of Good Research’, Minerva, 58, 115–37. Merton, R. K. (1968). Social theory and social structure. New York: Simon and Schuster. Porter, A.L., and Rafols, I. (2009) Is science becoming more interdisciplinary? Measuring and mapping six research fields over time. Scientometrics 81, 719. Powell, W. W., & DiMaggio, P. (Eds.) (1991). The New Institutionalism in Organizational Analysis. University of Chicago Press. Tuertscher, P., Garud, R., and Kumaraswamy, A. (2014) Justification and Interlaced Knowledge at ATLAS, CERN. Organization Science 25(6):1579-1608. Van Raan, A.F.J., Tijssen, R.J.W. (1993) The neural net of neural network research. Scientometrics 26, 169–192 . Whitley, R. (2011) ‘Changing Governance and Authority Relationships in the Public Sciences’, Minerva, 49, 359–85.

Using Hierarchical Stochastic Blockmodels to Examine Synthetic Biology Discourse on Twitter

ABSTRACT. Background and Rationale

Synthetic biology is a recently established field in which practitioners re-design and engineer organisms or parts of organisms with the intention of changing their behaviors or characteristics. The hope is that such genetically modified organisms or components can help address pressing societal challenges, such as pandemics, global warming, pollution, and food security. Synthetic biology, for example, was integral in the development of mRNA vaccines to combat COVID-19.

Synthetic biology, however, is not only practically consequential; it is also theoretically interesting. Theoretical frameworks, such as the Triple Helix model and Pierre Bourdieu’s field theory, treat social life as composed of different domains with varying levels of dependence between them, for example the government, academia, and industry in the Triple Helix model. Some researchers might be more focused on producing research to be consumed primarily by individuals in the same field, whereas researchers in other fields often target their research at solving problems in areas unrelated to their field of study, e.g., how to make more pest resistant crops or how to inform policy decisions about the allocation of funds to social welfare programs. Synthetic biology is positioned at the nexus of many fields: seeking to solve myriad societal problems identified by policy makers and other researchers and often relying upon the private sector to support and implement research products.

Social media studies of science encourages researchers to focus on more than scientists and scholarly objects on social media and suggests that this would be important for synthetic biology, where non-academic communication is particularly important in bringing together a variety of stakeholders. Such communication occurring between fields and sectors frequently occurs on social media, and synthetic biology is no different. This study poses the following question: in a social media environment where diverse stakeholders discuss synthetic biology as well as its applications and implications, what specifically is discussed? Answering this question is essential not only to developing a more encompassing study of synthetic biology, beyond traditional bibliometric and even altmetric data, but also to understanding more generally how the interdependence of certain scientific fields with other fields, businesses, and even sectors impacts the approaches needed to study them.

Data and Methods

Data used in the following analyses come from Twitter, a common platform for synthetic biologists and other actors who communicate about synthetic biology. The Twitter Application Programming Interface (API) allows users to programmatically query the full history of tweets. Because research indicates that synthetic biology stabilized as a research field around 2010, the data in this paper consist of tweets from 1 January 2010 through 30 September 2022.

The following query was used to obtain relevant tweets during this period: #synbio OR #syntheticbiology OR "synthetic biology" OR synbio OR (iGEM -igemcity) OR (#iGEM -(#igemcity)). Hashtags are a feature of social media platforms that allow users to connect their posts with ongoing conversations without having to respond to specific individuals. Querying phrases that are not hashtags identifies relevant posts, even if they are unconnected to broader conversations identified through hashtags. Synbio is a common label used as shorthand for synthetic biology. iGEM stands for the International Genetically Engineered Machines competition, which brings together teams of high school students, college students, postgraduates, and community lab members from across the globe. Manual review of synthetic biology Twitter content and discussions with subject matter experts indicated that iGEM constitutes a significant part of the synthetic biology discourse on Twitter. Tweets were preprocessed by removing emojis, numbers, URLs, hashtags, account mentions, retweets, and non-English tweets and by making all text lowercase. There are 396,154 tweets in this corpus.

Hierarchical stochastic blockmodels (hSBMs) were used to analyze the content of this tweet corpus. hSBMs may be considered an unsupervised method because they help users understand—without specifying any prior knowledge or thematic structure—the different topics discussed in a set of texts. hSBMs represent tweets as a bipartite network in which tweets and the words used within them are connected and then uses statistical approaches to community detection to uncover topics. Topics are conceived as communities in which words are densely connected to each other because they frequently co-occur in tweets. hSBMs do not require the user to specify beforehand a given number of topics, which is usually required for methods like topic models despite being a quantity of interest, and can represent topics as hierarchically nested.


The hSBM identifies four “levels” of topics. The most abstract level contains 25 topics, and the most concrete level contains 924 topics. The second and third most abstract levels contain 84 and 230 topics, respectively. The most abstract level has topics about the iGEM competition, biotech startups, global markets, genomics research, major news stories, molecular engineering research, and popular science writing, among others. Topics at the most concrete level are very specific. One example topic is specifically about sustainability and research containing specific organisms (e.g., pseudomonas), and another is about sustainability specifically regarding biofuels. A final example of a very focused topic mentions the market performance of certain synthetic biology companies. The intermediary levels unsurprisingly present topics that are less focused or concrete but still not as abstract as at the highest level. One finding common to all topic levels is that non-academic content is much more common than academic content.


The fact that many topics do not pertain to academic research and/or research objects (e.g., article DOIs) reinforces the call of social media studies of science to expand the scope of studies on social media beyond scientists and their immediate activities. Synthetic biologists, for example, rely upon tooling and equipment from industry, and companies often work closely with professors to launch new products. Crucially, social media discourse surrounding synthetic biology also includes attention to ethical, legal, and societal issues, like sustainability and dual-use research. Theoretically, this means that domains in which scientists’ activities are oriented toward other types of actors (e.g., businesses, policy makers) or solving applied problems cannot be fully understood by studying only scientific output. On a more practical level, this means that studies often need more than extant compiled data sources, e.g., altmetric statistics, because such sources do not contain data necessary to understand science-society dynamics.

How do scientists’ network reciprocity and proximity affect their scientific resource exchange and collaboration?

ABSTRACT. Academic scientists rely heavily on individuals in their network to access research inputs, such as knowledge, data, and biological materials (Katz & Martin, 1997; Bozeman & Corley, 2004; van Rijnsoever, Hessels & Vandeberg, 2008). Failure to receive those resources can significantly delay or even disrupt scientists’ research agendas. In the past decade, science policy researchers have sought to understand how network structure and composition, such as tie length, closeness, and network density, might shape scientists’ work and research outcomes, such as job satisfaction and productivity (Reagans & Zuckerman, 2001; Sotaro, Walsh & Baba, 2012; Siciliano, Welch & Feeney, 2018). However, few studies have looked at how network reciprocity and proximity might jointly affect their resource exchange behaviors.

Social exchange theory often postulates that reciprocity is one of the guiding norms for social interactions (Cook et al., 2013). Reciprocity is understood as being a mutual correspondence of characters or values in social relationships (Memoli & Sannella, 2017), encompassing the norms and expectations which influence behaviors of giving, receiving, and exchanging resources. However, scholars, such as Brandenberger (2018), have indicated that few studies examine reciprocity at the micro-level (e.g., two-mode network events or interactions). Relational proximity is another aspect of network that has been relatively neglected in extant science policy research. Social network theory distinguishes different types of proximity, from relational to functional or instrumental. Relational proximity often captures friendship ties that are developed between people who are similar on a variety of personal characteristics, including gender, race, age and professional experience (Ibarra & Andrews, 1993); while instrumental or functional proximity is often used to capture geographic locations or physical distance (Brandenberger, 2018).

Using a unique dataset of ego-centric network data collected among US academic scientists working in three fields (biology, entomology, and ecology) in 2017, this proposed research strives to understand the antecedents of scientific resource exchange and collaboration. In this paper we explore three questions: (1) how does network reciprocity affect the scale of resource exchange and the extent of scientific collaboration?; (2) how does the impact of relational proximity vary from functional proximity?; and (3) how do network reciprocity and proximity jointly affect resource exchange and collaboration? The study finds that, growing network reciprocity and proximity facilitate resource exchange by reducing the potential transaction cost and mitigating the potentially negative consequences caused by unforeseen problems during the exchange. Moreover, the impact of relational proximity might be different from functional proximity as it further cultivates tacit norms and routines that facilitate interactions and exchanges. Finally, the result will demonstrate that the impact of network reciprocity varies according to the extent of proximity.

Our study has both theoretical and empirical contributions. First, prior studies on scientific resource exchange and collaboration primarily examine the impact of reciprocity and proximity separately (e.g., Ynalvez & Shrum, 2011; Bozeman & Corley, 2004; Sotaro, Walsh & Baba, 2012). This research contributes to the literature by further investigating when reciprocity matters most, or not at all, depending on different levels of relational and functional proximity. Second, we provide a more nuanced examination of the different dimensions of network proximity by comparing the effect of relational proximity versus the functional one. Besides this, this study offers empirical insights for policymakers and academic scientists on how to facilitate the exchange of scientific knowledge or research inputs as well as other collaborative activities. Study findings will have important implications for the advancement of science and scientists’ careers as scientists’ exchange and collaboration will greatly affect professional outcomes, such as job satisfaction and productivity (Jha & Welch 2010; Reagans & Zuckerman 2001).

Reference Bozeman, B., & Corley, E. (2004). Scientists’ collaboration strategies: implications for scientific and technical human capital. Research policy, 33(4), 599-616. Brandenberger, L. (2018). Trading favors—Examining the temporal dynamics of reciprocity in congressional collaborations using relational event models. Social Networks, 54, 238-253. Cook, K. S., Cheshire, C., Rice, E. R., & Nakagawa, S. (2013). Social exchange theory. In Handbook of social psychology (pp. 61-88). Springer, Dordrecht. Ibarra, H., & Andrews, S. B. (1993). Power, social influence, and sense making: Effects of network centrality and proximity on employee perceptions. Administrative science quarterly, 277-303. Jha, Y., & Welch, E. W. (2010). Relational mechanisms governing multifaceted collaborative behavior of academic scientists in six fields of science and engineering. Research Policy, 39(9), 1174-1184. Katz, J. S., & Martin, B. R. (1997). What is research collaboration?. Research policy, 26(1), 1-18. Reagans, R., & Zuckerman, E. W. (2001). Networks, diversity, and productivity: The social capital of corporate R&D teams. Organization science, 12(4), 502-517. Memoli, R., & Sannella, A. (2017). Inclusion: The principle of responsibility and relational reciprocity. Italian Journal of Sociology of education, 9(2). Siciliano, M. D., Welch, E. W., & Feeney, M. K. (2018). Network exploration and exploitation: Professional network churn and scientific production. Social Networks, 52, 167-179. Shibayama, S., Walsh, J. P., & Baba, Y. (2012). Academic entrepreneurship and exchange of scientific resources: Material transfer in life and materials sciences in Japanese universities. American Sociological Review, 77(5), 804-830. van Rijnsoever, F. J., Hessels, L. K., & Vandeberg, R. L. (2008). A resource-based view on the interactions of university researchers. Research policy, 37(8), 1255-1266. Ynalvez, M. A., & Shrum, W. M. (2011). Professional networks, scientific collaboration, and publication productivity in resource-constrained research institutions in a developing country. Research Policy, 40(2), 204-216.

Exploring emerging trends in 5G industry from patent data using text mining techniques and network analysis

ABSTRACT. Background and rationale

5G wireless technology is speculated to dominate the communication market and is expected to stay highly competitive over the next few years. According to experts, the arrival of 5G networks will transform entire business domains and create an endless number of new ones (autonomous vehicle market, smart cities, smart factories, autonomous mobile robots, connected medical devices, etc.)(Palattella et al., 2016; Cheng et al., 2018). It is thus necessary to focus on the technology development of 5G wireless communication in order to understand the market trends and help managers and policymakers to plan their strategy regarding R&D activities. However, it is a daunting task to accomplish, since technology development is progressing very rapidly, and researchers struggle to obtain reliable data enabling them to identify key technologies. In this context, patent data represent a valuable solution, and it is a well exploited avenues to map technology development. Most of these studies are dominated by bibliometric and scientometric analyses that use indicators based mainly on structured bibliographical information. However, these indicators are increasingly called into question by scholars and experts for their low informative potential in predicting future trends and intra and inter-industry knowledge flows (Chung and Sohn, 2020). To overcome these limits, many authors have claimed that textual description and linguistic patterns efficiently provide information on technical attributes or knowledge contained in patent documents, which are more powerful in creating indicators about technology trends then bibliographical information. Thus, a stream of studies extended the well-known approach of text mining to analyze the textual content of patents. This article falls in this strand of studies. It combines text mining techniques and network analysis to reveal development trends of the fifth generation (5G) technologies for mobile communication through patent analysis. The paper addresses three questions: 1) What are the latent technological topics in 5G patents? 2) How they evolve in time? and 3) Which is the configuration between several domains of 5G and the main leaders of the industry? In this paper, we provide a more large and precise analysis of 5G technological trends, and we overcome some limits of previous studies. In particular, we use a large dataset, we develop a semi-supervised technique to filtering out irrelevant documents, we detect latent technological topics by using further textual fields then title and abstract and we perform statistical analysis to underline different technological development strategy between top players in 5G domain.


The method employed in this paper consists of four main steps: 1) Data collection and preprocessing; 2) Filtering of irrelevant patents; 3) Detection of the most recurrent topics and focus group with experts; 4) Network analysis of the relationships between topics and the most important leaders in the 5G industry; and 5) Further statistical analysis. First, we aim to obtain the most exhaustive dataset possible containing granted patents related to 5G inventions. To do so, we use the service offered by PatSeer, one of the world’s most comprehensive full-text patent collections covering more than 100 patent offices. We retrieve patents that: a) were granted between 2010 and 1st March of 2021; b) belong to a patent office among the G20 countries or one of the supranational ones; c) correspond to the most recent publication of his simply family, which groups documents covering the same invention; d) contain the full-text in the English language; and finally, e) contain a selected list of words in their title, abstract or claims. Our query identified 56,665 patents potentially related to 5G from 31 different patent offices. On the resulting corpus, we apply the most classical Natural Language Processing (NLP) steps to prepare unstructured textual data for further analysis. We get a Document-Term Matrix with 6,058 different unigrams and bigrams of word after having removed features which appear in less than 25 documents and more than the 50% of the corpus. Second, we filter irrelevant patents retrieved from PatSeer because of the ambiguity of some keywords of our query, such as “5G” or “millimeter-waves”. To realize this operation, we implement a semi-supervised machine learning algorithm. The method consists of three steps: 1) creation of the training dataset for the identification of related and unrelated 5G documents; 2) training and test phases of the algorithm; 3) application of the best model to the dataset. The first step is initially assisted by a density-based clustering, the DBSCAN algorithm, which is executed on textual data. This process brings to the building of the training corpus, containing 6,000 documents belonging to 5G and 6,000 which do not. In the second step of this filtering operation, we train and test several supervised algorithms (Neural Networks, Naive Bayes, Random Forest, etc.) with several parameters and a 10-fold cross validation. Finally, we select the Support Vector Machine (SVM) algorithm which achieves a F1-score of 93% and we apply the best model to the original dataset, identifying 34,017 patents related to 5G. Third, we used the Latent Dirichlet Allocation (LDA) approach to detect the most recurrent topics in our dataset. After testing several parameters, the model with 50 topics obtained the best topic coherence score (0.63). A focus group composed by five experts (2 professors and 3 PhD candidates) of the department of Electrical Engineering at Polytechnic Montreal was conducted to drive the interpretation of the results. Forth, we perform a dynamic network analysis by following the following procedure: 1) splitting of our dataset into three time periods and selection of the top 20 assignees for each of them; 2) creating of a leaders-topics edge table by leveraging the probability distributions of topics over documents belonging to each leader and time period; and 4) computation of different measures of nodes centrality for network analysis. Finally, we estimated a linear regression for each topic, linking the mixture of the topic over documents, as dependent variable, and regions of assignees as independent variables.


We stress a bipolar configuration of Chinese and American players. Actually, these two groups tend to be located on opposite sides of the network and to be connected with different groups of topics, underling different strategies in 5G technological development. This constitutes the most import result of our study. On the one hand, American assignees focus more on “frequency division”, “Wi-Fi”, “User Equipment (UE)”, “Computer program”, “Frequency spectrum”, “Voice video call”, etc. On the other hand, Chinese assignees invest more on “Communication terminal”, “Signal detection”, “Network security”, “Relay station” and “Network performance”. The coefficients have highly significant effect, with the exception of the following topics, which have a no significant coefficient: “BICM”, “Antenna”, “Synchronization signal”, “Downlink uplink”, “Latency”, “V2X technology” and “Physical control channel”. This suggests that it exist a common interest in developing those technologies, and this is independent from the origin of the assignee.


This paper develops a topic-based patent analytics methodology to investigate technological trends and explore potential opportunities in 5G using patent data. The LDA topic modeling technique was applied to extract latent technological topics. The topic-firm network analysis is proposed to realize a competitive analysis, which is useful in shaping technology strategies for 5G business solutions.


Cheng, J., Chen, W., Tao, F. and Lin, C.L., 2018. Industrial IoT in 5G environment towards smart manufacturing. Journal of Industrial Information Integration, 10, pp.10-19. Chung, P., & Sohn, S. Y. (2020). Early detection of valuable patents using a deep learning model: Case of semiconductor industry. Technological Forecasting and Social Change, 158, 120146. Palattella, M.R., Dohler, M., Grieco, A., Rizzo, G., Torsner, J., Engel, T. and Ladid, L., 2016. Internet of things in the 5G era: Enablers, architecture, and business models. IEEE journal on selected areas in communications, 34(3), pp.510-527.


Evolution of the co-authorship network in 5G technology: Bibliometrics and network analysis from 2005 to 2021.

ABSTRACT. Introduction The fifth generation (5G) of mobile communication network (which succeeds 4G) has been accompanied by an increase in the number of connected devices and thus an increasing consumption of mobile data worldwide. In addition to increasing throughput and bandwidth, the introduction of this technology will support future applications that will impose additional requirements, including low connection latency. While the literature abounds with scientific articles addressing 5G, there is a lack of studies exposing a more detailed view of scientific collaboration in this area. To fill this gap, this study aims to answer the following questions: 1. What is the trend in terms of scientific collaboration in 5G? 2. What are the structural properties of the co-authorship network and is there an evolution of this structure over time? 3. What are the marked trends in international and inter-university collaboration? 4. Which countries and universities are most central to the network? Question 1 aims to determine the importance of scientific collaboration in the 5G field by assessing the evolution of links between researchers and comparing the proportion of collaborative versus single-author papers (intensity of collaboration). In other words, this first step will determine who works with whom on 5G. In answering Question 2, the co-authorship network will be analyzed at the structural level to identify the existence of a small-world structure (deemed more efficient in terms of knowledge transmission) within the network and to measure the degree of cohesion over time. Question 3 first aims at determining the intensity of international (between countries) and inter-university collaboration. Second, it will measure the evolution of the countries’ collaboration with their main partners. The strong rivalry between Huawei and the other players in the game may not have the same impact in the scientific networks as researchers hedge their bets and play both sides. Question 4 will allow us to identify the most central countries and universities in the network, in other words who have the most influence. In sum, answering all of these questions will brush a broad picture of the evolution of the international network of scientific collaboration in 5G. Literature review Throughout the bibliometric literature on 5G, it is interesting to observe two strong trends. First, several researchers use bibliometrics to paint an international picture of 5G research in general. The idea is to collect all scientific papers on 5G and to focus mainly on the number of papers published, the authors, the countries as well as the most productive and most cited universities (Aslam et al., 2020; Mao, 2021; Semwal et Pande, 2021). The second category of papers focuses solely on a particular aspect of 5G. For example, Farouqi, Arshad, and Khan (2021) review the state of research on security and privacy related to 5G networks. Dixit et al. (2020) focus on papers dealing with the antennas needed to transmit and receive 5G signals. Thus, these studies have paid little attention to the collaboration in this field. To fill this gap, this paper aims a clearer picture of the evolution of the scientific collaboration network related to 5G Data and Methodology All papers were extracted from the Thomson Reuters Web of Science (WoS) database using the term “5G” in the title, abstract or keywords. The choice to restrict ourselves to the term “5G” is due to our desire to eliminate false positives, due for instance to some keywords related to 5G technology components (e.g., milimeter wave, edge-computing, or massive-MIMO) being not exclusively related to 5G. This first search yielded 43 155 scientific articles, from 2000 to 2021 which were then manually examined to eliminate those that did not correspond to the fifth generation of mobile communication. The final database contains 13,561 articles. A series of three-year staggered co-authorship networks was then characterized using SNA packages from R, an open-source statistical software, and Gephi, one of widely used Social Network Analysis software. The methodology therefore comprises two steps: 1. Analyse and understand the evolution of the collaborative structure of 5G co-authorship network the intensity of scientific collaboration, the existence of a small-world structure and the degree of cohesion); 2. Calculate the centrality of countries and institutions in the co-authorship network (degree centrality, betweenness centrality and eigenvector centrality). Preliminary findings Three main findings emerge from this analysis. First, and not surprisingly, 5G research is mostly performed in collaboration (very few articles are published by single authors). For each of the periods under study, more than 96% of the papers are the result of collaboration with an average of four authors per paper. In addition, the network is not highly fragmented since the main component represents more than half of the researchers during the first period (2014 to 2016) until reaching more than 60% in the last period (2019 to 2021). Second, the results indeed show that the co-authorship network possesses the properties of a small world in all periods. Thus, the structure of the 5G co-authorship network indicates an optimal structure for knowledge diffusion and information sharing. This contrasts with the cohesion measures obtained: The results show low density in each of the time periods and a negative trend over time for centralization and centrality by proximity. In other words, there is less and less cohesion around central researchers and it is increasingly difficult for researchers to “reach” others indirectly through the network. Turning now to collaboration, the results show that national collaboration is favored over international collaboration in the 5G domain and that this trend is increasing over the years. For example, it is noticeable that from the 2017-2019 period, China favors domestic collaboration over international collaboration, where it has been losing ground over the last three periods. This is clearly observable by the relative decrease in collaboration with its main collaborators and in particular with the United States. There are several reasons for this trend. First, the weakening of collaboration during the last periods of this study coincides with the arrival of the COVID-19 pandemic, this cannot be ruled out as a possible root cause. Indeed, entry restrictions imposed by the United States impacted collaboration as most research collaborations usually begin with face-to-face meetings, to establish the necessary trust relationship, rather than by video conferencing. Second, exacerbated political tensions between China and the United States from 2018 onwards may also contribute to explaining our observations. Chinese scientists are reluctant to travel to the United States because of foreign interference investigations. For example, the U.S. Department of Justice under the Trump administration launched the “China Initiative” program in 2018 to investigate alleged Chinese spying in research and industry. However, despite the set of measures instituted by the U.S. government, the trend of collaboration with China from the U.S. perspective is still growing: the results show an increase in U.S. collaboration with China that has stabilized in the last two periods (2018-2020 and 2019-2021). Our results also show an increase in collaboration with China from the points of view of Italy, Spain, the United-Kingdom, and Sweden. However, the political tension between the U.S. and China over Huawei’s role in 5G network deployment eventually reached Europe and the rest of the world. This may explain the decrease or stabilization, especially in the last period (2019-2021), of the collaboration between European countries, Japan, South Korea and Canada with China. Finally, with respect to the positioning of countries within the collaborative network, the results show that China, the United States, Great Britain, and France stand out from other countries in all three measures of centrality. Implications This article provides a description of the 5G co-authorship network. The results may be the basis for future work. For example, other studies could investigate in more detail the impact of public policies on collaborative trends at the international level.

15:15-16:45 Session 4E: Triple Helix
Advancing University, Industry, and Government Collaboration in Bibliometrics Building a network infrastructure for data infrastructure

ABSTRACT. Background and Rationale Applying an ethnographic approach to the research and application of bibliometrics, this paper synthesizes the scope and relevance of scientometric work with actor-network centered approaches from qualitative social sciences. In doing so, I draw on data collected within the Competence Network for Bibliometrics (Kompetenznetzwerk Bibliometrie, KB) in Germany. The KB is funded by the federal ministry for research and education and is the leader for the development of bibliometric tools in Germany. As a network with 22 high-profile member organizations including Fraunhofer and Max-Planck institutes, KB follows the research output from every field of study, reveals its impact, provides metadata, and makes research visible across disciplinary boundaries. It links partner research institutions and stipulates new disciplinary dynamics in the field of bibliometrics and scientometrics. Recently, the center has initiated an organizational transformation process from a center structure towards an agile network. Reasons for this are manifold, but predominantly rooted in the need for pooling bibliometric skills and making them available to the wider public including universities, government agencies and the industry. Design, control, and implementation of this transformation process is a complex organizational challenge given that the center now comprises of 22 consortium partners. This paper critically examines 1) this transformation process and elaborates on the scientific benefits it has produced so far and 2) the potential this restructuring hold for the fields of skills transfer and science communication with the goal of strengthening exchange between science and government agencies as well as industry. The study and application of bibliometrics is by definition concerned with quantitative measures. This paper promotes an innovative mix of applying ethnographic methods to a field that is shaped by quantitative approaches in order to uncover, and later utilize interpretations, analyzes and attitudes from network partners. Such feedback will be instrumental in structuring later stages of the transformation process.

Methods The aim of the on-going research presented here is to shed light on the benefits and functioning of the KB transformation process towards a network. Methods applied draw from ethnographic and related qualitative methods in order to gain a deep understanding on the 1) transformation process towards an agile network structure, 2) how the implementation procedures applied are viewed by network members and 3) how network members evaluate the changes made so for a) their work, b) their own productivity, c) the overall benefits for the whole network. Applied methods are participant observation in the virtual network meetings, different forms of interviews (online: open, theme-focused, structured) as well as in situ research in selected partner institutes in Germany.

Anticipated Results Anticipated results provide insights into how the transformation process is perceived by selected members of the network. Collected data consists of internal voices from the network and will inform the future implementation process of the organizational transformation of the KB. It also acts as quality assurance as it helps to evaluate and, if necessary, adjust strategy. Furthermore, data will also shed light onto how an agile network structure for a science consortium shapes and improves communication and general interaction between the sciences as well as government bodies and industry.

The Relationship between Government Innovation Funding and the Likelihood of Venture Capital, IPO, and Acquisition in Genomics

ABSTRACT. While governments seek to spur innovation and technology commercialization through funding programs, we have little insight into other outcomes that these programs may enable. This study examines the relationship between the U.S. government's SBIR program and the long-term success of participating firms.

Introduction Many governments support innovation and technology commercialization in firms through funding programs that target promising technologies. Two such programs in the US are the Small Business Innovation Research (SBIR) and Small Business Technology Transfer (STTR) program created to support R&D in small firms. The secondary objective of these programs is to help organizations cross the “valley of death” in which innovation is promising, but too expensive to pursue for private investors (Bonvillian, 2011). Research exploring the long term effects of such programs provide mixed results (e.g., Kapsali, 2011; Qian and Haynes, 2014; Lanahan et al., 2022; Dutta et al., 2022). Work has shown the positive impact of these programs such as increased commercialization of innovations (Audretsch, Link, & Scott, 2002), and increased commercial success of these innovations (Archibald, & Finifter, 2003), but long-term studies are rare. Given that the US government spends over a billion dollars each year on these programs, more insight is warranted. For example, we do not know the extent to which resources from SBIR and STTR programs influence the success of ventures in terms of VC, IPO and being acquired.

Theoretical Background - Summary The SBIR and STTR programs are two of the latest and biggest programs that the U.S. government has enacted to support small business development (Bonvillian, 2011). Through 2021, SBIR and STTR have cumulatively provided almost $60 billion in funding (Small Business Administration, 2022). While both programs fund R&D, the SBIR focuses on innovation in small businesses while the STTR focuses on innovation in public/private collaborations, particularly between small businesses and nonprofit research institutions. Both programs are coordinated by the U.S. Small Business Administration. The SBIR program supports innovation activities in small, often nascent organizations. Implemented in 1982, the SBIR program seeks to promote high-technology innovation in small businesses by funding R&D projects. Federal agencies in the U.S. government with an R&D budget over $100 million must allocate 3.2% of their budget to SBIR grants. The agencies then solicit proposals from small businesses related to the agencies’ needs. In 1982, the program provided over $38 million for R&D in small businesses through 789 awards. Through 2021, over 170,000 awards have been made totaling more than $53 billion (, 2022). The STTR program was established in 1992 to support innovation collaborations between firms and public organizations such as universities and government labs and facilitate the transfer of intellectual property from research institutions to the market. Federal agencies with R&D budgets over $1 billion must allocate 0.45% to funding STTR grants. Five federal agencies participate in the STTR program. Through 2021, the program has grown to provide over $5 billion with 16000 grants. The overarching objective of these programs is to support R&D activity that is often expensive, at times beyond the means of small, nascent ventures (Link & Scott, 2010; Bonvillian, 2011). Thus the programs enable these firms to better compete with larger firms with more resources, encourage technology commercialization and support economic growth (Small Business Administration, 2013). While basic science lays the foundation for this technology, government projects tend to build applications of basic science work, usually having a proof of concept to get government funding (Rogers et al. 2001). Thus, these projects are vetted by an experienced audience and are closer to commercialization than academic projects based on foundational scientific research. This can help firms build social capital in addition to helping them commercialize innovations (Gray et al., 2022). Additionally, the SBIR awarding agencies choose to fund firms that they believe are viable and potentially successful bringing their technology to market (Lanahan et al., 2022). Giga et al. (2022) found that these grants correlated to higher patenting rates for small businesses. Hypothesis 1: Technology firms supported by SBIR and SBIR grants have a higher likelihood of success than other technology firms.

Obtaining VC funding is an important milestone for founders wishing to complete an IPO. Raising VC funding is the sale of a firm’s equity to external investors, providing financing to help startups grow. In 2021, VC fundraising reached new highs with 17,000 firms raising over $300 billion (Pitchbook-NVCA, 2022). VC funding acts as a signal to external stakeholders on the value of the firm and its likelihood of growth. VC backing can lead to high growth and IPO (Chemmanur et al. 2021). Obtaining both government grants and VC will support a firm’s ability to succeed in the long term. Hypothesis 2: Technology firms supported by SBIR grants and VC have a higher likelihood of success than other firms.

Methods We selected the genomics sector of biotechnology firms in the U.S. Genomics is the study of a complete set of genetic information or an organism. 1983 is a starting point since it is the year of earliest relevant startup activity. This ensures that the data are not left-censored (Blossfeld & Rohwer, 2002; Yamaguchi, 1991). A database of all genomics companies started in the U.S. before 2019 was compiled by triangulating many sources including patents, government grants, industry lists, scientific publications, university websites, VC listings, directories, press releases, and articles. To be included, over 50% of its activity such as products, patents, R&D, or sales, must be related to genomics. The firm must also be for-profit and independent. This classification process parallels previous works that identified new technology firms (e.g., Schoonhoven, et al., 1990; Woolley, 2017). The database consists of 619 firms and 36 firms were dropped due to a lack of reliable data for the founding teams resulting in a final sample of 583 firms. Using press releases and deal listings such as Crunchbase, we determine the date of the first VC funding, IPO, and acquisition for each firm as appropriate. For each, the following variables were collected or constructed: a dichotomous variable of event attainment, year of event, and time between firm founding and event. Several firm and macro level controls were included. We used the event history analysis technique using STATA with maximum likelihood estimation, robust standard errors and Weibull distribution.

Findings In total, 47% of the firms obtained SBIR or STTR funding, 58% obtained VC, 18% had an IPO, and 30% were acquired. The models show that firms with government funding were less likely to obtain VC, go IPO or be acquired. However, when firms with both government funding and VC were almost three times more likely to go IPO than other firms.

Significance This study develops our understanding of the relationship between government support and other measures of firm success: VC, IPO and acquisition. The results indicate that if the government is seeking to support high-growth firms that are acquired or go public, agencies should work with other ecosystem institutions such as other investors to improve the firm’s likelihood of success.

Technology Transfer and Commercialisation in Nigerian Universities: Motivation, Barriers and Policy Options

ABSTRACT. Technology Transfer and Commercialisation in Nigerian Universities: Motivation, Barriers and Policy Options Background and Rationale Transitioning from resource-based to knowledge economy has placed premium responsibilities on the knowledge institutions to support innovation (Adelowo, Olaopa and Akinwale, 2017; Adelowo, 2018; Adelowo and Surujlal, 2020). New knowledge creation and dissemination are fundamental roles of higher educational institutions (HEIs) and research organisations (ROs) (knowledge institutions), although some large corporations sometimes establish research base. The core mandates of these institutions have consistently undergone changes over time to reflect different levels of economic and technological development. For instance, educational institutions are set up to supply critical human capital needs at all levels to provide leadership and appropriate governance to both administrative and technical systems in the society. Later, research and development (R&D) activities was added given the substantial knowledge pool existing in the institutions. Translating R&D outputs to useful products, processes and services in the industry has become parts of the mandates of knowledge institutions in the last few decades, and for this to happen, these institutions have to be entrepreneurial in their approach. The arguments for knowledge institutions to become entrepreneurial in their engagements was clearly articulated in the knowledge spill-over theory of entrepreneurship (KSTE) (Acs et al. (2013) and the need for universities to be responsive to societal challenges (Etzkowitz (1998). This implies that knowledge institutions possess talents and tangible research results that are capable of transforming the society or an economy. This argument becomes more obvious when economies with most innovative firms and regions, including global leaders in high-tech are considered. For instance, most of the top twenty leading innovative companies globally emerged from the developed and newly industrialising economies that have not only invested heavily in knowledge creation but also created a channels through which those knowledge outputs are transferred for economic renewal and societal benefits (BCG, 2019). These economies are also home to top ranked universities and research institutions, supplying critical skills and knowledge inputs to the industries in a coordinated manner. Therefore, for developing countries in Africa, particularly Nigeria to harness knowledge outputs for development, there is the need for the knowledge institutions to embrace entrepreneurship and innovation as parts of coherent policy to achieve both institutional and developmental goals. Parts of the process to embrace innovation and entrepreneurship include the creation of and, or strengthening existing mechanisms for knowledge transfer in these institutions, including technology transfer offices, technology incubators, knowledge parks, innovation hubs and other innovation infrastructure. Nigeria has since 2006 started establishing technology transfer offices in its knowledge institutions to serve as a collation hub for R&D outputs and to foster linkages with industry for the purpose of technology commercialisation among others. To date, about forty-three intellectual property and technology transfer offices (IPTTOs) have been created/established in different institutions across the country. Despite its existence for more than a decade, the performance of few of them have been recently assessed, suggesting improvement in the patenting activities among the scientists and researchers (NOTAP, 2019). Also, institutions without the IPTTOs have also demonstrated great potentials in creating commercialisable research results but without any documented (known) mechanisms for its management. Data and Methodology Therefore, this article x-rays the strategic approach(es) adopted to sensitize stakeholders and manage research outputs in Nigerian universities and research institutes using primary data collected through cross sectional survey conducted among thirty-one (31) intellectual property and technology transfer offices and one hundred and sixty-seven patentees. For the institutions without the IPTTOs, we also established the mechanisms adopted to manage tangible research outputs within them. The motivation and barriers to patenting and commercialisation were examined among scientists and researchers in the selected universities. Preliminary Results The results show weak management capabilities in the TTOs, as most of the managers and staff do not have adequate knowledge and training in IP management. Most of the staff (67%) of the TTOs, including the directors regarded themselves as permanent staff, however many of them have primary department or faculty they are affiliated to in the same university. The basic office infrastructure in the TTOs are fairly adequate while only few of them have automated their IP database. On major activities of the TTOs, IP sensitisation and training for faculty members, industry collaboration, facilitation of inter/intra departmental linkages and collation of R&D outputs are reported as the dominant ones. The TTOs are majorly financed by the host institutions and sometimes by the internally generated revenue. Financing TTOs from the institutions’ budgetary allocations is perceived as additional financial burden, given that universities and research institutes are not adequately funded. The Offices hardly receive funding from the National office for Technology acquisition and Promotion, an Agency of government that is responsible for establishing and funding IPTTOs in the country. The major challenges confronting the offices are inadequate funding, personnel, training opportunities and inability to sufficiently foster industry collaboration for research uptakes. Few of the TTOs have assisted their institution to develop robust IP policy at the time of data collection. From the inventors’ perspectives, key motivations reported for filling patents are promotion, prior knowledge of IP, expected monetary returns, career advancement and recognition/award/fame that is associated with it. Only a handful of inventors have commercialised their inventions, and that was done through private arrangements. Major barriers to inventions commercialisation are lack of industry collaboration, poor funding, inadequate innovation infrastructure, and lack of IP policy or incentives. The paper concludes with policy recommendations to improve innovation ecosystems in the universities.

References Acs, Z. J., Audretsch, D. B., and Lehmann, E. E. (2013). The Knowledge Spillover Theory of Entrepreneurship. Small Business Economics, 41 (4): 757–774. Adelowo C. M., Akinwale O. Y. and Olaopa O. R. (2017). Innovation and Knowledge Transfer in Nigeria. International Journal of Research Innovation and Commercialisation, 1 (1): 57-73. Adelowo, C. M. (2018). Factors influencing Academic entrepreneurship in Nigerian universities. International Journal of Innovation and Regional Development, 8(4): 337-357. Adelowo, C. M. and Surujlal, J. (2020). Academic entrepreneurship and traditional academic performance in universities: evidence from a developing country. Polish Journal of Management Studies Vol. 22, No. 1, pp. 11-25. Etzkowitz, H. (1998). The Norms of Entrepreneurial Science: Cognitive Effects of the new University-industry Linkages. Research Policy, 27 (8): 823-833.

Local innovation potential of universities: university-industry cooperation and bridging researchers

ABSTRACT. Most research-active universities tend to have a significant socio-economic impact on their city or metropolitan area, but their potential to generate such impacts in their local geographic area is not well understood.

The ability to generate impacts is now often linked to their innovation capacity or innovation performance, often framed within regional or national policy settings where research-intensive universities are expected to become more productive as ‘engines of innovation’ to the benefit of their hometown, local region, or domestic economy. Whether or not such a university manages to meet such expectations depends on externalities and local circumstances as well as in-house preconditions. In an attempt to disentangle the large array of internal and external factors, we introduce the place-based concept of university local innovation potential (ULIP) to address the question why some universities seem to be in a better position to realize their innovation potential than others.

The nature and composition of that potential is represented by an ecosystem model comprising of four R&D-based components, which covers relevant in-house organizational features as well as external factors of the university’s city of location.

Focusing on the human resources component within a university’s R&D-ULIP profile, this quantitative, empirical study is mainly devoted to examining the role of cross-sectoral academic researchers, i.e. those who are or were also active in the business sector. Building on our prior studies, we posit that such boundary-spanning ‘bridging researchers’ within a university are a key enabling factor of local university-industry research cooperation (Yegros-Yegros and Tijssen, 2014; Tijssen et al., 2020). We hypothesize that the presence of such bridging researchers is a key explanatory factor of ULIP.

Our internationally comparative study of ULIP profiles comprises a total of 520 large research-intensive universities in three countries: USA (210), China (204) and the UK (106). The empirical data relates to the years 2016-2019. Each university is represented by nine metrics-based variables. We focus our attention on the two ‘dedicated resources’ variables in the model that represent the presence of cross-sectoral, boundary-spanning academic researchers within these universities. To test our hypothesis, we conducted a series of statistical tests with generalized linear regression models. The statistical relationships between the model’s components and local university-industry research cooperation were assessed with quantitative data pertaining to each university’s output of co-authored research publications resulting from ‘local’ research cooperation between universities and R&D-active companies that located within a 50km range from the centre of the university’s city or metropolitan area.,

The results of the regression analyses indicate that bridging researchers are indeed a highly significant ULIP component in each of the three countries. We also find that, as to be expected, local university-industry research cooperation depends on both the university’s research profile as much as its geographic location (city and country). University patent production, however, appears to be a much less relevant explanatory variable.

The observed ULIP pattern across the Chinese universities differs markedly from the patterns in the USA and UK. China also exhibits a steep increase in the quantity of local university-industry co-publications owing to Chinese government policies to promote and support university-industry cooperation.

Literature references

Tijssen, R., Van de Klippe. W. and Yegros, A. (2020). Localization, regionalization and globalization of university-industry research co-operation in the United Kingdom. Papers in Regional Science (

Yegros-Yegros, A. and Tijssen, R. (2014). University-industry dual appointments: global trends and their role in the interaction with industry. Proceedings of the Science and Technology Indicators Conference (STI 2014), Leiden, Netherlands. 712.

15:15-16:45 Session 4F: Carbon, Batteries, Energy
Greening manufacturing: Technology intensity and carbon dioxide emissions in developing countries

ABSTRACT. Industrialisation is an important engine of growth and ‘catch up’, but also associated with harmful carbon dioxide emissions and hence with climate change. This poses a challenge for sustainable industrial development, particularly for late industrialisers on how to industrialise while also mitigating carbon dioxide emissions. This paper investigates the effect of technology intensity across manufacturing industries on carbon dioxide emissions: are medium- and high-technology manufacturing industries less emissions-intensive than low-technology manufacturing industries in developing countries? The paper analyses this relationship for a panel of 68 developing and emerging economies over the period 1990-2016, by adapting the environmental Kuznets curve and the stochastic effect by regression on population, affluence and technology approaches. Using two alternative measures of emissions and estimating generalised method of moments model, the results show that medium- and high-technology manufacturing industries are associated with lower carbon dioxide emissions than low-technology manufacturing industries. The results also show that these differences vary by the income levels of countries. These findings have important policy implications, suggesting that a shift towards more technology-intensive manufacturing production processes may be a more environmentally sustainable industrialisation path for developing countries.

The multilevel roles of the State in transformative innovation policy: analyzing the implications for emerging economies in the case of Uruguay and Colombia

ABSTRACT. Literature on Transformative Innovation Policy (TIP) is growing fast (Haddad, et al., 2022; Diercks, et al., 2019) and, although it has received some attention in Latin America, the role of the State in TIP in the context of emerging economies can be further explored (Ordóñez-Matamoros, et al., 2021). The context of Latin American emerging economies represents differentiated implications for the role of the State in TIP. For instance, gaps in governmental capacity for policy design and implementation, the differentiated types of challenges and priorities for Latin American countries in contrast to more developed economies in the Global North (e.g. socioeconomic inequality, violence, high levels of poverty, biodiversity loss, among others), the central role of politics that shape innovation policies in line with contested democracies, among other factors that define the role of the State in this context.

In order to better understand the implications of TIP in emerging economies we need to further inquire on how these specific features redefine the conceptualizations of the State in shaping innovation policies. We argue that a ‘performative’ perspective on the role of the State is needed, instead of the prevailing ‘monolithic’ approach that overlooks the interplay between different agents at multiple levels of TIP, which might lead to variegated outcomes in terms of transformative change.

In this research we inquire on what are the multilevel roles of States in TIP in Latin American emerging economies. In line with Borrás & Edler (2020), here we argue that TIP is a complex process in which the State plays diverse roles -even contradicting ones- depending on de facto governance settings. Therefore, we unpack these roles at three governance levels in the case of energy policy in Uruguay and the policy for social appropriation of knowledge in Colombia. We aim at disentangling the way in which the strategies of different actors shape the ultimate capacity of the State to design and implement (or fail) innovation policies with transformative potential at different governance levels.

Here we define transformative innovation policy (TIP) as “a set of public actions and instruments, through which governments mediate and mobilize resources towards more sustainable and inclusive sociotechnical systems via the promotion of knowledge and innovation production, diffusion and use with a long-term perspective” (Ordóñez-Matamoros, et al., 2021, p. 119). Considering the comprehensive ambitions of this policy approach, governments deploy differentiated forms of action and roles. While it can often be an active promoter of meta-governance for transitions (Kuhlmann & Rip, 2018), literature usually portrays this role as a reaction to failures in markets (Nelson, 1959), systems (Woolthuis, et al., 2005), or transformation processes (Weber & Rohracher, 2012). Despite its explanatory value, the failures approaches assume the role of the State as linear, reactive and limited to problem-solving (Turnbull, 2018).

As a response to that, we draw on Borrás & Edler’s (2020) typology of roles of the State, according to the drivers of sociotechnical change (State or non-State actors) and the modes of governance involved (hierarchical or heterarchical). Furthermore, we use these initial typologies in a multilevel heuristic, in order to grasp the complex interactions that take place in TIP implementation processes. We ground our reflection on the analysis of two cases: Uruguay and Colombia. First, we analyse Uruguay’s ‘Energy Policy 2005-2030’ approved between 2008 and 2010. This is generally regarded as a successful State-level policy for energy sovereignty, with successful results in a short period of time in terms of reducing energy costs and strengthening national energy industry. This is related to the scope of this transformation, the time span in which it took place, and the political consensus on which it was built. We explore this transformative process by focusing on the underlying institutional arrangements that shaped the State’s capacity in this case and the role of innovation policy therein.

Second, we analyse the Colombian Social Appropriation of Science, Technology and Innovation (SASTI) Policy, in place since the early 1990s. Here we observe an emerging direct approach to development that challenges the path-dependence dynamics that drive innovation policy exclusively towards increasing productivity and economic growth. In contrast to the Uruguayan experience, this case shows how governments do not necessarily have a purposeful approach on transformative innovation, but their role is rather shaped by the bottom-up agency efforts of institutional entrepreneurs, who nest transformative initiatives in spite of overarching business-as-usual policy environments.

These cases follow the case study research method (Yin, 2018). The data studied considered legal regulations, policy documents, academic and technical documents, and interviews with actors involved in the cases. It is worth mentioning that the two cases took a different scope and focusing unit of analysis. Thus, while the Uruguay case considered the transformative policy, the Colombia case was focused on the institutional entrepreneurs. Such differences do not affect the case analysis. Instead, it contributes to discussing from different viewpoints the state’s roles in the Global South, enriching the study.

Early findings suggest at least three possible sets of roles of the State. The first set depicts an optimistic or supportive role of the state. This category is constituted by the roles described so far by literature, as illustrated in section two. The second set is shaped by those roles that describe a neutral position of the State towards transformative processes. This neutrality means an indifferent position of the State towards transformative challenges. In other words, the State does not support nor block transformative processes if there are any in development. Finally, the State can perform a negative set of roles to interrupt or erode the transformative processes. At this point, the question of what are the conditions that explain the deployment of any of these three sets of roles by the State emerges.

This research contributes to a more critical understanding the governance challenges that the Latin American context represents for policies aimed at sustainability transitions and social inclusion, for in such settings it is not possible to assume that the State plays a specific and distinguishable role in addressing transformative governance demands.

Innovation-Led Oriented Policies: Investigating the Development of Low-Carbon Hydrogen Technologies

ABSTRACT. Background and rationale: Recent literature has focused on the importance and many potentials of innovation or mission-led policies to respond to social, environmental, and economic challenges (Mazzucato, 2018). This framework recently gained attention by defending the role of governments in setting the direction of technical change, promoting innovation, and the diffusion of new technologies. Thus, mission policies help create and shape new markets via targeted innovation-led oriented policies and can be considered an opportunity for countries with less mature R&I systems to accelerate the development of their capacities (Fisher et al., 2018). The main differences between mission-oriented policies (MOPs) and other typical innovation and industrial policies may be in terms of “scale (bigger), scope (broader) and target (more specific)” (Alves, Vonortas & Zawislak, 2021, p.81). MOPs often use a mix of policy instruments going beyond the mere realm of R&I policies and require horizontal policies cutting across governance levels (Fisher et al., 2018). Ultimately, mission-led policies are seen as a helpful way to employ a policy direction that is smart, inclusive, and green (Mazzucato & Perez, 2014). In fact, scholars do consider that “green growth” is the ultimate goal and the next big technological and market opportunity that can stimulate private and public investments (Mazzucato & Perez, 2014). A promising technology that falls under the umbrella of “green growth” is the production and adoption of low-carbon hydrogen. Producing hydrogen from renewable sources is seen as a global alternative for decarbonizing energy production and economic activities by 2050. In an attempt to close the gap between rhetoric and action, the International Energy Agency (IEA) has set a roadmap that establishes pathways to reach net zero emissions by 2050 that highlights the importance of hydrogen as an energy vector (2021). Similarly, governments from advanced and developing countries launched ambitious hydrogen strategies and directed economic stimulus funds to this area. Based on this background, we can consider that producing low-carbon hydrogen is one goal of a set of global policies that target green growth. Although MOPs have a major part to play in delivering better quality growth while addressing grand sustainable challenges, they can also face their own challenges in their design, implementation, and assessments. Alves, Vonortas and Zawislak (2021), emphasize that the mismatch between the expected goal and what is actually feasible (based on the available technological capabilities) creates what the authors call “a fuzzy boundary” that often leads to the unsuccessful implementation of missions. Another important aspect to consider is that, while setting goals and determining pathways, missions will hardly be reduced to a single development path or by a single technology. The establishment of a goal or a mission should consider the available innovation capabilities in order to promote a successful regime change or shape and create markets. We consider that MOPs may create an institutional incentive for regime change, catching-up, capability building, and eventually for market creation, but we must also consider that a MOP approach should acknowledge the many stages of development and diffusion of still immature technologies. Geels’ (2002) approach is helpful to understand the dynamics of regime change, the emergence of new technologies and their diffusion processes, and how they interact with extant sociotechnical regimes that are crucial for the societal sustainable transformation. One can consider that MOPs cannot be successful without the understanding and mastering of innovation capabilities and the characteristics of their technological path. A full understanding of these dynamics can directly impact the success of the implementation of the MOP framework for sustainable innovation.

Objective: In this article, we aim to analyze and discuss the effectiveness of applying the MOP framework in the context of innovation and diffusion of new technologies applied to low-carbon hydrogen and present a historical approach to the development of this sustainable innovation.

Methodology: We conducted an in-depth literature review of the recent history of the technological development of low-carbon hydrogen as an energy vector. A detailed mapping of global policies that are part of the mission approach applied to this topic was developed. This review covered papers, international reports, recommendations, and cooperation agreements related to low-hydrogen technologies. Significance of the study: This study contributes to academic research and literature by providing a detailed study that enhances the potential and limitations of the MOPs framework applied to the technological development of sustainable technologies. Our analysis demonstrates that MOPs can be helpful tools for addressing important societal problems and can also stimulate government and private actions that could not have happened otherwise. However, the detailed comprehension of the technological regimes and capabilities of a technology, which is the central discussion of a mission, is essential as this influences the implementation and the success of a MOP.


Alves, A. C.; Vonortas, N. S.; Zawislak, P. A. (2021). Mission-Oriented Policy for Innovation and the Fuzzy Boundary of Market Creation: the Brazilian Shipbuilding Case. Science and Public Policy, 48(1). doi: 10.1093/scipol/scaa059 Fisher, R., Chicot, J., Domini, A. et al. (2018) Mission-Orientated Research and Innovation: Assessing the Impact of a Mission-Oriented Research and Innovation Approach. Final Report

Geels, F. W. (2002) ‘Technological Transitions as Evolutionary Reconfiguration Processes: A Multi-Level Perspective and a Case-Study’, Research Policy, 31/8–9: 1257–74. IEA – INTERNATIONAL ENERGY AGENCY. Net Zero by 2050. A Roadmap for the Global Energy Sector. IEA, Paris, 2021.

Mazzucato, M. (2018). ‘Mission-Oriented Innovation Policies: Challenges and Opportunities’, Industrial and Corporate Change, 27/5: 803–15. Mazzucato, M.; Perez, C. (2014). Innovation as Growth Policy: the challenge for Europe. Working paper series, SWPS.

The role of external knowledge in industry development and sub-market formation: the case of lithium-ion batteries

ABSTRACT. Background and Rationale

The development of many existing clean energy technologies and products has relied on the integration of knowledge and components that originated in different technological fields and industrial sectors. Therefore, to be able to design better public policies that support and accelerate clean energy innovation required for the decarbonization of the global economy, it is important to understand how such external knowledge contributes to the emergence and subsequent evolution of new technologies and industries.

Existing literature on industry evolution has emphasized the role of technological breakthroughs or discontinuities in the evolution of different industrial sectors, which is typically associated with undermining the competence of industry incumbents or enhancing the position of entrepreneurial new entrants. In many cases, these discontinuities are the result of technology spillovers that contribute external knowledge from other fields to innovation in the industry.

In this literature, industry disruption associated with technological breakthroughs usually leads to the emergence, growth, and eventual dominance of new sub-markets—specialized markets within a larger market. However, the emergence of sub-markets does not necessarily require technological discontinuity. Another stream of the literature suggests that sub-markets may also emerge in response to heterogeneous user needs, especially those unmet ones, offering an alternative to a supply-side view of the industry life cycle. It is not yet clear what role, if any, external knowledge plays in the emergence of sub-markets that serve new or previously unmet user needs.

In this study, we address this gap by investigating how external knowledge has contributed to the co-evolution of lithium-ion battery (LIB) technology and industry composition (i.e., actors and sub-markets) since the early 1970s. LIB is widely considered to be one of the key clean energy technologies contributing to global decarbonization efforts. The original LIB technology architecture emerged from a series of major scientific and technological breakthroughs which were awarded the Nobel Prize in Chemistry in 2019. More recently, the LIB industry has been dominated by the demand for batteries for electric vehicles (EV). These industry characteristics offer an opportunity to investigate both the supply- and demand-side perspective on the role of external knowledge in industry evolution.


We track how external knowledge has contributed to the co-evolution of LIB technology and industry composition over time by process-tracing external knowledge sources, main innovating actors, and the subsequent impact on technology and industry development for a set of 84 important LIB-related inventions assembled through a systematic literature review on the LIB history. In addition, for more recent inventions with a yet-unclear impact on future technology and industry development, we focused on inventions associated with 18 prominent companies and start-ups currently identified as promising by LIB industry experts, using Crunchbase to track the disciplinary and industry background of their founders and investors.


Based on our findings, we divide the history of the LIB industry into four periods. The first stage, “incubation”, started in the 1970s with Stanley Whittingham’s discovery of electrochemical intercalation and concluded in the early 1990s with the commercialization of the first-generation LIB by Sony. In this period, the knowledge base of what we know today as the lithium-ion battery was formed by many scientific discoveries and technological breakthroughs, with most of them bringing external knowledge from other fields to solve critical scientific and technical problems emerging on a trajectory toward the first working LIB. Innovation during this period was mostly driven by researchers and inventors in large corporations, universities, and public research institutions.

A “stabilization” period followed the incubation stage during the 1990s when LIBs demonstrated superior performance compared to other rechargeable power sources and quickly started to dominate the market for consumer electronic devices. During this period, the LIB technology design stabilized, with the majority of LIB innovations being incremental improvements of different components or manufacturing processes developed by corporations to improve LIB safety, performance, and manufacturing cost. Notably, very few of these innovations involved external knowledge, indicating a decline in its importance for the LIB industry during this period.

The third stage of “expansion” started around the early 2000s when the market success of LIBs for consumer electronics attracted the attention of users and producers seeking alternative applications for LIB technology, such as power tools or electric vehicles (EV). Different and often unique user needs in these applications required adapting LIB technology to those needs, often relying on external knowledge about these applications. Furthermore, universities, also attracted by these opportunities and often supported by public R&D funding, leveraged their knowledge from other fields beyond LIB, such as nanotechnology or materials science, to experiment with alternative battery concepts or components that were previously abandoned or had not yet been commercialized but were now considered promising for new applications.

The fourth stage, “diversification”, started around the early 2010s with the onset of demand-pull public policies that supported rapid growth in the EV market. The resulting dramatic increase in the demand for LIBs created a new rapidly expanding sub-market of LIB for EVs, which is characterized by multiple emerging and competing LIB and post-LIB technological trajectories rather than a single dominant technology design. Two new types of actors entered the LIB industry in this period. One is incumbent car manufacturers as actively engaged downstream users who bring a wealth of external knowledge from the automotive industry. Another is a booming number of start-ups, often created as university spin-offs, that seek to commercialize LIB innovations and alternative battery concepts developed in the previous period.

Significance for Policy

By analyzing the history of the LIB industry and technology, we find that external knowledge has been instrumental in the emergence and evolution of this industry. However, its relative importance varied over time. Initially, external knowledge provided a crucial contribution to the formation of the knowledge base of the emerging LIB technology. It was significantly less important during the stabilization stage. In the following two stages of expansion and diversification, which were characterized by the emergence and rapid growth of a new LIB sub-market for EVs, external knowledge, again, became important for the adaptation of the existing LIB knowledge base for new applications driven by user needs and demands. Particularly in the most recent period of diversification, demand-pull public policies provided crucial support for the integration of external knowledge into the LIB industry knowledge base by incentivizing the entry of new types of innovating actors.

Our findings provide important implications for policies supporting clean energy innovation. Policy mixes should combine supply-side and demand-pull policy instruments to facilitate the generation, identification, transfer, and integration of external knowledge into clean energy technologies. For example, R&D funding policies, particularly at the incubation stage, should be designed in a way that makes relevant external knowledge easily identifiable and accessible to researchers and inventors, e.g. by encouraging multi-disciplinary team composition and cross-sectoral learning. Demand-pull market incentives, particularly during the expansion and diversification phases, can attract new entrants to the industry, e.g., incumbents from other industries that provide external knowledge about promising new applications and user environments, or university spin-offs and start-ups that ensure technology lock-in is avoided. Finally, policies that support industry coordination efforts, e.g., through public-private partnerships or road-mapping exercises, can help identify and close gaps in the industry knowledge base where external knowledge can be productively applied.

18:00-21:00 Session 5: Poster Session @ Gala Dinner
Open Innovation System, Absorptive Capacities, and Sustainable Economic Growth in Africa

ABSTRACT. The current context in African countries is characterized by increasing environmental constraints, unemployment, and poverty, whence the need to achieve sustainable development. Indeed, it will ensure the satisfaction of the needs of the present generation without compromising those of future generations. Thus, countries must achieve sustainable growth, in particular by fostering infrastructure development; promoting research, science, innovation, and technological development; and scaling up global and regional partnerships for development (UN/ECA, 2016a). The recognition of the importance of technical progress and research and development (R&D) in fostering growth has led to the development of neoclassical, evolutionary, Neokeynesian, and institutional theories. Indeed, while the old neoclassical theory of growth integrates technical progress in exogenous form for the improvement of long-term growth (Solow, 1956), new theories do it in the endogenous form with an emphasis on knowledge accumulation (Romer, 1986), human capital (Lucas, 1988; Romer, 1990), public spending on physical capital and social infrastructure (Barro, 1990), and technological innovation related to R&D introduced in different forms (Aghion et Howitt, 1992; Grossman et Helpman, 1989; Romer, 1990). Evolutionary theories focus on supply through the central role of technological change in firms and the generation of novelty (Nelson et Winter, 1982; Silverberg et Verspagen, 1994). Neokeynesians consider the role of demand (the degree of consumer orientation) in technical progress and growth (Kaldor, 1966; Pasinetti, 1994). Finally, the institutional current is based on the economic organizational efficiency in terms of taxation and legislation on property rights, to name just a few, as crucial factors for improved economic growth (North, 1994). In their view, institutions must play the leading role in terms of technological policy to foster innovation behaviors of the most important economic actors through their mode of interaction. Thus, several systemic approaches to innovation have been developed after the national innovation system (NIS). Approaches that provide information on countries' capacity to generate and benefit from innovation (Freeman, 1987; Lundvall, 1992; Nelson, 1993). Indeed, the NIS advocates the interaction between different economic actors such as industries, universities, research centers, public institutions, financial institutions, and end users. Among these actors, industries and governments are considered respectively as the center and the main actor of the innovation process regulation. However, in the context of a knowledge-based economy, the Triple Helix (TH) innovation system views university institutions as leaders when it comes to the innovation process (Etzkowitz et Leydesdorff, 1995 & 1997; Leydesdorff et Etzkowitz, 2000). This approach advocates interactions between university, industry, and government as the key to improving the necessary conditions for technological innovation generation. The extension of the Triple Helix to Quadruple Helix model (Carayannis et Campbell, 2009) by taking into account social contribution has led to the development of the endogenous growth model of the national innovation system (Afonso et Monteiro, 2012; Monteiro, 2013). However, the environment, according to its current constraints in addition to the need to consider additional factors for the explanation of the technological innovation process has led to the extension of the Quadruple Helix model to Quintuple Helix (Carayannis et Campbell, 2010) and Ntuple Helix (Leydesdorff, 2000) that serves as a framework for sustainable improvement of country performance. However, the analysis of African countries’ performance reveals the underdevelopment of their national innovation systems as compared to those of developed countries which remain characterized by weak interactions between different spheres (Casadella et Benlahcen-Tlemcani, 2006; Gu, 1999). The studies on innovation systems in Africa also show an important synergistic contribution at the foreign level as compared to domestic and global ones (Mêgnigbêto, 2015; Mezouaghi, 2002). In addition, African countries, particularly those in Sub-Saharan Africa have recorded average growth rates of 5% over the past 15 years. But this growth is mainly due to commodity prices instead of technological innovation. However, the latter occupies the top position in Asia, emerging countries, and other developed countries (NU/CEA, 2016b; WEF, 2016). In terms of sustainability, there is a very poor performance of African countries as compared to those of other parts of the world despite the high natural resources endowment that abounds the continent. In fact, of the 180 countries included in the 2017 global sustainable competitiveness index report, the top four in Africa are Ethiopia (64th), Ghana (68th), Côte d'Ivoire (77th), and Kenya (80th). There are also 8 African countries in the 20 and occupying the last positions in such ranking (SolAbility, 2017). Notwithstanding, studies show that the development of the innovation system in addition to absorptive capacities (i.e. the different factors that affect the ability of countries to take advantage of foreign technologies) favor the convergence of developing countries towards developed ones (Abramovitz, 1986; Albuquerque, 1999; Archibugi et Coco, 2004; Edquist, 2001; Fagerberg et Srholec, 2008; Fagerberg et Verspagen, 2007; Filippetti et Peyrache, 2011; Godinho et al., 2004; Lee et Lee, 2019; Viotti, 2002). Also appears that the opening of the innovation system with trade, foreign direct investment, and integration determines the improvement of developing countries’ competitive performance (Lundvall, 1992 & 2015). Thus, how can the development of the open innovation system and absorptive capacities contribute to sustainable growth improvement in Africa? Therefore, this study aims at contributing to sustainable economic growth policy reinforcement in Africa by analyzing the link between the opening of the innovation system, absorption capacities, and sustainable growth. Thus, the generalized moment’s method in the autoregressive vector panel has been used. The data was used to cover a total of 27 countries from 2007 to 2016. The results show that the Open Quintuple Helix’s innovation system and the absorptive capacities of firms and universities have a negative impact on the sustainable growth rate in Africa. Moreover, only the increase in firms' absorptive capacity really coevolves with the Open Quintuple Helix innovation system and the sustainable growth rate. Finally, an infrastructure public policy shock leads to an improvement in investments and a sustainable growth rate. However, this shock leads to a decrease in the availability of scientists and engineers, in the firm’s absorptive capacity, and the efficiency of the Open quintuple Helix innovation system in the long term. Thus in terms of sustainable growth policy, African countries must strengthen investment and especially technological innovation infrastructure development. They must also ensure that the training of scientists and engineers and the firm’s absorptive capacity can be turned towards the acquisition and development of clean or green technological innovation.

What drives wastewater reuse policy adoption and reinvention? A policy diffusion analysis

ABSTRACT. Reuse of water has the potential to alleviate water shortages, reduce wastewater treatment costs, reduce water system energy consumption, and recycle nutrients. Many states have their own guidelines for water reuse. These guidelines include but are not limited to: guidelines for treating and reusing water, design and operation of reuse facilities, and water rights. States make their regulations within nationwide guidelines. Some states allow surface irrigation of food crops where there is no contact between edible portions and reused water, and some states do not allow food crops irrigation. Wastewater reuse has great potential to contribute to sustainably managing water to combat water scarcity and even help support a transition to a circular economy. Studies to date have not investigated the characteristics that influence the type of guidelines adopted and reinvented. In this study, we address this gap. We hypothesize (1) that physical characteristics such as drought in the region, population density, and the importance of the sector the reused water is being applied in have a large impact on water reuse policy adoption. We also hypothesize that (2) political and social characteristics such as governance ideology, and geographical neighbor status have a significant impact on the adoption of the guidelines. And, (3), we hypothesize that circular economy initiatives increase the adoption of wastewater reuse policies. We test these hypotheses in the US based on US EPA data. The findings explain the diffusion of wastewater policy and the relations between states. Moreover, this study enriches our understanding of environmental policy diffusion.

A guiding framework for University-Industry partnerships to strengthen innovation and technology transfer ecosystem…a case study from India

ABSTRACT. The socioeconomic advancement of any country is largely dependent on the science, technology, and innovation ecosystem of the country. Scientific and technological advancements are crucial in bringing technological and economic progress. This knowledge from the universities should flow to the industry for knowledge translation to address current challenges faced by society and also for bringing societal upgradation. The universities are the knowledge generators, and the industry is the knowledge consumers. There is a need to build strong university-industry linkages to enhance the STI ecosystem of the country. Although India has shown a significant rise in innovation capabilities by attaining the 40th global innovation ranking out of 136 economies worldwide, top innovation ranks amongst the low-middle-income countries and central and south Asian countries (Global Innovation Index Report, 2021-22). Despite that, India seriously faces challenges in university-industry linkages (at present, India stands at 41st global positioning in this parameter of GII, 2020-2021). Therefore there is a need to bridge gaps between universities and industry.

One of the key impediments in university-industry linkages is the lack of a facilitation platform and channels for translating university knowledge to the industry as industrial expectations are not met, and the innovation dies in the valley of death as per the innovation life cycle. To cater to the problem mentioned above, a detailed analysis of the university research in terms of Technology Readiness Levels (TRLs) has to be carried out to bridge the gap between TRLs levels. University research projects generally fall from TRL 1-4, and industry to take up the project required TRL 7-9; hence there is an immediate need to develop partnership mechanisms between industry and university to bridge this TRL gap from university to industry. The current paper focuses on collating the challenges faced by academia and the industry in terms of Science, Technology, and Innovation (STI) diffusion from university to industry and the necessary facilitation and enabling mechanisms required to bridge the same. The paper aims to bring out the university-industry linkages framework that can be explored for stimulating the university-industry linkage culture in India.

This guiding framework is developed after undertaking secondary research, and interview sessions with experts from academia (Panjab University, Chandigarh; Chitkara University, Baddi; Indian Institute of Sciences, Bangalore; Indian Institute of Technology (IIT), Delhi and Indian Institute of Chemical Technology (ICT), Mumbai ) and industry, open questionnaire-based feedback from industry and academia and undertaking various academic case studies. This framework can be practiced by universities in India to set up strong interconnects with Industry and explore university-industry linkages for innovation and technology transfer. Interview questionnaires and case studies were used as primary instruments, along with desk research for collating the information to understand the challenges faced by universities and industries in collaborating. Further, the best practice framework outline is proposed that will play a significant role in connecting the university and industry.

A Guiding framework to develop University-Industry Linkages is developed by 6 step process that comprises of following steps: • Step 1 Mapping of potential academic institutes and industry needs and challenges that will open the scope for collaboration between university and industry: Potential academic institutes in India were mapped for Expertise available (Faculty/scientific staff, trained researchers); Research Infrastructure (Facilities and instruments available); Intellectual Property (filed and granted); Technologies/products developed; Technologies available for licensing/commercialization and R&D status (R&D projects with TRL levels, etc.). On the other hand, the industry perspective was mapped by sessions of interviews with select Industrial associations (regional and national); Industrial clusters (regional and national; sector-wise); government recognized Industrial R&D units, and Industry (big corporations, MSMEs, start-ups, etc.). • Step 2 Develop Communicating Channels between Industry and Academia (I-A Enablers) Individually: To develop the communicating channels following source points can be explored by the university as Alumni association; Dean/Departmental head; Research and Consultancy Cell; Technology Transfer/Commercialization/Licensing Cell, with Legal/IP Cell; Corporate Laboratories; Centers of Excellence (as per industrial needs); I-A Clubs; Industry involvement in academic activities such as curriculum design and review, teaching and training, research advisory, etc.; Industrial Chairs and Industrial Fellowships; Entrepreneurship Cells/Clubs, Research Parks, and Technology Business Incubators (TBIs). • Step 3 Match Making (Industrial Needs/problems & Academic expertise/solutions): Dedicated team of I-A managers/officers have to be administered for the required job. Scouting of potential academic partners as per industrial challenge and requirements This can be done through Open Innovation Challenges, Industry hackathons, Networking by a Liaison officer/manager at I-A cell, One to one meetings with industry counterparts, Industrial surveys (open/closed), and Communicating science. • Step 4: Drafting MoUs and Agreements: Once the potential Industry-academia linkage is established, exclusive agreements defining the role and responsibilities of each partner, along with timelines and deliverables expected, will be drafted. A team comprising the legal adviser, IP advisers, Tech Transfer advisers, and financial advisers will be formulated to assist the I-A cell in drafting agreements. The agreement should address: Project aims and detailed scope, Risks and responsibilities to manage, Rights and remedies, Project management, IP management, Regulatory norms, Expenses and payments, Supervision, Schedules, Exit, and termination. • Step 5: Collaborative Action: The collaborative research will be undertaken with defined goals and objectives, resource allocation, and deliverables sought as per the timelines. Regular monitoring of the progress of the research work will be carried out, and evaluation in terms of the timely delivery of the project objectives by an external committee will be undertaken. • Step 6: Technology Transfer: Deliverables from the collaborative research undertaken in technology transfer or licensing will be explored. The culmination of the project with technology commercialization from academic entity to industrial entity will be laid out, and future long-term engagement plans for the same will also be discussed.

The study recommends the establishment of dedicated entities for facilitating the translational research ecosystem. The dedicated team will undertake the task of Liasoning between the university and industry. This will require and will cater to human resource and capacity-building initiatives. Each university should establish such translational research facilitation bodies. They can be named Technology Transfer Offices, Industry-academia cells, etc. Specific training in leadership and management human resources should be carried out to facilitate university-industry linkages and technology transfer agreements between the university and the industry. A dedicated platform (preferably virtual) should be set up where academic expertise can be highlighted, and industry problems/challenges can be posted. Specific mechanisms for connecting industry with academia as per their needs and priorities should also be developed and managed by a dedicated office/team. Regular display of academic knowledge through exhibitions, fairs, tech displays, and repositories should be maintained by the university portfolio. This guiding framework is developed after undertaking secondary research, interview sessions with experts from academia and industry, open questionnaire-based feedback from industry and academia, and undertaking various academic case studies. This framework can be practiced by universities in India to set up strong interconnects with Industry and explore university-industry linkages for innovation and technology transfer.

An Analysis of Revealed Comparative Advantages in Scientific and Technological Disciplines in the Anglophone Caribbean Region, 1996-2020

ABSTRACT. Background and rationale

This work aims to analyze the situation of the Anglophone Caribbean Region (ACR) scientific system. Despite being close to main research hubs such as the US and having cultural ties with others such as the UK, Netherlands, and other European countries, the ACR falls behind in research productivity compared to several regions of the world, which are closing the gap with the developed countries in specific research areas.

The ACR has specific barriers that make it a worth investigating region in terms of developing a well-suited scientific system for its development path. The ACR comprises CARICOM countries, which face similar obstacles. After the 2012 world economic recession, ACR experienced low economic growth rates. Also, an extraordinary economic downturn during the pandemic took effect, especially for those more tourism intense ACR economies. An increase in the frequency of natural disasters and historically high public debt rates are also causes of endemic fiscal issues in the region, which restrict countries' investment in areas such as science and technology. With a few exceptions in the ACR region, unemployment is pervasive throughout these countries, especially youth unemployment, which unfortunately ranks comparatively among the highest in the world.

However, the ACR region has similar advantages for developing a scientific system that fits its unique characteristics. Many initiatives were applied to foster research areas throughout several ACR countries, such as the green and blue economy, climate-resistant infrastructure, big data and digital development, geothermal energy, and several medical research investments focused on tropical viruses that can spread faster internationally. Also, despite the obvious negative consequences, the COVID-19 pandemic opened a window of opportunity for the region with the explosion of remote work. Some ACR countries (especially Barbados) took advantage of the last by attracting specialists in IT areas who work remotely for the US or Europe to their unique landscapes. While the consequences of the previous strategy are unknown for their STI system, other countries could have followed a similar approach.

These shared barriers and opportunities in the ACR open up the need to investigate which research disciplines will require more effort for capacity building and those in which the region excels. However, despite the similarities in the region, ACR countries and even specific islands within countries have their specificities. In this regard, it is essential to understand where the research capacities are located and where policy initiatives are needed for their development.

Descriptively, this research aims to answer in which research areas the ACR demonstrated an increase in its capacities during the last decades. Also, it turns necessary to understand in which countries this development occurred: this capacity building in specific research areas occurred homogeneously in ACR, or is the process concentrated in some countries? Also, it is intended to answer why capacities developed in specific research areas, and others do not.

The development of research capacities is an evolutionary process; thus, time is a key variable for this research. It would be a worthless effort to understand ACR research capacities nowadays without understanding how did evolve and interacted with the local ecosystem. For these reasons, our approach embraces quantitative and qualitative information analysis, explained in the next section.


This research describes and explains the evolution of the ACR’s scientific output in the last 25 years. First, built on bibliometric information from the Scimago database for the 1996-2020 period, this study illustrates the ACR research output compared –in relative terms- to the world. For this goal, based on the well-known revealed competitive advantage approach, the inquiry establishes the research domains where a region or a country shows a revealed comparative advantage (RCA). Nevertheless, more importantly, it shows how the regional and country-specific RCA evolved during the last 25 years. The last exercise makes it possible to identify descriptively how the research capacities raised or declined during the period of interest.

The empirical tool helps identify an RCA in a research discipline if it has a greater participation in the national scientific production than the participation of that area in the world's scientific production. Published articles and citation measurements of the scientific output are presented for 27 research areas in this research. Specifically, the RCA is defined as the quotient between the discipline's participation in the scientific output of a country and the participation of this same discipline in world scientific production. Formally,

RCA = ((P_is/P_i)/(P_s/P))

Where P_is are publications in a research area s in country i, P_i is defined as the total publications in country i, P_s is the publications in subject area s worldwide, and P is defined as the publications worldwide. This indicator was also developed for citations, focusing on both cases comparing RCA for ACR and the specific countries of the region throughout the entire 1996-2020 period. In terms of its interpretation, a revealed comparative advantage in a country's discipline is found if the RCA's value is greater than 1 in a given year. Otherwise, it does not show a competitive advantage for that specific year. Secondly, using the same database, we identify the specific research institutions that excel in research disciplines previously identified as the more productive in ACR. To go a step further in explaining the reasons for that success in such a deprived context for developing research capacities, the third goal of this research is interviewing leaders of the ACR's prominent research institutions and groups to understand their evolution. The last exercise aims to discern if the experience of prominent research institutions or groups is replicable in other ACR contexts.

Results and limitations

In our analysis, the ACR has revealed at least competitive advantages in six of 27 research areas for 1996-2020. These areas include agricultural and biological sciences, business, management, accounting, environmental science, medicine, nursing, and social sciences. However, enormous differences can be found in specific ACR countries. Heterogeneity in the ACR is the rule in terms of research capacities. Nevertheless, in some areas, it is possible to find no research capacities in the region and virtually in all ACR countries: Mathematics and Engineering and a few related disciplines.

These results raise questions for the debate regarding policy decision-making: Should the region enhance deprived research areas in all the countries? Or should they cooperate in interchanging the knowledge and professionals in the disciplines that each country revealed a competitive advantage? The experiences of successful research institutions and groups are replicable in other ACR countries? How could the ACR address its un-competitiveness in such important areas as mathematics and engineering?

The main goal of this research is to have the first long-term diagnosis of research capacities in the ACR. However, our analysis is not excent of limitations. First, the RCA as an analytical tool has its weaknesses mainly because our source of information comes from scimago database: meaning indexed databases, generating biases that minimize local non-indexed publications. Secondly, the interviews focus on successful research institutions or groups, which biases our results towards successful experiences. It is difficult or almost impossible to identify unsuccessful groups and the reasons for their decline since most of them are difficult to locate (few publications) or have already stopped operating.

Using Zero Robotics as a study case for Intersectional Antiracist Technology Framework

ABSTRACT. Background and rationale: A novel theoretical framework named “Intersectional Antiracist Technology Framework” is developed by Dr. Danielle Wood and Dr. Katlyn Turner. This framework uses Systems Architecture to explain, evaluate and design approaches to incorporate Intersectional Antiracism within the Definition, Design and Distribution lifecycle phases of technology. The framework defines technology across four levels of scale including Concept, Artifact, Complex Product System and Complex Sociotechnical System. The framework seeks to demonstrate methods at each phase in the lifecycle of technology to employ an intersectional antiracist mindset and act towards promoting equity.

In this work, Zero Robotics (ZR) is used as a study case in STEM outreach activities to implement the Intersectional Antiracist Technology Framework. The framework provides a method to evaluate how the Zero Robotics program applies antiracist principles to the program design and execution. New insights and visions to the program design and distribution phases are being explored using the framework. Zero Robotics, an education outreach program led by Prof Wood at MIT, is the first U.S. space robotics competition since 2009. The program aims to engage young students from secondary schools in computer science and space technologies and prepare them for the future STEM workforce. Over 20,000 students and 4,500 educators across 15 countries have participated in the program.

Every year, there is a game challenge being designed for students to gain hands-on experience in engineering and coding. During the program, students learn about the fundamentals of science and robotics and practice their coding skills in the online simulation called the ZR Integrated Development Environment (IDE). The final competition is livestreamed from the International Space Station (ISS) and students interact with the astronauts and watch their code running in space. The program uses the space free flyer robots aboard the ISS called Astrobee, and it seeks a social impact in promoting STEM education and education equality. The ZR program lies at the intersection of Complex Product System and Complex Sociotechnical System scales in the framework which means that both technical and social factors should be considered when asking how the program can be designed with sensitivity to the identity of the program beneficiaries. Continuous development of the program is being made to broaden the participation of underrepresented groups and overcome the technical challenges from the transition to a new robot system.

Methods: New collaborations with Minority Serving Institutions, as defined by the U.S. Department of Education, Navajo Technical University (NTU) and California State University Long Beach (CSULB), are established to promote the participation of Hispanic and Indigenous communities. MIT, the founder of the ZR program, is committed to provide training to college students at NTU and CSULB on the IDE and ZR online simulation tools. NTU and CSULB serve as regional hubs for communities nearby and are responsible for outreach activities and local community support. Based on the principles in the Antiracist Technology Framework, this approach seeks to apply culturally sensitive implementation to ZR customized to the concerns of urban Hispanic students and rural Indigenous students. MIT learns from the experts at NTU and CSULB who are familiar with each local context.

In summer 2022, the first ZR Middle School (MS) competition using the new Astrobee system was held successfully after the transition to a new generation of space robots. It is also the first time for NTU and CSULB students to mentor and support middle school students in the space robotics program. The game challenge is an imagined story that motivates the code activities called “The Great Astro-Spelling-Bee”.

This tournament is a 5-week program and was conducted from June 27th to July 29th, 2022, with a final event live streamed from the ISS on August 3rd. Field days and local workshops are conducted in CSULB and MIT with campus tours and laboratory visits. The program experienced the design and distribution lifecycle phases as defined in the Intersectional Antiracist Technology Framework. Pre-program and post-program surveys from students and educators in the program are collected to evaluate the participation, design, and operation of the program. The method with respect to Indigenous data sovereignty to collect data from Native American students is in development.

Results: The presentation presents a graphical summary of the Zero Robotics program using the Systems Architecture Framework with an emphasis on Antiracist Technology Design. The graphical summary highlights the Context, Stakeholders, Objectives, Functions and Forms of the Zero Robotics Programs and shows how these system elements interact with identity-based features. By using the Systems Architecture framework, the MIT team leading Zero Robotics can identify gaps that need to be improved to increase the cultural relevance of Zero Robotics for the Hispanic and Native American communities. This includes providing language-relevant materials and asking survey questions in a manner that fits community culture. Responses to these findings are ongoing.

20 teams with 178 middle school students participated in the 2022 MS program, and 6 US states (California, Massachusetts, Illinois, Minnesota, Arizona, and New Jersey) and 3 Tribal Nations (Navajo, Hopi and Zuni) are involved. 40 of the middle school students are from Long Beach, Los Angeles, and Paramount Unified Districts. This improvement in diversity and inclusion is made from community outreach conducted by CSULB. Sixty-seven adults have supported the program operations in the summer, including 37 educators and 30 college students.

The pre-program survey with responses from 124 students shows that 60.5% of them in the program are male and 32.3.% are female. Moreover, 52.9% are Asian, 14.7% are Hispanic or Latino, 10% are Black or African American, 9.4% are White or Caucasian (using identity labels aligned with US Census conventions). The fact that females are underrepresented in the program is consistent with observation from the U.S. Census Bureau 2019 American Community Survey (ACS). In the survey, women only constituted 34% of the STEM workforce while 52% of the non-STEM workforce. Moreover, 91% of students expressed that they plan to go to college and concentrate in STEM majors, and 70.2% of students said they participate in the ZR program because of the programming aspect it offers. This implies that the majority of the students who come to the program already have some background in STEM fields.

The post-program survey to students has 31 responses. 77.4% of the students think the 2022 game is interesting, 71% expressed their desire to return next year and 77% would recommend ZR to their friends. The post-program survey to educators has 13 responses. On average, educators ranked ZR 8.7 out of 10 in terms of the impacts on students’ summer learning. 100% of educators thought the game was interesting and fun. 61.5% of educators would run ZR again and 76.9% would recommend it to their colleagues. These results give positive signals on the program impacts on students’ learning experience and the game challenge designs.

Significance: The “Intersectional Antiracist Technology Framework” incorporates aspects from science and technology studies, critical theory, design research, and systems engineering. Zero Robotics, as an impactful STEM education outreach program, is used as a study case to define and evaluate the framework. This framework is also used to explore solutions in the design and distribution lifecycle phases to improve social justice in education. Historical discriminations and inequalities in socioeconomic resources create barriers for certain groups to be engaged in STEM activities. These barriers shall be minimized to involve minority groups and equal opportunities should be provided to promote future workforce diversity. The outcome of this work is an evaluation of the current phase of the ZR program and an analysis of defining and applying the Intersectional Antiracist Technology Framework.

Policy innovations spawned by the advance of rooftop solar systems

ABSTRACT. The emergence of affordable smart meters, solar panels, electric vehicle charging, and stationary storage systems, has spawned a new generation of policy innovations. These breakthrough technologies have motivated the creation of new policies, such as electricity tariffs that blend energy, demand and capacity in novel ways; new rules and regulations about who can sell electricity to retail customers; buy-back rates for excess solar generation; and minimum bills and interconnection fees to help ensure coverage of fixed costs.

The challenge now is to design retail electricity rates that reflect "cost causation" and prevent inequitable cost shifts—two Bonbright principles of utility ratemaking. As solar penetration advances, for instance, there may be a shift of costs from technology participants to nonparticipants, which has raised concerns about the equity principle.

The "classic" net energy metering tariff (where excess solar generation is bought back at a retail rate) has spawned an array of alternative tariff configurations. There are now buy-all, sell-all rates, value of solar tariffs, and net billing, as well as instantaneous versus monthly netting. Each of these options reflects stakeholder aspirations for these evolving technologies and markets.

In this study, we explore the reciprocal relationship between technological advances and policy innovation using a combination of qualitative and quantitative analysis.

First, we review the new wave of state net energy metering (NEM) policies and identify the extent that retail tariffs are evolving in response to technological advances and the market penetration of rooftop solar systems. The case studies show that policy innovations have been spawned by technological advances, but that their design is also a reflection of the structure of regional electricity markets.

Second, we conduct a statistical analysis of the continental U.S. states and the District of Columbia to explore how policies with various configurations correlate with technological adoption. A review of the new wave of state net energy metering (NEM) policies and a statistical analysis of correlates of their adoption are conducted. These allow us to identify the extent that retail tariffs are evolving in response to technology advances and the market penetration of rooftop solar systems.

Our research finds that net energy metering policy encourages the adoption of residential solar photovoltaics, with varying effects depending on the policy design. Further, we document the impact of increasing levels of adoption of distributed solar on the design of a new wave of variants to the design of net energy metering policy.

A Preliminary Study of First-Generation Academic Scientists’ Career Experiences

ABSTRACT. Though first-generation status has been studied in undergraduate student populations, little attention has been paid to first-generation faculty even though 16% of individuals who earned STEM doctorates in 2020 were first-generation (NCSES, 2020). Despite facing barriers typically associated with first-generation status (e.g., less material resources, lower cultural capital), a significant number of first-generation students are successful in becoming faculty members themselves. However, the first-generation faculty literature suggests that first-generation individuals often rely on less knowledge of the career landscape. Additionally, social class advantages upper-class individuals in their job search and job search success (Fang & Saks, 2021). These findings suggest that though some first-generation faculty are successful in their career pursuit, they may have had more difficulty in applying for and finding a faculty position compared to continuing generation faculty. Therefore, this study examines differences in the career experiences of first-generation and continuing generation faculty in a sample of academic scientists. Measures of career experiences include the number of jobs they applied for, the number of interviews they received, and the factors they considered when applying for a position.

Net Zero? An assessment of the technological innovation research funding towards low-carbon transitions
PRESENTER: Abdulrafiu Abbas

ABSTRACT. Confronting climate change implies substantial and rapid acceleration into new innovative technologies that would promotes clean energy systems, zero-emission in energy systems, transport and mobility and industrial emission reduction that can reduce global emissions. This paper asked which countries have funded the most energy, transport, climate change, and industrial decarbonisation research? And why (i.e., what could have influenced the high/lower funding rate)? What are the disciplinary politics of funding? Which disciplines have received the most funding, and which countries have funded the most social science research? To what extent are funded projects interdisciplinary or trans-disciplinary? To answer these questions, this study catalogue and examines comprehensive global public research programs across some of the world's largest funders (and countries), including the most substantial funders of public research or countries responsible for the highest carbon emissions. It synthesizes data from 1990-2021 for 69 countries involving 163 research councils to examine and understand the global funding patterns on energy, transport, climate change, and industrial decarbonisation research. The study reveals the mean external cost for R&D funding is $2.3 billion, and the cumulative total amount of research funding from 16 countries that are the largest funders in the data was $2.268 billion from 97 research councils. Using percentile estimations of the cost of funded projects, the data reveals 69.4% from the United Kingdom, 8.11% European Union Commission, and 4.16% United States and others 18.9%. When we put the global context into the funding patterns, the data provides that the United Kingdom contributed about 40%, European Union Commission 27%, the United States 21%, and others 5% of the cost of the funded project. The analysis of global research funding across general areas reveals that Engineering and technology attract significant funds, with 28%, followed by social sciences and economics 27%, the Arts and humanities receive 18%, 16% was for Natural and physical sciences, and the least as shown in the data was 11% which is for Life sciences and medicine. The study discusses the results of the review, which are based on R&D funding as a game changer of deep transitions, before discussing how countries, research councils and academic institutions are being funded and how research results are creating a socio-technical change.

From Crisis to Survival: How Informal Businesses Harness Innovation to Evolve their Businesses

ABSTRACT. Introduction In South Africa, and many other African countries, the informal sector accounts for a significant proportion of economic activity. Informal businesses also localize goods and services for the communities within which they operate, providing accessibility to essential products and amenities that would otherwise be accessible only outside informal areas. Their understanding of local customer need and community dynamics often informs their pricing and payment terms. However, access resources such as funding, skills development and insurance that enable businesses to evolve and grow into sustainable micro-enterprises remains a challenge. For these reasons, informal businesses are particularly vulnerable to risk, which not only affects the owner, but employees and customers in the local community. One way in which owners respond to risk to stabilise, sustain, and even grow their businesses is through innovation. Innovation is crucial for informal businesses to compete and continue to provide cheap, flexible options for low-income households. A concern is that informal sector public policy and support programmes tends to be directed at formalisation, through business registration (Jorgenson, 2010). Our research shows that business registration is not an accurate indicator of graduation from informal to becoming a sustainable forma business. Business evolution is more accurately represented along a continuum of intermediate states rather than depicted as binary opposites of formal and informal. The argument presented in this paper is that, by supporting innovation in informal enterprises, progression along this continuum can be promoted towards building sustainable micro-enterprises. The emphasis should be on building capabilities to formalise.

Methods This study draws on a survey of innovation in informal enterprises, using methodologies adopted and adapted from informal sector research and the standard Oslo Manual, together with qualitative interviews and digital life stories of business owners in the informal sector. A total of 996 businesses were used to determine the varying degrees of informality in the informal sector, complimented by 48 qualitative interviews and thirteen digital stories. The research was conducted in a peri-rural area of the KwaZulu-Natal province in South Africa during 2019-2021. We present an analysis of events that prompt a response of innovation for informal business owners, which enable their businesses to survive or grow, and how these responses manifests in varying levels of informality over time. Importantly, the bottom-up approach adopted through the research design, which identified informal businesses by local community interviews and self-identification allowed for the inclusion of businesses that had at some point been registered as formal enterprises. The analysis adapts criteria developed by Mbaye and Gueye (2020) to determine the level of informality of informal businesses in the study. The criteria applied include registration status, business premises, number of employees, financial records, as well as a having separate bank account, and access to financial resources from formal financing institutions. The criteria were inversed for the analysis, meaning that if a business was not registered, it would be assigned as having met the criterion. Similarly, if the business did not operate from a separate premises, had no employees or financial records, etc., the criterion would be met, and the corresponding level of informality was assigned. The number of criteria met by each informal business corresponded to a level of informality ranging from 0 (formal) to 6 (totally informal). Findings and Discussion Our results show that every business in the sample met at least one criterion, meaning that none of businesses in the sample were considered completely formal. Most businesses in the sample met five criteria and were thus considered mostly informal. Using qualitative data, the study explored the relationship between innovation and levels of informality. The study found that negative events, which places the informal business under risk, acts as a catalyst for innovation as a response to mitigate or remove the risk to the business caused by the event. The nature of these events, which the study refers to as innovation events, include personal or financial difficulties such as crime, debt, illness, or the loss of a close relative. The data showed that these responses often led to business survival or growth, which causes a change in the level of informality of the business, either by increasing thereby becoming more informal, or decreasing by becoming less informal.

Approaches to support informal businesses are often directed at formalization through business registration, and support in the form of skills development or financing are often reserved for registered micro enterprises (CeSTII, 2021). In South Africa, registration of an enterprise requires an application to be submitted to the Companies and Intellectual Property Commission (CIPC), which can be submitted online or via participating banks. The cost of registration is R125,00 and registers the business with the South African tax authority, South African Revenue Service (SARS). It also requires financial reports to be submitted annually to the Commission. The cost and requirements for business registration are often unattainable for informal business owners, and programmes to facilitate registration do not consider the realities of businesses in the sector, such as those provided by the Small Enterprise Development Agency (SEDA). Policy support for informal businesses to move from informal to formal is underpinned by a binary and linear understanding of formality and informal business growth, referred to as the Vuvuzela Graduation Model. The model depicts formality in direct relation to business size. In our analysis, business size and registration status are merely two of the attributes considered in the evolution of informal microbusinesses. And the end-state we consider is whether businesses become sustainable economic entities.

An investigation of the qualitative data suggested that the nature of innovation events can be classified into the categories of financial or personal. Innovation responses ranged from actions taken to start the business, seeking additional avenues to market the business, to adding new goods or services for customers. We explored the determinants of the innovation response and found that competition, opportunity identification, social capital, having the existing skills, as well as family responsibility motivated owners to respond innovatively to crisis as enablers of business growth or to sustain its income stream. Thirteen of the seventeen innovation responses led to a change in the level of informality in the business, all of them resulting in decreased formality. The four innovation responses that did not result in a formality shift showed that business owners took action that had formality-shifting potential, such as learning new skills or adding new products or services to grow their business.

Nine of the seventeen business owners reported more than one innovation event. Of these, five business owners had innovation responses that led to a change in informality. The remaining four demonstrated potentially formality-shifting actions. The relationship between the change of informality after innovation event one and after innovation event two are not cumulative, meaning that the business does not necessarily become more formal after each innovation event. Our findings also show that potential formality-shifting actions do not necessarily shift formality after the second innovation event. The business may decrease, increase, or depict no change in formality at multiple points along the evolutionary trajectory towards complete formality. Even in the event of a business becoming totally formal, formality can shift, and decrease, depending on the outcome of the innovation response. Box 2 Mandla’s Flowers and Vegetables Conclusion Business size or registration is not an accurate indicator of growth or formality. Instead, the focus on formality as an outcome may very well be displaced. Many business owners use formal registration as a tactic for achieving contracts from public service organisations especially. An increase in the number of employees does indicate greater sustainability of the business in many cases. But this is very much sector-dependent, and the nature of employees, whether family or not, needs to be looked at to assess sustainability. Should not the long-term sustainable functioning of enterprises be the true goal?

Crisis events catalyse innovation in informal businesses. Informal sector researchers have in the past documented innovations as they perceived it. By and large these tended to focus on artifacts that were significantly enhanced or even transformed for use in informal businesses. It is only by using the concept of innovation systems that we are able to make the connection between innovation activities and the trigger events for business growth.

Purpose, progress and significance of innovative and challenging project in South Korea

ABSTRACT. Since its establishment in 1999 according to the Framework Act on Science and Technology, the Korea Institute of Science and Technology Evaluation and Planning (KISTEP) has devoted itself to improving the quality of life of the people through the development of science and technology, innovative growth, and resolution of social problems. From science and technology policy planning and future forecasting, Research & Development (R&D) budget allocation and adjustment, national R&D project research, analysis, evaluation, and performance diffusion, preliminary feasibility study in the R&D sector, and international cooperation in science and technology, it provides in-depth research results. Today, in the face of domestic and international challenges such as competition for technological supremacy, new infectious disease, and digital transformation, the role and mission of science and technology are becoming more important. As a think-tank of science and technology innovation policy, KISTEP is providing detailed support for science and technology-oriented government operation based on data.

The innovative and challenging project is being promoted from 2020 to lay the foundation for creating innovative results by shifting away from the past “Fast Follower” method to a bold and challenging “First Mover” research project. In order to preemptively solve the challenges of science and technology related to people's happiness, quality of life, and creation of future innovation-leading industries, we are discovering and promoting innovative and challenging cross-ministerial R&D projects centered on clear missions. It covers social issues directly related to people's happiness and quality of life, such as environment, safety (disaster, accident, public safety, security), aging, health care, and food, and the creation of future innovation-leading industries. Themes that can be achieved through step-by-step improvement of existing technologies are excluded as much as possible, and research topics that will solve problems challengingly and will have a ripple effect if successful are mainly explored.

For the successful operation of the project, a promotion team has been established and is being operated. The name of the promotion team is KARPA (Korea Advanced Research Program Accelerator).

There are a total of five detailed criteria for selecting a research topic. “Clarity of Objectives” examines whether the problem to be solved and the objectives of the project are clear and specific. "Challenge" examines whether the business goal is to be world-first or world-class. “Innovativeness” examines whether disruptive innovation can be derived through R&D. "Differentiation" is reviewed to see if it does not overlap with the government's programs that have already been promoted or are currently being promoted. "Ripple Effect" determines whether a large scientific, technological, industrial, and social ripple effect is expected if R&D succeeds. Among the five detailed criteria, "Ripple Effect" is evaluated as the most important.

Research themes are being discovered through various methods. First, in the case of demand-based research, we are conducting a demand survey for new projects for related ministries and submission of proposals for innovative and challenging research themes for researchers from industry, academia, and research institutes. In particular, for industry-university-research researchers, the CIA (Crazy Idea Accelerator) online forum has been established, where researchers freely present ideas and experts evaluate and develop them, constantly discovering research themes. In the case of top-down, top-down research planning is being carried out based on key scientific and technological issues that need to be resolved nationally. We are pushing ahead with top-down planning to avoid step-by-step improvement of technology and to discover high-impact research themes candidates that can solve national problems and create future innovation-leading industries. By comprehensively analyzing reports related to domestic and foreign future strategies and promising technologies, promising research themes are derived, and blank areas that are missing from existing R&D projects are derived. In the case of the excellent researcher base, candidates for research themes are discovered by recruiting researchers with excellent ideas and insights. In addition, we are jointly discovering new research themes through collaboration with major research management organizations that are leading the planning of new projects.

When a topic is selected, a planning report is prepared through a detailed planning process over several months. The best industry-academic-research experts in Korea participate in the planning process. Detailed planning of the project is carried out so that the R&D project can proceed in the most efficient and effective way.

When the project implementation is confirmed through the government budget deliberation process, a project management team is launched for efficient project management, planning and evaluation are carried out centering on the head of the project management team (= Project Manager, PM). Flexible research methods are applied as much as possible to the selection of the research team necessary to achieve the project goals, the execution of the research, the execution of the research funds, and the evaluation.

After the inauguration of the project management team, the progress of detailed tasks are monitored, and feedback and follow-up measures are implemented so that the project can be operated as originally intended. A PM sharing meeting was held with the participation of the PM of each project management team to share the research direction, themes, progress, and to discuss and to adjust differentiation and related matters.

Institutionally, we are preparing a R&D support system that can maximize the creativity and autonomy of researchers by applying a research environment that does not blame failure and a flexible research method through this project. We plan to explore a total of 20 projects over 4 years, reflecting the government's R&D budget and implementing it. Topics discovered so far include about 10 convergence and original research, such as stratospheric drones for constant disaster monitoring, hypertube technology development for high-speed transportation, and CAR-T source technology development for solid cancer treatment. The topics discovered in 2020 and 2021 have been officially launched after planning and budget deliberation, and R&D is being actively carried out.

The current innovative and challenging project is being operated as a pilot project, and the promotion system needs to be improved in order to maximize operational efficiency and performance. It is necessary to prepare an operational plan for the second innovative and challenging project, such as securing separate financial resources for innovation challenge-type R&D and establishing a separate institution for efficient operation and management.

In this presentation, the purpose, progress, and significance of the innovative and challenging project carried out to create a more challenging R&D culture in South Korea will be analyzed. We would like to introduce the contents of the R&D program of the innovative and challenging project discovered over the past three years, and suggest the direction of future operation, including policy implications and system improvement measures revealed in the process.

Changes and characteristics of science and technology policy due to COVID-19 -Focusing on the case of Korean government R&D investment-

ABSTRACT. Science and technology have played a leading role in preparing for a new era at each stage of Korea's economic and social development for the past half century. The government-led strong technology drive policy has become a driving force for Korea's major industries such as automobiles and semiconductors.

In this study, we will look at changes through science and technology, which is a sure preparation for the huge social changes caused by COVID-19 and an uncertain future, and analyze how the direction of innovation policies across social and industrial fields is changing due to COVID-19. .

Due to COVID-19, non-face-to-face and remote culture has spread, competitiveness in the bio field has become more important, nationalism has been strengthened, and risks have become routine.

Therefore, this study analyzes how these changes have affected Korea's science and technology innovation policy and suggests a direction of policy change.

The Korean government has presented the following major policy directions to strengthen the role of science and technology in a national crisis. In order to turn a crisis into an opportunity, the R&D model centered on the private sector was spread, and the industry's ability to respond to digital transformation and strengthen its self-reliance. In addition, a crisis response system in which industry, academia and research institutes cooperate based on science and technology has been established for crisis response.

First, in order to spread the R&D model centered on the private sector, a new R&D model was introduced and spread by delegating the full power of planning and management to private experts and supporting innovative system improvement such as competitive methods and internarional evaluation.

In order to accelerate the digital transformation, infrastructure investment through the Digital New Deal and the spread of various convergence services supported the smartization of industries (automation and intelligence) and enhancement of regional innovation capabilities. It is promoting digital transformation by supporting R&D for next-generation Data-Network-AI (DNA) core technology and spreading convergence to all industries. Expanding for the application of DNA technology to overcoming limitations in each industry and solving social problems, and apply digital innovative technologies such as cloud, IoT, digital twin, and block chain to various fields such as finance, medical care, manufacturing, and automobiles to create new industries and services.

To revitalize the non-face-to-face economy, we are strengthening support for core technology development and commercialization, and strengthening support for core technology development such as XR and 5G-based immersive content technology (high-quality voice and image quality technology, personal media technology, etc.) necessary for digitization of social and economic activities. . The government is also expanding support for the creation of non-face-to-face innovative services, focusing on areas with high non-face-to-face demand, such as education, distribution, and medical care. In addition, in order to improve efficiency and stability through digitalization of public infrastructure, it is preparing to improve the productivity of public infrastructure by securing the basic technology for smartization of aviation, road, railway, and city management and port and construction.

In preparation for periodic national crises that affect national life and national security, the crisis response system was strengthened in which the science and technology circles rapidly mobilize their capabilities to predict and respond in advance, and to suggest solutions to crisis situations. In each major crisis situation, government research institutes are designated as dedicated research institutes, and based on technical monitoring method setting and information sharing, proactive monitoring and crisis response R&D functions are strengthened and rapid emergency response research promotion system was established.

We are responding to global issues such as carbon neutrality in order to strengthen R&D for solving social problems based on social problems and to contribute to improving the quality of life. In addition, in order to promote structural transformation of society and economy, it is focusing on strengthening the industrial ecosystem and supporting technological innovation.

Responding to changes in the future social structure such as a super-aged society, strengthen R&D to improve the health and enjoy a pleasant life of the elderly population, systematically predict and manage disasters, and invest in the prevention, response, and recovery of accidents at construction and industrial sites It is also expanding R&D investment to secure public safety.

Before the corona pandemic, basic research, research for industrial development, and challenging research were emphasized by era, but recently, mission-oriented research to solve problems tends to be emphasized. As economic growth is slowing and investment in R&D is stagnant, it can be seen as a tendency to more clearly emphasize the direction of R&D.

As a concept that emphasizes the problem-solving function of R&D, mission-oriented research is emerging. It is a method of deriving a specific mission from a problem, identifying the science and technology necessary to achieve the mission, and conducting necessary research. Mission-oriented research is research carried out with the resolution of very difficult problems as its mission. It tends to be pursued with the goal of solving regional or national and social problems.

Bibliometric Analysis of Nanotechnology Applications for Coronavirus Treatment from 2000-2022

ABSTRACT. Introduction Nanotechnology is considered to be one of the most promising technologies of the 21st century. With applications in nearly every scientific discipline, scientists can use nanotechnology to measure, manipulate, and manufacture at the atomic, molecular, and supramolecular levels. Over the past 20 years, there has been major investment in global R&D funding for nanotechnologies directed towards the improvement of health diagnostics, drug delivery, nano-biopharmaceuticals, and vaccine development. In a race to handle the COVID-19 pandemic, many scientists turned to nanomedicine innovations to develop novel drugs. Thus, it is critical to understand the evolving role that nanomedical innovations have played in combating COVID-19 and other coronaviruses over the past 2 decades.

Rationale In our study, we conduct a bibliometric analysis of nanotechnology and coronavirus research in order to understand how these technologies have evolved over since 2000. Bibliometric analysis is a research method for exploring and analyzing large volumes of text scientific data, enabling scholars to decipher and map connections in the data. Given the volume of scientific literature produced over the past two decades in the fields of coronaviruses and nanotechnologies, we believe that a bibliometric analysis of coronaviruses and nanotechnology offers a comprehensive and efficient approach to analyzing data.

In the winter/spring of 2020, COVID-19, swept across the world, killing millions of people and shuttering almost every institution, business, school, and organization. Scientists immediately began finding cures and vaccines for this deadly virus. Soon scholars of science policy began researching trends in COVID-19 research. Over the past year, there have been many bibliometric analyses of COVID-19 and over the past 20 years, there are even more bibliometric studies of nanotechnology. However, there is limited bibliometric research on the applications of nanotechnology for combating COVID-19 and there are no bibliometric analyses of nanotechnology for coronaviruses in general, including the Severe Acute Respiratory Syndrome (SARS) and the Middle-East Respiratory Syndrome (MERS). The pool of publications we examine is a larger dataset than most bibliometric studies have examined with respect to nanotechnology and COVID-19. This study fills that gap by showing trends in nano-coronavirus research since 2000.

We have three hypotheses: 1) From 2000-2018, there would be a low volume of scientific publications that discuss nanotechnology and coronaviruses (nano-coronavirus research). 2) In 2020, we expect the amount of nanotechnology research on coronaviruses to spiked after the COVID-19 pandemic. 3) In response to the SARS and MERS outbreaks, we expect the global scientific production of nano-coronavirus publications to temporarily increase.

Methods This study uses PubMed as the source for nano-coronavirus articles. PubMed is a publicly accessible database of medical-related research. PubMed is hosted by the National Institutes of Health and contains over 34 million citations and abstracts. PubMed is a common database for bibliometric analyses related to medical fields.

Our first step was developing a search query to find nanotechnology and coronavirus articles. The team conducted an extensive literature review of nanotechnology and coronavirus bibliometrics articles to find keywords. After collecting the most relevant keywords, we tested the accuracy of each keyword to ensure it had the necessary recall and precision. Once we developed the keywords, the team downloaded the articles from Pubmed on July 18, 2022. We found 3,446 nano-coronavirus articles from January 2000- July 2022. The raw data was then imported to Biblioshiny Software to perform bibliometric analysis on publication trends. We analyzed the annual scientific production of nano-coronavirus research, the corresponding author’s country of origin, the multiple-country publication ratio, and the analysis of keywords within publication abstracts. The raw data generated by Biblioshiny was exported to Microsoft Excel through which figures were generated.

Results/Discussion Upon analysis of the bibliometric data, we found our first two hypotheses to be validated. Between 2000-2018 there was a low volume of nano-coronavirus scientific publications and by 2020 the volume of scientific production skyrocketed. However, we could not find evidence to support the third hypothesis. There was no rise in nano-coronavirus scientific production after the SARS outbreak (2002-2004) nor the MERS outbreak (2012-2016). Additionally, analysis of the corresponding author’s country of origin reveals that the US, China, and India are the top three leading countries in total nano-coronavirus scientific production. Unexpected findings in total scientific production between 2001 and 2022 include the United Kingdom ranked low at 20th and Iran ranking high at 4th.

Future Research Over the next few months, we are going to compare our keyword data for nano-coronavirus literature with nano-influenza scientific literature produced over the same time interval. Our initial hypothesis is that the two domains will have different patterns in research topics and keywords; however, in contrast to nano-coronavirus research that spiked in recent years, we expect nano-influenza scientific literature to have a steady rise in production. Moreover, we expect nano-influenza research to have been produced in countries that are traditionally regarded as powerhouses in general scientific research as opposed to countries such as Iran and Saudi Arabia. The findings of our complete study may be used to help policymakers allocate funding for emerging nanotechnologies that are designed to combat coronaviruses as well as other infectious diseases. Our ultimate goal is to ensure that countries are better prepared to handle scientific R&D in response to future pandemics and public health emergencies.

Facilitating STEM doctoral graduates’ innovation in the developing world: the case of South Africa, 2000 – 2018

ABSTRACT. 1. Background and rationale As the post-apartheid era and an era of increased globalisation coincided at the end of the 20th century, South African innovation policy looked develop an inclusive knowledge economy that would move beyond the country’s isolationist, extractive past. For the past two decades, policymakers in this country have directly subsidised local universities’ production of doctoral graduates with public funds. South Africa is by no means alone in working to develop its knowledge economy through public investments doctoral graduates, with the European Union, the UK, China and Brazil making similar investments in doctorate production. In spite of this significant mobilisation of public funds, no studies to date have assessed the extent to which South African doctoral graduates in any subject have contributed toward the knowledge economy and innovation envisioned as apartheid ended. There is also a dearth of evidence on factors that might facilitate doctoral graduates’ innovation, such as mechanisms for doctoral graduates to perform arbitrage between the higher education and business sectors. Of particular interest in this study were Science, Technology, Engineering and Mathematics (STEM) doctoral graduates, given the importance in delivering innovations of comparatively high economic value attributed to these subjects in South African and international innovation policy thought. 2. Methods used in this study A sequential exploratory mixed methods design was used, comprising both a large-scale survey and semi-structured interviews. The survey component was the first and the primary component executed, with the qualitative component following it to provide more complete and complementary nuance to the analysis of the survey data. The quantitative component of this study drew on data produced from a large-scale survey conducted on the population of STEM doctoral students that graduated from a South African university between 2000–2018. The questionnaire employed in this survey collected data on graduates’ employment history, career-related experiences, educational background, demographic attributes and innovation involvement after the doctorate. The final data set of valid respondents used for this study consisted of 2 225 respondents. In this study, a descriptive profile is provided of this data set’s respondents in terms of a number of variables related to their:

• Doctorate-granting university; • Outputs produced from their doctorates; • Individual characteristics, including demographic characteristics; • Employing enterprise after graduation

Cross-tabulations with chi-square tests and phi symmetric measures of strength were conducted to test this study’s hypotheses, which tested the relationships between different variables under the above four categories and their self-reported innovation involvement from the survey questionnaire. The qualitative component of this study followed the quantitative component, using 30 semi-structured interviews to ascertain the contexts in- and mechanisms by which the aforementioned quantitative relationships held. 3. Results Approximately one-third (34%) of South African STEM doctoral graduates reported no involvement at all in innovation, as defined as involvement in design, development or implementation of new or improved products, services and entrepreneurial ventures. The remaining two thirds were divided between little or moderate involvement (35%) and active or very active involvement (32%). Bivariate analysis revealed that business-sector financing and employment during the doctorate were facilitative of greater innovation involvement, and qualitative data suggests that this is a product of increased proximity between academic producers and business-sector users of innovative knowledge and technology. Qualitative data further pointed to the role that funding could play in overcoming the lack of cultural proximity between business-sector practitioners in certain industries and doctoral graduates in certain demographic groups. Bivariate analysis around outputs produced from the doctorate provided further insights. Graduates who produced patents, plant breeders’ rights and designs from their doctorates reported higher innovation involvement than those who produced other outputs, such as software or peer-reviewed journal articles. Qualitative data indicated that outputs created from doctoral research are highly embryonic, theoretical and not suitable for direct use by the business sector, particularly in South Africa’s relatively mature mining sector. Software produced from the doctorate is characterised as scaling poorly and being substandard by software engineering norms outside of academia. In cases where doctoral graduates chose to transfer their expertise through formal employment trajectories, rather than outputs such as patents, managerial positions emerged as very innovation-facilitative positions in which graduates could coordinate across business and higher education sector boundaries to shape the strategy of their employing business enterprises. In terms of the individual graduate, male graduates were proportionally more likely to report a greater degree of innovation involvement, relative to female graduates. Qualitative data pointed to the challenges faced by female graduates, with the most intensive years of childrearing often coinciding with doctoral studies and the early career after graduation. Beyond childrearing, workspaces have also presented female graduates with unaccommodating environments and workplace incivility, notably in the South African mining industry, which had historically banned women from working on mining sites until the latter half of the 1990s. Doctoral graduates who were older at graduation also reported a greater degree of innovation involvement than those younger at graduation, and qualitative data suggests that this is due to greater work experience and knowledge of practitioner needs for innovative solutions on the part of older graduates, relative to younger graduates. As for the enterprise that doctoral graduates pursue employment with after graduation, bivariate analysis revealed that employment in the business sector was facilitative of a greater degree of innovation involvement, relative to employment in the higher education and government sectors. Qualitative data reveals the relatively faster pace of innovation in terms of moving from ideas to viable products as a characteristic of the business sector, compared to the slow pace of publication in the higher education sector. However, the relative insulation of the South African higher education sector and freedom to pursue risky and novel research is cited as a valuable feature, given the economic and political shocks that have characterised the country’s government and business sectors.   4. Significance of this study This study is the first to account for the innovation brought about by STEM doctoral graduates from South African universities, spanning the years since the beginning of the twenty-first century, in addition to focusing on the variables that facilitate their innovation outcomes. This study elucidates the importance of proximity between doctoral graduates and industries with the capacity to evaluate and utilise the knowledge and technology they create. It also highlights the role that funding arrangements during the doctorate may play in bridging historic shortfalls in proximity between doctoral students and business-sector practitioners. This study provides funders, universities and industrial policymakers a comprehensive basis of evidence with which to optimise innovation and knowledge economy development outcomes specific to doctoral graduates in STEM. This study is of significance to South Africa as well as other NSIs who are faced with imperatives to innovate beyond extractive, resource-intensive economies, to shape more inclusive knowledge economies with greater participation across different demographics, or both.

Analysis of the Impact of Sovereign Wealth Funds on Defense Policy : The Case of South Korea’s Cluster Munition Industry

ABSTRACT. This study explores the unique case on how sovereign wealth funds (SWF) impact defense policies. Lately, SWFs have emerged as a critical actor in defense policy, as they are breaking the strong links between the government and the defense industry. SWFs directly affect defense companies, whose industries have mainly been determined by the requirements of their own military. As a result, SWFs are changing the business ecosystem and thereby influencing future domestic defense policies. This study examines the decision of Hanwha, the largest defense industry of South Korea to separate the unit producing cluster munition due to the ESG considerations of European SWFs, which provides an interesting case for the research on increasing influence of non-state actors on defense innovation policy.

“Cluster munition” is considered inhumane because it can attack large area targets and have a high rate of failure, which may cause civilian damage. Nevertheless, it is difficult to ignore their technical effects in terms of war capability. Under the security situation of South Korea, which is to confront North Korea, cluster munition is an important asset in South Korea's defense policy and a high priority of their weapon system acquisition program.

In this situation, the European SWFs have excluded investments in companies that violate the criteria of responsible investment. The criteria include environmental destruction, opaque management governance, and inhumane weapons manufacturers. Korea’s largest defense industry Hanwha, which is a Korean cluster munition weapon system manufacturer, has also violated this criterion. Hanwha accounts for 46 percent of Korea's defense industry's revenue. As SWF's investment in Hanwha as a whole continued to be excluded because Hanwha manufactures inhumane weapons, Hanwha stopped manufacturing cluster munitions. This is because the defense sector, which accounts for only 13% of the corporation's total revenue, could not cause the group as a whole to be stigmatized. Instead, the defense sector established a separate company that could manufacture the already developed cluster munition weapons and supply them to the Korean military.

This study is based on the signaling theory of third-party intermediaries. This theory argues that a signal presented by a third party can more easily gain the trust of other actors than the signals from the stakeholders. This is because the third party has a low possibility of information distortion. The third-party actor SWF invests according to its own standard, but this behavior can impact the investment standards of other investment funds.

This study will be conducted through interviews and archive surveys. We plan to interview government and government-funded research institutes such as the Ministry of Defense, Defense Acquisition Program Administration, Agency for Defense Development, and Defense Agency for Technology and Quality. In addition, military organizations such as the Joint Chiefs of Staff and Army Headquarters, and defense industries such as Hanwha, KDI (which is separate from Hanwha), the Korea Defense Industry Association, and SWF affiliates in Korea. Hanwha's corporate disclosure and investment data will also be analyzed in time series, and the government and public military data will be examined.

Weapon systems exhibit capabilities by integrating several subsystems, and system integration is easier for companies that have large internal capabilities. Hanwha is the representative SI corporation in Korea. The withdrawal of Hanwha from Korea's cluster munition means that the point centrality of the domestic cluster munition will be dismantled in the future, making it difficult to continue research and development and mass production.

This study is expected to provide insights into how foreign actors influence domestic defense innovation policies. Existing defense policies were decided in a top-down manner among limited internal actors. In particular, it is not common to be cross-bordered by actors other than internal actors in the subsequent mass production stage of the weapon system already deployed in the field after domestic R&D was completed. We believe the results of this study can be applied as a reference point when establishing defense policies for various countries in the future.

Are researchers in national research institutes in China satisfied with the block grant funding policy ?-evidence from Fundamental Research Funds policy implementation

ABSTRACT. Block grants and project funding have been the two major forms of government funding for the science and technology field in the worldwide level. (Wang et al., 2018) Similar cases occur in China.(Aruhan et al., 2019) Ever since the science and technology reform initiated in 1985, the competitive project funding mechanism based on peer review has been the chief funding model in China. This has contributed to a huge breakthrough in national innovation capacity and development in various research fields and disciplines, yet it has also triggered a series of problematic issues, particularly for the individual researcher, i.e. over-competing causing less time in research, insufficient stable funding causing heavy workload from project application, etc. To solve these problems, the Chinese government issued a series of stable funding policies, starting with Fundamental Research Funds (FRD) to national research institutes in 2006. As one key type of block grant in China, FRD is granted to two types of research institutions. national research institutes and research universities. Recently studies have examined the efficiency and performance of block grant funding policy in the national research institutes and universities in China(Chen et al., 2018; Zhang, 2019;Fu et al., 2016; Liang et al., 2017), yet an in-depth investigation based on individual perspective, namely how individual researcher feels about the policy implementation has not been taken into consideration. Based on the two-factor theory(also known as Herzberg's motivation-hygiene theory),this study first combines the FRD policy intentions and expert opinions and establishes an FRD block grant policy implementation satisfaction evaluation indicator system. The evaluation indicator system is semi-structured, containing one part of closed questionnaire-based survey and one part of open-ended question. The closed questions consist of eight questions, three of which belong to hygienic factors and the rest five belong to motivation factors. The open-ended part consists of one question investigating the researcher individual suggestions to improve FRD policy implementation satisfaction. Next, this study hands out the questionnaire-based survey to various national research institutes in China. Based on the sample of 4,016 questionnaires received, this study then divides the respondents into two groups, namely the high-satisfaction group and the low-satisfaction groups. By adopting research methods of descriptive statistics, multiple linear regression and text analysis, this study shows the following conclusions: a. The satisfaction of FRD block grant policy implementation is positively and significantly associated with the amount of FRD funding the researchers receive. b. For the low-satisfaction group, the satisfaction of FRD block grant policy implementation is negatively and significantly associated with the pressure of receiving the FRD funding. c. The satisfaction of FRD block grant policy implementation is negatively and significantly associated with the age and professional titles of individual researchers. Further, the paper conducts the full-sample and grouped analyses on the individual suggestions for FRF policy implementation, revealing that to improve block grant policy satisfaction, researchers anticipate a larger amount of investment and an improved allocation scheme based on demands and strategic plans. Last, the study puts forward relevant policy implications based on these research findings.

Balancing the Tradeoff between Regulation and Innovation for Artificial Intelligence: An Analysis of Top-down Command and Control and Bottom-up Self-Regulatory Approaches

ABSTRACT. The rapid development of AI technologies has propelled various countries to increase their research and development capacities in this domain as part of “the AI arms race.” At the same time, the widespread utilization of AI highlights the need for regulatory interventions. Despite the difficulty of the regulatory task and uncertainty associated with AI’s impacts, several countries have started “the race to AI regulation” and have come up with unique and innovative approaches to regulating this technology. The spectrum of regulatory proposals spans from hard laws and the prohibition of certain systems to industry self-regulation based on AI ethics. The most detailed hard law on AI is currently undergoing public discussion in the EU, and regulation for recommendation algorithms is already implemented in China. Meanwhile, the governance of this technology elsewhere is mostly conducted through soft law mechanisms, which include governmental strategies and frameworks, alongside private and non-governmental sector guidelines and codes of conduct, often realized in the form of ethics-based industry self-regulation. This spurs the ongoing debate about which of the two approaches better promotes consumer welfare. While strict regulatory requirements may better protect society against the risks of AI technologies, they also tend to hinder the pace of innovation. It is unclear to policymakers and researchers which approach (strict command and control or ethical industry self-regulation) maximizes consumer welfare, and under what conditions. The conceptual difficulty in addressing this dichotomy partly stems from the lack of a common framework that incorporates both sides of the argument. In response to this gap in the literature, this paper has developed a model to address the following interrelated questions: (1) What are the advantages and disadvantages of the two regulatory approaches? (2) What institutional factors influence the outcomes of the two approaches? (3) How should governments optimally balance the tradeoff between AI innovation and consumer protection in general? To empirically ground our conception of different levels of regulatory stringency, we first examine the regulatory proposals from the EU, the UK, the US, Russia, and China. Our document analysis shows that a more stringent approach to AI regulation is taken by China, the EU, and potentially the US (if the Algorithmic Accountability Act is adopted), whereas a more relaxed approach is taken in Russia and the UK. The proposed level of regulatory stringency depends on how much they prioritize stimulating AI innovation in the private sector. Having understood the trade-offs from the policy documents, we zero in on the regulation of AI systems that are developed by the private sector for commercial purposes. Unlike those developed by the state for national security or military purposes, the former

presents a more challenging case for regulators since they do not have direct control over the innovation, exploitation, and usage of such AI systems. We also set aside systems that are outright prohibited since the issue they present is one of legal enforcement rather than economic trade-offs. Thus, our primary interest lies in the grey area – the types of commercial exploitation that are within the legal boundaries, yet may be considered unethical once revealed to the consumers. Examples of such exploitation include the case of Cambridge Analytica, the usage of large language models for clickbait fake news generation, deepfake technologies for generating pornography, or the boosting of Amazon’s own products on its website. However, the logic behind regulating consumer-facing AI systems is intricate, not least because decisions regarding innovation, consumer protection, and frequency of usage are all decentralized among various stakeholders who pursue their own objectives. The understanding of the optimality of various regulatory approaches, therefore, calls for a systematic framework capable of analyzing the strategic interaction between various stakeholder groups. As such, we answer the proposed research questions by constructing a game-theoretical model which examines the complex incentive dynamics between innovation and consumer protection. It is important to acknowledge that our model is not designed to be explanatory for the differences in regulatory approaches chosen by different countries, because the countries’ choices could be irrational, affected by path dependence, or be the derivative of the political regimes in power. Instead, our model intends to contribute to the normative discussion on the optimal approach in regulating AI, and clarify the current academic and policy debates by pointing out how the optimal regulatory stringency is conditional on the institutional environments. The regulatory stringency chosen by the government is modelled by the probability that the exploitative practices of local AI companies are revealed to the consumers. This modeling choice is motivated by a unique challenge facing AI regulators. One key aspect of regulating AI is the difficulty of interpreting the workings of the black box systems, particularly what kinds of data are collected and what types of algorithms are used to extract valuable information by the companies. This fundamentally differs from industrial sectors where their social cost of production such as environmental pollution is relatively easily monitored and detected. In that sense, it is important that our model incorporates the possibility of revealing information to consumers, which will affect the behavior of consumers and their welfare in the end. After all, for unethical but lawful exploitation, it is consumers’ knowledge of such practices rather than top-down prohibition that acts as a disciplining device. Based on our game-theoretic analysis, we have developed an economic theory of how the welfare-maximizing level of regulatory stringency for AI depends on various institutional

parameters. Under high foreign competition, domestic innovation plays a relatively small role in serving consumers. On the other hand, consumers benefit most when they are not misled to underuse the highly competitive foreign AI systems. As a result, the prioritization of consumer protection should motivate a government to choose a high level of regulatory stringency under high foreign competition. Meanwhile, under low foreign competition (for instance, due to strong protectionist policies), the domestic AI industry can effortlessly win over local consumers from their foreign competitors. This means domestic firms can derive high marginal benefits in terms of market share from improving their algorithms. As a result, the robustness of domestic firms’ innovation incentives should motivate a government to also choose a high level of regulatory stringency under low foreign competition. Interestingly, under intermediate foreign competition, the government faces a delicate trade-off between consumer protection and innovation. Too stringent regulation stifles the innovation incentive of the domestic AI industry, whereas minimal regulation subjects the consumers to excessive exploitation. To maximize the actual consumer welfare, the government may strategically lower its regulatory stringency and turn a blind eye on some occasions. Across all institutional environments, however, minimal regulations are never compatible with maximizing actual consumer welfare. As such, the objectives of such regulatory design may be either rationalized by the prioritization of innovation, domestic producer surplus, or the perceived welfare of the consumers. In the latter, the government is primarily concerned with the image that this regulatory intervention produces without worrying too much about the actual protection of consumers – essentially using a loosely designed regulation as a PR tool. This suggests that further empirical studies should pay close attention to cases where governments are proposing very loosely defined regulations for AI.

Discovery patterns in the concepts entropy network and their impact on scientific research

ABSTRACT. The massive growth of global research activity in recent years has spurred an exponential increase in the amount of related scientific literature. Successful research in the face of the increasing complexity of modern scientific knowledge together with the diversity and depth of the studied problems requires an understanding of the related scientific landscape - the structure and interrelation of trends in science. The situation gets even more pronounced for the granting organizations on their way to developing efficient granting policies since they have to deal with many areas of science at the same time. Professional expertise remains the unique option to address the mentioned problem but it is often subject to personal opinion which, as consequence, requires a pool of experts engaged to perform single expertise. Seeking for appropriate experts willing to participate in the expertise becomes a challenging problem considering applied time limits. Therefore future scientific information management should belong to systems that will play the role of “virtual experts”, helping researchers and scientific foundations cope with the massive amount of research data and the high speed of scientific progress by facilitating manipulation, filtering, and combination of information. One of the features of such systems should be the ability to generalize scientific knowledge to identify scientific research trends and possibly even predict ones.

To satisfy mentioned requirements it is necessary to develop methods of data manipulation and processing that will accurately analyze the current state of scientific knowledge and have the potential to predict its future states as an intrinsic feature of such an analytic system. Available digital records open wide possibilities for statistical analysis of scientific documents and related metadata for topic modeling and evolution, knowledge mapping, citation indexing, etc. We have a lot of statistical tools to evaluate and measure the scientific outcome by calculating different scientometric indices. But while such measuring is efficient for evaluating the current scientific progress it is an open question of how to model future scientific discovery.

Recent studies by Wang at. al. use historical data on paper citations to predict later citations based on early patterns. It was shown that citations accrue to articles published in scientific journals over time according to a well-behaved log-normal distribution, with a rise in citations at the point of publication followed by gradual decay. The proposed model can be used to predict an article’s success and its development may lead to the methods of effective detection of promising research directions. But even in this approach selected articles needed a human expert to analyze their content to extract promising new hypotheses.

We propose to combine mentioned approach with a full-text analysis of scientific documents to identify promising research directions. The central element in our approach is a field-specific ontology based on a conceptualization of scientific knowledge. We use data collected by Science-WISE (SW) platform ( about frequencies of scientific concepts - the unique phrases that comprise one or more words and reflect certain basic notions of the scientific ontology. Investigating the dynamics of the entropic proximity measure of concepts we plan to find common patterns in the corresponding ontology graph at the moments preceding the appearance of new concepts as indicators for possible discovery.

The SW platform appeared as a result of a collaboration between physicists and computer scientists from EPFL and CERN. The SW uses modern methods of information retrieval, text analysis, and statistical data analysis to process large corpora of research publications and offers semantic recommendation systems, semantic bookmarking capabilities, and paper annotation. It is essential that the SW system excludes subjective statements about scientific ontology and provides evaluation and reflection made by the scientific community. In this study, we use an SW collection of concepts of high energy physics, derived with unique algorithms by processing and parsing the texts of scientific documents from ArXiv ( preprint server.

Technically, scientific concepts are the keywords and a few of them are usually added to the article metadata to describe the main directions of the study. The collection of concepts commonly related to a particular scientific area forms the specific scientific ontology used to encode scientific knowledge in the texts of the documents. Information about the occurrence of the concepts in a particular text may identify the topic of the article. Statistics on the usage of concepts in various documents can be used to calculate the proximity between concepts. To evaluate such proximity we propose to calculate the amount of mutual information between concepts. For the collection of concepts, their mutual proximity is stored as a symmetric matrix of normalized distances between each pair of concepts. Scaling of the matrix on 2D or 3D plane can produce the so-called entropic map of concepts. By calculating the matrix for a particular time interval and then gradually increasing it with a certain time step we can investigate the dynamics of the underlying complex network of concepts. We explain the algorithm in more details in our previous study, where we use information about the dynamics of a such network of concepts to identify the hot topic trends.

The collection of concepts that can be related to a certain topic is changed over time. New concepts may be introduced by the scientific society to identify a new method, solution, or discovery. If the idea behind the concept is successful, later it can be used in many scientific documents on the corresponding topic so the amount of its associated Shannon entropy will grow. This also will lead to stronger connections with other concepts and eventually, this new concept may become a center of a new topic.

Detecting the patterns in the concepts network before and after the moment the new concept was introduced we plan to identify the necessary conditions preceding the creation of the concept. As a possible features to identify a successful concept we use the evolution of its Shannon entropy and citation history of the document(s) where it was introduced first.

We anticipate that detected patterns in the concepts network associated with successful new concepts can be used to predict future discovery. The detailed analysis of the parameters of the concepts network at a time preceding the moment of discovery may serve as a source of information for future prediction mechanisms that would augment scientific ability, increase productivity, and multiply returns from science for society.

The Impacts of Network Multiplexity on Successful Scientific Productivity: The Case of US STEM Faculty

ABSTRACT. This study focuses on the impacts of multiplex social networks on successful science productivity measured as funded research grants. While previous studies have demonstrated the diverse impacts of collaboration networks on science productivity by looking at the size, diversity, closeness etc. (e.g., Gaughan et al., 2018; Melkers & Kiopa, 2010; Newman, 2001; Siciliano et al., 2018; van Holm et al., 2021), few have looked at the impacts of social ties beyond publications. A social network tie becomes multiplex when multiple functions of interactions are shared within a single relationship (Ferriani et al., 2013; Ibarra, 1995). We look at how ties become multiplex across research collaboration, data and material exchange, and socialization outside of work. Network multiplexity helps individuals take advantage of the diverse types of resources available in a single relationship as such ties are connected through difference exchanges and high level of trust (Coleman, 1988; Methot, 2010). At the same time, network multiplexity can harm one’s productivity as the costs of maintaining such a rich tie can be costly and time-consuming (Mayhew & Levinger, 1976; Methot et al., 2016).

In our study, we ask two questions: (1) How does network multiplexity influence successful grants of academic scientists? and (2) Does the relationship vary by gender? Understanding differential impacts of network multiplexity by gender is critical given that literature substantially demonstrates marginalization of women in receiving social network benefits.

We use a 2017 national survey on US academic scientists in three STEM fields (biology, entomology, and ecology) to explore the interactive effects of network multiplexity and gender on successful science production. Drawing from social network literature, we develop hypotheses on the reverse U-shaped relationship between network multiplexity and productivity and how the relationship varies by gender. Using linear regression model, we test theories controlling for individual characteristics, work characteristics and organizational characteristics. The results will demonstrate that 1) faculty who have more multiplex network within their social networks will be more productive, 2) the positive impact of multiplex network will be reduced after reaching a tipping point, and 3) the effects of multiplex network will be different between male and female scientists.

Our study has both theoretical and empirical contributions. First, we contribute to understanding the impact of social networks on scientific productivity by integrating literature from the fields of social network, gender equity and personnel management. Second, this study can provide implications for workplace socialization in higher education institutions. We expect our results to provide venues for STEM fields in academic institutions where they can find ways to connect faculty’s socialization and their productivity. The study will conclude with policy implications to improve faculty’s productivity by emphasizing the importance of social networks and the gender difference which connects to faculty retention.


Coleman, J. S. (1988). Social capital in the creation of human capital. American Journal of Sociology, 94.

Ferriani, S., Fonti, F., & Corrado, R. (2013). The social and economic bases of network multiplexity: Exploring the emergence of multiplex ties. Strategic Organization, 11(1), Article 1.

Gaughan, M., Melkers, J., & Welch, E. (2018). Differential Social Network Effects on Scholarly Productivity: An Intersectional Analysis. Science, Technology, & Human Values, 43(3), Article 3.

Ibarra, H. (1995). Race, Opportunity, and Diversity of Social Circles in Managerial Networks. Academy of Management Journal, 38(3), Article 3.

Mayhew, B. H., & Levinger, R. L. (1976). Size and the Density of Interaction in Human Aggregates. American Journal of Sociology, 82(1), Article 1.

Melkers, J., & Kiopa, A. (2010). The Social Capital of Global Ties in Science: The Added Value of International Collaboration. Review of Policy Research, 27(4), 389–414.

Methot, J. R. (2010). The effects of instrumental, friendship, and multiplex network ties on job performance: A model of coworker relationships. University of Florida.

Methot, J. R., LePine, J. A., Podsakoff, N. P., & Christian, J. S. (2016). Are Workplace Friendships a Mixed Blessing? Exploring Tradeoffs of Multiplex Relationships and their Associations with Job Performance. Personnel Psychology, 69(2), Article 2.

Newman, M. E. J. (2001). The structure of scientific collaboration networks. Proceedings of the National Academy of Sciences, 98(2), 404–409.

Siciliano, M. D., Welch, E. W., & Feeney, M. K. (2018). Network exploration and exploitation: Professional network churn and scientific production. Social Networks, 52, 167–179.

van Holm, E. J., Jung, H., & Welch, E. W. (2021). The impacts of foreignness and cultural distance on commercialization of patents. The Journal of Technology Transfer, 46(1), Article 1.

Implementing Research Impact Assessments: what data, methods and resources do funders need to do this well?

ABSTRACT. Background and rationale: Funder research impact assessments (RIA) are exercises which measure the research outputs, outcomes, and impacts of research projects, portfolios, or programs. They include three key components: (1) activities which funders undertake to assess impact of their investment such as reviewing publications stemming from funded research and asking researchers about their perceptions of project impacts; (2) methods used within these activities such as bibliometric analysis to capture citation data and surveys or interviews to capture data about research impacts from investigators or stakeholders; and (3) frameworks used to guide these activities such as the Payback Framework or the Canadian Academy of Health Sciences (CAHS) Framework. The field of RIA has seen remarkable growth over the past 30 years. Several RIA frameworks have been developed and applied by research funders around the world, new technologies exist to better capture research impacts automatically such as or Researchfish, and best practice guidelines have been established to provide researchers and funders with high-level advice about conducting RIAs. Recent literature reviews have found that while RIA is a valid and essential tool to measure the wider benefits of biomedical research, there are clear methodological limitations that may limit the utility of RIA exercises and their potential conclusions.

To usher in the next era of RIA and mature the field, future RIA methodologies need to become more transparent and easily implementable. Current advice for RIA best practices such as those developed by the International School of Research Impact Assessment (ISRIA) provides an important foundation of the fundamentals of impact assessment, and several literature reviews provide an overview of available RIA frameworks for evaluation. However, the current literature falls short of offering practical guidance funders can use to design a RIA, including how to select an appropriate RIA framework and related impact indicators, collect and analyze data, and reflect on post-assessment steps that can be taken to improve RIA culture within their organizations. Our aim in this poster is to understand what data funders are currently collecting when performing their own assessments, how they collect this data, and how they use it to assess their own research portfolios. We draw on a subset of literature from our larger systematic review (Abudu et al. 2022) to understand the steps that funders have taken to implement an indicator-based RIA of a project portfolio.

Methods: Within our systematic review, we identified all published studies (n=21) which funders explicitly identified an assessment or evaluation-based RIA framework to guide their RIA activities. We reviewed these studies to examine: 1) the operational steps funders took to perform their RIA; and 2) the variation in how funders implemented the same RIA frameworks. This process allowed us to explore the varying indicator-based “operational pathways” that funders employed in their analyses and spurred us to organize the methods and approaches funders listed (such as literature reviews, identifying a sample of projects, documentary review, bibliometrics, surveys, and interviews) into a logical series of steps needed to complete a successful RIA. We then applied this series of steps to the papers in our review to understand the areas where funders are succeeding in RIA implementation and where challenges and opportunities for future development lie ahead.

For each paper included in our review, we coded data on the aims of the framework used by the funder and why it was chosen for use; the data collected by the funder to assess the research portfolio; and information the funder provided about staffing needs or requirements, resources used, and lessons learned during the assessment. In addition, we extracted data on seven steps for assessment implementation: Step 1: Setting the Stage for Analysis; Step 2: Framework Selection; Step 3: Metric/Indicator Identification; Step 4: Primary Data Collection; Step 5: Data Synthesis & Analysis; Step 6: Communicating Results; and Step 7: Reflecting on Best Practices/Capacity Building. These steps drew upon existing principles from the 2018 ISRIA Statement, the Reed et al. 2021 Methodological Framework for Evaluating Research Impact, the World Bank’s Roadmap for Implementing an Impact Evaluation, and informal conversations with A. Kamenetzky, an experienced RIA implementor at the National Institute of Health Research.

Results: We found that funders are using a wide range of frameworks (n=12) to implement RIAs, and that even when the same framework is used to analyze a research portfolio, funders are using different data sources, methods, and resources to support their analysis. Approximately half (n=10) of studies in the review identified metrics and impact categories that they planned to measure during their assessments within the methods sections of their papers. Primary data collection activities were the most widely reported activities across papers included in the review, however only thirteen studies reported activities to synthesize or analyze the data they had collected. No studies reported on activities to communicate results of their RIAs beyond the included review paper or included reflections on lessons learned after preforming the RIA.

Building on the analysis from this review, seven essential steps for RIA implementation – from setting up the RIA to data collection to an end-of-assessment reflection of best practices and opportunities for capacity building – were developed to assist funders. These steps may help funders to implement their RIA more practically and rigorously. We encourage funders to: discuss the preparatory activities that they undertook to set up their analyses in greater detail within papers’ methods sections (step 1); carefully select an assessment-based RIA framework for their analysis, considering if a legacy framework is applicable to their portfolio (step 2); pre-select indicators for impact measurement and specify them within the methods (step 3); detail all data collection activities undertaken to gather new and existing data for the assessment (step 4), as well as activities undertaken to analyze and synthesize data (step 5); share planned or completed dissemination activities to share results within the papers’ discussion section (step 6); and report lessons learned during the assessment process more formally to support knowledge-exchange of best practices across funders (step 7).

Significance: RIA for biomedical research can take many forms. In order to increase transparency and improve methodological quality and reporting of funder RIAs, we recommend that funders follow seven key steps for implementing a RIA. Incorporating the seven implementation steps will help to standardize the RIA evidence base and allow for much-needed future empirical research to determine optimal data sources and methods to best evaluate funders’ research impact. We hope that our work, in tandem with future research can inspire funders to feel more confident that their impact assessments are methodologically sound, meaningful to end users of their research, and providing valuable insights for their organizations.


Abudu, Rachel, Oliver, Kathryn, and Boaz, Annette (2022), 'What funders are doing to assess the impact of their investments in health and biomedical research', Health Research Policy and Systems, 20 (1).

Adam, Paula, et al. (2018), 'ISRIA statement: ten-point guidelines for an effective process of research impact assessment', Health Research Policy and Systems, 16 (1).

Gertler, Paul J., et al. (2016) Impact Evaluation in Practice, Second Edition [online text], Inter-American Development Bank and World Bank <>

Reed, M. S., et al. (2021), 'Evaluating impact from research: A methodological framework', Research Policy, 50 (4), 104147.

Are Women's Works and Claims Received with More Skepticism?

ABSTRACT. BACKGROUND There is a wide body of research on the gender gap in academia, but the gender differences in the content of citations remain underexplored. Additionally, we do not know if social cues like gender would remain relevant to the reception of scientific papers in the midst of a global public health crisis. In the pandemic, there is a higher tradeoff between speed and quality of information. Due to the high-stakes nature of findings, scientists should be cautious to embrace new information. But there is also a high pressure to form a consensus rapidly, so they must make decisions in a short period of time. Thus, this provides a rare opportunity to observe how a new scientific finding is acknowledged and legitimized in the scientific community from its very start to its end. This also provides better data coverage. Old papers are sparsely covered in bibliometric data and more recent findings take a long time to be legitimized. In contrast, the covid-19 started recently, so the data coverage is better, and its legitimization process is expedited due to the urgent nature of relevant research.

HYPOTHESES Past research suggests that stereotypes and bias will be even more impactful under highly uncertain and urgent conditions like the pandemic. The evidence has been shown in research about decision-making in the Great Recessions. Furthermore, in Status Characteristic Theory, gender forms a hierarchy whereby men have higher status than women. Gender is a diffuse status characteristic that influences evaluations of the most task. Men are advantaged as higher-status actors and women are disadvantaged as lower-status actors. There are also empirical studies that support such predictions. As female workers are perceived as less competent and less committed to their work, the legitimacy of female professionals is often questioned. In other words, female professionals are likely to experience pressure to prove their qualifications repeatedly. Gender disparities in the patterns of citations are also prevalent, and female-authored articles are systematically less central than male-authored articles in network positions. This suggests that female-authored articles tend to gain less attention in academia. For these reasons, I hypothesize that (1) female-authored articles will be received with more skepticism. I also theorize the relationship between novelty and the extent of skepticism. Because novel works are likely to be atypical, audiences will be more skeptical of novel offerings. Therefore, (2) when scientists claim that their article is novel, the article will be received with more skepticism. Additionally, as female accomplishments are often received with more skepticism, the effects will be stronger for female-authored articles than for male-authored articles. In other words, (3) the positive effect of claims of novelty on skepticism will be stronger for female-authored articles than for male-authored articles.

METHOD I use citation context data (sentences surrounding in-text citations) in Microsoft Academic Graph (MAG), an open-access database that covers bibliometric data of 225 million publications. I quantify the extent of skepticism in the reception of scientific papers and analyze the gender differences. In doing so, I rely on a text-based measure of uncertainty cues in academic writing developed by Chen et al. (2018). They manually compiled 61 uncertainty cues from academic writings and expanded this set of vocabulary into 2200 uncertainty cues using the word embedding approach. I count the number of uncertainty cues in the citation context to measure the extent of skepticism and then take a mean across the citation context within a cited paper. I construct a sample of covid-19 relevant articles. Articles should be published between December 2019 – December 2021. Articles have relevant keywords in their titles like covid or coronavirus. About 10% of papers are covered by citation context data. The final sample is 9,259 covid-19 papers. The main independent variable of this study is the gender of the author. Gender is coded by using the first names of authors. I rely on Lariviere et al. (2013) and use their gender dictionary. Another independent variable is claims of novelty. I use a novelty dictionary I developed with my collaborators in Leahey et al. (2022). It consists of 16 words that are synonyms of new and novel. This dictionary is applied to 9,259 abstracts of covid-19 papers. I control for a count of uncertainty cues used by authors because it can affect the extent of skepticism by the audience. Other control variables are the number of authors, months since 2019 December, citation count of an article, citation count of a journal, citation count of a first author, paper count of a first author, citation count of the last author, and paper count of the last author. I also generated a field variable. Life sciences, natural sciences, engineering, social sciences, and humanities. Finally, my method of analysis is Poisson regression because my dependent variable is a count variable and approximately follows the Poisson distribution.

RESULT The results suggest that female last-authored papers are received with more skepticism. Additionally, the number of claims of novelty is inversely related to the extent of skepticism in the reception of the paper. Results also indicate the moderating effect of gender and claims of novelty. As the number of claims of novelty increases, the extent of skepticism decreases, but this effect holds only for male last-authored papers and not for female last-authored papers. In other words, male last-authored articles claiming novelty enjoy the benefits of the lower extent of skepticism.

I conclude with the implications for gender inequality in academia. This study sheds light on the understudied mechanism of the gender gap in academia. In general, the results show that senior women academics who tend to be the last author in the authorship order are disadvantaged.

Clusters and the Internationalization of National Innovation Systems

ABSTRACT. The story of industrialization in East Asia is often framed as one of strong states. States which make conscious policy decisions to mobilize resources, concentrate efforts on targeted industries, and build up appropriate infrastructures as they see fit for catalyzing economic development. East Asian economic development has been characterized by export-led strategies, industrial policy, educational investments, sectoral targeting, among other things. These strategies have been deemed massive successes in East Asian economies such as South Korea, Japan, and Taiwan. The global economy has changed much since the inception of these strategies. Production and value chains are increasingly more internationalized, trade barriers have largely lowered, and technology has developed rapidly – to name a few of these changes. The state has had to navigate these changes either through choosing to continue previous policy programs and investments, or to end/change/implement new policies to address the changing global economy. These decisions are made with information from a variety of sources including knowledge of the past, knowledge from observing other state’s behavior, and predictions about the future. The narrative that still exists today is that the state has maintained power over coordinating and directing economic growth within these countries. This means that the state is the primary decision maker and policy is thus not driven by outside actors such as firms. National innovation systems (NIS) or, “the elements and relationships which interact in the production, diffusion and use of new, and economically useful, knowledge ... and are either located within or rooted inside the borders of a nation state” (Lundvall, 1992), are the bedrock for economic growth in the modern economy. NIS are often domestically focused, especially presently due to concerns about intellectual property theft, research espionage, and other vulnerabilities which could result in a loss of competitive edge in innovation. Given this, there is much discourse within governments about the internationalization of various aspects of a state’s NIS. At what points is it appropriate to cooperate with international partners? Recruit international talent? Participate in research exchanges and data sharing initiatives? The tension here is amplified due to the internationalization of science as well – as knowledge production becoming increasingly intensive and requires the input of highly specialized researchers the need to collaborate to advance knowledge has grown, but in the same context as increased concern about the vulnerabilities that accompany collaboration in high-technology sectors. This creates tensions between firms who may seek internationalization to improve their quality and quantity of innovative resources, and states who may seeks to protect a country’s innovation capacity from competition. This paper considers a puzzle which exists in the context of this development and innovation landscape. Economic clusters (also known as industrial parks or innovation districts in this context) are key components of East Asian economic development and NIS, especially in the technology sector. These clusters are often supported and directed by the state to serve the needs of local firms, entrepreneurs, and to stimulate economic activity for the purpose of export. Clusters help firms improve their competitiveness, efficiency, and ability to innovate through concentrating key components of that industry in one location and pairing them with research institutions to produce knowledge and a capable labor force. This tension outlined above is perfectly captured in the way in which economic clusters operate. Clusters very often have international presences and linkages that support the activities which occur within them – and these international linkages are critical for accessing markets, resources, and increasingly knowledge. This paper aims to understand how clusters have contributed to (pushed for?) the internationalization of a strong state’s NIS. Exploring this phenomenon will provide insight into the following: (1) how interests of important economic entities can influence the behavior of strong states, (2) how NIS can open aspects to the international community, (3) the importance of clusters as players in state’s overarching innovation strategies, and (4) the manner in which international linkages can manifest within clusters. Other questions that can be imbedded within this work include questions of how the state learns from other entities in the economic landscape, the limitations to states ceding to the influence of other entities for decisions related to NIS, and ultimately the considerations of balancing the pursuit of innovation at the risk of IP theft/research espionage.

Inequalities in knowledge production in the emerging Bioeconomy? The case of German-Brazilian scientific collaboration

ABSTRACT. Significant investments have been mobilized for supporting research, development, and bioeconomy innovations in Brazil. Germany in particular, is among Brazil’s most prominent bioeconomy collaborators, specifically with regard to tropical forests However, our first explorative empirical study on German-Brazilian scientific collaboration networks on bioeconomy indicates that scientific collaboration is perceived as unequal. In this paper, we describe the insights gained from qualitative interviews with Brazilian scientists involved in international collaboration in bioeconomy with Germany and contextualize them in research debates on unequal North-South relations. Different strands of literature such as the political economy of inequalities in knowledge production and R&I state that North-South inequalities materialize in the international division of labor of knowledge production and the scientific labor. The result of the paper is the identification of some indicators of inequality in the production and knowledge generated in the collaboration. The indicators identified allow us to critically discuss bioeconomy collaboration as a case of unequal North-South relations which tends to keep Brazilian scientific development in a (semi-)peripheral situation.

Differential Effects of Work-Family Conflicts on Academic Scientists’ Networks

ABSTRACT. Background and Rationale

A growing literature underscores that female academic scientists have faced significant work-family conflict (WFC) challenges as they often take on more family responsibilities than their male counterparts (Feeney et al., 2014; NCSES, 2021). WFC occur when there is interrole conflict in which the role pressures from the work and family domains are mutually incompatible in some respect (Greenhaus & Beutell, 1985). These conflicts go in two directions for academic scientists, including work-to-family conflicts and family-to-work conflicts (Fox et al., 2011). This study focuses on family-to-work conflicts: if academic scientists spend more time caring for family, they will allocate less time to their work and research (Fox, 2005). As such, a higher WFC will undoubtedly affect academic scientists’ career trajectories (Newton, 2013) and academic productivity (Derrick et al., 2021). This study addresses two research questions: (1) What is the relationship between WFC and the structure of networks? If so, does this differ by gender? (2) In organizations with poor work climates, are academic scientists who have higher WFC more likely to develop external network ties than internal network ties?

Generally, scientific performance is a function of visibility—scholars have a higher scientific impact when they are more visible in the scientific community (Derrick et al., 2021). Engagement with one’s professional community helps to build ties, provide access to resources, and increase visibility. According to social capital theory, social network ties are associated with different types of resources that are beneficial to researchers’ career and professional development (Lin, 2002). The existing literature has suggested that female and male academic scientists do not have equal access to these essential network resources, thus leading to differences in scientific productivity (Bozeman and Corley, 2004; Bozeman and Gaughan, 2011; Parker and Welch, 2013). Less understood is the extent to which WFC reduce this visibility, resulting in less resourceful professional networks. The nature of these networks also matters. Gaughan et al. (2017) reveal that both the size and composition (i.e. structure) of networks matter, with male researchers having more pro-productivity instrumental networks while female scientists possessing more psychological support-based advice networks that decrease their academic productivity. It should also be noted that although women usually have higher WFC than men, parenting engagement patterns have changed to some extent in recent years, with some men taking on more family responsibilities than they previously did (Derrick et al., 2021). Given that differential WFC can affect academic scientists’ networking strategy to develop either more productivity-related instrumental networks or support-based advice networks, I hypothesize that:

H1a: Academic scientists with higher WFC will have fewer instrumental network resources. H1b: Academic scientists with higher WFC will have more advice network resources.

In addition, a competitive/stressful work climate is shown as a significant predictor of WFC for both female and male academic scientists, suggesting that a more hostile work climate will lead to a higher WFC (Fox et al., 2011). This climate in turn can result in less access or inclination to engage with more localized networks, preferring networks external to this less-than-welcoming environment (Pinheiro and Melkers, 2011). Although there is no clear evidence that women always have higher WFC than men, academic scientists tend to have differential WFC affected by their work climates, which are in turn related to their network patterns. We expect that academic scientists with higher WFC would be more sensitive to the work climate issues when choosing to develop their collaboration and advice networks inside or outside their organizations, as compared to those with lower WFC. Thus, I hypothesize that the effects of work climates on network patterns would differ by WFC.

H2: Poor work climates will result in a higher proportion of external network ties for academic scientists with higher WFC.

Overall, there is a research gap in studying the relationship between academic scientists’ WFC and their network structure and patterns. It is therefore important to understand how academic scientists with differential WFC develop their networks and related resources, and how differences may exist across genders and organizations.

Data and Methods

This study uses data from the 2011 NETWISE II survey, an NSF-funded national survey of academic scientists and engineers in four STEM fields (biology, biochemistry, mathematics, and civil engineering). The survey includes 4,196 valid respondents from 487 universities and colleges in the U.S.

This study uses data from survey questions about the structure and resources of social and professional networks, scientists’ perceptions of work-family conflicts, and the work climates of their organizations. The E-I index developed by Krackhardt and Stern (1988) will then be used to measure the proportion of a scientist’s network that is outside versus inside of their institutions. Additionally, five indicators that are related to organizations (material resources, reputation, participation, fairness, and personal support) will represent the construct of work climates (Pinheiro and Melkers, 2011). Factors examined by previous research that may affect scientists’ social network ties will be controlled (race/ethnicity, citizenship—being foreign-born, professional age, academic discipline, workload, marital status, the number of dependent children, partners’ faculty status, social potency, social closeness, etc.).

Anticipated Results

As for the first research question, I anticipate that academic scientists with higher WFC will have fewer instrumental network resources and more advice network resources. Also, gender (being women) will be associated with more advice network resources if consistent with previous research. In terms of the second research question, I expect that poor work climates will result in a higher proportion of external network ties for academic scientists with higher WFC.


This study will contribute to gender equity research in the STEM academic workforce by: 1) integrating social network theory to examine the effects of WFC on female and male academic scientists’ network structures that are related to their careers and productivity; and 2) providing implications for higher education policies. Based on the previous research, WFC as well as the structure and patterns of networks are all associated with researchers’ scientific productivity. Thus, for the next step, I hope to incorporate scientific productivity based on the findings of this study. This study will also inform higher education institutions about the importance of organizational work climates for the career advancement of academic scientists who have differential WFC. Eventually, this will help higher education institutions attract, recruit, and retain employees with a more equitable work environment.


Data that will be used in this study come from the National Science Funded Project: “Netwise II: Empirical Research: Breaking through the Reputational Ceiling: Professional Networks as a Determinant of Advancement, Mobility, and Career Outcomes for Women and Minorities in STEM,” a project funded by the National Science Foundation (grant no. DRL-0910191; co-principal investigators: Julia Melkers, Eric Welch, and Monica Gaughan and Program Officer Janice Earle).

Navigating the tightrope: Understanding Shenzhen, its contemporary contexts, and its future as a center for Chinese technology and innovation

ABSTRACT. Background & rationale

Over the last four decades, the city of Shenzhen in southeastern China has enjoyed exceptional success as the country’s first special economic zone (SEZ). Despite humble beginnings, Shenzhen has emerged as the locus of China’s technological revolution, evolving a dense and idiosyncratic innovation ecosystem founded upon export-led manufacturing, labor and capital flows, industrial policy, and a distinct entrepreneurial spirit. However, while more recent literature has remained largely positive on Shenzhen’s prospects in the near term, the last several years have seen sweeping socioeconomic, political, and cultural change threaten ambitions for the city’s innovation-centered development. Within these new contemporary contexts, Shenzhen’s historical model of growth has become increasingly unsustainable. As such, on March 11th, 2021, China’s National People’s Congress endorsed the 14th Five-Year Plan of the Chinese Communist Party (CCP). The plan, which is to guide development policy for the period spanning 2021 to 2025, represents an important economic transition and a remarkable departure from the traditional instruments of Chinese development since the country first began its economic reforms. Rather than the investment-led, hyper-globalized model of growth upon which Shenzhen has relied and the country itself has staked its developmental legitimacy, the 14th Five-Year Plan paints a new vision of China’s near future: one of self-sufficient supply chains, consumption-led growth, and (perhaps most sweepingly) technological and innovative preeminence.

Therefore, by virtue of its central place with regard to China’s technology and innovation, Shenzhen will be embroiled necessarily in the sweep of this most recent Five-Year Plan. In the forty odd years since paramount leader Deng Xiaoping designated the small urban center the site of one of China’s most consequential market-oriented reforms, Shenzhen has grown beyond all imagination—indeed, in the decades after the SEZ’s establishment, even the CCP’s wildest conjectures failed to estimate the city’s meteoric ascent. Today Shenzhen is the third-largest city in China, home to some 17 million residents and counting, and is the professed nucleus of the country’s domestically cultivated technology sector. Global heavyweights such as Huawei, Tencent, and DJI headquarter there, to say nothing of other major conglomerates like Ping An Insurance and SF Express (both of which, incidentally, are looking to rebrand as technology companies). The streets bustle with entrepreneurial life, and scores of ambitious professionals flock from across the country and the wider world to make their fortune. Since its nascent days, Shenzhen has doggedly stood at the forefront of China’s reform and opening-up, and its urban character has been shaped by decades of immigration, Party taglines, and a pervasive sense of economic opportunity. Key to understanding China’s technological, innovation-driven ambition on the world stage, then, is understanding the city of Shenzhen: its history, its economy, and how the city’s contemporary contexts and the variegated trends that inform them lay the groundwork for the next stage in Shenzhen’s ongoing evolution.


The two primary methodologies utilized for this research were literature review and qualitative, semi-structured interviews. The first stage of the study consisted of a detailed assessment of existing published work, noting broader trends in the archival research, and developing a conceptual framework with which to analyze Shenzhen’s innovation ecosystem. This assessment was done to provide a holistic understanding of the city’s history and political economy, and helped ascertain Shenzhen’s position within the wider Chinese and international technology industries. However, to obtain a more granular view of Shenzhen and its people, and, moreover, to identify the cultural and societal factors behind the city’s success, the second stage of the study included semi-structured, 60- to 90-minute-long interviews with academics, experts, and industry professionals. These interviews furnished the contextual background upon which we interpreted recent events in Shenzhen, and were instrumental in characterizing the sociocultural conditions of city life. Both methodologies helped mitigate the lack of available or trustworthy data over this most recent period of Shenzhen’s development, and are commonly utilized for research of this nature.


Our findings indicate that—contrary to assumptions of smooth, steady development over the next several years—innovation in Shenzhen will enter a “tightrope era” of pronounced uncertainty as the city seeks to transition from one model of growth to another. Virtually every factor of innovation that we identified as crucial to Shenzhen’s success will stall or has already begun to stall over the next several years. Shenzhen’s previously exceptional demographics will be haunted by low birth rates, flagging migration, and soaring housing prices and cost of living. Rising costs in its manufacturing sector have made Shenzhen less cost-competitive for firms who increasingly look outside Shenzhen or even China for their manufacturing needs. The city’s proximity to Hong Kong has become a liability as a harsh government response to protests there roiled investor confidence in what was once a sure thing, and even the distinctive shanzhai networks which built Shenzhen’s early technology industry have become less relevant as large corporations and reduced market competition become the norm in the city today.

The political and cultural indicators have fared no better. The progressive policies which had historically fueled the city’s entrepreneurial culture are no more as the government under Xi Jinping has demonstrated a willingness to crack down upon the technology sector, empowering regulatory authorities and intertwining CCP cadres ever closer with designated private enterprises. The cultural backlash is apparent as well, as public dissatisfaction with lifestyles like the 996 working hour system prevalent throughout Shenzhen bubbles to the surface: in the last two years, both the tang ping and bai lan movements have proliferated online, spurred on by poor labor conditions, long hours, and the significant societal pressure to overwork in China.

Even the international business environment has become far more hostile. The European Union labeled China a “systemic rival” and froze an investment deal in 2021. India has banned hundreds of Chinese apps from the country’s digital infrastructure. In 2018, the United States ignited a trade war with China during which several of Shenzhen’s major corporations were caught squarely in the crossfire, and the Biden administration’s 2022 ban on semiconductor exports has signaled no end to the conflict in sight. All of this is occurring as the government implements draconian “zero-COVID” measures to address the ongoing pandemic, and as the country faces a sharp downturn in its property sector (triggered by the default of the China Evergrande Group, a company that is headquartered—coincidentally—in Shenzhen).


Shenzhen’s symbolic, entrepreneurial, and innovative importance to China is difficult to estimate. It stands as a testament to the country’s enormous progress and its towering ambition, and the city’s macroeconomic trajectory serves as a guidepost toward understanding trends prevalent throughout all of China. As one of the most prominent engines of Chinese technology and innovation, how it weathers what we have coined “the tightrope era” will be indicative of how the entire country fares in the near future, and informs others countries that have since adopted China’s SEZ model of development of how they should or should not implement policy. It remains to be seen whether Shenzhen can navigate the myriad of challenges it faces, or if it is destined to fail, but the city’s future will have widespread implications for innovation, technology, and urban development, regardless of result.

An Analysis of Autonomous Vehicle Deployment Forecasts: Understanding a Technology Hype Cycle

ABSTRACT. Background and Rationale In the mid-2010s, companies producing autonomous vehicles (AVs) were projecting the widespread deployment of this technology to happen very soon, which did not occur. Current company projections focus on narrow AV deployments in limited areas and circumstances. As companies have missed their deployment goals and commercialization becomes more distant, enthusiasm for AVs has waned. Some companies are narrowing their focus to lower levels of automation, and none have commercialized high automation at a large scale. Investor enthusiasm has cooled, and several companies have gone out of business. The industry recently realized that this deployment would not happen as fast as projected.

The public, and in some cases, policymakers, have yet to recognize the lack of progress in AV development after such widely reported, bullish projections last decade. The lack of technological advancement can help clarify the federal, state, and local policymaking process. While many companies have identified the lack of policy progress as a barrier to development, the policymakers at NHTSA have argued that policy will not make sense until development reaches a practical level. Georgia Tech currently has a concurrent research project assessing state policies and reports. That research can help bridge the gap on how policy has responded to the stagnation witnessed in these projections. The research indicates that states produced legislation and reports with the early AV technology hype cycle. An analysis of industry projections and their evolution can give insight into the AV technology hype cycle and guide policymakers faced with a similar hype cycle in the future.

Methods While company projections are narrow today, Georgia Tech and the Center for Automotive Research collected all initial forecasts, including projected timelines and their fulfillment. The data consists of estimates from auto manufacturers, tech companies, parts suppliers, and startups. These projections can be aggregated and cleaned to illustrate the technology and innovation hype cycle. Additionally, the underlying data can provide a deeper understanding of trends throughout the industry to which governments seek to respond. The research team at Georgia Tech will utilize the data on these early projections for quantitative analysis of how projections have scaled back and which sectors missed their forecasts. The data will likely be split into two parts: the initial stage of the AV hype cycle (2014-2019) and the most recent stage of development (2020-2022). The comparative data analysis of the two phases will also enhance the understanding of how AV deployment has changed and the proper expectations for the future. Lastly, the data can be utilized to understand which sectors have given up on the deployment of automation.

Anticipated Results The current data shows that all sectors missed most of the projections early in the hype cycle. However, it is expected that auto manufacturers, startups, and tech companies missed their forecasts more than parts suppliers. There is also an assumption that results will show projections have begun to decrease in time and milestones. It is unclear which sector has the lowest success rate in its commercial viability. Still, it is likely that the companies like Waymo and Zoox, funded by big tech companies like Google and Amazon, have the most robust success rate. The results can help inform policymakers on the imminence of AVs and how to adjust expectations to industry projections properly. A more practical understanding of AV development will allow policies to occur at a rate more in line with their development.

Analyzing the Significance of Public Values in Artificial Intelligence Patent Documents

ABSTRACT. Background and rationale

Determining patent eligibility for artificial intelligence (AI) inventions has become increasingly difficult to predict since the 2014 Alice Corporation v. CLS Bank International United States Supreme Court case. The case brought about concerns about abstract ideas being unacceptable for patent protection, yet foundational to computer software and technological inventions. At the same time, there has been both growing public debate and increased scholarship and policy dialogue on public and societal concerns associated with the rise of AI and its applications. This poster will present the results and insights gained from a systematic literature review on the societal dimensions of AI invention and innovation, focusing on the public values expressed in AI patent documents. We build on public value frameworks to conceptualize public values expressed in patents as statements regarding the potential social objectives and benefits of an invention, beyond the private value to patent holders. Although the United States Patent and Trademarking Office (USPTO) has no formal requirements for expressions of public values in patent documents, patent applications often describe the broader objectives or problems that the invention aims to address. Such public value statements inform the context for understanding the potential utility of a patent.

The United States and other countries spearheading technological innovation have begun widely developing AI, even promoting its proliferation. This is evidenced in the United States by growth of R&D investment in AI research and in the rapid expansion in recent years in AI patent applications and grants. Generally, in return for the private protection of intellectual property rights, patented inventions (and the innovations that flow from) are expected to contribute to R&D knowledge and, broadly, to the construction of social and public value. Yet, this is not straightforward, and there is much discussion about the heightened dilemmas that AI raises. For AI innovations, there have been concerns about the reliability of AI, the potential for bias across machine learning algorithms, and the loss of human agency and control. On the other hand, AI developers frequently promise that their AI inventions will address not only data security and bias concerns but also health, environmental, sustainability and other major societal challenges. In our literature review, we aim to analyze the emergence of literature that addresses the dilemmas presented by AI, considering how public value impacts associated with AI are framed and discussed, and how public values might be mapped in patent documents.


The methods used for the analysis that will be presented in the poster begin with formulating strategies for literature searches in the fields of public value theory, the public value of science, technology and innovation, the societal and public value implications of AI, and the societal and public value implications and assessment of patents. An iterative and structured search for relevant documents will be conducted in key scholarly databases (i.e., Scopus, Web of Knowledge). It is recognized that these topics represent potentially large swathes of literature. Bibliometric text mining tools (such as VantagePoint) will be used to organize, describe, and synthesize these literature landscapes. This approach will assist in identifying central and influential works and discerning diverse perspectives and schools of thought. Annotations will be developed from reading key works and the thematic literature review will highlight the recent developments in AI patent applications. A report will be produced that summarizes the findings of the literature review and contributes to the conversation about public value implications of AI and the associated concerns. From this, a visually communicative poster will be developed that details the method, the landscape of ideas, and the findings and implications. This research stands on it own as a deliverable poster but will also feed into an international project (undertaken by researchers from Georgia Tech, the University of Manchester, and the SKEMA Business School) to determine how text used in patent applications for AI technologies references public and private values. The broader project is using a machine learning method to text mine large volumes of AI patents. The project involves conceptualizing the public value frameworks that are used in patent applications and analyzing the anticipated uses, from a public values perspective, of AI inventions. The project will contribute to the existing literature surrounding AI and public values as described in the curated literature review portion of this research.

Anticipated results and significance

The research for this poster will identify, bring together, and evaluate aspects of the literature concerning patent protection, AI, and public values. After completing a thematic narrative literature review and considering how public value terminology is presented in the literature, the study will inform the algorithm training used for text mining expressions of public values in AI patent documents. This research will contribute to frameworks that evaluate ethical practices and morality for emerging technological innovations. This research will help to analyze and describe the textual evidence from patent applications and provide an opportunity to further explore anticipated societal implications for inventions and the effects of AI patents on human life. This presents an opportunity to explore positive and negative externalities associated with the increase of AI patents for healthcare use, digitalization, and other societal domains and to develop new concepts and methods that can contribute to evaluating the harm reduction, equity, and transparency aspects associated with AI patenting.

The Quest for Strategies of Social Control: from State Policy Documents for Autonomous Vehicles

ABSTRACT. Background How will state-level administrators respond to the arrival of emerging technologies, such as autonomous vehicles (AV)? With the expectancy of new technology, state actors are likely to face Collingridge’s dilemma (Collingdridge, D. 1980): at the early stage of technological development, it is difficult to forecast the social impacts of the technology, while at the later stage after the technology matured, social control is costly. In other words, state actors, who are responsible for public roads, must learn and react in a situation of great uncertainty about how the emerging AV technology will be developed and how it will affect society. In the study, we explore strategies in which state actors respond to uncertainty, learn the technology and its social possible impacts, and adapt to innovation.

Method & Data In this study, we collect two classes of policy documents 1) state laws and executive orders and 2) the reports about technical and socioeconomic aspects of autonomous vehicle technology, which are commissioned by states, both from 2012 to 2021. We use content analysis and topic modeling to analyze the contents of these policy documents. The core text of each document was extracted for modeling, and the text was preprocessed using snowball stemming. We found a peak at about 30-35 topics; 35 topics were specified in the final model.

Anticipated Results In our analysis, we take two approaches: first, we examine how states legislate for the arrival of autonomous vehicles, and second, in state reports, we explore topics states share and specific patterns in the content. We find a remarkable similarity of texts, especially in laws across states, suggesting common sources outside the states and/or institutional isomorphism. Also, we find the extent to which states track actions in other states in the reports, which might be a process leading to isomorphism. By combining two approaches, this study will provide an overview of strategies that various states are taking in anticipation of autonomous vehicles on the road.

Reference Collingridge, D. (1980). The social control of technology. St. Martin's Press

Governing academic research and publication with public sector logics: recipe for failures?

ABSTRACT. Background and rationale Academic publication is central to higher education as it serves various purposes that affect and determine the life and career of academics. Commonly known as an output of knowledge production, academic publications also play multiple administrative roles in determining individual promotion, funding, and institutional performance (Ramsden, 1994). In addition to its centrality in determining individual academics' accountability, earlier studies have also shown that, to varying degrees, academic publication contributes to knowledge governance through evaluative systems (Geuna & Martin, 2003) and rankings (Marginson, 2014). With such multiple roles, various policies and programs have been designed and established to improve publication performance, such as collaborative research initiatives (Lee & Bozeman, 2005), academic writing interventions (McGrail, Rickard and Jones, 2006), and publication incentives (Andersen & Pallesen, 2008; Xu, 2019). However, it remains unclear how massive incorporations of academic publications as output and administrative means in various policies and programs improve academic research governance, especially with the increasing influences of public sector narratives. Against this background, this study argues that consistent narratives are essential to facilitate the implementation of government policies and programs. Accordingly, this study uses regulatory mapping to uncover official narratives underpinning various laws and regulations in constructing and advancing our understanding of academic publications' roles in academic research governance. More specifically, the study aims to address two objectives: first, to identify and map legal provisions governing academic research and publication. Second, to assess the cohesion of narratives and values of corresponding legal provisions across regulatory mechanisms in understanding implementation approaches. In doing so, the study will assess Indonesia's regulatory mechanisms on higher education and research enacted between 1999 and 2019. Methods This study reviewed 30 authoritative documents governing Indonesia's higher education and research enacted between 1999 and 2019 to generate insights on how academic publication governs and, conversely, being utilized in research governance. Given the complexities and densities of Indonesia's research governance, obtaining the authoritative documents involved accessing various ministries' and agencies' databases. Therefore, the mixture of authoritative documents used in this study arguably represents the broad interests of different ministries and agencies on academic publication while also showcasing the propagation of academic publication in various government policies and programs. The review began with identifying various provisions related to research and academic publication. Keywords used in this study include academic publication, journal articles, scientific publication, and more ambiguous references such as 'research output' to identify relevant regulatory provisions. In making sense of extracted information, the study employed 'horizontal review' (Molinuevo & Saez, 2014) and 'vertical' classification of the authoritative documents. The horizontal review was used to uncover and identify different dimensions and perspectives of academic publication. Meanwhile, 'vertical' categorization was essential to analyze consistency and coherence between narratives, objectives, and provisions between 'high-level' policy and 'lower-level' technical documents (Howlett, 2009). Such an integrated approach provided this study with a comprehensive tool to unpack the dynamics of governance and implementation. Preliminary result This study identified significant proliferations of New Public Management (NPM) logic and values in various provisions governing research and academic publication, confirming public sector narratives' influences on higher education (Ferlie et al., 2006). It is also found that the Government has introduced numerous programs to enable and accelerate academic publications, such as publication incentives. However, as NPM values, like efficiency and effectiveness, arguably do not resonate with higher education traditions and values, this study discovered profound applications of control and audit in various programs, implying a strong sense of distrust. While some might justify such restrictive approaches with accountability reasonings, distrusts are deeply embedded in the administrative and bureaucratic obligations and practices associated with funding and performance assessment, thus, potentially reducing or crowd-out motivations (Frey, 1997) in conducting research and producing academic publications. Research funding Indonesia's research funding has been incrementally increasing controls on the applicants and quality demanded using track records and required outputs that are exercised through frequent revisions of funding guidelines. While applicants' track records and research outputs were arbitrarily defined for years, Guidelines Edition XII, published in 2019, requires the applicant's publication track record as a condition to apply for funding and specifies the type of publication criteria. For some schemes, these arrangements go into more detail, such as the obligation to state Hirsch-index or to publish at top quartile journals indexed by Web of Sciences or Scopus. With such increasing specificities, the funding arguably creates higher barriers to applying, especially among academics whose work is not publication oriented. Lecturers' performance assessment system The performance assessment system audits academic performance by aggregately measuring the outputs of research, teaching, and community services. However, the system sends mixed signals about academic publications. Although the system substantially rewards academic publications, the pre-2013 system required a review process to determine the 'real value' of the academic publication. While such a provisional mechanism no longer exists in the current system, the bureaucratic review of academic publications and other research outputs to assess academics' performance continues. As a result, the system arguably discriminates against academic research with excessive audit and gatekeeping on academic publications. Significance This contribution of this study is trifold. First, it aims to advance behavioural public administration research, which remains underexplored, especially in the higher education and research policy contexts. Empirically, the study also serves as an empirical contribution to the study of Indonesia's higher education, which is rarely discussed academically despite being one of the largest and most dynamic higher education systems. Lastly, the regulatory mapping performed in this study provides a foundation for further policy developments.

How sectoral differences influence STI policies in catching up by emerging countries: A systematic literature review

ABSTRACT. Background The relevance of government interventions and the catching up of emerging countries has been the subject of a significant amount of study that has been published in academic circles. However, the majority of articles concentrate on making up ground on a national level. Studies on sectoral catching up, particularly those dealing with distinct industries in different nations at various stages of development, on the other hand, are fairly sparse. Furthermore, research has focused on industrial policies, whereas just a limited number of authors have specifically addressed STI policies. With the presumption that the type of government policies strongly varies based on sectoral characteristics and learning objectives during each phase of catching up, the purpose of this study is to examine the applicability of STI policies that account for variation in stage of development and different sectors. The textile and apparel and the pharmaceutical industry are selected for investigate because of the vast differences between their respective knowledge bases: the former is supplier-dominated and the latter is science-based. The research attempts to answer the following research questions. 1. What are STI policy instruments have been adopted in each stage of sectoral catch-up in textile and apparel and pharmaceutical industry? 2. How the adoption of STI policies varied between the catch-up process of different knowledge sectors: textile and apparel and pharmaceutical industry?

Methods This study conducts a systematically review of academic literature on the STI policies for catch-up of pharmaceuticals and textile and apparel considering two primary criteria: i) papers including a discussion of the government's STI policies for sector catch-up; and ii) the records need to have information indicating the stage of catch-up at which certain measures were implemented. A search of Scopus and Google scholar databases for articles published up to May and 22 to the textile and apparel industry. By utilizing the three stages catching up model of learning objectives, i.e. operational skills & process technology, design technology for existing products, and new product development technology, those articles were examined to explores relevant STI policies that facilitate catching-up process at each phase of development for textile and apparel and pharmaceutical industry.

Findings The results show that a variety of STI policy instruments have been utilized at each step of the catching-up process in both the textile and apparel industries as well as the pharmaceutical industry. Moreover, although certain policy instruments, such as strategies for developing a public education system or policies that promote quality and standards, are relevant in each phase of the catching-up process, the goals of these policies are fundamentally distinct from one another and depend on different sectors as well as specific learning objectives associated with the stages of development. When compared the textile and apparel and the pharmaceutical industry, despite the fact that the two sectors have distinct sectoral characteristics, catching up strategies for both follow a fairly similar pattern. In the early phase of catching up, education system development was a top priority and contributed significantly to the growth of industries in latecomer nations. During this time period, R&D funding has been limited. Only when nations enter a later phase of catching up do their governments starting to explore more ways to encourage R&D investment as domestic firms have accumulated basic and intermediate R&D capabilities. These strategies may include competitive financing programs, tax incentives, venture capital, and policies that facilitate access to capital for R&D activities. In addition, in last phases of development, policies promoting linkages within the sectors become a crucial tool for fostering the development of sophisticated R&D skills among domestic players. Still, there are differences in which the textile and apparel industry and the pharmaceutical industry diverge from one another in terms of STI policy implementation. For example, intellectual property law plays a vital part in the growth of the pharmaceutical business, but it has played a lesser role in the other sector. On the other hand, one distinguishing feature of nations that have successfully made the transition of their textile and apparel industry into the later phase of catching up is government’s efforts to increase awareness about the value of new products and scientific and technological innovation as well as elevating design as a cultural and artistic symbol.

Contributions This research review research on government's initiatives to develop the textile and garment sectors, as well as the pharmaceutical industry of latecomer nations, and discusses the lessons that may be learned from those studies. The objective of this study is to investigate implementation of STI policies for catch-up purposes in a manner that takes into account differences in both the phase of development and sectors. The findings will shed light on when and how and when the government should employ policies on STI to facilitate the growth of the textile and apparel and pharmaceutical industry.

Investigating the determinants of publishing in journals indexed in the Web of Science: The Case of African Countries

ABSTRACT. Background: The research contributions of African scientists are not well presented and discussed compared to other parts of the world. The US-EU-China-Russia rivalry is generally well-covered in the literature. Yet, Africa has contributed significantly to the world’s scientific productivity (around 7% in the past two decades). In terms of scientific quality, however, no sign of extraordinary performance is found in Africa. Hence, investigating the factors affecting publication quality would reveal important information for policymakers to improve publication quality in parallel with increasing the number of articles. In other words, publishing in prestigious journals is an important factor in evaluating the scientific productivity of each individual, institution or country because governments, public authorities, and international organizations seek to maximize the outreach and benefit of their spending.

To measure publication quality, it is pretty common and a well-accepted practice for researchers to use the papers’ number of citations to evaluate their research quality. But it should be noted that authors sometimes cite papers to highlight the opposite view in the literature (Seglen, 1997). In addition, Moed (2009) argued that citations reflect the impact on the scientific community instead of research quality. The quality of the journal in which the paper is published can complement the number of citations to understand the research quality better. A universal and ready-to-access indicator for evaluating quality is the paper indexation. Among others, the Thomson Reuters’ Web of Science (WoS) database provides good coverage of quality English language journals in the fields of Science, Technologies, Engineering, and Mathematics (STEM) as well as Health Sciences. With some considerations, Chavarro et al. (2018) highlighted the quality of WoS journals by comparing two journals from the same country, discipline, and language and showing that the journal’s “universalistic characteristics such as h-index may have a large positive effect” on the inclusion in WoS.

Research Question: What increases the likelihood of publishing in WoS journals for African STEM and health scientists? Do these factors have significant interaction with gender, implying a gap between men and women?

Data: Data for the study were collected using a web-based, self-administered questionnaire, a previous version of which had been pre-tested for the 2013 worldwide survey Friesenhahn and Beaudry (2014). Following this pre-test, corrections were made, and the questionnaire was re-tested in Indonesia, Malaysia, Singapore and Thailand in a further study in 2015. It was then shortened, adapted to the African context, and tested in Zambia in early 2016. In 2016, we administered a bilingual (French and English) version of the questionnaire to individuals across all African countries that had co-authored at least one scientific article in a journal indexed by WoS in the preceding ten years. In total, 7,515 individuals responded, constituting a response rate of approximately 10%, once multiple email addresses of some individuals were excluded. When we removed individuals not working in Africa and those who did not complete the questions relevant to this study, we were left with 6,875 valid and complete cases, of which 3,752 were in Science, Technology, Engineering and Mathematics –STEM– and 1,495 were Health scientists. The other 1,611 cases are in Social Science and Humanities (SSH), which are not included in our analysis since WoS journals generally focus on STEM and Health Science. Considering missing values in some explanatory variables, our final sample contains 4,676 observations. We matched the data with the articles extracted from Leiden University’s Centre for Science and Technology Studies (CWTS – based on the Web of Science database) in-house database for the publications with at least one author with an African affiliation. To identify African scientists, all publications that had at least one African affiliation were extracted from the database and disambiguated by the Leiden scientists. The resulting database provides a unique author ID for each scientist. As databases are never perfect, we manually checked records for which we had multiple similar names or multiple email addresses for the same surname to disambiguate further. Method: The dependent variable of our model is the dummy variable of having at least one publication indexed in WoS in the three years preceding the survey (i.e. 2013–2015). Using a Logit regression on this dummy variable, we estimated the factors most affecting the likelihood of having a publication in WoS. Results: The regression results suggest some robust arguments. The more research articles (a proxy of research activities), the more amount of funds (a proxy of research support), and the more experience of international collaboration (a proxy of networking and knowledge spillover) may result in a higher likeliness of publishing in WoS. Other academic activities, measured in the number of hours, are also examined. Supervising, research, and fundraising activities positively, while admin and consulting activities negatively affect the mentioned likeliness. Regarding the researchers’ country, the ones in South Africa have more publications in WoS, indicating some institutional support there. The findings are empirically in line with the presumption that care responsibility decreases the probability of publishing in quality journals like the ones indexed in WoS. Regarding age/experience, mid-career researchers show higher publications in WoS. There may be a common perception in Africa that women are less capable researchers than their male colleagues. Still, the results show that when women are as funded as their male colleagues, they publish slightly more than them, implying women scientists use the fund more efficiently. In other words, when women have access to the same funding, there is no measurable gender gap in publishing in WoS. Another interesting finding is about the interactive effect of gender and level of career path (early-, mid-, and late-career). Considering the six groups out of the assumed interaction, women in the mid-career stage have a higher probability of publishing in WoS. This finding suggests that women “catch up” later in their careers, mainly because of their early careers’ high and unbalanced care responsibilities. Similar to this finding, Prozesky (2008) suggests that women are “slow” or “late” starters but late achievers in academia. Looking into other interactive effects, there is no difference between men and women, indicating that women are as efficient as men in producing scientific articles from available resources. In other words, there is no intrinsic difference between men and women African scientists in publishing papers.

References: CHAVARRO, D., RÀFOLS, I. & TANG, P. 2018. To what extent is inclusion in the Web of Science an indicator of journal ‘quality’? Research evaluation, 27, 106-118. FRIESENHAHN, I. & BEAUDRY, C. 2014. The global state of young scientists. Project Report and Recommendations. Berlin: Global Young Academy. MOED, H. F. 2009. New developments in the use of citation analysis in research evaluation. Archivum immunologiae et therapiae experimentalis, 57, 13-18. PROZESKY, H. 2008. A career-history analysis of gender differences in publication productivity among South African academics. Science & Technology Studies, 21, 47-67. SEGLEN, P. O. 1997. Citations and journal impact factors: questionable indicators of research quality. Allergy, 52, 1050-1056.