ATLC2023: ATLANTA CONFERENCE ON SCIENCE AND INNOVATION POLICY
PROGRAM FOR FRIDAY, MAY 26TH
Days:
previous day
all days

View: session overviewtalk overview

08:30-10:00 Session 11A: Career Roles: Trajectories
08:30
Gender bias in team formation: The case of the European Science Foundation’s grants

ABSTRACT. This paper investigates gender bias (if any) when teams are formed. Data from the European Science Foundation are used to estimate if female scientists have the same opportunities as their male colleagues to join a team when applying for funds. To assess gender bias, we construct a control group of scientists with the competencies for being invited to join the team by the project leader, the researcher in charge of forming the team for the grant application. By comparing the proportion of female scientists in the control group with the one in the observed teams, we find a gender bias against female scientists only when the project leader is a man, while we do not observe any gender bias when the project leader is a woman.

08:45
Project Roles over the Career Trajectory: The misalignment between performed labor and desired career preparation

ABSTRACT. Background and Rationale Team based project organization plays a critical role in the advancement of scientific knowledge, and the allocation of research work within teams affects the training of graduate students and postdoctoral researchers, which has implications for their ability to pursue and success in scientific careers. While a majority of new doctorate holders rate academia as their top choice for a post-PhD career (Woolston, 2019), an increasing number of recent graduates find the pathway inaccessible (Cyranoski et al., 2011; Doctorate Recipients from U.S. Universities: 2018, 2019; Schillebeeckx et al., 2013). This disconnect may be exacerbated by the division of labor on research teams where the supporting research roles that early career researchers occupy may not necessarily be valued during hiring and promotion decisions (Leahey et al., 2010; Milojević et al., 2018; Robinson-Garcia et al., 2020). While the devaluation of supporting research work is well-established, the extent to which scientists’ strategic decisions about their next career steps are affected by their ability to dedicate time to more prestigious research tasks is unknown. Do scientists allocate their time based on what they value most or what they feel will advance their career within in academe? We surveyed academic scientists in the fields of Biology and Biomedical research in the United States about their usual and desired roles on project teams, and the extent to which they see their current work as contributing to their desired career. The respondents’ perceptions of the roles that they have performed during their training period will shed light on the extent to which researcher expectations and career decisions are affected by their positioning on project teams and seniority. Understanding the distribution of labor on research teams can inform policy decisions that incentivize the types of work that will advance scientific innovation and provide opportunities for early career researchers.

Methods To prepare a list of potential survey takers, we identified a sample of peer-reviewed scientific papers in the Web of Science that were published in the period from 2018 through 2020 in Biology or Biomedical Research according to the paper disciplinary classification that was developed for the National Science Foundation. We focus on these fields due to ubiquity of team-based science in these areas and their discussion of the issues around research training in a team science environment (Alberts et al., 2014; Bourne, 2013). The final sampling frame contained 138,387 unique researchers with email addresses and affiliations in the United States. We emailed each author a link to participate in the survey and a request to forward the survey and message to collaborators who are currently or who had been graduate students or postdocs in the past two years. We collected 5,088 total responses to our survey, with 4,150 fully completed surveys. Our survey asked researchers at different career positions to rank the prominence of 14 different research-related tasks as defined by the CRediT taxonomy (Allen et al., 2019). After parsing the survey responses, we were able to calculate the proportion of researchers who rank each of the 14 CRediT tasks among their top three tasks. We clustered the roles into Lead, Direct Support, and Indirect support based on a previous classification (Xu et al., 2022), and performed a linear regression to identify which factors and project roles were associated with a higher degree of confidence for pursuing desired career, and which roles were associated with higher numbers of publications.

Results Most notably, we found that while publishing higher number of lead publications is weakly associated with greater a sense of preparedness for pursuing an academic career, there is virtually no correlation between publication counts and a sense of preparedness for pursuing a non-academic career. Nonetheless, the number of lead publications is very often a prerequisite for employment in university contexts. 85% of graduate students and postdocs report that that at least one of the top three roles they spend time performing is a lead role; in fact, the most common role for graduate students and postdocs, was “Writing the original draft” (22%), followed by the direct support roles, “Formal Analysis” (21%) and “Investigation” (18%). However, we found that the degree that early career researchers spend time performing lead roles was not associated with higher numbers of lead publications for early career authors. This counterintuitive negative result suggests that for early career researchers, spending more time on lead roles alone may not actually lead to a higher number of lead author papers. We argue that the reason for this may be that the particular lead author tasks that early career researchers perform were the most time-intensive task on a per-paper basis: Writing the original draft of the manuscript. On the other hand, as senior researchers perform lead roles, their most common lead roles were conceptualization and designing methodology, tasks that are less time-intensive on a per paper basis. For researchers targeting non-university settings, receiving credit for conceptualization is positively associated with their sense of preparedness for their intended careers.

Significance A common conception of doctoral training is that it is designed towards preparing for academic careers. The focus on scientific publications reinforces this conceptualization. Furthermore, establishing oneself as a lead author is essential for retention in academic careers (Robinson-Garcia et al., 2020). This work, however, demonstrates a misalignment between lead author labor and output. Furthermore, few doctoral students obtain academic positions. Whether training for these students is appropriate for their career needs is unclear, especially as trainees may have less autonomy to select the roles that most directly prepare them for their career. Senior researchers, policy makers, and academic institutions must take into account their students' future career trajectories (desired and real) in the distribution of labor.

References Alberts, B., Kirschner, M. W., Tilghman, S., & Varmus, H. (2014). Rescuing US biomedical research from its systemic flaws. Proceedings of the National Academy of Sciences, 111(16), 5773. https://doi.org/10.1073/pnas.1404402111 Allen, L., O’Connell, A., & Kiermer, V. (2019). How can we ensure visibility and diversity in research contributions? How the Contributor Role Taxonomy (CRediT) is helping the shift from authorship to contributorship. Learned Publishing, 32(1), 71–74. https://doi.org/10.1002/leap.1210 Bourne, H. R. (2013). A fair deal for PhD students and postdocs. ELife, 2, e01139. https://doi.org/10.7554/eLife.01139 Cyranoski, D., Gilbert, N., Ledford, H., Nayar, A., & Yahia, M. (2011). Education: The PhD factory. Nature, 472(7343), 276–279. https://doi.org/10.1038/472276a Doctorate Recipients from U.S. Universities: 2018 (No. 20–301; Special Report NSF). (2019). National Science Foundation, National Center for Science and Engineering Statistics. https://ncses.nsf.gov/pubs/nsf20301/ Leahey, E., Keith, B., & Crockett, J. (2010). Specialization and promotion in an academic discipline. Research in Social Stratification and Mobility, 28(2), 135–155. https://doi.org/10.1016/j.rssm.2009.12.001 Milojević, S., Radicchi, F., & Walsh, J. P. (2018). Changing demographics of scientific careers: The rise of the temporary workforce. Proceedings of the National Academy of Sciences, 115(50), 12616. https://doi.org/10.1073/pnas.1800478115 Robinson-Garcia, N., Costas, R., Sugimoto, C. R., Larivière, V., & Nane, G. F. (2020). Task specialization across research careers. ELife, 9, e60586. https://doi.org/10.7554/eLife.60586 Schillebeeckx, M., Maricque, B., & Lewis, C. (2013). The missing piece to changing the university culture. Nature Biotechnology, 31, 938. Woolston, C. (2019). PhDs: The tortuous truth. Nature, 575, 403–406. https://doi.org/10.1038/d41586-019-03459-7 Xu, F., Wu, L., & Evans, J. (2022). Flat teams drive scientific innovation. Proceedings of the National Academy of Sciences, 119(23), e2200927119. https://doi.org/10.1073/pnas.2200927119

09:00
Understanding Career Transitions of Applied Researchers to Universities. Evidence from Germany

ABSTRACT. Introduction This paper analyses the conditions and factors influencing non-university researchers' career transitions back into academia. In knowledge-intensive economies, research is performed in various institutional settings, in private and public organisations and covers a multitude of different purposes. Research in some key scientific research areas like artificial intelligence or big data is nowadays no longer driven by public research alone, but is intricately connected to public applied research and industrial R&D. Thus, research careers are increasingly forged in diverse knowledge production environments. However, previous scholarship on researcher mobility has focused predominantly on single-sector careers within universities. Inter-sectoral career transitions, in particular those of applied researchers from public research organisations (PROs) and industry into higher education, are still poorly understood. Given the variety of knowledge-intensive sectors where research careers are de facto built, this is a serious limitation.

Conceptual Approach In order to explain why some researchers may choose to transition back to universities, we draw on the notion of instutional proximity. Although all researchers, regardless of their domain of occupation, adhere to fundamental rules concerning the scientific method, in practice, their work is shaped by sector-specific institutional logics. Since career scripts associated with professional success differ across institutional domains, over the course of their career researchers develop distinct intellectual and social capital portfolios tailored to the respective institutional referents and optimised to enable career advancement within their sector. Opportunities and desirability of cross-domain transitions are determined by the possibility for researchers to advance their careers within the institutional referents of a different sector. Institutionally proximate sectors share formal and informal rules, goals, systems of meaning and a professional language. As proximity decreases, institutional referents begin to differ more strongly and the transaction cost of knowledge exchange and collaboration increases. Therefore, the higher the degree of proximity between domains, the more similarities there will be between expectations of successful research careers, the higher the synergies in the researchers' portfolios of skills, capacities and networks. Cross-sectoral transitions between institutionally more proximate domains will therefore have lower barriers. However, hiring decisions will also likely differ in specific situations, e.g. for early career hires vs senior hires since these types of researchers bring different kinds of value to universities.

Empirical Approach Germany offers a rich context for the study of cross-domain transitions due to the functional differentiation of its national research system. We focus on three pillars of the German system: universities, industrial applied research sector, and applied PROs. The latter are represented by Fraunhofer-Gesellschaft in our study. In academia, performance evaluation and, consequently, career advancement, depend in similar measures on scientific reputation, visibility within relevant peer community, and strong research metrics. Corporate researchers tend to work in an environment, which is functionally separated from academia and shaped by its own, dominant institutional reference system that is characterised by profit-orientation and intellectual property protection. Industrial researchers are less likely to publish research papers, apply for public funding or present their work at conferences due its proprietary nature. PROs are characterised by a degree of institutional hybridity. They are involved in producing scientific knowledge, but are also in contact with user environments. In PROs, institutional logics from various societal spheres co-exist and overlap, meaning that they may complement or contradict each other. Fraunhofer-Gesellschaft, the research organisation with 76 institutes in Germany, embodies key characteristics of an applied research PRO. Its mission is "to partner with companies to transfer original ideas into innovations" that benefit society and strengthen the economy.

Hypotheses Barriers to research career transitions into academia are high due to the idiosyncratic features of the academic career scripts. However, some parts of the academic sector look for high potentials that bring in complementary skills, which academics struggle to acquire. Therefore, in some cases cross-domain transitions into academia take place. We offer several hypotheses regarding the factors that might ingluence them. First, a researcher's publication activity is likely to matter as high-impact scientific publications are the cornerstone of scientific credibility: H1. Applied researchers who publish more and in more acclaimed journals are more likely to transition to universities. Second, there will likely be sectoral differences among transitioning researchers. Since institutional referents of applied PROs overlap with those of universities', we propose: H2a. The rates of transition of applied PRO researchers are higher than of applied industrial researchers. However, institutional proximity will be likely to have dynamic effect. Early career applied researchers should be attractive to hire for universities because of similarity in capacities. At senior career stages, capacities and networks of applied researchers become very different that those of academics regardless of the sector. Therefore: H2b. Differences in transition rates between applied PRO and industrial applied researchers decline with scientific age.

Method We use publication data from Elsevier Scopus ranging from 2010 to 2021 to identify instances of career transitions of 18,595 German authors who are affiliated either with industry or with Fraunhofer-Gesellschaft in the beginning of their career, but that later acquire university affiliations. We identify 927 individuals who have transitioned to universities. Based on this data, we create a panel dataset at the author-year level comprising a total of 30,430 observations. We split the data into a Fraunhofer sample and an industry sample. We define our dependent variable University Affiliation as a binary variable taking the value of 1 if a researcher is affiliated with a university at an author-year and 0 if no university affiliations are identified. The change from 0 to 1 measures acquired university affiliations or co-affiliations and therefore, we interpret them as instances of career transitions. We include the following independent variables in the analysis: Publication Stock; Mean Journal Impact Factor; Scientific Age. The following controls are included: co-publications between Industry researchers and university researchers; co-publications between Fraunhofer researchers and university researchers; affiliations with Max-Planck institutes and Helmholtz Centres; Female Gender. We estimate the models using probit regressions.

Results Around 8% of Fraunhofer researchers and around 3% of industry researchers transitioned to universities at some point in their careers. The three hypotheses are supported. Fraunhofer researchers transition to universities more frequently due to their greater institutional proximity and their competences presumably have a better fit with the academic system. Transitioning to universities is an easier - and hence more viable - career choice for them than for their peers in industry, especially during their early years. Later on, researchers moving to long-term contracts in Fraunhofer institutes are likely to begin to align more strongly with non-academic incentive systems and the number of transitions converges to those of industrial researchers.

Significance The lack of academic appointments of external researchers indicates missed opportunities, because recruitment of professionals from outside academia can offer tangible benefits to universities. Researchers with cross-sectoral experience are able to facilitate attitudinal alignment, knowledge exchange between domains and act as drivers for collaboration. Thanks to their external experience, they can identify original research questions, new impulses for application-driven research directions, which can lead to novel outputs characterised by both scientific excellence and societal relevance. Lowering the barriers to cross-sectoral transitions will improve the quality of knowledge production process in academia. Our findings further provide an indication of how different degrees of domain proximity influence the scale and the driving factors of cross-domain career transitions.

09:15
Academic Skill Variety Among Scientists and Engineers

ABSTRACT. Background and Rationale:

When pursuing graduate education at the Master’s degree level, students with Bachelor’s degrees in STEM fields can make the choice to return for a Master’s degree in their field of education, or pursue a Master’s degree outside of their field of study which may include a business degree. This choice is becoming more popular. In 2022, looking at the entrance into top MBA Programs on their corresponding websites by undergraduate majors showed that 41% of Harvard’s incoming class into their MBA program had a Bachelor’s degree in a STEM field. 42% of Georgia Tech’s, 45% of MIT’s, 33% of University of Pennsylvania’s, and 37% of Stanford’s MBA class had a Bachelor’s degree in a STEM field. This is becoming a common phenomenon.

Companies large and small have started requiring more diverse entrepreneurial skill sets among their engineers (Nichols & Armstrong, 2003; Rover, 2005). Mixing knowledge from the STEM disciplines with leadership, communication, and business skills can be very valuable for graduates for use in a variety of career paths (Winkler et al., 2015). Employers in engineering fields are wanting engineers with deep knowledge in their discipline who also have a broad array of cross-disciplinary knowledge (Mohd-Yusof et al., 2015).

Giving students a broad skillset helps them to be able to solve problems innovatively (Bodnar & Hixson, 2018; London et al., 2018; Wheadon & Duval-Couetil, 2016). STEM graduates need business skills to be able to take ideas from the ideation stage to actual innovation/production, whether they start their own businesses or not (Atkinson & Mayo, 2010). It has been found to be a valuable ability in engineering to be able to mix opportunity recognition skills with technical engineering skills (Bekki et al., 2018, Hixson & Paretti, 2018). Commercialization of innovation needs multidisciplinary skill bases (Boni et al., 2009).

In the sciences, multiple studies of skill disparities between education and needs of the workplace have found that graduates often lack skills in communication, project management, teamwork, problem-solving, critical thinking, and interpersonal skills (Jang, 2016; Lu et al., 1999; Radermacher & Walia, 2013; Tang et al., 2001). Verzat et al. (2009) highlighted the problem that while the STEM community is looking for “bilingual” graduates who have both technical and business capabilities, most STEM programs fall short of offering what industry is demanding (Dym et al., 2005; Eskandari et al., 2007). According to Atkinson and Mayo (2010): “Few business schools teach design. Few engineering schools teach market engagement. So, business majors lack engineering design, engineering majors lack market context, and science majors lack both.” In a study of 99,000 STEAM workers, Stenard (2021) found that STEAM workers regardless of entering wage workers or entrepreneurship reported “management or supervising people or projects” as their primary work activity. The activity of sales, purchasing and marketing was also found to be a top skill used by STEAM workers in their everyday jobs.

There is large body of research that suggests that individuals who have diverse skills and experiences are more likely to enter self-employment (Astebro & Thompson, 2011; Dobrev & Barnett, 2005; Lazear, 2004, 2005). However, there is mixed results as to how the skills are developed and whether skill diversity leads to successful self-employment or not. Lazear (2004, 2005) argues that it is a purposeful investment of skills, while Astebro and Thompson (2011) argue that those who have a taste for variety and enjoy doing many different things in their experiences often become entrepreneurs.

In a study of the literature on skill variety, Krieger et al. (2018) found that while Lazear is often cited with starting the research on skill variety using the diversity of coursework by students, only a few studies have considered academic skill variety (Cho and Orazem, 2014; Hsieh et al., 2017; Lazear, 2004; Orazem et al., 2015). A majority of the current research focuses on work-related measures of skill variety instead of academic measures. This paper answers the call from Krieger et al. (2018) for more studies on academic skill variety by studying the outcomes of those who diversify their degree field between their undergraduate and graduate degrees.

Based on prior findings that human capital and skill variety in particular impact self-employment outcomes (Stenard and Sauermann, 2016), this study looks into whether academic skill variety at the degree level impacts career outcomes, including self-employment outcomes.

Methods and Results:

This study uses data from the National Science Foundation’s (NSF) Scientists and Engineers Statistical Data System (SESTAT). The sample used includes 15,000 scientists and engineers with Master’s degrees, to examine the relationship between academic skill variety, career choices, and salary. In doing so the author distinguishes between the different reasons why people pursued their degrees in the first place to analyze whether it was a purposeful investment in skills or if it was for a taste for variety, such as a career change. The field an individual studied for their Bachelor’s degree and then the field they chose to pursue their Master’s degree in is observed. Therefore, one can see whether someone who had a technical undergraduate training switched and pursued a Master’s degree in business or if they continued to pursue more technical skills. This allows a new way to analyze academic skill diversity that has not been done in previous studies. The data also allows the analysis of variables that are usually unobservable. In addition to looking at skill diversity and particularly the types of skill variety, the data also allows an analysis of the intentions for obtaining the Master’s degree, whether it is for skill development or change in career interest reasons, etc. This allows the creation of a proxy for jack-of-all-trades via skill development intentions and taste for variety at the same time via a change in career interests. In addition, self-employed wage outcomes are compared between those who pursued additional specialized skills with those who pursued additional diverse skills.

This research finds that those who are pursuing a Master’s degree for career advancement or a change in their field are more likely to attain a Bachelor’s degree outside of their field of Bachelor’s degree education, while pursuing the degree for skill attainment is negatively and significantly related to pursuing a degree outside of the field of their Bachelor’s degree. Academic skill variety is not found to be significantly related to self-employment. Academic skill variety is positively related to salary generally for scientists and engineers, but not for those in self-employment.

This research contributes to the literature on STEM careers and human capital by providing new insights into the nature and implications of academic skill variety. This paper highlights that academic skill variety can occur for different reasons. Academic skill variety also has implications for work outcomes such as wages for workers in the STEM fields.

08:30-10:00 Session 11B: Realities of Scientific Work II
08:30
Gender and attrition in the changing nature of scientific work

ABSTRACT. Under-representation of women in science has been a longstanding phenomenon (Bayer & Astin, 1975; Etzkowitz et al., 2000; Larivière et al., 2013; Long & Fox, 1995; Preston, 2004; Zuckerman & Cole, 1975). Not only do women face more structural obstacles when entering science (Cimpian et al., 2020; Leslie et al., 2015; Reuben et al., 2014; Zuckerman & Cole, 1975), but they are also more likely to leave science and pursue non-scientific careers (Etzkowitz et al., 2000; Huang et al., 2020; Preston, 2004).

Prior studies examine gender inequality in science in terms of non-universalistic evaluation standards (Jappelli et al., 2017; Long et al., 1993; Long & Fox, 1995) or training opportunities (Moss-Racusin et al., 2012; Sheltzer & Smith, 2014; Stockard et al., 2021), family obligation (Cech & Blair-Loy, 2019; Fox et al., 2011), unequal positions in social network (Fox, 2006; Leahey, 2007; Xie & Shauman, 1998), and returns to their accomplishments (Ghiasi et al., 2015; Hofstra et al., 2020; Lerchenmueller & Sorenson, 2018; Reskin, 1976; Van der Lee & Ellemers, 2015).

Although those prior studies point out important factors that are attributed to perpetuating gender inequality in science, they often view gender inequality among scientific workforce in isolation from the changing nature of scientific work. As science is increasingly becoming a team-based, collaborative activity and the size of teams gets larger and larger (Adams et al., 2005; Wuchty et al., 2007), scientific work becomes increasingly bureaucratized, which is characterized by hierarchy, standardization, and division of labor (Pugh et al., 1968; Walsh & Lee, 2015; Weber, 1978). In particular, division of labor generates specialization in supporting roles (Hackett, 1990; Hagstrom, 1964; Walsh & Lee, 2015). At the same time, the increasing demands for productivity generate a growing need for supporting scientists whose primary role is supporting the leading scientists of projects or labs (Lee & Walsh, 2022). Therefore, the changing nature of science has been generating more and more supporting scientists whose work is critical to the production of science but who holds a more vulnerable status with unstable career paths than the leading scientists, e.g., academic principal investigators (PIs) (Lee & Walsh, 2022; Walsh & Lee, 2022). Milojević et al. (2018) show that a growing fraction of researchers spend their careers only in supporting roles, measured by those who have never been lead authors in their publications, and that these supporting scientists tend to have shorter careers than lead scientists.

This change in the composition of the scientific workforce may also exacerbate gender inequality in science. If those who have a more vulnerable status may get pushed into more vulnerable positions, female scientists may be more likely to be supporting scientists, and further have higher attribution than male scientists. Therefore, in this study, we empirically explore gender and attrition in science in terms of different roles in research collaboration, i.e., leading scientist vs. supporting scientist.

Using data on publishing careers from 1971 to 2012 from selected natural and social science fields (2,268,176 scientists in total), we find that women were historically more likely to spend their careers as supporting authors. However, we find this gap has been converging over time and recently flipped in natural sciences. Interestingly, this convergence has been overwhelmingly driven by a decreasing share of lead authors among men. Furthermore, statistical analysis using survival models shows that both women and supporting authors are more likely to leave publishing careers. However, among women, the supporting role has an especially high exit risk in social sciences, while the lead role has an especially high exit risk in natural sciences. In other words, female supporting authors are relatively worse off in social science than in natural sciences. We argue that natural science fields have provided a more rationalized and standardized career path for supporting scientists than social science, which is relatively more beneficial for the group that has traditionally a lower status, i.e., female supporting scientists.

We will further examine the primary tasks performed by male and female supporting scientists and other characteristics such as their novelty at the beginning of their careers and see how these may explain different representation and different rates of attrition. The results of this study will help us understand gender inequality in science in the context of the changing nature of science. Future research can examine how traditional drivers of gender inequality such as non-universalistic evaluation standards, training opportunities, and family obligation can explain representation and attrition in science between male and female scientists in different roles.

08:45
Leadership emergence and team performance: A network approach to analyze idea-generation discussions

ABSTRACT. 1. Background and Rationale In recent years, both industry and academia have placed greater emphasis on creativity and innovation and are actively promoting and supporting such activities. In this trend, research on leadership and teamwork to enhance creativity and innovation is burgeoning. However, research in this area is still in its infancy, both theoretically and empirically, and has yet to integrate existing fragmented knowledge into any unified framework(s) (see Hughes et al. 2018 for a detailed review). This study aims to remedy this situation by collecting unique and rich empirical data capturing teamwork and analyzing them using multiple methods, including conversation analysis, questionnaire survey, and those developed in network science. Our particular focus is on the process of leadership emergence within teams and its impact on team performance. Most empirical studies analyze the quality of leadership when the leader is already in the team (e.g., Klaic et al. 2020, Wang et al. 2013). On the other hand, the constructivist approach, which views leadership as emergent through interactions among team members, has also attracted attention (Knights & O’Leary 2006). However, which interaction patterns may lead to leadership emergence has been insufficiently investigated. In this study, we address the following challenges: 1. To experiment with idea generation by forming teams of people who have never met each other before, without pre-determining leaders or their roles 2. To systematically analyze conversations within teams to identify and compare communication patterns of leadership emergence and idea generation 3. To measure the degree of creativity/innovativeness of each team and examine its relationship to the above patterns Regarding 2), a typical method for capturing the contents of a conversation is to assign a tag(s) to each utterance that succinctly represents the semantic content (e.g., Choi & Richards 2012). However, many existing studies use tag information only for atomistic analysis – e.g., frequency of occurrence – failing to identify temporal patterns in the chains of tags. Concerning 3), due to the difficulty of measuring the quality of leadership and teamwork, the typical existing method has been through questionnaires, which can only provide qualitative, subjective evaluations. In response, we argue that our uniquely collected data and network analysis approach can solve these issues. Some of the authors are active members of the executive committee of the entrepreneurship education programs at a university in Japan, designing, implementing, and facilitating various idea-generation programs. Taking advantage of this, we analyze discussion log data collected in some of these programs that capture the business idea generation process of multiple teams. Teams are formed by committee members and consist of members who do not know each other beforehand. Teams develop and present their new business idea at the end of the program. The presentations are then scored by a panel of judges, including professional investors and entrepreneurial education experts. That is, quantitative evaluation values for the teams’ deliverables can be observed, which has the advantage of allowing us to discuss the relationship between teamwork and performance (Challenge 3)). In the analysis of conversation logs (challenge (ii)), we employ a network science approach in which conversation is represented as a network of tags and their chains. By doing so, we can apply network analysis methods to identify significant temporal patterns in the conversations. The research questions of this study are summarized as follows: ・Through what conversation patterns does leadership emerge? ・How does leadership differ from team to team? ・What is the relationship between different leadership types and team performance?

2. Data and Methods The data analyzed in this study were collected during a program held in 2021 in which Kayano city, Suwa city, and six universities in Japan collaborated on generating innovative business ideas that would lead to the revitalization of these cities. A total of 28 participants (university students and city employees) were divided into six teams and engaged in discussions over a three-day period. Team members were fixed, and no one was assigned to act as a leader. All team discussions were recorded and transcribed using the conversation analysis tool Trint (https://trint.com). The noise and errors were then manually removed/corrected. In addition, tagging of each utterance was conducted. The appropriate classification of tags depends on the content and purpose of the conversation. In this study, the set of tags was determined by referring to existing studies on idea creation and discussion (Coursey et al. 2019, Alloatti et al. 2021). Participants were asked to complete a questionnaire regarding their individual impressions of team members, such as “who do you think is a leader?” and “with whom do you find it difficult to communicate?”. The questionnaire results are used in our analysis to complement the conversation analysis. Finally, the judges’ evaluation scores on each team’s presentation (assessed based on the quality of the idea, feasibility, and whether it is worthy of investment) were collected and used to capture team performance. As mentioned above, network science methods are utilized as the conversation analysis technique. Temporal motif analysis (e.g., Paranjape 2017) is applied to the network of tags to identify distinctive patterns in the process of leadership emergence on each team.

3. Results and Anticipated Results So far, we have analyzed the differences in tag appearance patterns across teams, and the following findings were obtained. First, there was a bias among team members in the number of times they spoke. Existing research suggested that differences in speaking time lead to an advantage within a group (Mast 2002) and that those who speak up more often become leaders (MacLaren et al. 2020), but we found that this is not necessarily true. Indeed, people who speak more often tend to be perceived by others as leaders. However, our analysis of the tags revealed that those who frequently ask questions and confirm things with other members are more likely to be perceived as leaders, even if they speak less often. Second, we found that the smaller the percentage of “opinion” in the utterances of those perceived as leaders, the higher the team performance tended to be. In other words, in teams with flat initial relationships (like our cases), a style of leadership that does not exclude others from activities was well accepted by team members and contributed to team performance. We are currently analyzing temporal network motifs (=relational structures) consisting of multiple tags, to extract some distinctive patterns in the process of leadership emergence, and examining the relationship between the extracted patterns and team performance. By analyzing the data from the first day and the last day and comparing the relationship between the two, we will identify differences in team conversation patterns before and after the emergence of leadership. The findings of this study are expected to contribute to the understanding of the ideal leadership styles and the promotion of its establishment, which in turn will enhance innovation in teamwork. We also hope that through this study, we can provide new and effective methods for teamwork analysis, particularly conversation analysis.

09:00
STEM collaboration practices: how academic patent networks influence knowledge flows

ABSTRACT. Universities have an important role in fostering science and technology (S&T) and contributing to public science and economic growth through innovation, knowledge transfer, and commercialization. STEM faculty at research intensive universities are encouraged to pursue patents for their research and often utilize networks to gain the necessary resources and to transfer knowledge. The structure and composition of information and collaboration networks are important for knowledge flows, as they impact the production of novel S&T knowledge, can reduce research infrastructure costs, and increase the usefulness, influence, and flow of new ideas. A common way to measure direct network collaborations in the patent literature is to look at co-inventors on patents without identifying knowledge flows between academia and industry. Instead, this paper specifies when network collaborations include both academic and industry co-inventors across different disciplines to see how the networks affect knowledge creation and flows. By using this approach, I assess the effect of the public versus private divide and identify how the combination of the two sectors matters for knowledge flows and the advancement of public science. Using data from a national survey of university scientists and engineers matched with publicly available patent network and citation data, I test variation in patent citations by patent network structure and composition. The results have implications for network practices, knowledge flows, and university policies to encourage academic and collaborative patenting.

09:15
Towards the measurement of epistemic disagreement in science

ABSTRACT. # Background: Healthy disagreement among scientists drives the creation of new knowledge and is a necessary precursor to consensus. Yet in spite of their prominence in histories and theories of scientific progress, disagreement has received little empirical attention. A rigorous quantitative accounting of disagreement and consensus would have many far-reaching impacts, such as informing better models of science and innovation, supporting tools that improve the accessibility of the scientific literature, and contributing to new consensus-aware approaches to science policy. However the complexities of disagreement have so far made quantification difficult, instead confining its study to time- and effort-intensive historical analyses and surveys which, although rich in detail, cannot generalize across the sheer scale and heterogeneity of science.

This abstract describes results of an ongoing project that aims to address these challenges by providing a rigorous foundation for the study of disagreement in science. First, we aim to provide a measure of disagreement. Leveraging the increasingly-availability of full-text content from scientific publications, we describe a methodological approach to generate and manually-validate cue-phrases that reliably signal that an in-text citation sentence represents valid instances of disagreement. We then use this approach to quantify the extent of disagreement across more than four million publications in the Elsevier ScienceDirect database, and investigate the rate of disagreement across fields of science. Building on this initial study, we also chart a path towards a more thorough and holistic study of disagreement through the curation of an extensive dataset of exemplar scientific disagreements alongside the development of theoretically-grounded and validated indicators of disagreement.

# Methods We adopt a holistic notion of disagreement that can occur when one paper references another, characterized by three distinct types:

1. Paper-level disagreement: when one publication offers a finding or perspective that is (at least partly) incompatible with the perspective of another (even though there may be no explicit contradiction). 2. Community-level disagreement: when a citing publication, without explicitly disagreeing with a cited publication, instead mentions controversy or lack of consensus in the larger body of literature

We identify disagreement in a corpus of in-text citation sentences (citances) extracted from English-language research articles published between 1980 and 2016 and indexed in the Elsevier ScienceDirect database. A set of preliminary signal and filter terms were derived through an iterative process of manual labeling, validation, and discussion between all the authors both in-person and virtually. Signal terms were chosen such that citances containing them were likely instances of disagreement; the presence of filter terms further increased the likelihood. Queries were constructed from all signal/filter term pairs.

Two coders manually labeled 50 randomly-selected citances returned by each query as valid or invalid instances of disagreement. Our operationalization of disagreement was robust, with 85.5% agreement between coders. The validity of each query was calculated as the proportion of citances labeled as valid by both coders. Only the 23 most precise queries, with at least 80% validity, were selected to form the disagreement indicator, resulting in about 500,000 citances, the incidence of which in a field forms our novel indicator of disagreement.

# Results Disagreement accounts for 0.32% of all citances in our corpus. Examining the rate of disagreement across a high-level categorization of scientific disciplines reveals that rates reflect the “Hierarchy of Sciences” proposed by Auguste Comte. Namely, the lowest rates of disagreement can be found in the physical sciences (0.15%) and math & computer sciences (0.06%), whereas the highest rates of disagreement are in the social sciences & humanities (0.61%)—differences that likely stem from the complexity of the subjects they study and which methods are possible. This interpretation is further supported by examining more fine-grained sub-fields, which reveal that fields such as paleontology have higher rates of disagreement than other subfields in the natural sciences, likely owing to their dependence on contested and incomplete historical records rather than controlled experiments.

We also investigate how disagreement varies across a range of contextual factors. In the interest of brevity, here we provide a high-level summary of some of these findings: The rate of disagreement has steadily declined between the year 2000 and 2016 in physics & engineering, whereas rates in other fields either remain stable or only marginally change.

- Papers are more likely to issue a disagreement against a newer paper rather than an older paper. This effect is especially prominent in the social sciences & humanities. Disagreement is mostly likely to occur at the beginning of a paper, mostly likely in the introduction or literature review.

- Citation to another person’s work is 2.4 times more likely to be disagreement than is self-citation (defined as any overlap between the authors of a citing and cited paper), though this difference is lowest in the social sciences & humanities (1.6 times greater).

- We find little difference in the rates of issuing or receiving disagreement between men and women first and last authors. Receiving a disagreement citation has no notable effect on citations in the years following the disagreement. However, papers that issue a disagreement citation tend to receive more citations than papers that do not, suggesting that critically engaging with prior work is associated with higher citation impact.

- Our approach provides a strong first-step in a quantitative understanding of disagreement. There are limits, such as reliance on a non-exhaustive signal and filter terms, yet despite them, we show the approach to be robust, transparent, and act as a strong first step towards more general indicators of disagreement.

# Next Steps: This study serves as the foundation for the further development of data and indicators for the study of disagreement and consensus across science. Namely, we aim to develop the largest-ever dataset of scientific disagreements, drawing from the method listed here, alongside identifying disagreements through explicit editorial notices, such as published comments, and identified from social networks like Twitter and PubPeer. Then, we will develop a series of theoretically-validated indicators of disagreement at the level of scientific topics, evaluating each based on their ability to identify topics containing the previously-identified disagreement. These indicators will be wide-ranging, including the cue-word based approach used here, but also analyzing features of article metadata, the complexity of their textual content, and the position of the paper in the broader citation network. To showcase their applications, the best-performing indicators will be applied towards studying the causes, consequences, and trends of disagreement in key topics during the COVID-19 pandemic.

# Significance: The significance of this work can be understood across four main dimensions. First, these indicators will facilitate empirical analyses of the temporal evolution of consensus formation and provide vital insights into theories of scientific progress. They will also prove crucial to research in adjacent fields such as the public engagement with science, in which public controversies can be compared against their corresponding scientific consensus. Second, these indicators will also find applications in industry, particularly for scientific search engines and discovery tools. Third, these indicators will contribute to consensus-aware science governance, giving funding agencies, journals, and other institutions the tools to guide scientific fields towards debate or consensus. Fourth, our indicators of consensus mark an important conceptual shift in the science of science—moving beyond measures of performance and instead developing indicators of scientific certainty, a feature of knowledge itself.

08:30-10:00 Session 11C: Global & Regional
08:30
New industrial and innovation policies in the context of recent productive and innovative global transformations

ABSTRACT. 1. Introduction The paper is based on the main results of the research project called "The global dynamics of production and innovation and the role of territory and national states: challenges for the development of the Health Economic Industrial Complex in Brazil" developed under the project "Challenges for the Brazilian Public Health System (SUS) in the national and global context of social, economic and technological transformations”. The project was supported by Fiocruz (Oswaldo Cruz Foundation), one of the most important public research institutions in Brazil.

Some of the global transformations in production and innovation in the last twenty years are related to the deepening of the financialization process and the reduction of the world trade and world GDP growth rates. These trends have characterized the global economic scenario since the international crisis of 2007/08 and were exacerbated by the Covid 19 Pandemic crisis. The changes in global production and innovation dynamics have provoked significant impacts on the public policies focused on productive and innovative development of developed countries and China.

In this sense one of the most important change in the role of state since the international crisis of 2007/08 was the increase in protectionism in the industrial and innovation policies of the most developed countries. It was observed that the governments of most countries, especially of the G-20, have significantly increased the use of barriers (tariff and non-tariff) to minimize the impact of changes in the global dynamics of production and innovation and of international crises on productive structures and to protect domestic companies.

Another policy instrument increasingly used by the most advanced countries since 2016, especially since the Covid 19 pandemic, is related to the control of foreign capital in strategic economic activities related to national security. The sectors related to health (pharmaceutical products and medical equipment, for example) are increasingly subject to control of direct foreign investment entry.

While openness to foreign direct investment has not been eliminated outright, in recent years many governments have established or strengthened mechanisms to increase control of foreign direct investment (OECD, 2020). The most common explanation for doing this has been related to national security, but it can be observed that the protection of domestic firms and the strengthening of the domestic productive base have also been stimulating more developed countries to increase control of foreign capital inflows (especially foreign direct investment).

In addition to using trade policy and increasing control over the entry of foreign capital to protect local industry and domestic firms, the different governments of the developed countries have also adopted policies aimed at stimulating innovation, especially since 2016, aimed at the development and diffusion of new digital technologies (the so called Industry 4.0). In these cases, public resources are articulated with the measures to protect local industry and with other mechanisms, such as the use of the public purchasing power. Those policies are mostly restricted to locally owned companies, excluding companies controlled by foreign capital.

Another important change observed in recent years in the industrial and innovation policies of developed countries and in the strategies of multinational companies is related to attempts to stimulate local companies to return to their countries of origin important parts of their productive systems previously re-located for countries with low costs, mostly in the Southeast Asia. This process was part of the globalization of production and expansion of global value chain. Such practices, treated in the literature as "reshoring", were already a policy objective of some countries since the middle of the last decade, but received a greater impulse with the crisis of the pandemic, that demonstrated the importance of local productive and innovative capabilities in some strategic activities.

2. Objectives

The paper has three main objectives:

(i) To analyse the characteristics of industrial and innovation policies of developed countries, with respect to: increase in protectionism, increased in the use of instruments aimed at controlling the entry of foreign capital and stimulus to the generation and incorporation of technologies associated with Industry 4.0. (ii) To analyse the reshoring strategies (mechanisms aimed at the internalization of stages of the production process previously displaced to countries with lower production costs) adopted by multinational companies and governments of some developed countries from the mid-2010s on strengthened after the crisis of the Covid 19 pandemic. (iii) To suggest inputs for the design of an industrial and innovation policy in developing countries considering the recent transformations in the global dynamics of production and innovation and in the public policies focused on industrial and innovation development of developed countries.

The context of change in the role of the state in the production and innovation dimension of the developed countries has to be considered in the design of industrial and innovation policy in Brazil. Traditionally, international agencies and institutions suggest that developing countries should liberalize their economies and reduce tariff and non-tariff barriers to increase their competitiveness in the global economy.

In this sense, this paper has the objective of contributing with inputs to new designs of industrial and innovation policies in developing countries (especially Brazil), which take into account the new global trends in terms of productive and innovative dynamics and the new characteristics of the industrial and innovation policies from developed countries and contribute to economic and social development.

3. Methodology The paper uses the innovation system approach to analyse the systemic character of the recent innovation policy adopted by developed countries and to reflect upon the design of new industrial and innovation policy in developing countries.

The methodology of the paper is based on two main axes. The first will be based on the mapping of references about the changes in the role of the State since the international crisis of 2007/08, as well as about the main characteristics of the industrial and innovation policies of the most developed countries. In this case, bibliographical research in international information sources was carried out. The main international information sources to be consulted are the Organization for Economic Cooperation and Development (OECD) and the United Nations Conference on Trade and Development (UNCTAD).

The second axis of analysis of this research project is based on the use of databases to analyse in greater detail the use of protectionist instruments (tariff and non-tariff barriers) in industrial and innovation policies of selected developed countries. For this, the Global Trade Alert database will be used (www.globaltradealert.org). This database gathers, since 2009, information on protectionist and liberalizing measures in relation to international trade adopted by a set of developed and developing countries. This information allows an analysis of the growth of protectionism in the context of industrial and innovation policies, as well as the main instruments used to protect domestic industry.

The main expected result of the paper is to suggest analytical inputs and proposals for the design of a new set of public policy for productive and innovative development in developing countries. This new industrial and innovation policy must consider the international context of increasing protectionism for the domestic firms of developed countries and the need of new approaches for industrial and innovation policy. The paper concludes that these new approaches should be focused on solving national challenges and seek to articulate social demand with the economic development in developing countries.

08:45
The Division of Scientific Labour: An empirical view on contributions to global science from the periphery

ABSTRACT. Acknowledged in the contribution dynamics of authorship is an asymmetry between the Global North and the Global South (also referred to as the ‘scientific periphery’) in the production of new knowledge. Marginson (2022) points to a dominant global discourse that frames science as a centre-periphery hierarchy. Baber (2003: 621) refers to a perpetuation of ‘an unstated but real global division of intellectual labour’. Others argue that theoretical and methodological innovations are considered legitimate tasks for Global North scholars while data collection is the designated role for those from the Global South. This paper examines the contribution of scientists from the Global South by relying on multi-authored scholarly publications to assess their changing contributions to global science over time. The focus on the contributions of scholars in the Global South in an asymmetrical global science system provides a unique and empirical perspective. It is a perspective that accounts for the networked nature of science that shapes the relative contributions to the global scientific network and provides important insights for funders and policy-makers seeking to establish equitable partnerships between the Global North and the Global South in global science.

09:00
Technological Capabilities and Co-Invention Networks for Regional Diversification: Evidence from Brazil

ABSTRACT. Purpose: This paper investigates the role of the regional endowment of technological capabilities and co-invention network structures in the emergence of new regional technological specializations. We consider a region specialized in a technology when its share of patents in a given field is higher than the share in the country.

Literature Review: Regional technological diversification has been addressed in the literature as a path-dependent process where new technologies congruent with the extant local endowment of capabilities are more likely to be successfully implemented in a region (Hidalgo et al., 2007; Neffke et al., 2011). The rationale is that regional diversification is like a branching process whereby the chances for the successful development or adoption of new technologies rise with the cognitive proximity to those in which the region is already specialized. This, for instance, is the core idea behind the “smart specialization” movement in the European Union. A significant literature has already explored this argument, emphasizing the endogenous dynamics of the regional diversification process. They empirically show that the regional endowment of knowledge and other capabilities is crucial to developing new economic specializations (Balland et al., 2019; Neffke et al., 2011). In parallel, another stream of literature has been pointing out the benefits of inter- and intra-regional co-invention networks for regional innovation performance (Bathelt et al., 2004). However, the impact of these network configurations on regional technological diversification has not yet been widely discussed in a branching process context. While some recent attempts on this matter have concentrated on external linkages (Balland & Boschma, 2021; Whittle et al., 2020), the role of co-invention networks at the inter- and intra-regional levels remains unclear.

Methodological Procedure: We use patent data from Brazil (2000 – 2019) to design two types of networks: one based on inter-regional and the other based on intra-regional co-invention linkages. We then conduct an econometric analysis at the microregional level to examine the effect on regional technological diversification of four main variables: centrality degree, betweenness centrality, transitivity, and technology flexibility. Centrality degree is the number of connections a region establishes with others, capturing the region's access to external knowledge. Betweenness indicates how well-positioned a region is in the inter-regional co-invention flows in Brazil, meaning that the region intermediates knowledge flows between others. Transitivity shows how well knowledge disseminates inside a region based on the network’s cohesiveness. Technology flexibility captures the average cognitive proximity between the regional technological capabilities and the technologies in which the region is not specialized.

Findings: Our results indicate that the regional technological endowment and co-invention network structures at both the inter- and intraregional levels are essential for diversification. High inter-regional betweenness is shown to be important in all contexts. High intra-regional transitivity is shown to be important in most contexts. Network centrality is statistically significant and positively affects diversification when interaction terms regarding the regional development level are included. Our results suggest a substitution mechanism between technology flexibility and betweenness, on the one hand, and GDP per capita and betweenness, on the other. This effect also holds when we change betweenness for centrality degree. It means that less developed regions with lower technological capability endowments benefit from establishing inter-regional co-invention linkages in general, regardless of control and intermediate positions. Our analysis corroborates with findings from two strands of literature. First, the literature on evolutionary economic geography which has empirically shown that the diversification process is path-dependent and is strongly impacted by the relatedness between different sectors, technologies, and products, as the current knowledge and capabilities regionally available will support the development of new ones. Second, the literature on regional innovation networks, in which the flows between physically close and distant agents contribute to enhancing knowledge recombination and production, encouraging learning and innovation.

Policy Implications: Our results can enrich regional diversification policies. For well-endowed regions the issue is what to aim for and with who. Less well-endowed regions face additional stress. Designing place-based policies can be especially challenging for them due to fewer knowledge and technological capabilities which, in turn, narrows the options for policymakers regarding diversification targets. Adding network-related aspects broadens the scope of policy by including practices that favor the connections of actors within and across regions.

REFERENCES Balland, P. A., & Boschma, R. (2021). Complementary interregional linkages and Smart Specialisation: an empirical study on European regions. Regional Studies, 55(6). https://doi.org/10.1080/00343404.2020.1861240 Balland, P. A., Boschma, R., Crespo, J., & Rigby, D. L. (2019). Smart specialization policy in the European Union: relatedness, knowledge complexity and regional diversification. Regional Studies, 53(9). https://doi.org/10.1080/00343404.2018.1437900 Bathelt, H., Malmberg, A., & Maskell, P. (2004). Clusters and knowledge: Local buzz, global pipelines and the process of knowledge creation. Progress in Human Geography, 28(1). https://doi.org/10.1191/0309132504ph469oa Hidalgo, C. A., Winger, B., Barabási, A. L., & Hausmann, R. (2007). The product space conditions the development of nations. Science, 317(5837). https://doi.org/10.1126/science.1144581 Neffke, F., Henning, M., & Boschma, R. (2011). How Do Regions Diversify over Time? Industry Relatedness and the Development of New Growth Paths in Regions. Economic Geography, 87(3). https://doi.org/10.1111/j.1944-8287.2011.01121.x Whittle, A., Lengyel, B., & Kogler, D. F. (2020). Understanding Regional Branching Knowledge Diversification via Inventor Collaboration Networks. Papers in Evolutionary Economic Geography.

09:15
Monitoring and evaluation of regional agricultural innovation in emerging economies: assessing the conditions for further operationalization in the case of Casanare, Colombia

ABSTRACT. Background and rationale

Science and innovation are core practices for agricultural development, and the extent to which knowledge is successfully produced, transferred, and applied often defines the trajectories of agricultural development processes. This subject has been captured by the concept of Agricultural Innovation Systems (AIS), which describes the interactions between multiple agents within a given institutional setting that shapes innovation processes, in this case for agricultural development (Rajalahti et al., 2008; World Bank, 2012; also see Tropical Agriculture Platform).

In emerging economies, the dynamics that take place within AIS are often shaped by variegated knowledge sources at the regional level, especially in rural agricultural systems where overlapping institutional settings and diverse actors perform a role from situated practice. In such settings agricultural scientific colleges are still weak (Goyeneche, et al., 2019), making it difficult to assess AIS on the base of scientific metrics.

This is the case in Colombian regions, where knowledge is subject to hybridization processes. In such settings local and scientific knowledges interact in the different phases and domains of the AIS. Therefore, the coexistence of multiple emergent patterns and diverse degrees of sophistication makes it difficult to track the specific features and configurations shaping the AIS’ capacity and performance.

The growing institutionalization of the Colombian AIS demands further consolidation of mechanisms for monitoring and evaluating (M&E) agricultural innovation to enhance informed decision-making (Ordóñez-Matamoros, 2013) at the regional level. This practice generates knowledge about the system, which favors strategic direction towards the objectives set to achieve innovation (Hekkert et al., 2006).

However, in a context of emerging practices, such measurement needs to transcend the traditional approaches that tend to overemphasize bibliographic scientific output, innovation patents, and results-based management, usually of a linear and short-term nature, to assess better the variety of practices that take place in regional AIS.

While there have been theoretical contributions to develop logical models for M&E mechanisms of the Colombian AIS, considering the particular institutional trajectory of the country in that regard (Londoño, Ordóñez-Matamoros & Uribe, 2021), further research needs to advance on the operationalization of such frameworks in order to assess their relevance in regional contexts.

This paper reflects on a pilot project implemented in the Casanare region in Colombia, assessing the conditions under which a M&E mechanism should be operationalized as a tool for the strengthening of regional AIS.

Methods

We rely on the framework of Londoño, Ordóñez-Matamoros & Uribe’s (2021) to examine the scope and limits of the proposed monitoring and evaluation model, as well as the conditions for its regional operationalization and possible challenges for its scalability to other regions.

Empirically, the paper builds on primary data from three workshops conducted to involve local actors in the definition of a set of indicators for M&E in Casanare’s AIS, as an instrument to make the M&E process effective. The first workshop aimed at identifying, validating, and formulating indicators for monitoring and evaluating science, technology, and agricultural innovation. The second workshop maintained the same general objective but focused on the validation of indicators. Finally, the main objective of the last workshop was to start the process of collecting information for the calculation of indicators. This last workshop collects information from the 47 most relevant actors of the AIS and is essential to see the effectiveness of the mechanism evaluated.

The initial battery of indicators was built by revising different sources of secondary information. Among them, national and departmental policies on agricultural innovation, laws, research products, and an advance that the National Government has been implementing in the form of a national survey of agricultural science and technology. Furthermore, the direct involvement of the authors in this project offers multiple lessons that allow a better interpretation of the gathered information.

The case of Casanare offers a relevant context in which agricultural innovation can be regarded as an alternative to extractive industrial practices that pose challenges for sustainability transitions at the regional level. Casanare is known for its oil extraction economy, as well as extensive livestock farming and monoculture crops of palm oil, which entail diverse social and environmental tensions.

Anticipated results

In Colombia, there is a theoretical and methodological approach to how the M&E mechanism for agricultural innovation should work. However, the operationalization of this remains a challenge for the country because it depends on the environment and the characteristics in which this mechanism is developed, and Colombia is a country composed of diverse regions that have different populations and geographical and productive characteristics. Therefore, the results will allow validating this methodological and theoretical instrument and show whether the country can implement a standardized M&E model for all departments based on knowledge of the internal and critical relationships of the system. Consequently, these show the requirements of capacity and coordination in various management levels of the tools of follow-up and monitoring to go deeper into its territorialization and homogenization.

Significance

This research adds to the growing literature on AIS, with a special interest on the regional scale and M&E practices, which remain underexplored. The regional scale in this case suggests possible pathways for the implementation of national AIS and policies in that realm, while considering emerging practices and interactions between actors whose knowledge is built upon multiple sources beyond the scientific. This research suggests possible ways to cope with complex emerging practices for monitoring and evaluation purposes, so that M&E mechanism in AIS can better inform policymaking.

This paper also contributes to a better understanding of innovation policy and practice in emerging economies. The Colombian case offers an interesting landscape of growing institutionalization of AIS at the regional level, along with multiple sustainability challenges that can be better addressed by means of agricultural innovation. However, such endeavor can only be fulfilled with relevant information and data on the performance of AIS, and M&E systems are a necessary condition to this.

08:30-10:00 Session 11D: Policy Considerations
08:30
How are GVC-oriented policies different? The Implications for ST&I Policies

ABSTRACT. Global value chains (GVC) have taken the policy world by storm with virtually all leading international organizations dedicating flagship publications to the topic. Yet, despite the huge enthusiasm about GVCs, there remains substantial ambiguity about their policy implications. In this article, we argue that the novelty of GVC-oriented policy lies in the elevated role that it gives to tasks, linkages, and firms. Such novelty pushes policymakers to adopt new task-based industrial policies, global connectedness policies and corporate due diligence policies that are challenging the current rule-based trading system.

A first novelty of GVC-oriented policies is the shift of attention from industries to tasks. Global production allows countries to functionally specialize in finer-grained value chain stages instead of entire industries. Yet the disproportionately large value-added that is captured by intangible-intensive tasks (e.g., R&D and marketing) presses policymakers to develop public policies that attract and retain the latter. In both developed and developing countries, policy makers are thus adopting a mix of task-based industrial policies – including ICT infrastructure investments, tax credits and subsidies – to expediate the upgrading of local industrial activity into more intangible-intensive tasks. For example, the Malaysian government has invested heavily in technology centers like the National Applied R&D Center that houses labs offering a wide range of instruments and infrastructures for advanced ICT testing services accessible to GVC suppliers at subsidized rates.

Second, GVC-oriented policies emphasize the role of linkages. GVCs link domestic and foreign tasks, and these linkages influence domestic firms’ economic and social performance. On the positive side, they act as powerful conduits for accessing foreign knowledge and resources that can be leveraged to improve technological and operational capabilities. Decent work parameters imposed by foreign value chain partners can also incentivize firms to improve local labor conditions. On the negative side, these linkages can reduce economic resiliency by transmitting foreign economic shocks to domestic firms. A focal concern of policymakers is thus how to properly regulate, deepen, and strengthen GVC linkages so that they can promote economic and social upgrading while guaranteeing economic resilience. On the one hand, such global connectedness policies aim to improve the access to knowledge and information that is transferred through linkages. On the other hand, they focus on strengthening the absorptive capacity of local firms. For example, for many years the Chilean government has promoted networks of suppliers to enhance inter-firm collaborations and reduce the gap between lead-firms and their input providers.

GVC-oriented policies finally elevate the status of firms. Multinational enterprises (MNEs) have enormous power in GVCs. Sitting at the apex of a hierarchical chain, they have the muscle to select which firms are included or excluded in GVCs, to determine the terms of supply-chain membership and to allocate where, when and by whom value is added. It is this power that allows MNEs to not only enhance the efficiency of the GVC, but also to promote social and environmental conditions extraterritorially. They can remediate poor labor and environmental conditions in global supply chain factories, for example, by pledging to work only with suppliers and sub-suppliers that adhere to strict codes of conduct and collaborate with them to reach this goal. Policymakers not only see this power as a challenge but also as an opportunity. By properly harnessing MNE power within GVCs, governments can use domestic policies that target MNEs to foster changes along GVCs. Several countries, for example, have started to impose strict corporate due diligence principles on MNEs that apply along their entire GVC. For example, the French government in 2017 implemented the Duty of Vigilance Law to impose a legal duty to exercise human rights due diligence.

These new policy directions challenge current trade policy practices in several ways. First, the intricate relations between tasks, linkages and firms require policymakers to move beyond traditional silo thinking where each Ministry pursues its own objectives independently. Now more than ever Governments need to develop coordinated trans-Ministerial efforts to synchronize their trade, innovation, industrial and social policies. Second, the global scope of GVCs requires countries to develop new supra-national policies to avoid distortionary arbitrage strategies that are rampant in GVCs. The recent G20 agreement for a minimum global corporate tax rate to close cross-border tax loopholes is a good example of new steps in this direction. Third, the rule-based multilateral trading system needs to modernize to tackle challenges that GVC-oriented policies pose to the foundational principle of non-discrimination. For example, corporate due diligence policies that apply throughout GVCs represent a legal obstacle to “national treatment” if they lead to the discrimination of imports from non-compliant businesses.

Taken together, even if GVCs reinforce several traditional trade policy elements, they in other areas force a fundamental redesign in policy thinking by accentuating the role of tasks, linkages and firms. We argue that these changes are so profound that the set of GVC-oriented policies has become a truly different concept that challenges the rule-based trading system.

08:45
Policy considerations for AI and genome editing shaping humanity

ABSTRACT. Introduction

Science policies (or lack of) have the potential to maximise opportunities as well as expose society to risks presented by emerging technologies. The integration of artificial intelligence (AI) and biotechnology, whilst in its infancy, presents significant opportunities and risks, and proactive policy is needed to manage these emerging technologies. While AI continues to have significant and broad impact, its relevance and complexity magnify when integrated with genetic editing in particular; applying AI elements such as machine learning and deep learning can foster substantial benefits as well as daunting risks that range from ethics to national security. Yet, this combined field has not been adequately studied from a policy perspective. The components need clear definitions and analyses with respect to their practical combined implications for society. The study of genome editing, and its technological advances have been facilitated to a large degree by their underpinning bioinformatics platforms, which has allowed screening and detection and gene manipulation to become common practice. Application of AI to this technology will catapult the revolutionary potential of genome editing from ‘hypothetical’ to imminent. However, application of AI to this rapidly advancing field poses severe risks from potential weaponization and bioterrorism as well as opportunities to improve health and wellbeing. Both genome editing and AI technologies are being pursued at scale in various global markets and the genome and AI research agenda is moving towards a progressively permissive landscape, with AI and genome editing technologies being brought together to cut costs and time. For instance, DNAnexus, a cloud based commercial platform, offers products utilizing machine learning techniques for mutation predictions, diagnostics, and gene target optimisation. Verge Genomics, an AI driven therapeutics company, is bringing together human genome and machine learning to create predictive models for drug development with its first ALS drug just entering the clinical market. It is just a matter of time before these developments are further exploited with AI tools, not just to modify the human genome to cure disease but to develop resistance to diseases, enhance vision, create superhuman strength, and pave the way for a species evolution. There clearly is an urgency to addressing policy issues surrounding these emerging technologies. Given this gap, we are investigating the policy implications of the application of AI to genome editing in humans, in particular technology governance as a cross-cutting theme. This analysis will help regulators take stock of advances with AI-facilitated genome editing and develop a future scenario-focussed framework to protect human interest by considering the implications of these technologies being pursued at scale and globally. Our methods outlined below will ensure that this is not just an academic exercise, by engaging relevant sector and policy experts and ensuring that the outputs reach a variety of audiences.

Methods

Landscape assessment: We will use software-assisted horizon scanning for assessing the state-of-the-art of AI and genomic tools, creating a typology for their use, and assessing where they have been or integrated. We will complement this approach with interviews with up to 10 subject-matter experts. We will conduct a desk-based review of the current regulations surrounding genome editing and AI applications in the life sciences as a frame of reference for understanding the areas in policies that are unclear or uncertain. Landscape assessment will enable us to identify the most important factors that affect the trajectory of AI and genomic tools. Futures framework: Landscape assessment will form the basis for how the relationship between AI and genomic editing is currently configured, and a futures framework will then be used to explore how this relationship might change over time. We will use a scenarios-based futures exercise, where the landscape assessment will facilitate crafting of illustrative stories that outline how AI and genomic tools might remain the same, improve, or change in some other way based on the effects of the factors identified in the landscape assessment. A basic scenarios-based table-top exercise (TTX) will be conducted based on the trajectory of the landscape over the next 25 years, bringing together key stakeholders in genomics, AI, and regulatory sectors, to contemplate the various policy dimensions of these technologies. The TTX will identify opportunities, risks, and policy issues, with special consideration of technology governance challenges as a cross-cutting issue. The proposed approach will serve as a blueprint for future areas of research in multiple sectors where various policy lenses could be applied (e.g. impact on agriculture and food security). Going through such scenarios with stakeholders will enable us to characterise how well current policy addresses these scenarios and what changes may be needed taking this further than just an academic exercise. The project team will use MURAL software for scoring the scenarios, to enable the prioritisation of the most important policy responses or changes required. Key risks, opportunities, policy implications, and actions will be identified as outputs of the exercise.

Anticipated results

The current regulations and discourse are not appropriate to respond to the melding of AI and genome editing technologies and to regulate them at scale. Using a combination of landscape review and futures methodology, we will create a novel frame of reference for policymakers and the public. This work approaches the understudied sector of the confluence of AI and genome editing, and it explores technology governance as a broader policy challenge that could be applied to other sectors and technologies. The outputs of this work will be consolidated into a short report, summarizing opportunities, risks, and policy considerations for countries investing in R&D, thus supporting responsible innovation. The evidence will provide a steppingstone for a roadmap for policymakers and will serve as a mechanism for engagement with potential clients in Europe and the US such as DARPA, NATO, DoD, and more. We will pursue further dissemination via development of a podcast with sector experts delving into the output produced and assessing their views. This will further our research and help reach policy makers and the wider public to generate interest and a public discourse as well as policy considerations. We will actively engage our networks and potential clients to discuss our findings and create bespoke policy briefs and concept notes as a follow up activity.

09:00
The Evolving Role of White House Science Advice: An Assessment of the Membership Balance and Policy Impact of the President’s Council of Advisors on Science and Technology (PCAST)

ABSTRACT. Overview This presentation offers a comparative analysis of the President’s Council of Advisors on Science and Technology (PCAST) across presidential administrations, beginning with President George H.W. Bush through the first two years of the Biden administration. PCAST – a federal science advisory committee appointed by and advising the president – will be assessed for its changing balance of member social and professional perspectives and its role and impact in federal policymaking over the last thirty years. The goal is to understand the changing balance of PCAST’s expertise and the evolution of its advisory role and across time, as well as to employ this analysis to develop recommendations for PCAST’s organization and operations in future administrations.

Background and Rationale PCAST was created by President George H.W. Bush in 1990, although its origins date back to the Truman administration and the early Cold War. PCAST is co-chaired by the president’s “science advisor,” its only federal member, and consists of roughly 30 independent scientists, engineers, industry leaders, and other research professionals. PCAST is charged to advise the president – either directly or through the science advisor – on all matters related to science, technology, and innovation (STI).

While PCAST’s function has not changed significantly since its creation, its membership demographics, policy areas of focus, and influence on White House policymaking has shifted considerably with each presidential administration. Most recent research studying scientific experts in U.S. government decision-making focuses on “science for policy” technical advisory bodies within federal regulatory agencies. PCAST, however, is “policy for science” presidential committee responsible for guiding White House policy development and implementation across a broad range of issues that rely on STI data and analysis, including national security, public health, industrial competitiveness, environment and energy policy, and science education. PCAST’s proximity to the president, its high-profile membership, and the historical influence of its predecessors make it a unique case study for examining the role of scientists and other research experts working at the nexus of science, policy, and politics. Empirically evaluating scientific advice remains a challenging research question. This presentation offers data and a preliminary analysis of PCAST’s changing member perspectives and its evolving role and impact in White House STI policy decision-making. The goal is to understand how and why professional or social groups were represented on PCAST’s roster across time and how PCAST policy recommendations are developed and implemented in broader STI ecosystem through time.

Methods This research employs two analytical frameworks to understand PCAST’s membership demographics and the nature of its advisory function over time, from 1990 through 2022. The first framework establishes a measure of PCAST’s diversity in the context of its governing legislation, the Federal Advisory Committee Act (FACA). FACA mandates certain transparency requirements for committee operations, as well as guiding language for ensuring committee membership is “fairly balanced” in represented interests and technical expertise to address its intended function. As PCAST members are appointed based on both their scientific knowledge and personal achievements, member demographics are categorized into “professional perspectives” and “social perspectives.” PCAST membership will be compiled from the Federal Register and each individual will be coded by their professional (i.e., scientific expertise and career experience) and social perspectives (i.e. race or ethnicity, gender, and geographic location) to be collected from public biographical records. Data will be supplemented with public statements from PCAST members—such as oral history interviews and public statement from current and former White House officials—to provide comprehensive picture of how each president and their science advisor approached membership balance and if that balance is representative of the national STI workforce at the time of service.

The second analytical framework offers a methodology for examining PCAST’s intended function and its impact in White House policy development and implementation. Although FACA ensures PCAST’s formal independence from the administration, PCAST remains responsive to White House policy priorities through the leadership of the science advisor and sometimes direct engagement with the president. PCAST, in its alignment with administration priorities, could possess an “instrumental” role, providing substantive policy analysis that advances the public interest, or a “symbolic” role, legitimizing or otherwise supporting administration decisions – or a combination of both. To make this determination, each president’s PCAST will be analyzed based on four criteria: its leadership structure (e.g., Does the science advisor serve as its sole chair or co-chair?); its process for consensus building (e.g., How are policy recommendations vetted and agree upon?); the long-term impact of policy products (e.g., Do PCAST studies translate into concrete policy actions?); and the image of PCAST presented through official government press releases and its reception in national media (e.g., How and how often are PCAST members and their work represented in news outlets?). Anticipated Results

The two frameworks will help to provide a complete picture of PCAST’s shifting membership balance and its role in White House STI policymaking beginning with the George H.W. Bush presidency. With respect PCAST’s diversity, the council’s exceedingly broad charge to address all policy related to STI raises the question of how best design a representative council membership inclusive of all individual and career backgrounds and scientific disciplines. An initial analysis of PCAST membership demographics demonstrates a trend toward increased inclusion of both professional and social perspectives with time. However, the data indicate a persistent underrepresentation of both categories with respect to the national STI workforce and overall U.S. population. In particular, women and minoritized racial and ethnic groups are underrepresented in PCAST in most administrations. Notably, individuals of Asian descent were significantly underrepresented with respect to that group’s increasing participation in the national STI workforce. Additionally, individuals with expertise in the social sciences, while consistently represented in most PCASTs, are not included in PCAST at the level of active U.S. STI professionals.

Tracking the trajectory of PCAST research products from ideation, to development, dissemination, and implementation on downstream policy decisions is a challenging task. The second framework will provide a method for beginning to understand how PCAST has interfaced with high-level stakeholders in the STI community to deliver policy recommendations, and how those recommendations translated into actions or informed government decision-making. The analysis will allow for a deeper understanding of the de facto nature of PCAST’s independence and the limits of working within both FACA and the highly political context of the White House.

Significance This study presents the first longitudinal analysis of PCAST membership and policy products. PCAST stands as one of the U.S.’s most visible scientific bodies, unique in both its position within the federal STI policymaking apparatus and the preeminence of its membership. The anticipated results will offer an empirical basis for developing best practices for PCAST’s membership selection and operations. As PCAST is involved in high-level development of federal STI policy, optimizing its activities and policy products to increase its future impact will in turn work to advance the overall health of the U.S. national STI enterprise.

09:15
Policy levers to traverse the “Valley of Death” – which to pull, when, and how hard?

ABSTRACT. Summary To meet the demands of the ongoing green energy transition, policy makers need to know how to induce market transformation towards low-carbon technologies. However, the widespread diffusion of new technologies is hampered by a variety of barriers: e.g. high upfront costs, uncertainty around performance, etc. There are abundant policy tools designed to overcome these barriers, but relatively little is known about which policy tools to deploy when, for how long, and how much. A comprehensive approach to designing a policy mix could illuminate the path forward to improved efficiency in policy support by taking advantage of synergies between individual policy tools. Here, we model the owner-broker interaction in green building technologies in a variety of randomly generated policy contexts. We then search through this parameter space for patterns in the combinations of policies – their sequence, timing, duration and strength – that reliably yield market transformation. Our findings contribute to a growing literature on policy mixes, and to the body of policy knowledge derived from computational social science approaches. This paper is a work in progress with preliminary results. Full results are expected in time for the conference.

  Introduction / Background Governments have set aggressive timelines for net-zero decarbonization that requires widespread energy and sustainability transitions (H Res. 109; Weyant, 2011; Mallapaty, 2020; Dooley, Inoue & Hida, 2020). However, despite a clear acknowledgement of the need for energy and sustainability transitions, the institutions and policies that would drive these transitions remain unclear. The economics and innovation literatures call for a better understanding of “transitions management,” acknowledging the role of emissions pricing, cost reductions for low-carbon technologies, and information barriers that slow the diffusion of nascent technologies (Meadowcroft, 2009). The “valley of death” describes the challenges of moving a nascent technology from basic science to widespread market penetration. The valley of death is often approached as a cost problem. Large private R&D investment results in high per-unit prices, and low demand at high prices prevents scaling up and decreasing marginal costs of production (Nemet et al., 2018). The dominant perspective on reducing these costs, and thus traversing the valley of death, centers on market mechanisms such as subsidies, carbon taxes, or cap and trade schemes. However, the implementation and effectiveness of these approaches has been, at best, incomplete (Geels et al., 2017). Politicians have been unwilling to move on these strategies or set emissions prices that are too low, and as a result, socially efficient externality pricing covers only a small portion of global greenhouse gas emissions (Geels et al., 2017; Weyant, 2011). We contend that the failure of conventional pricing tools to pull nascent low-carbon technologies into widespread diffusion is the result of a policy mix that ignores the underlying information barriers that drive the valley of death. We frame the issue of high per-unit costs as stemming from “knowledge gaps” and information barriers that substantially increase risk and subsequently preclude significant investment from capital markets (Blackburn et al., 2020; Jaffe et al., 2005; Weyant, 2011). Many policy instruments, including public R&D investment, pilot projects, demonstration projects, public procurement, tax breaks and rebates, regulations and mandates, and other market instruments, have been shown to foster learning effects and reduce information barriers, risk, and costs (Bossink, 2002; Bossink, 2015; Edler, 2007; Aghion et al., 2009; Blackburn et al., 2020; Geels et al., 2017, Jaffe et al., 2005; Weyant, 2011). Despite the variety of tools and levers that can be pulled to induce market transformation, each instrument deployed in isolation may not generate enough learning or overcome sufficient informational barriers to drive the sustainability transition (Rosenow et al., 2017). The question of how hard and in what combination to pull these levers remains pressing (Nemet, 2010). We propose that synergies are likely to exist between various policy tools, and that coordinated efforts to implement carefully designed policy mixes can leverage these synergies to bridge the valley of death. Analyses of the interactive effects, temporal factors, and institutional dynamics of policy mixes are likely to shed light on how governments can design effective transitions management strategies (Flanagan et al., 2011) Here, we develop a theory of the valley of death with a particular focus on information issues that drive high costs, mechanisms that can be used to overcome these information issues, and specific policy tools that can be implemented to traverse the valley of death.

A New Approach: Building a Bridge with Policy Mixes Multiple market failures and barriers contribute to the technology VoD, thus, a “policy mix” of several instruments – some pushing from the supply side, some pulling from the demand side, and other acting to reduce informational barriers directly – are likely needed to bridge the VoD (Rosenow et al., 2017; Flanagan et al, 2011; Raven and Walrave, 2020; Fischer and Preonas, 2010; Weyant, 2011; Lehmann, 2012). Deploying a combination of cost-focused and information-based instruments has been shown to reduce the cost of transition management for emissions reductions compared to a single policy (Fischer and Newell, 2008). The design and implementation of policy mixes to spur nascent technology diffusions is beginning to take hold on the international stage (OECD, 2009). An attractive feature of the policy mix approach is that the effects-in-isolation of individual policies are unlikely to be additive when implemented together – rather the expectation is that synergistic interaction effects, temporal dynamics, and institutional dynamics will contribute to the success of the policy mix (Flanagan et al., 2011)

Methods Overview To demonstrate the framework for understanding the relationship between context and policy levels, we deploy several methods sequentially. First we build an agent-based model (ABM) of building development driven by owner-broker interactions in which we operationalize both contextual variables and policy levers. Second, we simulate the first two (of three) scenarios: a no-intervention scenario the establish a baseline and a random intervention scenario to explore the space of policy interventions. Next, we conduct an unsupervised cluster analysis on the cases with successful outcomes from the random intervention scenario to find groups of self-similar “pathways” to success. Next, we simulate a third scenario with strategic policy interventions based on the pathways to success identified in the previous step. Finally, we compare the distribution of successes across the three scenarios.

Preliminary Results Based on 10,300 simulations of our model, we identify solutions that improve the probability of successful market transformation. The random intervention scenario improves the expectation of successful outcomes over the no intervention baseline scenario from 15.2% to 22.3%. However, it does not do so systematically. The details of each random context are not used to inform policy design – this reflects a tremendously naïve approach to policy design. We improve upon the policy design by strategically connecting a policy prescription to a specific context, and in doing so demonstrate a further improvement to outcomes: increasing the expectation of success to 28.1%. These results can help inform how policymakers and other stakeholders can strategically deploy policy tools to accelerate green market transformation.

08:30-10:00 Session 11E: Evaluation 2
08:30
Learning to evaluate transitions and missions policies

ABSTRACT. The ‘turn’ in policy towards addressing societal challenges calls for a radical rethink of how to do research and innovation policy. Our paper focuses on evaluation in this new policy environment. Challenges for evaluation include the increased scope of the policies needed, devising theories of change that are challenge- rather than push-based, tackling complexity, and new roles for evaluation in the life cycles of policies that tend to be longer than those of traditional research and innovation interventions. Policymakers and evaluators alike need to embrace and incorporate lessons from the fast-growing literature on transitions and missions. The paper will synthesise results from our own and others’ research on evaluating transition- and mission-like interventions, deriving operationally relevant conclusions for evaluators and policymakers.

08:45
Meta-evaluation of Science, Technology and Innovation Policy: evidence from an emerging economy

ABSTRACT. The growing institutionalization of innovation systems in emerging economies has led to a more prominent role of the State in policy design and implementation to foster knowledge and innovation production, diffusion and use in different realms of society. In this context, Science, Technology and Innovation Policies (STIP) and instruments require the constant evaluation of its performance and results, in order to better inform the role of governments therein.

In emerging economies, the increasing number of policy documents contrasts with the limited capacity for the implementation of national STIPs, which is often related to the lack of use of policy evaluation results and its derived lessons and recommendations (Kuhlmann & Ordóñez-Matamoros, 2017). One of the reasons for that is the single-intervention approach of policy evaluations, which do not often reflect their implications for broader innovation systems. In that regard, translating policy learning into effective policy change demands more comprehensive lessons on the strengths and challenges of STIP with the use of STIP evaluations. In other words, the body of knowledge produced by evaluations may clarify what we know and do not know about STIP, in order to enhance STI policymaking and governance.

Meta-evaluation frameworks help us to systematically assess the results of evaluations in order to learn more about how STIPs perform within systems (Edler, Ebersberger, & Lo, 2008). Rather than focusing on the intrinsic technical merit of single interventions, these approaches emphasize the broader contribution of STIPs to innovation systems (Arnold, 2004). While such approaches are frequently used in mature innovation systems (Edler, Cunningham, Gök, & Shapira, 2016; Collin, Sandström, & Wennberg, 2022), these could be further operationalized in emerging economies, for instance in Latin America (Bin, Andrade, Vasconcellos, & Salles-Filho, 2019), where more knowledge is needed on the systemic role of STIPs.

With that in mind, this research presents a meta-evaluation of STIP in the context of Colombia, as a relevant and illustrative case of an emerging economy. It assesses the design of and results produced by the STIP evaluations conducted in the country. Specifically, this work has a three-fold purpose: first, to derive lessons learned from the STIP evaluations performed in Colombia; second, to understand and explain the factors that facilitate or hinder the conduct of successful evaluations, where success depends on a) whether the evaluations achieved their objectives and b) whether the evaluation achieved any impact; and third, explore the value or merit of different types of approaches and methods that are used to evaluate. In this regard, some key research questions are:

• Did the purposes of the STIP evaluations clear and achieved? What lessons can be derived from evaluating the effectiveness of STIPs? • Have the evaluation approaches been useful or not in understanding the underlying rationale of STIPs? • What methods have been used to understand the interventions? • What factors are determinants in the impact and use of evaluations?

This paper follows a meta-evaluation approach, i.e. “an evaluation of evaluations” or a systematic analysis of a series of public policy evaluations with the purpose of informing the scope and quality of the ways that the merit of public interventions is assessed (Scriven, 1991, p. 95).

To that aim, empirically we examine different public repositories of evaluations of the Colombian national government, including the Ministry of Science, Technology and Innovation, the National Planning Department and its evaluation system, and the Colombian Observatory of Science and Technology.

A total of 36 evaluations have been identified so far, conducted between 1997 and 2021. The basic relevant information of each evaluation report is extracted and systematized in a table, including categories such as the type of evaluation, the type of STIP evaluated, the date of publication, the time frame observed by the evaluation, the authors and its institutional affiliation, the purpose of the analysis, the evaluation methodology and the type of methods, the main results and conclusions, among others. Furthermore, a grounded literature-based framework is being developed to derive additional categories that allow a better assessment of the evaluations. The main themes of evaluation reports will be codified under these categories by using traditional content analysis software (e.g. Atlas.ti, NVivo).

The expected results include a complete data base with systematized and processed information on STIP evaluations in Colombia. This will allow to eventually build a STIP Evidence Gap Map (EGM) that graphically shows what we currently know and ignore about the STIP process and performance in the country. According to Snilstveit et al. EGMs provide thematic collections of evidence structured around a framework which schematically represents the types of interventions and outcomes of relevance to a particular sector. By mapping the existing evidence using this framework, EGMs provide a visual overview of what we know and do not know about the effects of different programs (2016, p. 120).

Preliminary findings show an evolution on evaluation approaches in the Colombian National STI System, beginning in the 1970’s with mechanisms such as expert panels to assess the merit of STI projects and instruments ex ante. During the 1980’s and 90’s peer-review is the main mechanism to evaluate STI policy, and by 1997 results and impact evaluations are put in place to assess the socioeconomic effects of STI programs. During the 2000’s the previous mechanisms are further institutionalized, along with institutional evaluations of actors in the National STI System, and by the 2010’s decade more evaluations are conducted using more standardized criteria, professionalizing the role of the evaluator, and complementing measurement mechanism of scientific capacity, performance and output.

The evolution of STIP in Colombia reflects what the body of literature on STIP evaluation stresses regarding the ever growing and changing nature of evaluations: as general STIP frames change (Chaminade & Edquist, 2010; Vehlo, 2011; Schot & Steinmueller, 2018; Kuhlmann & Rip, 2018), so are STIP evaluation approaches, which seem to reflect on the former, offering at least four broad streams on evaluation practice.

A first approach is inspired by the scientific tradition of peer review in which the value of policies lies in their intrinsic (scientific and technical) quality. Second, there is an approach derived from new public management thinking interested in the broader socioeconomic impact of research and innovation programs. Third, a systemic perspective concerned with the performance or 'state of health' of innovation systems, considering the multiple variables that compose them (Georghiou, 1999; Arnold, 2004; Edler, et al., 2016).

Fourth, with recent innovation policy approaches such as Mission-Oriented Innovation Policies (MOIP) (Mazzucato & Semieniuk, 2017) or Transformative Innovation Policy (TIP) (Schot & Steinmueller, 2018), there is a renewed interest in formative evaluation (Magro & Wilson, 2013; Molas-Gallart, Boni, Giachi, & Schot, 2021), oriented towards learning and reflexivity in experimental processes for transition to sustainability (Luederitz, et al., 2017; Mickwitz, Neij, Johansson, Benner, & Sandin, 2021).

This research intends to produce relevant evidence aimed at STI policymakers, program managers and evaluators to inform STIP process and its phases of design, implementation and evaluation, while contributing to the broader debate on the effectiveness of STIP in Colombia. Results can also be relevant for regional projects interested in STIP governance and evaluation, where there may be lessons for cross-learning between Latin American countries (Bin, Andrade, Vasconcellos, & Salles-Filho, 2019), while dialoguing with STIP meta-evaluations conducted in developed countries (Collin, Sandström, & Wennberg, 2022).

09:00
Evaluating Sectoral and Associative Policy: The Case of the FSAT Program in Argentina.

ABSTRACT. The objective of this paper is to study the effects of the micro policy of sectoral funds on the innovative and economic performance of firms. For this purpose, we are going to study the High-Tech Sectoral Fund (FSAT), one of the instruments of the Argentine Sectoral Fund FONARSEC), implemented by the National Agency for the Promotion of Research, Technological Development and Innovation (I+D+i Agency) of the Ministry of Science and Technology of Argentina. The period of analysis is between the years 2010 - 2017. Evaluations of policy instruments in general focus on conducting impact analyses, either from a market failure or systemic problems perspective. In turn, these studies also mostly focus on studying horizontal programs that do not promote particular sectors. This research analyzes a sectoral fund whose goal is also to strengthen the link between the scientific and technological sector and the socio-productive sector in order to contribute to the solution of social and economic problems. In particular, we will take as a reference the policy of support to isolated innovation projects to evaluate whether the promotion of public-private partnerships generates a better innovative performance. This type of analysis becomes relevant due to the scarcity of studies that analyze the impact of vertical innovation policies. In particular, given the challenge that the covid-19 pandemic implied for policy in general, and science and technology policy in particular. Currently, the policy has shifted to a much more focused scheme, and this requires knowledge about the results of this type of policy. This paper thus contributes to policy by shedding light on the results of a vertical instrument. The policy of sectoral funds is a recent milestone in Argentine science, technology and innovation policy. In particular, in relation to the way in which technological innovation was supported in companies. Three elements make it possible to dimension this change. First, innovation policy shifted from horizontal instruments to vertical instruments that sought to provide support focused on sectors, regions of lesser relative development, or technological platforms. Secondly, there was a shift from isolated support for companies to innovate, to instruments focused on public-private partnerships. Finally, the size of the non-refundable contributions was substantially larger than that of the pre-existing horizontal instruments: the minimum non-refundable contribution was USD400,000 and could reach up to USD10,000,000. This policy was manifested in the Argentine Sectoral Fund (FONARSEC), composed of three instruments: Sectoral Technological Innovation Funds (FITS). Regional Technological Innovation Funds (FITR) Sectoral Funds for High Technology (FSAT). FITS was designed with a focus on sectoral development, while FITR has a regional focus. Finally, FSAT has a focus on intensive technology platforms, such as biotechnology, nanotechnology and ICTs. Within this framework, this paper aims to analyze the impact of FSAT. To this end, we will study the causal relationships between access to financing, the formation of a public-private partnership and the trajectories of the firms. We will analyze whether the trajectory of their innovative effort and/or market performance improved as a result of their participation in the program. Understanding the impacts of the instrument on the companies will make it possible to identify the public policy criteria that have worked and those that need to be focused on in order to improve. The methodological approach is based on an econometric study that will determine whether the trajectory of a firm that accessed the instrument was different from one that did not. For the evaluation, a panel of data will be constructed with a balanced structure and information for two moments in time, before and after the FSAT considering the average of three years. The identification strategy will combine two impact evaluation techniques. First, a Propensity Score Matching will be implemented to identify those firms in the control group that, due to their observable characteristics, are not similar and comparable to the beneficiary firms. Then, once the database has been cleaned, a Differences in Differences estimation will be implemented to estimate the trajectory that would have been observed in the absence of FSAT. By comparing the two trajectories, the observed and the counterfactual, we will be able to identify the Average Treatment Effect attributable to FSAT, controlling for unobserved and fixed factors over time. The results confirmed an impact of FSAT on the trajectory of the beneficiary companies. On the one hand, the evidence showed that the growth in spending per employee on innovation activities was higher thanks to FSAT. In addition, the growth in employment and total sales was more intense thanks to the program. No evidence was found on the trajectory of spending per employee on R&D activities and firm productivity. The first result could be reflecting the positive synergies of the public-private linkage promoted by FSAT. R&D activities were undertaken by academic research groups, which took advantage of the funding received to incorporate scholarship holders, train researchers, incorporate equipment and improve their infrastructure. This allowed companies to rely on the capabilities of these groups, and to undertake the rest of the innovation activities by increasing their investment effort. The second result may reflect the short time that has elapsed since the end of the FSAT. Previous evidence suggests that on average it takes 6 years to begin to observe impacts on the productive efficiency of companies participating in innovation support programs. Future research plans to conduct an evaluation that takes into account these possible dynamic effects.

09:15
An Integrated Approach to Innovation Policy Analysis: The Case of Romania

ABSTRACT. Background and Rationale: The effects of policies to support innovation must be addressed in the context of the entire policy mix (Flanagan et al. 2011). Therefore, it is important to establish whether the collection of policies that may influence innovation activities do so in a coherent and constructive manner (Falk 2009). At the same time, the actual manner in which policies are designed, implemented and their governance context will also either facilitate or constrain the ability of such policies to deliver on expected goals. To address the needs of countries to improve their innovation policy approaches, World Bank has developed a public expenditure review framework (PER) for innovation policies (Correa 2014). This framework proposes and integrated approach to the science technology and innovation policy environment of a country. It consists of four stages, namely, policy mix analysis, functional analysis, efficiency analysis and effectiveness analysis. The first three stages have been fleshed out into detailed analytical methods and applied in several countries around the world providing useful insights for governments to improve their practices. This paper illustrates the application of the methodology with the specific case of Romania. Methods: In Romania, the policy mix analysis and functional analysis were conducted of the set of policies for supporting and enhancing innovation, especially in the context of its entrepreneurship ecosystem. Each one of these components of a PER have specific empirical strategies and instruments to carry out the analyses. The policy mix requires cataloguing all the policies and identifying a full range of features of each one. These include the institutional context of the policies, period the policies were active, their stated general and specific objectives, intervention mechanisms, intended beneficiaries, supported activities, budgets, including their origin, among others for a total of more than 100 possible items. The functional analysis uses a model of the policy intervention with 31 categories covering design, implementation and governance. For each category, an assessment of performance is coded on a Likert scale using criteria derived from a broad base of recommended practices found in scholarly journals, agency reports and consultations with experts. Data and Results: The data for the policy mix analysis is obtained from the government documents describing each policy instrument and government databases that contain budgets and other administrative information. Consultations with relevant ministries and agencies are often required to complete information needed for all the categories in the policy mix catalog. The functional analysis uses the results from the policy mix, documents obtained during the first phase and is complemented by in depth interviews with the relevant public officials who are responsible for managing the instruments in the analysis. The information on each policy instrument is used to address each of the categories in the model and assess on a likert scale the degree to which they reach best recommended practices in each area. In the case of Romania, the results of the policy mix analysis show that, first, the current portfolio of policies is highly concentrated, with significant resources devoted to a few instruments and a ‘long tail’ of much smaller instruments which are likely to be sub-scale. Second, most instruments have multiple top-level objectives, such as productivity improvement combined with job creation. This may suggest a lack of clarity in the design of instruments. Third, there are gaps in intermediate-level objectives, especially regarding improvement of management practices and linkages with foreign firms and market access. Fourth, there are gaps in support for entrepreneurs, especially regarding early-stage, pre-profit, startups and individual entrepreneurs - as well as the intermediary organizations which support such startups, such as incubators and accelerators. Some ‘entrepreneurship’ instruments may potentially reinforce the role of incumbents rather than encouraging the growth of innovative startups. Fifth, grants are the dominant intervention mechanism, which may be appropriate in a relatively under-developed innovation and entrepreneurship ecosystem, notwithstanding their potentially distortionary effect. However, it is important to remember that not every element of the ecosystem can necessarily be resolved through the addition of public money: regulatory reform is also needed and factors such as cultural perceptions of innovation and entrepreneurship are also important. In the case of the functional analysis, the results of the analysis show, first, that Instrument-level evaluation could be improved. This is currently stronger at the Priority Axis level of EU supported instruments. Improved evaluation at the instrument level would also support rationalization of the portfolio, by providing information about which instruments should be scaled-up. Second, instrument design is consistently weak in the articulation of a theory of change. Developing and articulating these models should enable better accounting of inputs and activities by managing authorities, as well as improved targeting of the instrument. It should also prompt ideas for alternative instruments and mechanisms, which would assist with rationalization of the portfolio and the development of a wider range of mechanisms. Third, administrative burdens for many beneficiaries are still high. There has been a significant effort by managing bodies to simplify the application processes for support scheme and reduce the bureaucracy for applicants. However, there are still indications that these processes remain overly complex for applicants and could be further simplified. Fifth, bureaucratic burdens for administrators are also significant. In particular, multiple auditing of instruments appears to impose a significant burden on administrative teams. Bureaucratic friction is likely to be increased by the structure of many instruments which have both a Managing Authority and a separate Intermediate Body; this may be a necessary condition of some European funding but is unlikely to be optimal.

References Correa, P. (2014). Assessing Public Expenditures on Science, Technology and Innovation: A Guidance Note. World Bank Group. Falk, R., 2009. The Coherence of the Instrument Mix. Report Nr. 8 in the context of the Study: Evaluation of Government Funding in RTDI from a Systems Perspective in Austria. : Vienna. Flanagan, K., Uyarra, E., Laranja, M. , 2011. Reconceptualising the ‘policy mix’ for innovation. Research Policy, 40 (5): 702-713.

08:30-10:00 Session 11F: Innovation for Public Purposes
08:30
Applied research to develop cancer drugs, basic research to succeed

ABSTRACT. 1.Background and rationale Turning scientific research into innovation is a considerable challenge in the medical field (O’Connell & Roblin). Current developments in scientific research have not been mirrored by the same level of progress in drug development (Pammolli et al., 2011), especially with regard to cancer diseases, the most dangerous type of non-communicable disease (World Health Organisation, 2017). Even though research on genetic alterations in human cancers has led to a better understanding of molecular drivers of cancer diseases, and this knowledge should provide more useful drugs, the effectiveness and success rate of cancer drug developments are remarkably low (Hutchinson & Kirk, 2011; Begley & Ellis, 2012). Previous investigations have also confirmed that most medical research organisations focus on publishing novel scientific research instead of developing new drugs (Venditto & Szoka, 2013). This state of affairs is the motivation for the present research to explore the knowledge transfer from scientific research to drug development in cancer diseases. In academia and industry, all parties support the idea that new drug developments rely on the improvement of scientific research. However, how scientific research can be transferred into drug development is still being discussed. Some scholars support that scientific organisations should participate in drug development directly, as their scientific research helps them better understand pre-clinical results and match the patient conditions with in vitro tests (Van Dongen et al., 2017; Haeussler & Assmus, 2021). However, others believe publishing scientific research and developing drugs both require a great deal of effort, and that organisations do not have enough resources to cover both areas well (Du et al., 2021). Thus, this investigation will explore to what degree the basicness and the scientific impact of research influence organisations to engage in drug development and their effect on its success. Given the limitations of knowledge and resources, some organisations prefer to co-operate with others to develop drugs jointly rather than in isolation. Knowledge spillovers can be generated via co-operation activities (Hájek & Stejskal, 2018; De Noni et al., 2018). According to Smith (1994), the definition of knowledge spillover is the process of knowledge transfers from producers (knowledge sources) to users (knowledge receivers) through sharing, interaction, and the exchange of knowledge. In a co-operation network, knowledge spillover is always produced as a phenomenon in which the existing research efforts of co-operators may allow a given organisation to achieve results involving less research effort on their part than they would otherwise require (Jaffe, 1986). In this paper, research co-operation networks are built to observe and analyse the knowledge spillover through co-operative relationships. However, the efficiency of knowledge spillover is different with regard to basic and to applied knowledge, and it is still not clear whether the scientific impact of research can effectively spill over through the co-operation network. Thus, this paper also explore the spillover effects of basicness and the scientific impact of research on the success of drug development in the co-operation network 2. Method Since cancer diseases are very dangerous and the efficiency and success rate of cancer drug development are remarkably low, it was decided to choose two typical cancer drugs for the sample: alkylating and immunological cancer drugs. Variables were built with reference to the process of drug development through information from publications, clinical trials and FDA-approved drug products in oncology. Publications data was collected from PubMed, since it is the optimal publications database in the biomedical field (Falagas et al., 2007). The U.S. Clinical Trials Registry was chosen due to collect clinical trials data. For the purposes of this study, only the clinical trials from 2005 to 2018 were considered. The final database contains 104 drugs, with 250,257 publications and 14,345 clinical trials. After matching publications and clinical trials, there were 29,723 organisations with publications, of whom 832 had develop clinical trials in alkylating cancer drugs and (or) immunological cancer drugs. The co-operation networks were built according to the records of “Sponsor/Collaborators” in ClinicalTrials.gov. There are 591 organisations in clinical trials co-operation networks. There are two dependent variables, Engagement of drug development and Success of drug development. For a given publishing organisation, Engagement of drug development means the organisations decide to develop clinical trials. This takes a value of 1 if that organisation develops clinical trials, and 0 otherwise. The second dependent variable is Success of drug development, which is calculated by the number of FDA approved drugs or Phase IV clinical trials drugs. The first independent variable is Basicness. This is calculated according to the Triangle of Biomedicine, which maps PubMed papers onto a graph to determine the basicness of the organisations’ scientific research (Weber, 2013). The basicness of a given organisation is measured by the average of the basic scores of their publications. A higher score means it is more basic. The second independent variable is Impact. The scientific impact of an given organisation is calculated by the average of the Relative Citation Ratio (RCR) of publications. Other factors which influence the engagement and success of drug development are monitored: Publications are used to reflect the number of scientific research articles; Organisation Type to reflect the types of organisations; Country to reflect the location of organisations; Drug, Clinical Trials, Participants and Biomedical Percent to reflect the factors of clinical trials; Degree Centrality and Betweenness Centrality to reflect the characteristics of co-operation network; and NIH, Other U.S. Fed, Industry and Other Funding to reflect the funding sources of organisations in clinical trials development. Since the dependent variable, Engagement of drug development, is a dummy variable and there are only a few values equal to 1, rare events logistic regression was run to test the effect of basicness and scientific impact on the engagement of drug development. The zero-inflated negative binomial regression and spatial Durbin model were used to test the direct and spillover effect of basicness and scientific impact on the success of drug development. 3. Results The results show that although engagement in drug development is an important concern, only a few medical research organisations (less than 3%) actually engage in drug development. The lack of applied research is the reason behind this lack of engagement. Applied research fosters organisational engagement in drug development. In drug development process, basic research increases the success rate of drug development. The scientific impact of research not only stimulates the organisation into engaging in drug development, but also provides novel solutions to increase the success rate. In the co-operation network, applied research is easier to transfer and is more successfully exploited than basic research, thus the spillover effect of the basicness of scientific research is negative on the success of drug development. An efficient way to obtain frontier research from co-operators is through co-operation; thus, the spillover effect of the scientific impact of research is positive on the success of drug development. This investigation improves understanding about which scientific research — basic or applied — leads to engagement and the success of drug development in cancer diseases, and whether high-impact organisations also develop drugs. Methodologically, this study offers a new approach that overcomes the huge computational effort to empirically test these links, exploiting the spatial Durbin model to test the spillover effect in the co-operation network.

08:45
How does artificial intelligence contribute to public values? A large-scale analysis of AI patents

ABSTRACT. Background

New technologies, as they are applied and scaled-up, especially if they represent disruptive developments, invariably raise concerns about broader public implications, distributional effects, and governance. In recent years, attention has turned not just to the societal consequences of new technology deployment but also to the anticipation (and mitigation) of potential consequences in research, design, and early emergence phases. In parallel, frameworks for considering the implications of new technologies have evolved from expert technology assessment to more diverse (and arguably more encompassing) constructions such as public values mapping and responsible research and innovation.

Advances in analyzing scientific literature outputs and patent documents can now shed light on the emergence, significance, networks, and trajectories of rising topics in research, invention, and innovation. Building on this work, we add a new dimension by identifying and analyzing the public values expressed in patent documentation. We put forward an approach to recognize early signals as to the public values expressed in patents as a new technology emerges and scales. Resulting insights can contribute towards assessing the societal implications of technological development.

For empirical exploration, we focus on artificial intelligence (AI) - a domain that has received growing investment from the private and public sectors and has substantially increased its patenting activity in recent years. Science, technology, and innovation (STI) scholarship has drawn attention to ethical, privacy, bias, control, and distributional consequences of AI applications, as implemented by private sector organizations and governments. AI patenting studies have also examined relationships with business development, economic growth, regional innovation clusters or global innovation dynamics. However, the social sciences have yet to comprehensively probe the relationships between AI patenting and public values and how AI patents express public value claims.

Given this knowledge development opportunity, the paper addresses three research questions. First, what kinds of public values are embedded in AI-related patents? This involves delineation of the AI patenting domain, the conceptualization of public values, and the operationalization of an approach that can identify and contextualize public values within a patent text corpus. Second, how are public values expressed in patents distributed across time, geographies, and areas of AI applications? AI patenting has developed extensively over recent decades, spreading across countries, sectors, applications, and assignees. There is rising public debate about AI’s societal implications, with – varying by country and sector – policy interventions and the introduction of good practice codes. Public debate has also grown on sustainable development goals and global challenges. We anticipate that the public value rationales for AI inventions may in some ways evolve to reflect these public debates. Third, how do public values expressed in AI patenting speak to different dimensions of AI governance and policy? Here, we seek to distill insights and recommendations that can inform AI governance and policy, including implications for early anticipation and responsible innovation.

Methods

We use a peer-reviewed method combining a systematic keyword-based search with patent classification codes to select AI-related patents. The method is comprehensive in that it captures a range of approaches relevant to the domain, including deep learning, machine learning, supervised leaning, and other advanced techniques. We also conceptualize and operationalize an ontology of public values, building on available literature and prior work by research team members. To address the core research questions, we iteratively tune a BERT (Bidirectional Encoder Representations from Transformers) language model. This a machine learning method for natural language processing developed by Google and now in widespread use. We use BERT to classify USTPO AI patents (comprising patent applications and grants filed by US-based and non-US entities) according to whether and how public values are expressed. Text expressing public values can be found in patent background text, although it is also in other patent parts. An AI patent may express more than one set of public values, while not all AI patents express identifiable public values. A topic model identifies AI patents’ main public value themes (e.g., healthcare, privacy, discrimination, environment, etc.). Model performance is tested and iterated.

Results and significance

Research is advanced in identifying and classifying public values in US AI patents, drawing on patent records contained in PatentsView. We have developed a database of more than 100,000 US AI full-text patents records with application filing dates between 2005 and 2021. We have identified more than 4.6 million patent text sentences which potentially contain expressions of patent values. We anticipate completing the testing and refinement of the model by the end of 2022, with analysis of results in early 2023. The resulting consolidated database of public values expressed in US AI patents, along with other associated patent information (including filing year, award status, patent class, inventors and assignees, address information, forward and backward citations, and whether public funding is attributed) opens multiple opportunities for investigation. This includes addressing our core research questions related to what kinds of public values are expressed in AI patents and variations by time, sector, and inventor/assignee geographies, and policy implications. We will probe factors or motivations that might lead inventors to express or address public values in their inventions or to recognize social, legal, and ethical implications. One of these might be the presence of public funding; another could be the continuous v. discontinuous characteristics of the patent (using methods related to emergent word combinations), where we might expect the latter to be more associated with attention to public values.

While we anticipate that the approach presented in the paper will present a methodological advance, a key goal of the paper is also to advance frameworks for conceptualizing and operationalizing public values that may have broad applicability. While social scientific analysis of patents and patenting practices began to emerge in the 19th century, and began to significantly increase in the 1950s, much of the social sciences literature on patents is focused on their legal and organizational aspects, complemented by management and economics of innovation scholarship on business strategy and private economic value considerations. Nonetheless, the early 1970s did see the growth of articles looking at the implications of patents to society, with a focus on developing countries. It was argued that, until then, not enough was known regarding the distribution of benefits of patent systems. Critical concerns were raised about the benefits of patenting to the economies of developing countries. Social sciences scholarship on patenting grew more rapidly and diversified in focus from the 1990s, with recent increased interest in overlooked topics, such as the impacts of patents on human rights or gender inclusion. Scholarship more generally on public values in science and technology has also grown, although there has been only limited work on public values in AI patents. As well as providing responsible innovation and policy insights about how AI patents are expressing public values, we hope that the concepts and approach elaborated in the paper will stimulate debate and open new pathways for further work on the relationships between innovation in emerging technologies and public values.

09:00
Biomedical research and its propensity for spillover: a comparison of investigator initiated and NIH funder-initiated grants

ABSTRACT. Extended Abstract:

Context:

The allocation of funding across biomedical research in the US National Institutes of Health is a long-standing issue of policy interest. Previous analyses of this issue have often focused on the responsiveness of funding to disease burden. Critics of the agency’s priority setting process have repeatedly called for better alignment between funding and disease burden, and patient advocates for specific diseases for more funding for their causes.

In response, opponents of planning have argued that research in one area frequently leads to advances in others. Research can open up new lines of enquiry and may be particularly fertile in terms of spilling over into new fields; this may even turn out to bear relevance to unexpected disease burdens. There is limited empirical evidence informing these debates, and it has been difficult to gain estimates for the frequency and ubiquity of research yielding publications in unexpected fields.

One reason why is that, until relatively recently, there was no formal reporting categories for NIH funding. This changed with the 2008 “Research, Condition, and Disease Categorization” (RCDC) effort, through which the agency (on the request of Congress) developed standardized methodologies to classify funds by area (Sampat et al 2013). The NIH notes: “RCDC provides consistent and transparent information to the public about NIH-funded research. For the first time, a complete list of all NIH-funded projects related to each category is available” (NIH 2011). The NIH began with 212 categories starting in 2008, and currently has 293 covering a range of diseases, scientific fields, and other categories. NIH grants awarded during this period are linked to any of the categories they match, using NIH’s own algorithms, and this information is made available the public grant database NIH RePORTER.

RCDC provides categories for grants. To identify grant outputs that are not in the same categories as the funding inputs, we also need to measure and categorize the outputs. NIH RePORTER includes information on all publications resulting from its awards. However, these are not mapped to RCDC categories. Accordingly, we develop a machine learning algorithm that maps text to RCDC categories and apply the algorithm to the abstracts of all publications resulting from NIH grants.

Methods and empirical approach:

In this paper we provide new evidence to inform these debates, examining the extent to which grants in one set of broad scientific or disease areas lead to publications in others. We use the NIH’s “Research, Condition, and Disease Categorization” to identify categories for NIH grants awarded between 1985 and 2015. We develop a machine-learning model to map text to these categories and use this model to also categorize publications resulting from these grants.

We began by gathering all 895,981 NIH funded projects and abstracts from 2008 onwards, and lists of their resulting publications, from the NIH RePORTER database. RePORTER (through its ExPORTER tool) provides unique identifiers, PMIDs, for all publications resulting from NIH awards. Using PMIDs, publication abstracts were then gathered from NIH’s National Library of Medicine through PubMed. Our final dataset comprises 97,336 unique core projects and 954,755 unique publications attributed to those projects.

As noted, RCDC categories are assigned to NIH awards, not publications. In order to examine category crossing, we need to assign both awards and publications to the same categorization scheme. To link publications to RCDC categories, we applied machine learning algorithms, trained on NIH project abstracts and their RCDC assignments, which we then used to assign RCDC codes to publication abstracts. Using different vectorizers and parameters, we applied eleven algorithms. To evaluate their performance, we set aside ten percent of project abstracts - not used to train the algorithms - and compared the NIH’s assignment of RCDC to project abstracts with that of the algorithm. The performance of our selected algorithm compares favourably with other notable benchmarks.

After selecting our machine learning algorithm, we deployed the algorithm to assign RCDC categories to publication abstracts. This allowed us to see how frequently each RCDC category in each grant abstract occurs with each RCDC category in each of the corresponding publication abstracts. Moreover, this allowed us to explore the extent to which publications with RCDC categories not identified in their grants could be considered as spillovers. To assess this, a randomly drawn and blinded sample of grants and publications was manually assigned ICD-10 categories (International Classification of Diseases version 10) by two independent raters.

We then explore whether the propensity to yield such spillover publications varied on whether they emerge from RFA (request for applications) funder-initiated grants, or from R01 and other investigator-initiated grants. The ability to identify potentially highly fertile research areas, those that have a high propensity to yield publications in unexpected fields, may depend on whether they are initiated by the principal investigator, or a wider set of stakeholders co-ordinated by the funder.

Results and significance (so far and further anticipated findings):

We found grants in a given set of focal categories frequently generate publications in one or more different non focal categories. We anticipate being able to develop this into a dependent variable, ‘publications with unexpected categories’, and place it within a regression framework, either as a dichotomous dummy variable or as a continuous count variable.

To validate the dependent variable(s), our randomly drawn and blinded sample showed substantial corroboration with ICD, with few false positives and false negatives. That is, we find that unexpected publications (those with RCDC categories not assigned to their grants), are also similarly unexpected with ICD (those with ICD categories not assigned to their grants) – and vice versa.

Along with suitable controls, we will examine grant mechanisms as one of key independent variables of interest. Our preliminary results suggest that RFA funder-initiated grant mechanisms are at least as fertile as R01 investigator-initiated grants in generating outputs in unexpected categories, if not more so.

We expect our results will offer evidence on the frequency of spillovers, and on whether certain grant mechanisms may carry a greater propensity for spill over than others. Such results would relate to one of the central questions of the Atlanta 2023 conference “Should policies stoke innovation fires as the engines of growth or direct them to solve environmental challenges?” and could be significant for those interested in research targeting and research policy missions more generally.

09:15
Problematising the role of STI policy in agricultural innovation in developing countries: The case of South Africa

ABSTRACT. Background and rationale

Agricultural innovation policies are defined as those policies that are designed to improve a country’s agricultural innovation capabilities. In most countries, agricultural innovation is usually managed by several government agencies, but principally by those in charge of agriculture or science, technology and innovation. While Science Technology and Innovation (STI) policies at the national level may primarily focus on the establishment of an enabling environment for innovation in the country, sectoral agricultural policies on the other hand may focus on encouraging and helping farmers to enhance their outputs.

From a developing countries point of view, agriculture has been recognised as a sector that may contribute to economic growth by creating jobs, food security, and promoting the preservation of bio-diversity and scarce natural resources. While agriculture plays a significant role in the socio-economic developments of these countries, the sector’s contribution to the economic growth in Sub-Saharan Africa has largely stagnated and in some cases, significantly decreased in recent years due to a myriad of factors and challenges including, high and rising input costs, lack of support, droughts caused by climate change, water scarcity, crop or livestock diseases, and global competition. For example, in South Africa, agriculture’s contribution to the GDP has decreased from a total share of 7.7% of gross domestic product (GDP) in 1969, to approximately 3% every year in the last ten years (DAS, 2012). According to the OECD, South Africa’s agriculture sector is among the least supported in the world. Moreover, the OECD estimates that South Africa’s Producer Support Estimate is currently 3,2%, versus 4,6% for Brazil, 7,1% for the US (OECD, 2021). A recent FAO report, maintains that agriculture and aquaculture sectors will need to innovate not just to enhance the efficiency with which inputs are converted into outputs, but also to conserve precious natural resources and minimize waste in order to adapt to future global issues (FAO, 2017 ).

Realising these problems, many sub-Saharan developing countries including South Africa, have moved swiftly to make a number of policy adjustments, notably in the areas of agriculture and innovation. For example, one of the key objectives and justification of South Africa’s innovation policy has been the boosting of declining economic growth through innovation and the increase of productivity as well as competitiveness. More specifically, the new government white paper on Science, Technology and Innovation (DSI, 2019) as well as the decadal plan both outline important innovation policy priorities which puts innovation at the centre of seeking solutions to South Africa’s main pressing challenges of poverty, rising inequality, unemployment and economic growth.

Spielman and Birner argue that innovation policies are typically classified into three types, namely: (a) policies aimed at establishing and strengthening the formal structures and institutions required to collect and apply new and existing knowledge; (b) policies that encourage and enable the development of new innovations among farmers and other stakeholders; and (c) policies that incorporate and coordinate governmental, private businesses, and other stakeholders that are involved in innovation processes. Given that agricultural innovation has grown more intertwined with national innovation priorities in recent years, it has naturally become vital to integrate it into national innovation programs. Improved national and sectoral policy coordination between agricultural and innovation policy priorities can help policymakers close gaps and focus on sectoral challenges that are usually overlooked in both national STI and agriculture policy circles.

Objectives

The objective of this study was to investigate how policy problems in national STI policies and sectoral agricultural policies have been defined in order to stimulate innovation in the agricultural sector, to enable agricultural businesses to participate in the innovation process. In addition, the study examined the assumptions behind the policy measures offered in the STI policies in order to identify asymmetries in agricultural and STI policy coordination. An efficiently coordinated agricultural innovation policy would ensure that policy makers avoid making inaccurate assumptions about important agricultural policy priorities which may prove to be costly, and may inadvertently result in inefficient policy instruments that do not address the fundamental needs of agricultural sector stakeholders. As a result, the purpose of this study is to try to bridge this knowledge gap.

Methods This study used the critical discourse problematization framework (CDPF), an approach which merges two complimentary techniques to policy analysis, namely: (a) the critical discourse analysis and (b) policy problematization approach to problematise agricultural innovation in existing innovation policies. Policy problematization can be defined as the process of breaking down specific policy issues into problems that need to be solved. This approach recognizes that all policies have inherent problems and suggests that these problems have implications for how citizens are treated by governments and how they are conditioned to understand the social world and their roles as citizens. The main goal of studying problematizations, therefore according to Bacchi, (2012, p. 2) “... is to dismantle taken-for-granted fixed essences and show how they have come to be”. While traditional policy analysis techniques use quantitative methods to seek for causal explanations for why policies succeed or fail, on the contrary, qualitative discourse approaches look to analyse policy in terms of how dominant narratives emerge and come to define as well as shape policy practices. As such, to investigate the research problem, the study took a qualitative research approach, while using secondary data sources including key copies of the national innovation policy documents, national development plans and agricultural policies. This method was commensurate with other policy analysis research studies that employed secondary data (see Bacchi, 2015; Makoza 2013; Midgley, 2013). The documents were sourced from the South African Government departments of Science and Innovation as well as the department of agricultural land reform and rural development (DLDR). The researchers purposefully selected materials that were (a) related to the national STI policy and national development (b) produced as part of policy development process. Additionally, the set of documents used for the analysis enabled to examine the discourses, practices, and contextual settings that shaped the development of the problems that the national innovation and agriculture policies aimed to solve (Bacchi, 2009; 2011; 2015).

Contributions and implications

This study contributes to the debates on agricultural innovation policy by shedding light on the importance of policy coordination between national STI policies and sectoral agricultural policies in developing countries. It uses the case of South Africa’s experiences with problematization of the role of STI policy in agricultural innovation to further show how problematization may be used to identify asymmetries in priorities and policy coordination issues that hinder agricultural innovation. These coordination issues may hinder the development of appropriate agricultural innovation policy instruments intended to stimulate innovation and address challenges faced by the agricultural sector.

10:30-12:00 Session 12A: Challenges in Innovation Policy
10:30
Covid-19 and Beyond: Arguments for a New Innovation Policy

ABSTRACT. Background and rationale The Covid-19 pandemic has shown the vulnerability of nations, global institutions, and humanity at large to cope with this disease. It has underscored the need for a Great Reset that calls for a revisit by nations, the global institutions of the premise and assumptions of the growth discourse, and the translations based on them. This need for a reseat has long been echoed by low and middle-income economies in their struggle for addressing developmental challenges such as health, and climate change, for achieving the target set for SDG goals, etc. The pandemic has shown the vulnerability of coping with the pandemic even in OECD economies and exposed various fault lines in these economies. A large consensus is emerging that contemporary growth models which underpin contemporary policy framing are not resilient enough to cope with the various shocks. New innovative approaches during the pandemic have played a major role in fighting the pandemic against adverse circumstances. They have challenged the dominant models of innovation which are centered on growth models. The paper is motivated by the issues raised above to look more closely into the research and innovation systems. Primarily the paper centers on to what extent the conceptual/theoretical framework that guides present policy discourse and implementation needs a revisit. The paper draws from these to make a more informed argument, suggestive pathways in the context of innovation and innovation policy Method The paper is exploratory research drawing from a large body of scholarly work across different strands of research within innovation and STS, and development studies. It has selectively drawn from a huge body of literature covering COVID-19 and past-pandemic discourse. The challenges of climate change and green growth, and issues of technology transfer therein, were also examined to see to what extent they echo the struggles and challenges faced in containing the pandemic. Results A ‘New Normal’ emerged for fighting the disease. Collective global response, local networks, frugal innovations, open sharing, exploring new daring pathways, novel public-private partnerships, etc. that in some ways the pandemic compelled to take, provided solutions and hope to mitigate the severe impact of this pandemic. Research became more OPEN with the global sharing of research findings, genome sequencing, data, tools, and techniques which gave leads for drug/vaccine development. Many innovations happened in the face of institutional voids and resource constraints (frugal innovations) that were characterized by being low-cost, affordable, and ‘know-how’ that is easily transferable and replicable demonstrating an alternative pathway to prevention and costly healthcare. Face masks, sanitizers, and PPE kits were made from reuse of existing material, make-shift hospitals, containment zones, low-cost oxygen cylinders, and ventilators that were not as sophisticated but could provide critical make-shift support that became life savers in the face of the weak health infrastructure. The development of Covid-19 vaccine/drug was more complex and difficult due to high levels of uncertainty surrounding the virus i.e. its effect on the human body, its rapid mutation, pathways it chose to enter and target different functions, etc. Inspite of these challenges, different treatment options, and vaccines were available within a year or so which in normal development takes between 10 to 15 years. This was possible due to various novel strategies and radically different approaches adopted. Repurposing Strategy “old drugs, new use”, Global coordinated among them, the WHO ‘Solidarity Trial’ and Access to Covid-19 tools (ACT) accelerator and new platforms for drug/vaccine development, the proactive regulatory system can be cited as key enablers in bringing new vaccines within the unimaginable time frame. Discussion and Conclusion The covid-19 exposed various fault lines. However, drawing from the stories of resilience, unprecedented response, and some extraordinary moves from countries to fight this pandemic have also suggested new ways forward. Many of these were transient measures to address the alarming challenges. Frugal solutions are more accurately felt now not only in Global South but also in the Global North. The pandemic also highlighted the need to move towards a more inclusive innovation system framework, which would be open to incorporating local expertise and informal knowledge systems beyond the actors and institutions of the formal economy. As spaces of frugal innovations vary widely and are of variable quality, their adoption and scalability call for innovation systems to strengthen these frugal innovations and create institutional mechanisms for supporting them. The innovation systems on the other hand need to see that adherence to formal standards should not lead to eroding the essential essence of these frugal innovations. Greater attention is also needed for other forms of innovation, such as user-led and open innovation (see e.g. von Hippel, 2005). Innovation Systems (IS) approach has given a more nuanced understanding of factors that leads to the failure of the research-innovation ecosystem that hinders the creation, translation, and diffusion of technologies. The attention to different forms of failures in IS approach is distinguished under ‘system failure’ (Woolthuis et al. 2005) and augments the ‘market-failure’ rationale for policy intervention. Infrastructure, institutional, network, and capability failure are the key domains covered under System failure. Many instances were observed wherein the STI ecosystem could not perform due to these failures. Many other types of failures were observed which have been pointed out by Weber and Rohracher (2012) namely Directionality, Demand articulation, Policy coordination, and Reflexivity failure that could be seen impeding the institutions effectively during the shock exhibited by the pandemic. Informal innovations have been largely neglected within the IS framework. Greater attention is also needed to other forms of innovation, such as user-led and open innovation (von Hippel, 2005). The post-pandemic transformation calls for rethinking development more broadly. Long-term transformative change in our industrial society needs to expand the criteria of “success” in economic development beyond growth and profit to include inclusiveness, equity, trust, and sustainability. The important role of public funding behind successful Covid-19 vaccine development has largely remained invisible (Cross et. al., 2021). This is also true of many vaccines and drugs in the market including blockbuster drugs. The voices that are suppressed when price reduction is demanded by activists, by low and middle-income economies in the access and equitable debate have high merits. Inflating the pricing of patented drugs which results in deadweight loss, costing of drug development which suppresses the huge public funding, equity and access are issues that are raised in these debates and calls for globally coordinated action plans and sensitivity of high-income economies to regulate the big drug MNCs which operate as a cartel to inflate prices. The covid-19 pandemic has shown the importance of ‘commons’ that calls for shared resources managed collectively in providing economic activity and livelihoods, mutuality, and building trust and solidarity. Amartya Sen (Sen, 1999) who had cautioned the uncritical acceptance of the notion of ‘growth identical to development’ makes one rethink the new policy framework which is just, equitable and inclusive and at the same time creates/promotes innovation and entrepreneurship.

10:45
Ethics, Justice, and Policy: On the Dangers of Innovation

ABSTRACT. Background and Significance “That’s not ethics! It’s innovation! And it’s dangerous!” These phrases, addressed by an engineer to a philosopher – me – have served as a kind of Zen koan, which means they call for a response. What was it that I, the philosopher, had said to provoke such a reaction? I had talked about going beyond merely following rules, codes, or laws and making ethical decisions rooted in one’s autonomy. How else could one decide, after all, if one did not own one’s decisions? Would simply following the rules not take the decision out of decision making? What sense of ‘innovation’ underlies the contrast with ethics, conceived as rule following? Must all innovation violate a rule or rules? And what danger comes with suggesting that people ought to make their own (and ownable) ethical decisions? Of course, one could make the wrong decision. Making wrong ethical decisions could be dangerous, too, in the sense that doing so could lead to other bad consequences. But making correct ethical decisions also often means facing danger. Whistleblowers, to choose but one example, often face reprisals, even if they do everything right. We could argue that anyone who punishes whistleblowers is themselves wrong for doing so; but that hardly removes the danger that accompanies whistleblowing. So, what is ethics? What is innovation? And what makes innovation so dangerous?

Methods This presentation addresses these questions by considering three cases.

1. NSF’s new TIP Directorate In 2022, the US National Science Foundation (NSF) established its first new directorate in 30 years – the Directorate for Technology, Innovation and Partnerships (TIP). According to NSF’s website, the point of the TIP Directorate is “maximizing NSF’s impact.”

"NSF has advanced the full spectrum of fundamental research and education in all fields of science, technology, engineering and mathematics, or STEM, for more than 70 years — from foundational, curiosity-driven research that has led to new knowledge about our world, to use-inspired, solution-oriented research that has directly impacted people's everyday lives. At every stage, investments across this spectrum have been deeply intertwined.

NSF's TIP Directorate doubles down on the agency's commitment to support use-inspired research and the translation of research results to the market and society. In doing so, the new directorate strengthens the intense interplay between foundational and use-inspired work, enhancing the full cycle of discovery and innovation." (https://beta.nsf.gov/tip/latest, accessed 10/31/22)

In what sense is the new innovation directorate itself an innovation? How might it be dangerous? Is NSF wrong to double down on impact?

2. The Ethics of Community-engaged Research NSF is increasingly recognizing that community-engaged research – itself arguably an innovation – raises ethical concerns (https://www.nsf.gov/geo/opp/arctic/ace/community.jsp, accessed 10/31/22). Although Indigenous communities may have their own policies governing the ethics of community-engaged research, and some countries (such as Brazil) have laws governing researchers’ engagement with Indigenous communities, there is no policy governing the ethics of community-engaged research as a whole (i.e., beyond Indigenous communities). To put the point differently, there is nothing resembling the Common Rule, which governs the ethics of human subjects research, for community-engaged research. This section of the presentation discusses an effort to fill this gap.

3. Incorporating Social Justice into Engineering Ethics Education This section of the presentation discusses an effort to integrate considerations of social justice into an engineering ethics class. In collaboration with an engineer, the author has re-designed an engineering ethics class around the idea of social justice to test whether doing so may allow us 1) to reorient the focus from professionalism to include social and cultural impacts; and 2) to integrate discussions of ethics and social justice in novel ways through the engineering context. For reasons to be addressed in the presentation, ‘ethics’ and ‘social justice’ are treated as separate subfields within the discipline of philosophy. Although it suggests an innovative approach, perhaps bringing these subfields together is dangerous (for the students in the classes, for the instructors, and for society once engineers educated in this way are unleashed on society). Although all three cases relate to the idea that innovation is dangerous, this last case is the one most obviously related to the fear that innovation in (or in place of) ethics is dangerous.

Results All of these cases represent nascent examples of the interplay between ethics, justice, and innovation. We have yet to see how any of them play out. For this reason, it is too early to report definitive results.

Conclusion To claim that innovation is dangerous and for that reason should not replace ethics misconstrues ethics, innovation, and danger. We should avoid the contrast between ‘safe’ ethics conceived as rule-following and ‘dangerous’ innovation conceived as rule-breaking. We should also avoid the contrast between ethics and innovation, since innovation within ethics is not only possible, but also desirable. Moreover, innovation can be ethical, as well as dangerous. In fact, we will be better off if innovations incorporate concern for ethics from the beginning.

11:00
Revisiting Research and Innovation Futures 10 years after: Devising policy insights against the backdrop of actual developments

ABSTRACT. Background and rationale Introduced in the year 2000, the European Research Area has become the overarching framework for research and innovation policy (R&I policy) in the European Union. In a nutshell, it aims to better harmonize national and EU-level R&I policy in order to avoid duplication and contribute to the formation of critical mass in research and innovation in Europe. Even if the support for the ERA framework has varied with the changing mandates of the European Commission, it was renewed and updated first in 2010 and most recently in 2020 with a view to the time horizon 2030. The future of R&I, of European R&I policy and of the European Research Area was also the focus of two forward-looking projects funded by the European Framework Programme for R&I between 2012 and 2014: RIF (Research and Innovation Futures 2030: From Explorative to Transformative Scenarios) and VERA (Visions of the European Research Area 2030). Both projects explored alternative scenarios of how practices, organizational forms and institutional frameworks of research and innovation might evolve up to a time horizon of 2030, and what implications these scenarios may entail for R&I policy. The focus of analysis differs somewhat between the two projects, with RIF putting more weight on the practice and organization of research and innovation in a global context, and VERA focusing on the practices, organization and governance of R&I at the European level, but both projects focused at the future of European R&I policy and the European Research Area in particular. In this paper, we analyse the findings of these two projects with a twofold intention. First of all, we want to compare in what regards the two projects cohere in terms of anticipating a departure from strongly held beliefs in what should characterize European R&I policy in the early 2010s, i.e. whether the projected future scenarios differ or resemble each other in terms of their underlying principles (what we call “key tenets”). Secondly, we compare the scenarios with what was actually decided in 2020 as the guiding orientations for the European Research Area in 2030, i.e. at the time horizon that the projects addressed.

Approach and method The methodology of the paper is based on four main elements. Starting point are the fundamental principles of European Research Areas policy (key tenets) in the early 2010s. As a second step, and using these key tenets as a reference point, a meta-analysis of the RIF and VERA scenarios was conducted in order to analyse whether the scenarios maintained or departed from these key tenets. The insights from both projects are used in combination in order to have a more reliable and comprehensive basis of emerging ideas in relation to the future of R&I (policy) in Europe. The third step consists of analyzing the most recent update of ERA policy targeting a time horizon of 2030, and again focused on the five key tenets of ERA policy (and on possible new ones). Finally, by comparing the findings of RIF and VERA with those of the actual evolution of ERA policy, we are able to assess whether the two foresight projects were able to anticipate the actual evolution of ERA policy as part of the portfolio of scenarios developed.

Results The first result of our approach consists of the identification of fives key tenets of ERA policy in the early 2010s: • Key tenet 1 “Scientific excellence is a central pillar on which European RTI policy builds and from which major long-term benefits are expected.” • Key tenet 2 “Public funding of basic and frontier research is justified by market failure arguments, and is not questioned.” • Key tenet 3 “(Academic) scientific knowledge is claiming primacy over other forms of knowledge production.” • Key tenet 4 “The integration and/or coordination of resources at European level are a pre-condition for effective and efficient ways of organizing research by avoiding duplication of efforts, concentrating on harmonized roadmaps, and ensuring critical mass.” • Key tenet 5 “The main purpose of R&I is to create jobs and growth (in the old industrial economy).” The analysis of the five (RIF), respectively four (VERA) scenarios indicate first of all, that the emerging changes and weak signals in the ways of doing and organizing research and innovation point to the possibility of much more transformative changes ahead. We conclude that we should not take current wisdom on the fundamental beliefs in what makes up good R&I at face value, but continuously explore and question whether these beliefs are still valid. Secondly, RIF and VERA are quite coherent in terms of the projections and underlying principles of what might characterise R&I in 2030. The scenarios from the two projects do not provide identical views on how the five key tenets might change in the future, but the majority of projected future features are similar. Moreover, three pairs of scenarios feature similar characteristics and implications. As the projects drew on similar conceptual foundations and applied different, but mutually compatible methods for scenario-building, finding this congruence of internal scenario logics supports the overall consistency and validity of the two scenario sets. As regards the most recent revision of ERA Policy 2030, several of its novel elements were already identified in at least one of the two foresight projects. This also underlines the usefulness of foresight in anticipating sensible futures, even if they were regarded as “unrealistic” at the time of development. We also look into how the results of the RIF and VERA were taken into account in the design of ERA policy 2030, which would support the ambition of the ERA Policy 2030. In parallel, looking at the fundamental principles ("key tenets"), which remained untouched in the policy development so far, can give focus to policy discussions today. These principles should be revisited whether they are indeed still "things we can take for granted" or would need political attention and policy change as well.

Significance of findings The paper provides two significant insights into the anticipatory quality of the scenario development approaches used: (1) Scenario approaches with scenarios co-created by stakeholders and experts are suitable to develop disruptive future projections that question fundamental beliefs (i.e. they can go well beyond incremental change); (2) While disruptive projections in scenarios shall serve at the time of scenario-making mainly as tools to challenge current thinking and implicit assumptions, and thus do not need to be realistic, from a hindsight perspective as we take it here, it is relevant to know how future projections have materialized in order to assess the further ambitions of policy development and identify open issues that still should be on the political menu.

11:15
Innovation policy making from the practitioner's perspective – A morphological design process for policy instruments – The case of Germany

ABSTRACT. Background and rationale

Tackling grand societal challenges requires new conceptual foundations encouraging a debate on the need for modified innovation policy approaches. While the discussion on mission-oriented innovation policy originated as an observation of an empirical phenomenon, only a few theoretical underpinnings have been undertaken so far. Some research focuses on the definition and operationalisation of missions and objectives of policy-making. Other scholars, including our previous research, attempt to improve the analysis of policy instrument mixes and their agile coordination, implementation and adaption in the course of dynamic innovation processes. Missions and associated failures are often rather broadly conceptualised and have only limited insights into the instrument design processes of political practitioners. A practice-oriented conceptualisation of types of innovation policies needs to encompass more analytical dimensions of importance for the policy design. Such policy intervention points are particular areas in the socio-technical system or its environment along with the dynamics of transformation processes (such as the stimulation of niches or regime destabilisation). At these points, suitable policies are supposed to facilitate or support transformative change (Kanger et al., 2020). However, a situation analysis is needed for innovation policy practitioners, which requires identifying and prioritising technologies, relevant target groups, and specific functional strengths (or problems). In that way, the strategic directionalities and the relevant operative knowledge are provided as a basis for the instrument design processes and policy mix orchestration.

Our previous research on the agility of innovation policy has shown that successful instrument selection and design processes depend highly on clear and operationalized strategic objectives (Weber et al., 2021). To guarantee the practicability of recommended policy instruments, policies need to be developed in the context of existing policy mixes and to include the perspective of polity (Which innovation policy actors and structures of the innovation policy system are concerned?) and the required dynamic adjustment processes of policy instruments (politics) to guarantee necessary adaptions to the dynamics of the addressed innovation processes.

Research questions and objectives

The proposed research is developed within the "Innovation Policy Orchestra" project funded by the German Ministry for Research and Education. It aims at considering the complexity of design features and implementation practices of innovation policy instruments, including the role of the implementing policy actors and the dynamics of policy processes for adjustment and orchestration of instruments in the course of the addressed innovation processes. The project addresses the following research questions: • What kinds of analytical dimensions does the literature discuss for a situation analysis of policy intervention points in innovation systems, processes and mission-oriented socio-technical transformation? • What kinds of policy instruments and their specific design features can be identified to develop practical policy recommendations addressing the results of the situation analysis? • How can the situation and the instrument analysis be combined into a morphological design process that includes the orchestration of existing and new policy instruments? The objectives of the research project are two-folded. For the practice of innovation policy, a comprehensive morphological policy design process shall be developed, such that innovation policy designers, for example, in ministries and executing agencies, have a viable method to operationalise innovation policy strategies and the instrument selection and design. For innovation policy research, developing a holistic innovation policy design process helps to understand the practical challenges of innovation policymakers, which have been discussed as a crucial research gap in the literature. Existing typologies of innovation policy instruments, such as supply- and demand-sided instruments or technology-specific and neutral instruments, provide only limited insights on the instrument selection, as not all possible instruments and design principles are included. Through the in-depth discussion on instruments and their impact mechanisms, we aim to develop a holistic overview of instruments and design features to improve policy design processes for scientific policy recommendations and for the practitioners of innovation policy-making. Though new trends, such as the mission orientation, have been discussed in the literature, there is still a lack of practicable policy recommendations to implement an agile mission-oriented innovation policy. However, such policy recommendations require considering the specific national contexts of the innovation policy system. Therefore, the objectives of the proposed research project focus on the case of Germany. To share experiences and discuss the relevance of our research for other countries, we are motivated to present our research at your conference.

Methods

The proposed project follows a qualitative case study research design. In the first step, two structured literature reviews were conducted to collect the criteria for the situation analysis and the possible instruments and design features for the instrument analysis. As the literature reviews have demonstrated important research gaps, notably for polity and politics, our research project uses semi-structured expert interviews to consider the practitioners' perspectives on innovation policy. We are currently conducting interviews with the German Ministries for Research and Education, Economic Affairs and Climate Protection and executing agencies. Together with relevant policy documents, the interviews are transcribed and coded in an iterative process such that evidence contributing to the research questions can be structured and added to the morphological design process.

Preliminary results and significance

For structuring the morphological design process, we follow Zwicky (1948) and others using matrices as a creativity tool to analyse and solve complex problems. The morphological approach facilitates decision-making by providing relevant analytical dimensions and their specifications. For example, the analytical dimension of "target group" is subject to different specifications, such as SMEs or research institutes, which can be selected to know what kind(s) of target groups are relevant to address by policy-making. Two matrices have been developed, one for the situation analysis and one for the instrument analysis. The interviews aim at collecting the tacit knowledge and experiences from policy practitioners in order to get to know tacit and unknown design criteria, practical decision rationales and aspects from the policy design and implementation process itself. This new empirical perspective adds to the existing conceptual debates on operationalising mission-oriented innovation policy-making. As a result, our instrument analysis considers the entire scope of innovation policy instruments and design features by assigning the different kinds of possible policy inputs (financial, guidance, information) to specific activities to be addressed by the instruments (physical installations/infrastructures, knowledge generation, capacity building, market formation, innovation system building). Both dimensions for the assignment of instruments are subject to specific design features, which we have collected from the literature and the qualitative interviews. The design features include relevant aspects of polity (Who of the innovation policy system is deciding and who is implementing an instrument? What kind of political structures and existing rules and regulations need to be considered?) and of politics (How are learning cycles based on monitoring and evaluation processes institutionalised?). We aim to finalise the interview analysis and the matrices in the spring of next year to present the results at the conference.

References

Kanger et al. (2020): Six policy intervention points for sustainability transitions: A conceptual framework and a systematic literature review. Research Policy 49(7), 104072. Weber et al. (2021): Agilität in der F&I-Politik. Konzept, Definition, Operationalisierung. Study on the German Innovation System 8-21 for the German Expert Commission for Research and Innovation. Zwicky (1948): Morphological astronomy. The observatory 68, 121-143.

10:30-12:00 Session 12B: Rethinking National Innovation Policy Frameworks
10:30
Spheres of the context and institutional framework as conditioning factors for the accumulation of technological capabilities

ABSTRACT. Background and rationale There is already a wide literature on processes of accumulation of technological capabilities (ATC) at company and national level. This literature has been enriched by evidence collected through the use of qualitative and quantitative methodologies in many countries, sectors and types of company. Technological capabilities are understood as abilities to make effective use of technological knowledge, and reflect the mastery that companies have of technological activities. These capacities differ between companies and are at the base of their innovative activity and competitiveness. Companies build technological capabilities gradually, over time, through learning processes. The literature on developing countries and emerging economies shows an advance in knowledge about the nature of learning, its characteristics and evolution processes. In these countries, companies tend to adopt adaptive TC strategies, instead of strategies aimed at leading processes to move the technological frontier based on R&D activities. Progress has been made in the characterization of the innovation systems in these countries and in the identification of their weaknesses (Dutrénit and Sutz, 2014; Cassiolato, Lastres and Maciel, 2003), as well as in the analysis of how the instability of the environment in the which companies compete influences their decisions (Katz, 1985; Freeman, 1995, 2001; Vera-Cruz, 2004, 2006; Arza, 2007, 2013; Dutrénit et al., 2019, 2021). However, there are gaps in knowledge about the impact that a set of spheres of the context (economic, environmental, scientific, technological and sociopolitical) may have on these accumulation processes, as well as the mechanisms used in decision making by companies in the face of changes in these spheres. Likewise, the institutional framework also influences the decisions of companies (Lall, 1992; Campbell and Pedersen, 2007; Hitt, 2016), but little is known about how institutions affect ATC decisions. The objective of this document is to explore the incidence of the economic, environmental, cultural, scientific and technological, including science, technology and innovation policy, and sociopolitical spheres, as well as the institutional framework, on the ATC processes, and propose an analytical framework for the analysis of this relationship. We start from a critical review of the theoretical perspectives of the National Innovation System from developing countries, the ATC and the role of the context in it, the contributions on subsystems of society by Freeman (1995) and institutional studies. Two fundamental premises are raised. First, business agents make decisions individually or collectively to ATC, this accumulation is divergent between companies and productive sectors and implies a gradual process of change in which learning plays a crucial role. Second, the decisions of business agents are influenced by the different spheres, the institutional framework and its specific institutions, and the decisions of other agents. The degree of complementarity, coupling, maturity and quality of the institutions affects the processes and degrees of ATC. Some agents will have more conditioning than others and this affects their behavior and decision making regarding ATC.

Methods This paper reviews the existing literature and proposes an analytical and conceptual framework to analyze the incidence of different spheres of the context on ATC processes. This analytical framework will be the basis for the empirical work, based on case studies of Mexican companies.

Results or anticipated results The main result is an analytical proposal on the incidence of various spheres of the context, such as economic, environmental, cultural, scientific and technological, including science, technology and innovation policies, and socio-political, as well as the institutional framework, on the ATC processes, and propose an analytical framework for exploring this relationship. On the one hand, from the perspective of the NSI, important analytical categories are established on the processes and activities in the creation, dissemination and use of knowledge, the articulation between the different agents involved (such as higher education institutions, companies, government agencies, among others), learning processes, public policy instruments, and the incentive structure, among other specific categories. Analyzes of the role of the context in ATC processes and Freeman's (1985) analysis guide us more specifically to consider analytical categories such as context, institutions, system scales, and importantly spheres or subsystems. Freeman argues that these spheres have relative autonomy and economic growth and development depend on them; this paper explores their influence on ATC processes. On the other hand, from the perspective of institutional studies, analytical categories such as formal and informal rules, laws, contracts, property rights, institutional arrangements, cooperation and coordination make it possible to explain the processes of change and institutional evolution. This institutional framework contains three different types of institutions: the norms of the companies, the relative rules of the NSI and the meta-rules that emanate from the spheres, which enable or restrict the decisions of ATC.

Significance The main result is an analytical proposal on the incidence of various spheres of the context, such as economic, environmental, cultural, scientific and technological, including science, technology and innovation policies, and socio-political, as well as the institutional framework, on the ATC processes, and propose an analytical framework for exploring this relationship. Usually there has been a tendency to analyze ATC processes with little attention to the influence of different spheres of the context on these processes. Some works have analyzed the role of some spheres, especially the economic one, in a general way or through some variables (e.g. interest rate or the exchange rate), but our knowledge about this is very partial. That limits the recommendations we make for STI policy. Defining more clearly the influence of different spheres of the context on the ATC processes allows us to rethink the recommendations for the STI policy, and for other areas of public policy.

10:45
Technology Sovereignty: empirical implementations of a conceptual framework

ABSTRACT. Technology Sovereignty: empirical implementations of a conceptual framework

Background The global economy is in crisis mode due to the pandemic, the Ukraine war and the energy crisis. Even before that, an ever growing shift in the global economic focus already begun in the early 2000s, particularly in the area of research- and knowledge-intensive products as a result not only of the globalization of markets, but also the globalization of knowledge and supply chains. Supply chains could no longer be regarded as secure and dependencies were questioned. Protectionist tendencies in numerous countries, a politicization of the global economy - not least as a result of a newly evoked system competition between China and the "Global West" - as well as a specialization of individual countries resulting, among other things, from cost considerations, which in turn has led to major dependencies in individual technologies, had already presented many companies with major challenges before the current crises. The multitude of challenges at the business level aggregate into challenges and starting points for innovation and economic policy action at the macroeconomic level or at the level of sectors. The question of technological sovereignty - defined as the preservation of options for action based on one's own competencies and stable networks - is of particular importance against the backdrop of the above-mentioned challenges. However, internationally integrated supply, production and ultimately innovation chains have created dependencies and links which, against the backdrop of the economic and political challenges mentioned above, can restrict or even jeopardize secure innovation processes and thus competitiveness. Changing markets and new competitors also have an impact on both technological sovereignty and competitiveness, and thus ultimately also on most countries' security of supply.

Concept With its policy paper , Fraunhofer ISI had suggested a concept for Technology Sovereignty (TS) and a first empirical implementation for its monitoring. That paper defines Technology Sovereignty as the freedom to choose between different options instead of being directly or indirectly single-dependent. The definition makes a strict distinction to autarky and protectionism. It stresses the need for collaboration and exchange as well as knowledge and competence building for securing a country's technological sovereignty. In a second - scientific - paper , we offered a theoretical underpinning derived from economics and sociology, when we emphasized Technology Sovereignty as being defined by agency at the state-level (government action). This makes TS subject to policy action, required to be taken by government, industry and science in a country alike. The theoretical and the first empirical implementations of the concept revealed, however, a need for fine-grained (at the technological level) demarcations of crucial inputs (natural resources, pre-products, system components) at the level of technologies or technological system. This paper aims to suggest an empirical approach that will, in addition, be implemented for a set of selected technologies, building on networks and collaboration structures of national economic/innovation systems. With reference to the basic concept for defining, measuring and evaluating technological sovereignty presented by Fraunhofer ISI, this paper compiles and evaluates key statistical indicators for describing and assessing technological sovereignty versus dependency.

Methods To be able to enter at the level of technologies, IPR data (patents, trademarks) are used as one of the main data sources. The development or stock of competencies in key technology fields is an essential building block in securing the technological sovereignty of nations. Hence, bibliometric analyses - i.e. not only the counting of scientific journal publications, but also the use of citations and thus an assessment of the visibility and relevance of corresponding contributions - provide an indication of the intensity and quality of knowledge generation. In addition, the feasibility of a selective analysis of trade data is examined. While patents can be used to implement the delimitation of specific technologies in a fairly targeted manner, it is generally necessary to switch to more highly aggregated product groups (e.g. 'vaccines' in their entirety) in the case of trade data. We develop a technology-specific set of networks at the level of companies and research institutions with the help of co-patents, co-publications, and trademarks. For this purpose, we will resort to technology definitions that can be demarcated for all these indicator dimensions. In a next step, we aggregate each of these networks to the level of countries. At this level, we will also take into account the ownership of companies from BvD Orbis as well as inventor versus assignee country of patents, so that a potential international command of national technological competences can be modelled as well. The resulting networks will be merged in a multilayer network analysis. This will be set at the technology or at least technology field level, allowing a focus on essential technologies. The goal is to determine the extent to which countries have control over and/or access to particular technologies. With this multi-layer network analysis, we hope to be able to work out who collaborates with whom (and who does not) and also what the ownership relationships are. Inputs (value chains) cannot be directly mapped in this way, but the linkages between countries (based on links of companies and research institutions) are able to show technological collaboration.

Results As the project is still in a very early stage, comprehensive results are not yet available. We expect this analysis, however, to provide better insights into the interdependencies and also the network density itself. If there are many unconnected players in a technology field, then there are at least potential alternatives in the global network - hence a larger freedom of choice and room for policy action. From bibliometric and patent data analyses we already learned that international collaboration and exchange has increased in the past years. These kind of analyses, however, have employed individual indicators only or analyzed them separately, while an integrated assessment of a country's Technology Sovereignty based on multiple dimensions has not been attempted in this way, to the best of our knowledge.

11:00
Obstacles to Innovate and the Role of Public Funding: Evidence from Chile

ABSTRACT. Background and rationale Public funding for Science, Technology and Innovation (STI) is generally assessed in terms of its pertinence for the successful termination of inventive outcomes. Nonetheless, its projected influence on other key factors that might also shape innovation performance across companies remains largely unexplored. These include the role played by such subsidies in stimulating likelihood to innovate and even in fostering a higher resilience to sort out those financial and non-financial obstacles usually encountered by firms along the road. The primary purpose of this paper is to fill such existing gap.

We therefore study firm-level elements configuring propensity to engage on innovation strategies (product and process innovation), together with the adaptability of these in the presence of barriers to innovate, both for the case of firms subject to public funding as well as for those that are not. Chile’s Innovation System is herein referred as a case study due to its recent noteworthy performance as one of Latin America’s inventive hubs. The 9th to 11th module of the country’s innovation survey (spanning the 2013-2018 period) constitute our main source of micro information.

Methods Our econometric strategy is based upon a Multinomial Logistic Regression analysis (MLR). Two type of model specifications are derived from such approach: a baseline model solely accounting for innovation probability determinants, and a second multinomial regression where each of these latter elements are individually interacted with a given obstacle. In line with the MLR method, our dependent variable (inno_outcome) is built by classifying manufacturing firms into four categories that signal the execution of different innovation alternatives: (1) firms exclusively practicing product innovation; (2) those following process innovation only; (3) companies being able to implement both (top inventive firms), and (4) those other without either strategy. This latter fourth group of companies (labeled as potential innovators) are included in the form of a baseline sample within each of our econometric specifications.

Individual expenditures on intramural and extramural R&D, the use of intellectual property rights, the total amount of labor employed by the establishment (proxy for size), cooperation agreements with other firms along with the presence of scientific ties with research institutions comprise the set of independent variables explaining probability to innovate. Obstacle-related MLR estimators are then generated by interacting each of these with the following barriers: financial difficulties (lack of funding, prohibitive costs to innovate), knowledge disincentives (scant information about the availability of highly skilled personnel and/or latest technologies), network restrictions (failure to cooperate with other economic units) and, market obstacles (prevailing uncertainties with regard to future demand, market being dominated by well-established producers, etc.). Given our research objective, this empirically strategy is applied for two types of firms (Chilean firms with public subsidy and those without).

General results Alleviating size-related limitations to innovate, strengthening the relevance of intramural R&D funding and, encouraging the signing of more cooperation agreements with other firms (in spite of the presence of monetary and non-monetary barriers) represent the main supplementary benefits that stem from access to government support. While extramural R&D and intellectual property protection are found as critical and generally resilient elements to increase likelihood to innovate for the average inventive firm, our research notes that publicly supported (less innovative) companies fail to also utilize these two as pivotal instruments for the completion of scientific outcome (their respective regression coefficients are non-significant). Surprisingly enough, possibility to innovate (for all type of firms) seems to be reduced by the presence of scientific ties with research institutions. Such negative impact is neither drastically deepened by the presence of innovation barriers nor improved by public funds.

Policy implications These general results have strong implications for those developing economies seeking to evaluate the effectiveness of their innovation policy beyond mere indicators of R&D investment and their resulting output. Our empirical evidence illustrates additional indirect effects that can originate from granting public resources to less innovative companies. Alleviating barriers to size, nurturing cooperation skills coupled with the advancement of superior abilities to efficiently exploit intramural R&D investment, irrespective of obstacles of any sort, encompass some of the extra benefits that can be induced by government funding. Areas of innovation policy enhancement are also here put forth. To properly rely on intellectual property protection, extramural R&D as well as on collaborative projects with research institutions, less innovative companies require additional and tailored specific instruments of government support. Public funding, alone, does not seem to have a profound effect on these latter set of factors across Chilean innovators.

10:30-12:00 Session 12C: Informetric Methods
10:30
Quantifying science in policy : applying govscienceuseR research software to describe the science of Environmental Impact Statements

ABSTRACT. Motivation: In policy discourse, it is common to hear phrases such as “listen to the science” as a way for agencies to solve policy problems. But to build stronger connections between science and policy it is first necessary to understand current patterns of scientific information use in practice. How can we quantify the kinds of science that are used to support policy decisions? And what can these quantifications of science tell us about the process of policymaking?

Approach: This paper makes two contributions to research on the science-policy interface. Methodologically, we debut the open-source R package collection 'govscienceuseR' as a method for quantifying the scientific information referenced in policy documents. Policy documents are typically published as PDFs, and do not follow common structures or indexing rules. Thus, existing tools and software widely used for academic bibliometric analysis do not work for most policy documents.

In brief, govscienceuseR is a set of computational tools for extracting and indexing references from policy documents and other unstructured texts. These tools provide an accessible workflow for policy researchers to input documents and get an output of references classified into a general grouping (e.g. academic journal, government report, conference paper) and predicted matches to an index of journal articles and reports. Empirically, we run this tool using Environmental Impact Statement (EIS) documents filed between 2013 and 2020. Starting from the EPA’s ‘E-NEPA’ repository, we scraped project metadata and downloaded over 11,000 available documents prepared from 2013 to 2021.

Using the EIS documents, we implemented the protocol for extracting, classifying, and matching references defined in the govscienceuseR packages. First, using the referenceExtract package, we extracted potential references with the help of the anystyle.io bibliography parser. Using the referenceClassify package, we removed misidentified portions of text, identified which references refer to different publication sources, and disambiguated different tags that refer to identical features (e.g., PNAS and Proceedings of the National Academy of Sciences). Using this package we also employed a pre-trained (on a sample of extracted references) artificial neural network classifier to probabilistically tag references by category (academic journals, government agencies, and conference proceedings). Finally, using the referenceSearch package, we matched extracted references to records in a global bibliometric database (Open Alex, the successor to Microsoft Academic Graph) to disambiguate the references and link to metadata such as reference counts, journal attributions, and institutional sources.

Preliminary results: Across the 11,000+ documents and over 1.5 million extracted potential, govscienceuseR tools helped pare these data down to approximately 500,000 classified references. Of these, around ~13.5% are from an academic journal, while ~86.5% are from grey literature sources such as public agencies and policy research institutes. Our ongoing analysis will summarize the distribution of scientific references observed in policy documents on multiple dimensions including: (1) scientific prestige; (2) journal format and focus; (3) organizational producers; and (4) topics and methods. Answering these questions in the context of EIS projects, helps us better understand the kinds of science that are used to justify environmental policy decisions.

10:45
Informetric methods for studying the diversity of the scientific workforce: Towards a state-of-the-art

ABSTRACT. Background and rationale Evaluative processes struggle with the notion of diversity of a scientific workforce (Walsh et al., 2019). Despite overwhelming evidence on the need for diverse teams in terms of division of labor (Robinson-Garcia et al., 2020), ethnic mix (Freeman & Huang, 2015) or gender (Díaz-García et al., 2013; Maddi & Gingras, 2021) among others, bibliometric methods have traditionally been focused on the development of impact and productivity indicators. Recently, the development of machine learning algorithms, new data sources and strong calls for action favoring an open and diverse scientific ecosystem, have given room to a stream of studies focused on studying different aspects of career trajectories, diversity of profiles or biases in science. In this paper we attempt at reviewing recent advances in the development of novel informetric approaches and methods to study diversity in science. Specifically, we focus on those related to the scientific workforce.

This focus on diversity of the scientific workforce differentiates this work from other reviews focused on individual level indicators (Gauffriau, 2020; Wildgaard et al., 2014), as we deliberately ignore productivity or impact indicators to focus on methods and approaches aimed at characterizing individuals and the context in which they work. For this, we first review the changes with regard to data sources which have made possible such approaches. Here we focus on two specific aspects: 1) the expansion of author identifiers, and 2) the improvements made with regard to author name disambiguation algorithms. Then we revise the different proposals made by organizing these approaches into three groups: 1) informetric methods related to individuals’ characteristics (e..g, gender, age, mobility); 2) methods related to individuals’ context (e.g., career trajectory, social engagement); and 3) approaches related to team dynamics (e.g., author position, contribution statements). We then conclude by identifying the main gaps in the literature and pointing towards potential areas to explore. Data sources The development and expansion of individual level-metrics is closely linked to that of data sources. Their inclusion of metrics, the launch of author profiles and the improvement of the bibliographic metadata have expanded both the popularity of certain metrics and the possibilities for quantitatively studying individuals’ academic activities. Here we review the main milestones on how data providers have contributed to expand the use (and misuse) of individual level metrics (Haddow & Hammarfelt, 2019), while at the same time increasing the opportunities for more detailed and fine-grained analyses. We specifically focus on two phenomena that are key to understanding the renewed interest of scientometric studies on individuals: the expansion of author identifiers and the development of name disambiguation algorithms. Individual characteristics The context surrounding the researcher should be considered to offer a more holistic view of the individual level performance (Ràfols, 2019). This section discusses works which have studied intersecting individual categories (age, gender, ethnicity or national background, or cultural identity) influencing the academic career and their activities, aiming to clarify which indicators have been used to study the context and personal features of the researcher. Context Two factors have been essential for the study of individuals’ context. First, the development of author identifiers and name disambiguation algorithms. These have allowed the study of changes in the bibliographic metadata associated with publications of a single individual. These changes are monitored by studies using bibliometric methods to study career trajectories. For instance, we find studies focusing on geographic mobility (Moed et al., 2013), changes between sectors (Jurowetzki et al., 2021) or topic mobility (Yu et al., 2021). The second development has been the launch and expansion of new data sources. For instance, altmetric data allows mapping interactions of academics with other stakeholders (Robinson-Garcia et al., 2018), open access data allows the study of open practices (Ramos-Vielba et al., 2022), and author registries allow looking into individual funding (Costas et al., 2021). Team Dynamics There is an abundance of literature with regard to team dynamics. Since Derek de Solla Price published his seminal work ‘Little Science, Big Science’ (1963), a field has grown in relation with team science (Hall et al., 2018). Here we discuss on approaches related to individuals’ and their role when collaborating. We focus on four specific aspects: the use of author order, contribution statements, disciplinary differences and types of collaborations. Conclusions The present study represents a first attempt at compiling recent developments in the field of scientometrics with regard to the development of new approaches and methods to characterize the diversity of the scientific workforce. These new developments represent a unique opportunity to better study and understand how new scientific knowledge is produced from a sociological point of view. From an evaluative perspective, the present review has important implications, as it allows us to identify major methodological gaps and limitations when approaching research evaluation from a quantitative perspective. We conclude by suggesting an agenda for further development and potential areas of interest in which these methods can be applied. References Costas, R., Corona, C., & Robinson-Garcia, N. (forthcoming). Could ORCID play a key role in meta-research? Discussing new analytical possibilities to study the dynamics of science and scientists. In A. Oancea, G. E. Derrick, & N. Nuseibeh (Eds.), Handbook on Meta-Research (pp. 1–20). Elgar Publishing. Díaz-García, C., González-Moreno, A., & Jose Sáez-Martínez, F. (2013). Gender diversity within R&D teams: Its impact on radicalness of innovation. Innovation, 15(2), 149–160. https://doi.org/10.5172/impp.2013.15.2.149 Freeman, R. B., & Huang, W. (2015). Collaborating with People Like Me: Ethnic Co-authorship within the U.S. Journal of Labor Economics, 33(S1), S289–S318. https://doi.org/10.3386/w19905 Gauffriau, M. (2021). Counting methods introduced into the bibliometric research literature 1970 – 2018: A review. Quantitative Science Studies, 1–52. https://doi.org/10.1162/qss_a_00141 Haddow, G., & Hammarfelt, B. (2019). Quality, impact, and quantification: Indicators and metrics use by social scientists. Journal of the Association for Information Science and Technology, 70(1), 16–26. https://doi.org/10.1002/asi.24097 Hall, K. L., Vogel, A. L., Huang, G. C., Serrano, K. J., Rice, E. L., Tsakraklides, S. P., & Fiore, S. M. (2018). The science of team science: A review of the empirical evidence and research gaps on collaboration in science. American Psychologist, 73(4), 532–548. https://doi.org/10.1037/amp0000319 Jurowetzki, R., Hain, D., Mateos-Garcia, J., & Stathoulopoulos, K. (2021). The Privatization of AI Research(-ers): Causes and Potential Consequences -- From university-industry interaction to public research brain-drain? (arXiv:2102.01648). arXiv. https://doi.org/10.48550/arXiv.2102.01648 Maddi, A., & Gingras, Y. (2021). Gender Diversity in Research Teams and Citation Impact in Economics and Management. Journal of Economic Surveys, 35(5), 1381–1404. https://doi.org/10.1111/joes.12420 Moed, H. F., Aisati, M., & Plume, A. (2013). Studying scientific migration in Scopus. Scientometrics, 94(3), 929–942. https://doi.org/10.1007/s11192-012-0783-9 Price, D. J. de S. (1963). Little science, big science. Columbia University Press New York. Ràfols, I. (2019). S&T indicators in the wild: Contextualization and participation for responsible metrics. Research Evaluation, 28(1), 7–22. https://doi.org/10.1093/reseval/rvy030 Ramos-Vielba, I., Robinson-Garcia, N., & Woolley, R. (2022). A value creation model from science-society interconnections: Archetypal analysis combining publications, survey and altmetric data. PLOS ONE, 17(6), e0269004. https://doi.org/10.1371/journal.pone.0269004 Robinson-Garcia, N., Costas, R., Sugimoto, C. R., Larivière, V., & Nane, G. F. (2020). Task specialization across research careers. ELife, 9, e60586. https://doi.org/10.7554/eLife.60586 Robinson-Garcia, N., van Leeuwen, T. N., & Ràfols, I. (2018). Using altmetrics for contextualised mapping of societal impact: From hits to networks. Science and Public Policy, 45(6), 815–826. https://doi.org/10.1093/scipol/scy024 Walsh, J. P., Lee, Y.-N., & Tang, L. (2019). Pathogenic organization in science: Division of labor and retractions. Research Policy, 48(2), 444–461. https://doi.org/10.1016/j.respol.2018.09.004 Wildgaard, L., Schneider, J. W., & Larsen, B. (2014). A review of the characteristics of 108 author-level bibliometric indicators. Scientometrics, 101(1), 125–158. https://doi.org/10.1007/s11192-014-1423-3 Yu, X., Szymanski, B. K., & Jia, T. (2021). Become a better you: Correlation between the change of research direction and the change of scientific performance. Journal of Informetrics, 15(3), 101193. https://doi.org/10.1016/j.joi.2021.101193

11:00
Identifying Hot Topics Based on Export Activity

ABSTRACT. Background and Rationale: The identification of hot or emerging topics is of interest to a broad variety of stakeholders. As such, it is one of the primary drivers of bibliometric analyses, especially those that attempt to characterize research areas through clustering of document sets. Once clusters are obtained, the hot or emerging clusters are most often identified as those that are growing the fastest or those with the youngest average age. In other words, most current studies identify hot topics as a production-based phenomenon where such topics represent ‘areas of knowledge.’

Our previous work has also looked at the growth rate in the number of papers in a document cluster as an indicator of emergence. Using our comprehensive detailed global model of the research literature (Scopus, 52.9 million documents, 98,767 clusters) we improved upon a method to predict which clusters of papers would experience high growth (productivity) over a future 3-4-year period (Boyack & Klavans, 2022). Among the 81 independent variables tested – which included author-, semantic-, application-, network- and gender-based cluster level features – the best predictor was related to historical growth rate, a production-based variable.

More recently we have begun to investigate the citing and cited relationships between clusters. Other studies that examine citations between groups of documents have typically been framed as investigating ‘knowledge flow’ or ‘knowledge diffusion’ and have been done at a highly aggregated field or discipline level. This is consistent with the notion of document clusters as ‘areas of knowledge’. We view clusters of documents differently. To us, each detailed cluster represents a research problem, and each paper is contributing a micro-solution to the overall problem. However, some micro-solutions are used outside the cluster. When a paper from one cluster cites papers in another cluster it represents the exchange of a ‘micro-solution’ between clusters. We refer to this as the export of a micro-solution from the cited cluster to the citing cluster. Concurrently, it is also an import to the citing cluster from the cited cluster. For example, when deep learning algorithms first appeared, they heavily imported micro-solutions from research on neural networks. Once deep learning was established, it began exporting micro-solutions to a variety of applications, for example facial expression recognition and online fashion recommendation. When considered in bulk, these historical imports and exports of micro-solutions may provide insights into how a research problem evolves over time.

This has caused us to reflect on the definition of ‘hot topics.’ Are hot topics those that experience extreme growth in production (micro-solutions to the problem at hand), or are they those that have high degrees of exports to other topics? This study pursues the notion that it is just as important to predict topics that will have exceptional growth in export activity (as enablers of broad application) as those that will experience exceptional growth in production. In order to explore this issue further, we conduct two experiments that are described in the following sections.

Experiment #1: In the first analysis, we chose to use the 71 document clusters from our 2014 study (Small, Boyack, & Klavans, 2014) because they had been shown to be emergent and had been validated with external information. However, these document clusters were identified in an older model. We thus took the papers from those clusters and found them in a model used in a more recent study (Boyack & Klavans, 2022). In most (57/71) cases, at least 60% of the papers from the original set of emerging document clusters remained together in the new model. Thus, the clusters in the new model remain relatively true to the emergence identified in 2010. In several cases (e.g., for graphene oxide clusters), multiple of the original emerging clusters had merged in the new model.

We then looked at the publication trends of these 57 documents clusters using the method for characterizing imports and exports that were described in a third study (Klavans & Boyack, 2022). In that study, we defined an import or export link between clusters as cases in which a paper cites at least 4 papers in another cluster. These instances were summed over pairs of clusters to identify import and export patterns by year. We then calculated production, import and export shares for each cluster (in reference to the entire model). This process led to several observations. 1) Production activity peaked within several years of identification as an emergent cluster for all but five clusters. In roughly half of the cases the peak lasted a single year and was followed by a decline in production. In the other cases the peak time was flat for several years. 2) Import and production activity were roughly parallel and higher than export activity up until the year of emergence. 3) Export activity dramatically increased after the year of emergence and continued to grow at a very high rate until several years past the peak year.

The overall story here is that, in general, production grows along with import of micro-solutions from other clusters as a cluster is emerging. Once it has emerged, production growth continues but is dramatically outpaced by export of micro-solutions, especially once production in the cluster starts to tail off. This provides evidence that cluster exports continue to be important well after the time that a cluster would no longer be considered hot from a production standpoint, and that tracking and prediction of such exports may be very important from a decision-making perspective.

Experiment #2 To follow this up we are in process of reproducing the recent study in QSS (Boyack & Klavans, 2022) in which we published a method of predicting which clusters in a large-scale detailed model of the research literature would experience exceptional growth. However, in this experiment we will use export activity as the dependent variable rather than production activity. Dependent variables will be the same 81 features used in the previous study plus two new variables – historical trends in import and export activity for a document cluster.

Significance: We expect to find that the sets of clusters that achieve exceptional growth in production and in exports will be partially overlapping and partially unique. Examination of those clusters that are unique to the set of exceptionally growing exports will help us to know if we should be predicting exports in addition to or rather than production. Vetting of those clusters with expert will enable us to understand better the potential policy implications of exports.

References: Boyack, K. W., & Klavans, R. (2022). An improved practical approach to forecasting exceptional growth in research. Quantitative Science Studies. doi:10.1162/qss_a_00202 Klavans, R., & Boyack, K. W. (2022). Predicting trends in research using research community imports and exports. Paper presented at the 26th International Conference on Science, Technology and Innovation Indicators (STI 2022), Granada, Spain. Small, H., Boyack, K. W., & Klavans, R. (2014). Identifying emerging topics in science and technology. Research Policy, 43, 1450-1467. doi:10.1016/j.respol.2014.02.005

11:15
Identifying and Characterizing Translational Research Strategies

ABSTRACT. Rationale: How might one identify translational pathways and evaluate translational research strategies? A translational research strategy is, in essence, a story about how one plans to commit resources over time, and across many related research problems, with a specific goal in mind. Translation, from the Latin trans (across) and latus (to carry or bear), simply means to carry or bear something from one place to another. Translation is generally defined as the process by which initial observations are turned into tangible interventions and applications. Central to this definition is the concept of ‘process’. Translation is not about where something is carried from or where it is being carried to. Neither research topics, communities, nor problems are translational in and of themselves. Rather, translation is about the connections between topics, communities, and problems. Translational research is about the carrying across of (micro)solutions from one problem to another—from one community of researchers to another. A translational research strategy is therefore an exercise in pathfinding. From the perspective of strategy, a ‘good’ translational strategy is one where the organization has expertise in specific research problems and the micro-solutions that link these problems. In other words, the path is navigable. A ‘bad’ translational strategy is one where the problems are stated vaguely and/or there is no way to evaluate whether one has expertise in the relevant set of micro-solutions. This study proposes a systematic method by which translational pathways can be identified using a large-scale detailed model of the scientific literature. It also focuses on storytelling based on linkages as a means of evaluating potential translational pathways from a strategic perspective.

Background: Most bibliometric analyses focus on characterizing research areas. This is often done by identifying a (small) set of papers, clustering those papers, and characterizing the actors and content of the clusters. Relatively little work has focused on the linkages between clusters even though translation is inherently an inter-cluster phenomenon. Moreover, most of these studies have little chance of accurately characterizing translation simply because their data do not cover the entire potential translational pathway.

We overcome these difficulties by using a global model of science, one in which 19 million documents from PubMed have been grouped into 29,013 clusters. Each cluster represents a research problem to which a group of researchers is contributing micro-solutions. (No single study solves the research problem, but each study contributes a piece of the solution—a ‘micro-solution.’) The clusters represent intellectually distinct problems along the basic-to-applied spectrum. For instance, one cluster may represent a problem in understanding the role of R-loops in DNA damage while another addresses the role of silica exposure in lung cancer among coal workers. When a paper from one cluster cites papers in another cluster it represents the potential exchange of a ‘micro-solution’ between clusters. This is an example of what is ‘carried from one place to another’. When considered in bulk, we refer to this phenomenon as the importing and exporting of micro-solutions between sets of research problems.

Consider, for example, significant events in the history of the COVID-19 mRNA vaccine. Between the initial observation that started it all—the discovery of mRNA in 1961—and the emergency authorization of the mRNA vaccine in 2020, many different groups of researchers with many different backgrounds and goals contributed over many years. But while the story of the mRNA vaccine would not exist without them, it isn’t solely about them. It’s about those discoveries and micro-solutions carried from one community of researchers to another. Thousands of pathways or stories like this one have occurred across the research landscape. More importantly, many currently incomplete potential pathways exist. Progress in translational science can be accelerated and improved by identifying these potential pathways in a strategic manner.

Method and Results: We have identified one translational pathway of strategic importance to the University of Michigan Medical School (UMMS) by exploring the PubMed model of science manually. UMMS has a specific competency in the general topic of chronic pain. We started by identifying a cluster in the PubMed model on pain-related aversion – a basic science cluster in which UMMS has noted expertise. We then identified other clusters that heavily cite the pain-related aversion cluster, finding that the strongest linkage was to a cluster focused on the problem of pain anticipation. This cluster is also one in which UMMS has a strong presence. We then carried on this process for three additional steps, leading to a five-node pathway.

While we were able to easily ascertain the particular focus of each cluster, to tell the story of translation one must have a more detailed understanding of the topical focus of the links between clusters. We thus identified the subset of papers that comprise these citing pathways and summarized their content. For instance, the link from pain anticipation to pain-related aversion is based on 320 citing instances and can be summarized as “the anticipation of pain prioritizes awareness and attribution of somatosensory sensations as painful…”

From a strategic standpoint, this pathway—which has been validated by administrators at UMMS—is important because, while the institution currently has faculty expertise in the first four nodes in the pathway, it has almost no activity in the final (bedside) node. In other words, UMMS discoveries are being translated to tangible patient health impacts by other institutions, potentially depriving UMMS researchers of valuable insights into the clinical applications of their work and precluding them from the clinical collaborations that could make their future work more impactful. If UMMS wants to develop expertise along a complete pathway, it will need to add research in that area to its portfolio, either through recruitment or collaboration. This is key information for UMMS.

While this translational pathway was identified manually, our next step is to develop methods to reproduce these results algorithmically. Once done, we will apply the method to other areas, to identify additional translational pathways relevant to the UMMS strategic vision. These will be validated by experts in-house. The algorithmic method and additional results will be presented at the conference in May.

Significance: In summary, we argue that these intellectual links, which we can quantify through citations, represent dependency relationships. Research communities are formed around difficult to solve problems and these problems cannot be solved in intellectual isolation. These research communities therefore rely on the continuous exchange of micro-solutions, the record of which can be found in article references. These exchanges represent literal instances of translation: the carrying across of information from one community to another. Ensembles of these exchanges become translational pathways—a directed process network of dependency relationships that we call “translation”.

Significantly, by examining the papers that comprise these dependency relationships between communities, we can summarize and understand them, effectively describing the way in which solutions from one community are used by another. The entire translational network, if visualized with all the edges, would look like an impenetrable hairball. We would, in effect, be attempting to describe all of science. The methodological solution is therefore to distill the narrative by screening out nodes and links, both simplifying and strengthening it, so that effective translational strategies can be identified and characterized.

10:30-12:00 Session 12D: Private Sector Funding
10:30
Utilizing the national resources for financing transformation: experiences from four Latin American countries

ABSTRACT. Background and Rationale: The sustainable transformation of the economy is increasingly a relevant topic globally, including for the Global South. This topic has particular relevance to natural resource-rich countries because these countries are challenged with distinctive pressures globally and locally. These are 1) market pressure to produce more extractive resources (in particular, mineral resources required for renewables) by the Global North for the energy transformation; and 2) growing local tensions between the extraction of natural resources and local environmental sustainability and society. It is urgent to identify a new pathway toward their own transitions, leveraging natural resources.

The management of revenues from natural resources, in this context, plays a critical role in transforming the economies of resource-rich countries. Traditionally, the revenues--mineral royalties--had been levied and these were used to stabilize the macroeconomic fluctuations (e.g. stabilization fund, sovereign wealth fund, natural resource fund) or economically compensate the people who live in the localities (usually poor communities) where extraction activities take place. These revenues, however, were rarely invested actively for innovation or knowledge for transforming the existing structure of the economy.

In the public policy literature, the importance of the dynamic capability of the public sector had been identified as a critical factor in adapting society to the new context. The management of finance and how to invest in knowledge would also require such dynamic capability of the government, whichever level it is situated either at national or local. The existing literature attempt to classify the types of capabilities and actors necessary for this purpose; however, there are still limited cases from the Global South.

Several Latin American countries, on the other hand, had introduced policy that earmarks certain parts of natural resource revenues for science, technology, and innovation purposes. In this paper, we call these initiatives as Natural Resource for Knowledge Fund(NR4KF). The mechanism, in distinctive ways, is introduced with the purpose to strengthen knowledge and innovation capacities in the country, especially in remote regions. The cases we study in depth are Bolivia’s direct tax on hydrocarbons (IDH), Chile’s Innovation for Competitiveness Found (FIC), Colombia’s Science, Technology and Innovation fund (FCTel), and Peru’s Canon and Mining royalties. These mechanisms are introduced in distinctive moments between 2004-2012, allowing sufficient time to observe the outcomes of these initiatives.

The purpose of examining above mentioned Latin American cases are to identify the types and moments of policy interventions and their design that are needed to avoid various difficulties that are associated with natural resource revenues. We consider that studying the case of NR4KF in the four Latin American countries mentioned above would give some insights into the challenge of transformation in the Global South, especially for those with natural resources, and provide policy suggestions.

Methods: Building upon the literature on the management of natural resources, industrial policy, and public management, we developed a framework for the ideal design criteria for NR4KF. These consist of four static design criteria (clear statement of purpose, rule-based design, multiple stakeholder governance, and transparency), and three dynamic design criteria (monitoring and evaluation system, Institutional ma/managerial capacity building, coordination with external actors). This framework is used to compare four Latin American NR4KFs.

In order to understand the mechanism, procedure, and outcome of NR4KF in each country, the following information is collected and analyzed: 1) gray literature on legal and policy; 2) interviews with policymakers involved in the process, and 3) data on the flow of NR4KF and STI performance indicator. Point 3) was done to identify the short-term outcome of STI activities to indicate some aspects of transformation.

Results or anticipated results, significance:

This study compiled rather a unique set of data, the revenue flow to STI areas, connecting back the natural resources. The observation of complied data on the flow of NR4KF in relation to STI performance indicators (number of scientific publications, exported products) did not show strong relationships. This can be interpreted as the lack of impact in providing finance for transformation, or simply the time span in which the observation was made was not long enough to mark the difference.

As the quantitative measures to assess NR4KF fail to show a clear outcome of NR4KF, the qualitative comparative analysis was conducted to identify differences in mechanism in detail based on the NR4KF design criteria.

While the static design principle of NR4KF ensures consistency, and transparency, and diminishes the possibility of corruption, the rigid rule-based design hampered the effective use of the funds. For example, in Bolivia and Peru, due to the shortfall of capacities in research management, universities appear unable to cope with sudden increases in the inflow of research funds, especially lacking the means to convert them into research output. This problem is compounded by the fact that the funds cannot be used for salaries to hire researchers. The lack of flexibility in management practice and capacity to manage it effectively was especially noteworthy in the regions, where the funds are allocated. As the result, a higher proportion of funds is left unused and even if these are used, funds are used on items that may not effectively lead to knowledge outcomes leading to transformation (such as physical infrastructure on research i.e. ICT equipment and research facilities). This means that the dynamic capability of knowledge institutions needs to be built in tandem with the creation of the static rule-based design of the fund.

In contrast to the above, dynamic design criteria can be used to examine institutional abilities to adapt to the new “evolving “policy goals. Under this category, the striking finding across the cases is their absence. For instance, monitoring and evaluation systems are mentioned in the design but not implemented. Even with the system installed, the political cycle often interrupts projects preventing them from undergoing an adequate evaluation process to ensure policy learning. In other words, nurturing the dynamic capability of the state would require a well-functioning policy setting, a big challenge for the Global South.

This paper illustrated the capability of transformation requires the dynamic capability of public institutions, through a comparative analysis of NR4KF examples in four Latin American countries. The results would have the following implications: one is to show in detail both static and dynamic capabilities are necessary for the resource-rich Global South to develop to enable transformation; second, if transformation were to be implemented effectively, the capacity building at implementing agencies is critical prior to mobilizing large sum of finances and, the third, long term aim for policy should be combined with a short-term policy. This will provide the means to maintain a good balance between flexibility in rule-based design and space for dynamic interaction.

10:45
The Role of Speedier Public Funding for Small High-Tech Firmsentrepreneurial finance; seed funding; SBIR; NIH; public venture capital; entrepreneurial timing

ABSTRACT. Technology-based entrepreneurial firms face considerable barriers in commercializing their innovations. Time, in particular, is a scarce resource for start-ups (Levesque & Stephan, 2020). New firms operate under high risk and severe capital constraints as they try to outpace competitors and generate economic profit. Research has shown that start-ups’ capital structures influence firm outcomes and development speed (Hecchavaria et al., 2016). This is especially the case during the early stages as a company tries to move from proof of concept to early seed stages when the pot of money available to firms is quite small and their ideas are quite risky. Hearkening to Benjamin Franklin’s famous quote, “time is money” quite literally to an entrepreneur. The persistent qualm of the entrepreneur is that they rarely have enough of either.

This is where endogenous growth theory suggests government has a role. Government intervention is seen as necessary to reduce barriers in helping firms develop technologies, based on the idea that the creation of new knowledge is a public good that the private sector will not effectively support due to the high risk entailed (Mazzucato, 2011). Government enters at an earlier, riskier stage than private funding is willing to enter. But public funding operates under different institutional norms than private funding and we posit in this paper that these differences have implications for how the funding impacts firms’ subsequent milestones. Qualitative evidence from interviews with entrepreneurs indicates that federal funding through the United States (U.S.) Small Business Administration’s (SBA) Small Business Innovation Research (SBIR) and Small Business Technology Transfer (STTR) programs may be too slow to attract high quality applications relative to alternative sources of early funding (NASEM, 2022). This slowness can be attributed to the nature of the routinized bureaucratic forms that federal agencies follow (Perrow, 1986) which make the minute-by-minute decisions of an angel or venture capitalist difficult, if not impossible, to replicate. Evidence from Europe indicates start-ups that receive public venture capital (VC) sooner have higher sales-growth (Grilli & Murtinu, 2015). These questions have not been investigated in the U.S. context.

In this paper, we ask whether shorter time periods between proof-of concept and seed stages in public VC allow firms to achieve greater technology and financial milestones. We exploit a unique programmatic shift in the U.S. SBIR and STTR funding to answer this question. Owing to the concerns that important technologies were lying dormant, the federal government began experimenting with new models to support high tech firms more swiftly by offered “Fast-Track” awards, which allow firms to bypass Phase I (averaging $150,000) and move directly to Phase II of the program which offers significantly more funding (averaging $1 million). We take an exploratory approach to examine performance differences between firms who secure these special awards, called Fast-Track awards, and comparable firms who received the regular Phase II award after a Phase I. We isolate our setting to SBIR/STTR awards made through the National Institutes of Health (NIH). This allows us to examine a range of firm outcomes.

First, we present a descriptive overview of these special awards, including an exploration of the Fast-Track program to understand which firms are likely to receive them. Next, we estimate a series of two-way fixed effects panel regressions to assess whether speedier SBIR funding increases the likelihood that a firm will a) produce clinical trials or b) medical devices as the result of the funded application; c) obtain patents; d) attract subsequent funding from private sources; and e) be acquired compared to the control group (extensive margin). Finally, we use a unique dataset on state match programs in a differences-in-differences analysis to examine the impact of this exogenous windfall funding (Myers & Lanahan 2022) on the Fast-Track recipients (intensive margin). We use NIH’s RePORT data archive for detail on award type and corresponding patents and clinical trials, as well as the USPTO PatentsView database. We construct further outcome measures using the SBA SBIR award database as well as information on VC, angel investment, and other private funding support from Crunchbase. While numerous studies document the impacts of this program on follow-on financing (Lerner, 2000; Howell, 2017), patents (Myers & Lanahan, 2022), and employment (Lanahan et al., 2021), consideration of the influence of the timing of funding rounds is largely absent from the public VC literature.

We find that among firms that receive a NIH SBIR/STTR Phase II or Fast-Track award, the marginal Fast-Track dollar is associated with a greater number of follow-on clinical trials and patents. We also find that among Fast-Track award winners, those that receive an exogenous increase of funding through variably available state match programs have even greater number of clinical trials than those that do not receive the state match funding, but receive less private venture finance. These are important outcomes for small, young firms and illustrate that faster SBIR funding can be beneficial if firms are pursuing clinical trials and patenting as part of their development process.

This paper makes three contributions. First, we contribute to the field of entrepreneurship by responding to recent calls for a “time-based lens for entrepreneurship research” (Lévesque & Stephan, 2020). We offer new insight into the importance of the timing of public, early-stage, proof-of-concept, and seed funding. Second, we contribute to the literature on public VC by exploring a previously understudied method of hastened public funding. Finally, we draw practical implications for entrepreneurs and for policy. We provide new understanding about how the SBIR program may be meaningfully enhanced by quickening the pace of funding, if the SBIR wants to increase products, technologies, and follow-on funding as a goal.

11:00
The commercialization of DoD-SBIR patents

ABSTRACT. The paper proposes a novel, web-based approach to innovation policy evaluation. The approach overcomes one major limitation affecting current evaluation methods, namely the tracking of invention commercialization. We implement it to study the impact of the U.S. DoD-SBIR program on technology commercialization. We start by identifying the universe of USPTO patents that acknowledge support by the program, and construct a set of control patents as benchmark. We then track whether these patents are mentioned in relation to commercial products in virtual patent marking web pages. We interpret the latter event as signal of commercialization. Finally, we compare the commercialization probability of SBIR-funded and control inventions. The results support the view that the SBIR program is quite effective at stimulating the commercialization of federally-funded scientific discoveries. The effect is particularly strong for grants covering development expenditures and Phase II awards.

11:15
Corporate science and IPO

ABSTRACT. IPOs allow companies to raise new capital, but this money comes with strings attached. Disclosure requirements and pressure from the shareholders affect your scientific strategies and consequently your scientific output. I test the causal impact of going public on firms’ scientific output, using data on 1,919 US IPO firms, which is the population of IPOs from 1994 to 2010 with at least one patent. My empirical strategy involves a treatment group of firms that successfully completed an IPO and a control group of firms that filed for an IPO but then afterwards decided to withdraw their filing. Identification is achieved instrumenting the IPO decision with stock market average returns in the two months after the filing. Preliminary results show a positive effect of IPOs on scientific output, measured as patents and scientific publications.

10:30-12:00 Session 12E: Tech Transfer
10:30
Innovative Funding Mechanisms to Enhance Federal Technology Transfer and Public Private Partnerships

ABSTRACT. The Federal Government spends more than $150 billion on research and development. This funding has fueled the innovations and technologies that have transformed our societies. However, the pathways from research to impact can be arduous and complex. Way to effectively engage, work with, and leverage resources from the private sector, including small businesses, for example through public private partnerships, continue to be critical to the maturation and commercialization of new technologies and the supply chains necessary in their development.

The Federal Government continues to look for innovative mechanisms to engage with non-Federal stakeholders to enable and improve engagement and opportunities with the private sector. STPI has been working with DoD and DOE to better understand the lessons learned and identify ways to enhance the use of a few of these mechanisms--1) establishing an agency-affiliated foundation, 2) using the other transactions authority (OTA); and 3) using the partnership intermediary agreement (PIA) authority). STPI will discuss cumulative lessons learned for planning and implementation of these mechanisms working with two major research and development agencies--DoD and DOE--and how they inter-relate to complement one another. Currently, there is a dearth of published information regarding the use of these mechanisms to enhance Federal research and development and technology transfer goals. Published information tends to be focused on a single agency if at all. This presentation will build on prior studies published by STPI, including Opportunities to Advance Department of Defense Technology Transfer with Partnership Intermediary Agreements (https://www.ida.org/-/media/feature/publications/o/op/opportunities-to-advance-department-of-defense-technology-transfer-with-partnership-intermediary/p-20450.ashx).

10:45
Varieties of Regional Innovation Systems around the World and Catch-up by Latecomers

ABSTRACT. This study identifies the characteristics and types of the regional innovation systems (RIS) of regions and cities in emerging economies in comparison to those in advanced economies. It uses the citation data of the US patents filed by 30 regions. Some RIS variables are newly developed, and they include intra-regional, inter-regional, and inter-national sourcing of knowledge and local ownership of innovation. The cluster analysis of these variables enables us to identify four major types of RIS around the world and link them to regional economic performance. The four types are, in the descending order of their per capita income levels, as follows: large, mature RIS characterized by a combination of long cycle technology specialization and high local ownership (Group 1), mixed RIS characterized by a long cycle and low local ownership (Group 2), “strong catch-up” characterized by short cycle and high local ownership (Group 3), and “weak catch-up” characterized by short cycle and low local ownership (Group 4). Groups 3 and 4 include only the regions in emerging world. They similarly specialize in the same short cycle time of technologies (CTT)-based sectors but show different records of economic performance. The key differentiating variable is the degree of local ownership of knowledge, which can be a basis for increasing domestic sourcing of knowledge and sustained catching up. Another important variable is decentralization, of which the level is lower in the strong catch-up group than in the weak catch-up group. In this Group 3, catching up is led by big businesses. Several cities experiencing upgrading, like Moscow, Beijing, and Shanghai, also show an increasing trend of local ownership and centralization.

11:00
Government Funding and the Development of Innovation and Entrepreneurship Ecosystems

ABSTRACT. Background and rationale

Access to funding is known to be critical for the emergence and consolidation of new innovative ventures. There are several studies mentioning funding as one of the pillars on which Innovation and Entrepreneurship Ecosystems (IEEs) are based, although there is still a lack of systemic knowledge about funding sources and their effects on ecosystems (Frimanslund et al., 2022). Although there are several types of private funding, such as venture capital and business angels, here focus on discussing the role of government funding for the development of ecosystems, since public funding dedicated to support scientific research - either applied research aiming at innovation directly, or more basic research - functions as trigger for innovation and the creation of new knowledge-intensive companies. Thus, the purpose of this article is to show how non-reimbursable funding from public sources affects the dynamics of IEEs, particularly in the context of emerging economies, where other sources of funding tend to be scarcer.

Literature Review

It is important to distinguish between the two groups of grants from public research support agencies that promote the development of IEEs. The first, direct, consists of programs aimed at applied research that focus on innovation, such as the Small Business Innovation Research (SBIR) in the US and Fapesp’s Innovative Research in Small Businesses (PIPE) in Brazil. Such initiatives are based on two fundamental principles: (i) the existence of market failures in the process of funding entrepreneurship and innovation; and (ii) the expectation of socialization and economic gains arising from the success of the companies supported. In these cases, public sources of funding not only generate liquidity for the ecosystem (Autio & Ranniko, 2016) but also act as indicators of the potential of selected projects. The relationship between knowledge-based firms and public research institutions can increase their innovative potential, which reinforces the role of public institutions as relevant agents in the entrepreneurial ecosystem. Also, public research funding improves the systemic conditions of IEEs, as it provides early-stage funding for technologies that will be the basis of new products and services. The second group, indirect, involves all support for scientific research, including basic research, which eventually may translate into the generation of innovation and new ventures. The prerogative of this group is supported by the knowledge spillover theory of entrepreneurship, which identifies the production and circulation of knowledge as assets that are strongly attached to territorial spaces and support the development of entrepreneurial and innovative activities. The approximation of new and existing enterprises with universities fosters gains in terms of technological capabilities (Audretsch et al., 2020). Furthermore, academic research itself can create opportunities for economic exploration, making universities relevant cradles for the emergence of new innovative ventures (Fischer et al., 2019). In both cases – direct and indirect – the role of public funding in ecosystems in incipient stages has a more relevant relative weight. This is due to the lack of maturity in the configuration of the IEEs, in which the presence of private capital still falls short in achieving traction.

Method

The methodology is divided into two fronts: 1) Literature review. The research envisages carrying out a literature review on public funding and the development of Innovation and Entrepreneurship Ecosystems (IEEs). The search was carried out in Scopus and Web of Science, selecting scientific articles from 2015 to 2022. The search occurred by topic, ordered by relevance. From the initial list of articles identified, the most relevant ones were selected according to the research objectives. 2) To advance in the development of methods, metrics, and indicators for the analysis of public funding in IEEs, we proceeded with the following steps. a. Develop an approach for the specific case of Unicamp. b. Work with Unicamp’s licensing and technology transfer data. Licensing cases: Connect companies with supporters by crossing the name of inventors/technologies/dates with Fapesp’s global base. Spin-offs: Connect companies with supporters by crossing the name of inventors/technologies/dates with Fapesp’s PIPE base. Data gathered from companies in the science park: Check the relationship between companies in Unicamp’s Science and Technology Park and Fapesp’s support. We carried out a survey and subsequent triangulation between the unstructured data provided by the Unicamp Innovation Agency (Inova) with the structured data provided by Fapesp. This enabled the analysis of the direct and indirect connections between public funding and IEEs.

Results

This article contributes to academic research and literature in two different ways. First, the confirmation of the gap identified in the literature concerning the lack of studies focused on the role of government funding for the development of innovation and entrepreneurship. Second, our analysis demonstrates the relative centrality of public funding within the context of an IEE embedded in a developing country. This will allow us to developed a more nuanced picture of the role played by public sources dedicated to research in shaping the dynamics of such ecosystems.

Implications & Significance

This article brings a few important lessons. First, we came to the conclusion that it is important to strengthen government funding to promote both applied research aiming at innovation and basic research capable of generating new ideas. Second, it is critical to assess the role of public funding in generating new ventures in selected IEEs through mechanisms capable of measuring this effect; for such, we recommend the assessment of licensing and technology transfer data, university spin-offs, and companies located in science parks.

REFERENCES

Audretsch, D. B., Belitski, M., & Cherkas, N. (2021). Entrepreneurial ecosystems in cities: The role of institutions. PLoS ONE, 16(3). doi: 10.1371/journal.pone.0247609 Autio, E., & Ranniko, H. (2016). Retaining Winners: Can Policy Boost High-Growth Entrepreneurship? Research Policy, 45(1), 42-55. doi:10.1111/caim.12338 Fischer, B. B., Moraes, G. H. S. M., & Schaeffer, P. R. (2019). Universities' institutional settings and academic entrepreneurship: Notes from a developing country. Technological Forecasting and Social Change, 147, 243-252. doi:10.1016/j.techfore.2019.07.009 Frimanslund, T., Kwiatkowski, G., & Oklevik, O. (2022). The role of finance in the literature of entrepreneurial ecosystems. European Planning Studies. doi:10.1080/09654313.2022.2055962

10:30-12:00 Session 12F: Societal Impact of Research
10:30
Disentangling the societal discourse on covid-19 in Belgium: scientists communicating in the written press during the recent public health crisis

ABSTRACT. Introduction Scientists were omnipresent in the media during the covid-19 pandemic. More than ever, the recent public health crisis required health scientists to become public spokespersons. In Flanders, the northern Dutch speaking region of Belgium, a few academics became known as “the virologists” in a very short period of time, and could count on various media bringing their up-to-date knowledge to the wider public. But not only academics from the medical sector were featured. The covid-19 pandemic became the new issue of choice for academics from various non-medical disciplines as well, with a number of them using the written press to voice their critique of the predominant medical discourse to the general public. While counting media coverage provides valuable insights into the "amount" of media coverage, the question of what discourses both medical scientists and non-medical scientists employed remains underexplored. That the public debate is also an arena where knowledge, ideas and interpretations from different scientific disciplines compete for attention – alongside “scholarly” circuits of communication – is often overlooked. This gap is likely informed by the persistent methodological problems we face when studying scientific communication beyond the academic sphere. The recently developed neural language processing tool at Sentometrics however, allows for a close analysis of the scientific discourse of academics in the written press regarding the covid-19 pandemic. Advanced textual indicators provide insight into the nature and relevance of these interactions, and improve understanding of the lexicon, sentiment, sequence and timing of scientists communicating during the recent public health crisis. Methods Researchers active at the five universities funded by the Flemish Community in Belgium were the focus of this study. Using data from Flanders Research Information Space (FRIS), an open source database operated by the Flemish government, active researchers (n = 35277) were used for analysis, supplemented with additional data on gender, career position, number of publications, sector and discipline. The names of active researchers were queried in the Belgian media database GoPress, collecting the occurrence of their names, after correction of namesakes (false positives). We excluded academics with a political mandate. We focused on Dutch speaking newspapers and magazines in Belgium, because of the highly concentrated Flemish media ecosystem. The time period of the newspaper articles was from Jan. 1, 2019 to Dec. 31, 2020. The year 2019 functioned as a control year, to analyze the distribution of media attention of academics in the written press before the covid-19 pandemic. In the near future, the analysis will be supplemented with data from the year 2021. The covid-19 pandemic was the overriding theme of 2020 and 2021 news in Belgium. First messages on covid-19 began to appear in January 2020 in Belgium, followed by a proliferation brought by the first casualties in March 2020. Quickly, the federal government sought assistance from several expert consortia, which included several academics. These quickly became the public face of the collective response to the pandemic. They consulted regularly to assess the public health crisis, advising the federal government to tighten, maintain or ease the measures, communicating publicly about them in written media. Preliminary results A first question presents itself as to which individual academics appeared in the written press. Only 7.6% of academics (2640 of 34669) appeared once or more in the Flemish written press of 2020. Moreover, five professors (four medical professors and one biostatistician) got 30.34% (6817 of 22467) of all media mentions that year for all academics at Flemish universities, pointing to high skewedness in the data. These “Big Five" became known as “the virologists” and took the lead in providing the public with actual information on the virus and its societal consequences. Their daily presence in several media rendered them “celebrity scientists” almost instantaneously, with initial praise and adulation from the public but, over time, also hatred and sometimes even death threats. In terms of sector - and disciplinary differences, academics from the medical sector evidently received relatively the most media attention in 2020 (49,92%; 11212 of 22462 articles). Whereas in 2019 it was social scientists who received the most media attention (41.0%; 5800 of 14130 articles), in absolute terms they surprisingly remained almost as prominent in 2020 (26,73%; 6004 of 22462), but lost media attention in relative terms. In relative terms, almost all non-medical disciplines lost media attention in 2020 compared to 2019, apart from academics from Psychology and cognitive sciences, who saw a very slight increase in relative popular attention (from 3,6% of articles in 2019 to 3,7% of articles in 2020). The covid-19 pandemic was therefore not just a medical affair; all kinds of academics brought their expertise into the public arena, interpreting, explaining, advocating, promoting the impact of the public health crisis on their area of expertise, some criticizing or expressing their dissatisfaction with the "purely medical" discourse of the virologists. A few academics took the lead in this, representing the “critical” opposing voice in the debate. Significance Results show how the written press provided an immediate opportunity for academics to publicly assert the policy relevance of their field and discipline, not “organically” from the traditional scientific publication cycle, but more directly from the stage they were given in mainstream media. While counting media coverage provides valuable insights into the "amount" of media coverage, the question of what discourses both medical scientists and non-medical scientists employed remains unanswered. Future analysis aim to unravel which context-specific lexicon academics from both medical and non-medical employed during the covid-19 pandemic, improving our understanding of the impact of proactive and reactive science communication of academics in the written press.

10:45
A Toolkit for Demonstrating Societal and Economic Impact of University Research

ABSTRACT. Background and rationale The demonstration of the societal and economic impact of a university remains challenging owing to a paucity of data that connect the education and research activities of the institution to short- and long-term outcomes. We have pursued the connection between education (PhD alumni) and economic (employment) outcomes and the connection between grant income (research) and patenting (innovation) to demonstrate how actionable insights can be developed for one institution (Cornell University).

Methods In this study, we have linked and enriched data available within Cornell University with datasets relating to career outcomes from an online social network and research publications to demonstrate the post-graduation career trajectories of doctoral holders. We have also linked research grants, publications, and patents awarded to Cornell researchers to explore gaps in translation activities from the funding inputs to the research and innovation outputs of the university more directly. This work was conducted under an approved IRB protocol requiring all personally identifiable information to be held securely and only aggregated deidentified results distributed beyond the authorized project team.

Results Using a cohort-based approach, we have determined the current employment of all PhD alumni from two Cornell colleges with a high degree of precision and recall. Across the cohorts studied (graduates from academic years 2000, 2005, 2010, 2015 and 2020) we were able to show that 79% could be unambiguously identified in a popular online social network using just their full name and known former affiliation to Cornell University. Of these, 1 in 10 alumni leave the US shortly after graduation while more than 78% remain in the US up to 22 years post-graduation. Of those that remain in the US, alumni are predominantly based in a small number of north-eastern and western states with high background rates of PhD employment. Half of Cornell PhD alumni are currently working in industry, and another one-third in education; the remainder are largely in government or non-profit organizations. For graduates working in industry, government and non-profit sectors, ‘scientist’, ‘engineer’ and ‘researcher’ roles are most common. Additionally, we anticipate that the exploratory work linking research grants to publications and patents will be able to help identify different patterns of funding sources tied to innovation outcomes based on research focus or department at Cornell. By comparing these links across departments at Cornell, we expect to uncover gaps between funding and levels of engagement in innovation activities and understand whether these differences can be partially explained by relative focus on basic to applied research activities.

Significance Our analysis of PhD alumni career outcomes indicates that the value of the research-based education and training that the higher degree reflects is retained in the wider economy irrespective of whether the individual pursues an academic career path or exits the sector. In the case of Cornell University, those that do leave academia tend to remain within the US, gravitate towards parts of the country where knowledge-based work is abundant, and occupy roles that capitalize on the knowledge and skills they developed during the course of the PhD. The anticipated findings from the work exploring linking Cornell grants to innovation outcomes will help identify existing practices within departments that could benefit from Cornell resource investment, enabling the university to focus on specific academic units in supporting translation activities for societal and economic impact.

Conclusion We have shown in this work that the societal and economic impact of an institution such as Cornell University can be demonstrated by combining data available within the university with other publicly- or commercially-available datasets. To that end, the economic benefit of PhD alumni who remain within the country and take up employment beyond academia that utilizes their doctoral training and skills can be effectively demonstrated, and the innovation outcomes of research-oriented grant income can be connected directly through advanced mapping approaches. Taken together, we suggest that this is the beginnings of a toolkit that other institutions may wish to develop and adopt for their own specific circumstances and that can inform and monitor university strategy and policy-making.

11:00
Societal impact of research and public policy: a bibliometric assessment

ABSTRACT. Suggested topic area: Societal impact

Background and rationale: The literature on societal impact of research has been pointing to a diverse range of ways to identify, analyze, and measure how scientific knowledge flows from its production spaces to engage with and influence society (Viana-Lora & Nel-lo-Andreu, 2021; Smit & Hessels, 2020; Bornmann, 2013). Part of this debate is related to the impact of research on public policy design, implementation, and evaluation processes (Boaz et al., 2009; Newson et al., 2018). In general, such studies adopt a diversity of theoretical and methodological approaches that basically follow two paths, which are not mutually exclusive: forward tracking, starting from research to identify its impact on policy; and/or backward tracking, starting from policy to identify the use of scientific research. In both paths, one of the possible approaches refers to the use of bibliometrics and altmetrics techniques to capture and analyze the use of research in policies and measure its impacts (Tahamtan & Bornmann, 2020). In addition, the debate on the societal impact of research - in its various social dimensions and spaces, including policy - allows us to put into perspective and debate the role of stakeholders directly involved in the production of scientific knowledge, especially universities and funding agencies.

Objective: Considering this debate, our study aims to explore the potential of bibliometrics to identify and analyze, in an exploratory way, how policy documents have been created based on research funded and promoted by the São Paulo Research Foundation (FAPESP) in two main areas: society and environment. We assume herein the inseparability between the social and environmental spheres. Considering the environment and society as elements that mutually affect each other, it is common for research funding institutions to include the analysis of environmental impact within the social scope. The Higher Education Funding Council for England (2019), for instance, considers that health and environment are part of the definition of social impact by affirming that it means “[a]n effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia” (HEFCE, 2019: 118). Regarding FAPESP's financial support for projects, in the field of life sciences, which accounted for 45% of the disbursement made by the institution in 2020, the health area was responsible for more than 25% of this investment, totaling more than BRL 247 million that year. Thus, as for societal impact, we have decided to focus on health-related research funded by FAPESP. Likewise, amongst the several projects supported by FAPESP in the environmental field, climate change-related research has been receiving most of the grants, which reflects the global trend of counteracting the emission of greenhouse gas and warming of the earth’s surface. In other words, we will seek to understand how policy organizations have been using research funded by FAPESP in these two major socio-environmental problem areas, namely health and climate change. This study answers the following research question: to what extent is it possible to measure the impact of research funded by FAPESP in the areas of health and climate change by exploring bibliometric resources?

Methodology: Here we make use of Overton, considered the largest and most comprehensive database that tracks how research (papers) are cited in policy documents produced by government agencies, think tanks, nongovernmental organizations, and intergovernmental organizations in 182 countries (Fang et al., 2020). The research explores the following path: from the policy documents found under the umbrella of "health" and "climate change", we have identified the papers written by authors from Brazilian research institutions. After this initial screening, we have selected only the research funded by FAPESP. Such procedure allows bibliometric data to be obtained from both the funding-related articles and the policy documents that use them. Afterwards, an exploratory analysis was conducted, focusing on four aspects: (i) the characterization and type of political organization that cited research funded by FAPESP, including country of origin; (ii) the sub themes of impact within health and climate change most related to research funded by the agency; (iii) how the research is being used, considering the debate about political appropriation and mobilization of research; (iv) and basic bibliometric analysis of the identified papers. Although the study cannot be considered comparative in nature, the methodology enables the establishment of similarities and differences between the impacts considered (health and climate change).

Significance of the study: By using Overton, this study explores ways to connect funding to the impact of research on public policy, attempting to dialogue with the literature on the social impact of research, as well as empirically bringing about novel analyses on one of the main Brazilian research and innovation funding agencies. On the one hand, the focus on the funding agency is justified by its prominent and influential role in stimulating the societal impact of research within the scientific community; on the other, funding agencies are embedded in a context of political and social pressures to demonstrate the societal impact of funded research, also seeking for political and societal legitimization (Curry et al., 2020; Watermeyer, 2019). Moreover, FAPESP is considered one of the most important research and innovation funding institutions in Brazil. Regarding the chosen impact themes, it is known that health and environment are fruitful fields for the dialogue between research and public policies (Newson et al., 2018; Bornmann et al., 2016), as well as relevant themes to society, closely related to the Development Sustainable Goals (SDGs). Thus, we discuss herein, from the data and analyses carried out, the role of funding agencies in promoting studies capable of politically impacting health and the environment.

References Bornmann, L. (2013). What is societal impact of research and how can it be assessed? A literature survey. Journal of the American Society for information science and technology, 64(2), 217-233.

Viana-Lora, A., & Nel-lo-Andreu, M. G. (2021). Approaching the social impact of research through a literature review. International Journal of Qualitative Methods, 20, 16094069211052189.

Boaz, A., Fitzpatrick, S., & Shaw, B. (2009). Assessing the impact of research on policy: a literature review. Science and Public Policy, 36(4), 255-270.

Curry, S., De Rijcke, S., Hatch, A., Pillay, D. G., Van der Weijden, I., & Wilsdon, J. (2020). The changing role of funders in responsible research assessment: Progress, obstacles and the way ahead. RoRI Working Paper,

Newson, R., King, L., Rychetnik, L., Milat, A., & Bauman, A. (2018). Looking both ways: a review of methods for assessing research impacts on policy and the policy utilisation of research. Health research policy and systems, 16(1), 1-20.

Smit, J. P., & Hessels, L. K. (2021). The production of scientific and societal value in research evaluation: a review of societal impact assessment methods. Research Evaluation, 30(3), 323-335.

Tahamtan, I., & Bornmann, L. (2020). Altmetrics and societal impact measurements: Match or mismatch? A literature review. El profesional de la información (EPI), 29(1).

Bornmann, L., Haunschild, R., & Marx, W. (2016). Policy documents as sources for measuring societal impact: how often is climate change research mentioned in policy-related documents?. Scientometrics, 109(3), 1477-1495.

Fang, Z., Dudek, J., Noyons, E., & Costas, R. (2020). Science cited in policy documents: Evidence from the Overton database. In Altmetrics conference. http://altmetrics. org/wpcontent/uploads/2020/11/02_submission_Fang_Dudek_Noyons_Costasaltmetrics20. pdf.

11:15
Literature review and scientific mapping on the economic, policy, and societal impact of research

ABSTRACT. Background and rationale Over the past years, research funding agencies have demanded studies on the impact of the research financially supported by them to verify how the creation of knowledge relates to improvements in economic, social, and policy-related areas (Aiello et al., 2021; Sandes-Guimarães, Velho & Plonski, 2022; Milat, Bauman & Redman, 2015). More importantly, research funding agencies are concerned about demonstrating the relevance of scientific research and studies to address societal problems, i.e., about producing evidence of the impact of the funded research. This movement is important because it promotes higher accountability and advocacy directed to different types of stakeholders and it demonstrates the effects of investing in research. This is also part of a larger discussion that investigates the science of science, or research on research, that aims to identify how scientific research can be made more open, inclusive, and impactful. Considering the above, through a brief literature review, we have come across multiple impacts deriving from research. In order to conduct our study, we decided to group them into three clusters, namely, economic, policy, and societal. This division allows us to identify similarities and differences between the many impacts of investing in research according to their specific contexts. In addition to understanding aspects related to the literature on research impact, thinking of the societal impact of research is increasingly connected with discussions on Equity, Diversity and Inclusion (EDI) in academia. In this sense, we consider that incorporating EDI indicators in the literature review together with traditional bibliometric indicators is relevant as it sheds light on aspects that are often neglected when discussing the state of the art of a certain topic. In other words, this research approach allows us to identify the status quo and marginalized discussions of relevant topics by including indicators of genre, race, location, and language.

Objective This study aims to show an overview of the economic, policy and societal impact of scientific research from 2015 to 2022 through the elaboration of a scientific mapping, as well as critically discussing it. This scientific mapping considers four main areas: a) thematic evolution, b) methodological approach, c) performance based on traditional bibliometric indicators; and d) performance based on new bibliometric indicators (EDI). It is important to note that this study is part of a larger project to investigate and develop research impact indicators for several initiatives carried out by the São Paulo Research Foundation (a.k.a. Fapesp) in Brazil.

Methodology Through a bibliometric analysis carried out in the Web of Science and Scopus databases, we provide the scientific mapping concerning the three broader impacts mentioned herein, namely economic, policy, and societal. It is important to clarify that to all impacts considered, basic bibliometric and literature review techniques will be applied. Common to all impacts is the inclusion of EDI metrics, as well as the identification of articles that address the role of funding or funding agencies in impact evaluation. However, considering the vastness of sub-themes implicit in each of these impacts, we resorted to a few methods to delve deeper and focus on each more effectively. The economic impacts will be evaluated based on three pillars, namely: university-industry collaboration, focusing specifically on an output indicator of university-industry co-authorship in scientific articles; the relevance of funding for the development of innovation and entrepreneurship ecosystems; and the importance of funding for the creation and development of small business programs, such as the SBIR. Regarding policy, we intend to identify how the literature on policy impact of research addresses the flows of knowledge from academia to policy (forward tracing method) and the use of scientific knowledge by policymakers (backward tracing method). Taking into account the research objectives, the following aspects will be emphasized: methodologies for evaluating the different flows and use of knowledge between science and policy-making; the debate on the role of funding and funding agencies in promoting the impact of research on policy and ways to assess and evaluate this, including the incorporation of new bibliometric and altimetric methods to measure impact; and the thematic dimension (public policy area reached) of the articles identified, looking for trends in knowledge areas that present greater predominance on the impact of policy research. As for the societal impact, we had to make a more delimited cut, as the literature informs that the social impacts encompass several fields, like health, the environment, public policies, and education. However, it is important to state that the social and environmental areas are relevant and frequently cited when it comes to the societal impact of research. Thus, by assessing the main thematic projects supported by Fapesp, we found that the health area is the social field that has received the highest financial support in recent years, which emphasizes its relevance as a field of research and project development. On the other hand, considering the relevance of environmental issues, we verified that the environmental-related area that receives the most funding from Fapesp is ​​climate change, which reflects the general concern with global warming, expressed and emphasized by reports like those by the Intergovernmental Panel on Climate Change (IPCC). For these reasons, we chose health and climate change as the societal impacts to be further assessed in this study.

Significance of the study The analytical exercise of identifying and developing a scientific mapping of the economic, policy, and societal impact of research allow us to identify similarities and differences between recently developed impact assessments. The intercambiation of methodological approaches is a rich contribution of this study as new ways of assessing a certain impact could be limited to a field of knowledge and have not yet overcome the disciplinary barrier; this may be useful when applied to other areas of knowledge. Another relevant contribution of this study is that the scientific mapping provides us with an outlook on research opportunities and points out where contributions can be made to the field. In addition, evaluating the state of the art in each dimension based on traditional performance indicators gives us an overview of the status quo of a topic of academic interest which is a well established way of providing information and evidence of how impacts are being assessed in recent literature. However, as we previously outlined, funding agencies are concerned about ways to make research practices more open, inclusive and impactful and an effective way of addressing and investigating this topic is to consider alternative indicators while performing a literature review. In that sense, we provide a relevant contribution by stimulating the incorporation of EDI indicators when conceptualizing a study, and also by developing new bibliometric indicators based on EDI.

References Aiello, E., Donovan, C., Duque, E., Fabrizio, S., Flecha, R., Holm, P., Molina, S., Oliver, E., & Reale, E. (2021). Effective strategies that enhance the social impact of social sciences and humanities research. Evidence & Policy, 17(1), 131-146. Milat., A. J., Bauman, A. E., & Redman, S. (2015). A narrative review of research impact assessment models and methods. Health Research Policy and Systems, 13(18). Sandes-Guimarães, L. V., Velho, R., & Plonski, G. A. (2022). Interdisciplinary research and policy impacts: Assessing the significance of knowledge coproduction. Research Evaluation, 31(3), 344-354.