1、HAMAD AL-IBRAHIMThe Potential for Artificial Intelligence Assistance in Funding ResearchThis document was submitted as a dissertation in March 2024 in partial fulfillment of the requirements of the doctoral degree in Public Policy Analysis at the Pardee RAND Graduate School.The faculty committee tha

2、t supervised and approved the dissertation consisted of Steven Popper(chair),Osonde Osoba and Luke Matthews.The external reader was Jonathan Grant.This dissertation was supported financially by the Anne and James Rothenberg Dissertation Award,and by the University of Qatar.DissertationPARDEE RAND GR

3、ADUATE SCHOOLFor more information on this publication,visit www.rand.org/t/RGSDA3387-1.About the Pardee RAND Graduate SchoolThe Pardee RAND Graduate School has specialized in graduate-level policy education since its founding in 1970.It is the original public policy Ph.D.program in the United States

4、 and the only program based at an independent public policy research organization.To learn more about the Pardee RAND Graduate School,visit www.pardeerand.edu.Published in 2024 by the RAND Corporation,Santa Monica,Calif.is a registered trademark.iii Abstract This dissertation investigates the potent

5、ial of Artificial Intelligence(AI)in transforming decision-making processes within funding agencies,which include governmental,quasi-governmental,and private organizations that finance research and innovation.These agencies are crucial in directing scientific inquiry and innovation by funding projec

6、ts that meet their strategic and societal goals.The study seeks to determine how AI can improve the efficiency,transparency,and objectivity of funding allocations,posing the question:How can AI be effectively integrated into the decision-making frameworks of funding agencies to optimize outcomes?The

7、 research methodology combines a thorough analysis of AI applications across various sectors with detailed interviews involving stakeholders from funding agencies,AI experts,and funding recipients.This mixed-method approach provides a broad perspective on the current integration of AI and its challe

8、nges.The dissertation progresses through several chapters,each offering unique insights into AIs role in funding agencies.Chapter 2 analyzes the existing processes and challenges within funding agencies,incorporating a landscape analysis and insights from stakeholder interviews to identify areas whe

9、re AI could offer improvements.Chapter 3 discusses AIs capabilities and applications in sectors like education,healthcare,and finance,examining their implications for funding agency decision-making.Chapter 4 introduces a strategic AI framework specifically designed for funding agencies,emphasizing t

10、he need for transparent algorithms and advanced explainability tools to ensure clear AI-driven decisions and build trust among stakeholders.The findings highlight AIs potential to enhance the peer reviewer assignment process and optimize proposal management through learning models.The study stresses

11、 the importance of combining AIs computational power with human expertise and maintaining ethical considerations.It also points out the necessity for agencies to adapt AI solutions that are sensitive to the changing research landscape and societal needs.In conclusion,the dissertation argues that AI

12、can significantly improve the effectiveness and fairness of funding decisions when thoughtfully integrated.This research contributes to the discussion on AI applications in public sector decision-making,offering valuable insights for iv policymakers,AI developers,and funding agencies.It advocates fo

13、r leveraging AIs benefits while carefully addressing its challenges to improve public funding mechanisms.v Contents Abstract.iiiFigures and Tables.viiChapter 1.Research Funding for the 21st Century.1Addressing Challenges in Funding Agencies.2Dissertation Aim and Scope.3Research Methodology.4Disserta

14、tion Structure.5Chapter 2.Comparative Analysis of Research Funding Agencies.6Methodology and Data Collection.8Comprehensive Analysis of Organizational Decision-Making Processes in Grantmaking.15Challenges with the current process.31Evolution and Adaptations in Response to the Challenges.34Insights a

15、nd Solutions from Grant Management and Scholarly Publishing.35Conclusion.41Chapter 3:The Role of Artificial Intelligence in Research Funding Decision-Making:Capabilities,Challenges,and Ethical Aspects.43Artificial Intelligence and Decision Theory.44Sectoral Exploration of Artificial Intelligence in

16、Decision-Making.45Challenges in AI-Driven Decision Making.46Literature-Based Insights into AIs Role in Strategic Decision-Making.50Technological Landscape and AI Applications in Funding Agencies.55Potential Commercial Tools for Decision-Making Support.61Identifying Gaps and Harnessing AI Capabilitie

17、s.62Conclusion.67Chapter 4:Designing an AI Framework for Enhanced Decision-Making in Funding Agencies.68Integrating AI in Funding Agency Decision-Making.68Operationalizing AI:Unveiling Peer Review Decisions and Enhancing Decision-Making Processes.73Targeted AI Solutions for Funding Agencies.107Concl

18、usion.123Chapter 5:Final Remarks.124Restating Research Problem and Objectives.125Summary of Key Findings.125Discussion on AI Integration.125Reflection on Methodology.126Practical Implications.127Theoretical Contributions.127 vi Limitations and Future Research.128Final Thoughts.128Appendix.Implementi

19、ng AI Models for Enhanced Funding Decision Analysis.130Abbreviations.139References.141 vii Figures and Tables Figures Figure 2.1.Methodology for data collection.9Figure 2.2.Decision Making Process Employed by Funding Agencies.15Figure 3.1.Trunks Decision-Making Framework Heading 7,Figure Title.50Fig

20、ure 3.2.A Visual Guide to AI-Enhanced Strategic Decision-Making Tailored to Funding Agencies.51Figure 4.1.The Suggested Decision-Making Process.70Figure 4.2.Data Collection&Review Synthesis.75Figure 4.3.SHAP Value Impact Analysis on Predictive Model Features.79Figure 4.4.Peer Reviewer Insights Quant

21、ification.82Figure 4.5.Membership Function Visualization for Categorizing Levels of Innovation in Fuzzy Logic.86Figure 4.6.Analyzing the Spread of Proposal Evaluations:A Histogram of Fuzzy Logic Scores.87Figure 4.7.Using Fuzzy Logic and Member Function for Funding Decision.88Figure 4.8.Strategic Ali

22、gnment and Outcome Prediction.94Figure 4.9.Final Selection and Funding Allocation.102Figure 4.10.Decision Tree Analysis:Visualizing Strategic Funding Categories Based on Proposal Metrics.104Figure 4.11.Network Analysis of Proposal-Reviewer Matches Using LLM Similarity Weights.114Figure 4.12.The Data

23、-Driven Decision-Making Cycle Enhanced by SHAP for Model Transparency and Insight Generation.116Figure 4.13.Feature Impact Analysis:SHAP Values Indicating Importance in Funding Decision Model.117Figure 4.14.Illustrating Feature Contributions to Predictive Decisions with SHAP.118Figure 4.15.LIME Appl

24、ication to Explain a Funding Decision.119 viii Tables Table 2.1.The 30 Shortlisted Funding Agencies.11Table 2.2.,The List of Interviewees.14Table 2.3.Review processes followed by organizations.23Table 2.4.Overview of Grant Evaluation Processes and Criteria Across Entities.29Table 3.1.Overview of Sof

25、tware Tools Employed by Global Funding Agencies.57Table 4.1.Analyzing Fuzzy Logic Insights:A Detailed Breakdown of Nuanced Funding Recommendations vs.Peer Reviewers Binary Decisions.89Table 4.2.Comparison of SVM Model Funding Predictions with Peer Reviewers Decisions:Evaluating the Alignment in Fund

26、ing Outcomes.90Table 4.3.Strategic Alignment and Outcome Prediction Output.97Table 4.5.TF-IDF Cosine Score Matrix Between Proposals and Reviewer Expertise.111Table 4.6.Embeddings Cosine Similarity Scores Matrix Between Proposals and Reviewer Expertise.112Table 4.7.LLM Similarity Scores Matrix Betwee

27、n Proposals and Reviewer Expertise.112Table 4.8.Unsupervised Topic Classification.122 1 Chapter 1.Research Funding for the 21st Century Innovation is widely recognized as a pivotal driver of economic development and prosperity.Studies consistently emphasize the positive correlation between innovatio

28、n and economic growth,highlighting its potential to boost productivity,create employment opportunities,and foster inclusive economic development(Fagerberg,Srholec,and Verspagen,2010).According to Lutz Bornmanns analysis,there has been an estimated 8-9%annual growth in research activity levels every

29、9 years(Bornmann and Mutz,2015)attributable to various factors such as the competitive nature of innovation and the structure of academic promotion(Rawat and Meena,2014).Furthermore,the significant role played by funding agencies cannot be understated.These agencies,including public and private enti

30、ties as well as non-governmental organizations(NGOs),contribute significantly to the surge in research activities by supporting research endeavors.Each funding entity operates with its distinct processes and procedures governing funding allocation and decision-making.This will be discussed in detail

31、 in chapter 2.President Donald J.Trump stated:Theyre giving away approximately,as I understand it recently,more than$32 billion a year,32 billion.And so,weve been looking at that for a while,and were going to be having some statements to be made about that.$32 billion a year.Its a lot of money,and w

32、e want to make sure its being spent wisely.1 The statement highlights the substantial allocation of public funds towards research and underscores the imperative of ensuring prudent utilization of these funds.However,the complexity and opacity inherent in the decision-making processes within public f

33、unding agencies have raised concerns regarding accountability,transparency,and efficiency.Criticism of the business model of federal funding agencies has stemmed from the lack of transparency and the extended duration required to observe tangible impacts.In 2011,Arizona State University president Mi

34、chael M.Crow criticized the NIHs discovery model and advocated for a multidisciplinary approach with a focus on outcomes beyond academic outputs(Crow,2011).Amidst the broader landscape of funding innovation,challenges abound,encompassing issues related to selection criteria,evaluation conduct,and de

35、sired outcomes.Ensuring fairness and efficacy throughout the decision-making process amidst these challenges is complex.Most notably,reliance on traditional peer review methods introduces potential biases and inconsistencies in evaluating research proposals.Additionally,the opaque nature of decision

36、-making procedures in research funding can impede transparency and accountability,1 https:/ 2 giving rise to concerns about equitable funding distribution and the extent to which intended research outcomes are realized.Addressing Challenges in Funding Agencies Building on the recognition of innovati

37、ons critical role in economic prosperity,it is essential to delve into the operational intricacies faced by funding agencies.These entities,pivotal in propelling research and development,grapple with a spectrum of challenges that can impede their efficacy and transparency.Through examining processes

38、 like peer review and considering both micro and macro-level hurdles,a clearer picture of these impediments emerges.The peer-review process,while long-standing and esteemed and regarded the gold standard for evaluating research proposals(Lasker,2018),encapsulates several of these challenges.Despite

39、its intent to uphold scientific integrity,the process often falls prey to issues of bias,inconsistency,and lack of transparency raising concerns about its reliability as a quality control mechanism(Shamseer et al.,2017;Tennant et al.,2017).Instances of data fraudulence and breaches of confidentialit

40、y further underscore the vulnerabilities within this system.For example,the U.S.Office of Research Integrity reported that 94%of misconduct cases from 228 identified articles between 1994 and 2012 had issues related to data fraudulence(Steen et al.,2013).In 2017,NIH announced that it would re-review

41、 several applications due to violations of confidentiality rules by reviewers.According to the director of NIHs Center for Scientific Review(CSR),a single reviewer reviewed 60 proposals that violated some aspect of the confidentiality rules(Brainard,2018).Beyond the monumental task of peer review,fu

42、nding agencies confront micro-level challenges that affect their day-to-day operations.These include the need for meticulous review of proposals,data validation and analysis,and the logistical complexities of coordinating peer reviews.Such tasks not only demand significant time and resources but als

43、o present portals for introducing errors and delays as evidenced by the considerable global expenditure on peer review processes(Cornelius,2012).One study estimates the worldwide cost of peer review to be around 1.9 billion annually and 15 million hours(Chauvin et al.,2015).At a broader scale,macro

44、challenges loom,encompassing issues like decision-making transparency,incentive structures for peer reviewers,and the potential for inherent conflicts within the system.These factors collectively influence the strategic direction of research teams and the allocation of funding,with implications for

45、the overall effectiveness and integrity of the innovation funding landscape(Young,2009).Considering these multifaceted challenges,the question arises:Can artificial intelligence(AI)offer solutions?With the growing interest in leveraging AI to automate and enhance decision-making processes within fun

46、ding agencies,it becomes imperative to explore AIs potential to address both micro and macro challenges.While AI offers promising avenues for improving efficiency and objectivity,its integration must be approached with caution with adopters being 3 mindful of the limitations and ethical consideratio

47、ns inherent in deploying AI systems(Yuan et al.,2021;Bolander,2019).Dissertation Aim and Scope In light of the significant role innovation plays in economic growth and the challenges identified within the funding mechanisms that support such innovation,this dissertation seeks to delve into the trans

48、formative potential of AI in reshaping the decision-making processes of funding agencies.Amidst the array of challenges,from operational inefficiencies in peer review systems to broader issues of transparency and accountability,this research will critically examine the potential of AI to offer groun

49、dbreaking solutions.This endeavor is particularly timely considering the growing need for more efficient,transparent,and equitable funding mechanisms.The objectives of this investigation are to:analyze the decision-making framework within funding agencies.scrutinize the existing challenges within th

50、ese decision-making processes,pinpointing both micro and macro-level obstacles that compromise efficiency and transparency.evaluate the current integration and future potential of AI applications in this context,assessing the effect that AI could have on enhancing decision-making processes.consider

51、the practical and ethical implications of implementing AI-enhanced systems,with a focus on addressing risks such as algorithmic bias and ensuring the responsible deployment of AI technologies.This research aims to map out the landscape of funding agencies and their decision-making procedures,identif

52、ying key areas where AI could be most beneficially applied.This includes a thorough examination of AIs role across various stages of the funding process from the initial review of proposals to the assessment of outcomes.More specifically,the dissertation addresses its main theme by answering several

53、 research questions:Problem Statement:How can funding agencies enhance their decision-making process by utilizing the potential capabilities of AI?Research Question 1:How is the decision-making framework within funding agencies currently structured,and what are its primary characteristics?Research Q

54、uestion 2:What challenges and limitations are inherent in the existing decision-making processes of funding agencies?Research Question 3:In what ways can the capabilities and applications of AI be harnessed to enhance decision-making processes within funding agencies,and how might AI address specifi

55、c challenges faced by these agencies?Research Question 4:What are the key obstacles associated with the implementation of AI in the decision-making processes of funding agencies,and what strategies can be employed to mitigate these challenges?4 Research Methodology This research will undertake an ex

56、ploratory approach combining primary and secondary data to address the formulated research questions.The backbone of the analysis will be a literature review that provides a secondary data source for addressing Research Questions 1-4 and establishing the dissertations theoretical framework.Primary d

57、ata collection will encompass a comprehensive landscape analysis examining over 400 funding agencies to identify prevalent practices and challenges.This extensive survey will inform the creation of a shortlist of case studies,selected from the larger data base based on predefined criteria to ensure

58、relevance and diversity,for in-depth investigation within this dissertation.In addition to the landscape analysis,the research included semi-structured interviews with key stakeholders including decision-makers from U.S.public funding agencies and international agencies with experience in AI impleme

59、ntation.Interviews were also conducted with founders of startups offering innovative solutions to the identified challenges in funding agencies.These interviews aimed to enrich our understanding of the intricacies of decision-making processes within these agencies and to explore the potential and cu

60、rrent applications of AI in analogous contexts.In conjunction with the literature review that identified potential AI capabilities applicable within funding agencies and those already employed in other sectors,the landscape analysis extended to investigating the degree to which funding agencies curr

61、ently utilize software solutions to streamline their operations.This inquiry assessed whether these software tools possess AI functionalities conducive to enhancing decision-making processes.Furthermore,the analysis will include an examination of available commercial solutions that could potentially

62、 address existing capability gaps within funding agencies.In our examination of the current decision-making frameworks within funding agencies,we will identify and assess existing AI capabilities that,while possibly utilized in other sectors,hold potential for implementation in the context of fundin

63、g agencies.This comprehensive analysis and landscape review of technological solutions is designed to function as a gap analysis.Through this,we aim to pinpoint specific areas within the decision-making process where enhancements are feasible and propose a suite of potential AI-driven solutions tail

64、ored to address these identified gaps.Our goal is to offer actionable insights and options for each stage of the decision-making process,leveraging the insights gained from both the landscape analysis and the technological capabilities review.A key aspect of this project has been developing AI showc

65、ases to demonstrate the potential of artificial intelligence to offer innovative solutions to the challenges identified in funding agency processes.Presented in Chapter 5,these showcases will illustrate AIs capabilities,including intelligent proposal-reviewer matching,topic modeling for proposal cla

66、ssification and categorization,enhancing explainability of predictive models,and network analysis to uncover hidden patterns and connections within the review process.5 The intention behind these showcases is to demonstrate in practical terms how AI technologies can be applied to enhance specific as

67、pects of the funding agencys decision-making framework,potentially leading to improvements in efficiency,transparency,and overall effectiveness.These showcases have been developed based on the insights gathered from the landscape analysis and stakeholder interviews(Chapters 3 and 4),providing practi

68、cal examples of how AI can be utilized to improve funding agency operations.Dissertation Structure The following chapters of this dissertation will illuminate different aspects of the research questions,progressively building an understanding and providing insights into AIs transformative potential

69、in the realm of research funding.Chapter 2:Comparative Analysis of Research Funding Agencies directly addresses the first and the second research questions by detailing the existing decision-making frameworks within funding agencies.This will be complemented with insights from stakeholders,enriching

70、 the understanding of the first,second,third,and fourth research questions by bringing in perspectives from stakeholders involved in the funding process.This chapter uncovers the real-world challenges and expectations regarding AIs role in improving decision-making efficiency and transparency.Chapte

71、r 3,The Role of Artificial Intelligence:Capabilities,Challenges,and Ethical Aspects,explores the third research question by discussing the capabilities of AI technologies pertinent to the decision-making processes in funding agencies.It also delves into the challenges and ethical considerations invo

72、lved in AI integration,providing a balanced view of AIs potential and limitations.Chapter 4:AI Showcases illustrates practical applications and potential solutions to the identified challenges.It showcases how AI can be leveraged to address specific issues within the funding process.This chapter dem

73、onstrates AIs practical utility,contributing to answering the third research question.Chapter 5:Conclusion Synthesizes the findings from the preceding chapters,offering conclusions that directly relate to all four research questions.This chapter reflects on AIs transformative potential in funding ag

74、encies,summarizing the studys contributions and suggesting pathways for future research and implementation.6 Chapter 2.Comparative Analysis of Research Funding Agencies This chapter investigates the decision-making processes and challenges within public and philanthropic research funding agencies,fo

75、cusing on the key question:What are the decision-making processes in these agencies,and what challenges do they face?It provides an overview of methodologies and data collection techniques used to analyze these agencies,primarily from high Research and Development expenditure countries,and delves in

76、to the stages of their grantmaking processes,highlighting key challenges faced and potential AI-driven solutions.This discussion sets the stage for a comprehensive examination of global research expenditure trends and the diverse roles played by different funding entities.Over the past decade,global

77、 research expenditure has significantly increased,driven by a range of organizations dedicated to advancing various stages of research,including basic,applied,and translational studies(Research Professional News Intelligence,2020).In this context,the term agencies refers to entities that specificall

78、y fund research activities,encompassing government bodies,private sector firms,non-profit organizations,and academic institutions.Diverse organizations orchestrate this expenditure,aiming to advance knowledge and contribute to the public good.The Organization for Economic Co-operation and Developmen

79、t(OECD)identifies four primary sources of R&D funding:the business enterprise sector,the government sector,the private nonprofit sector,and the higher education sector(Frascati Manual 2015,OECD,2015).Government entities have traditionally been the primary funding source for public research to advanc

80、e scientific knowledge.These entities fund various organizations and individual researchers through block or competitive funding programs.However,the business enterprise sector has also increased its research funding in recent years,accounting for nearly three-quarters of the total expenditure on R&

81、D performance in the OECD area since 2009(OECD,2022).Nonprofit organizations have also played a pivotal role in supporting societal and technological development in developing nations,from implementing existing technologies to developing innovative technological solutions,such as in the case of COVI

82、D-19,Human Cell Atlas(Human Cell Atlas).Despite their critical role,funding from private nonprofit institutions represented only 1.4%of all R&D spending between 2015 and 2018 in 37 OECD countries(OECD,2021).This study primarily focuses on organizations whose main mission is grantmaking for the publi

83、c good.We propose a broad categorization of these grantmaking or research funding organizations into two types:A)Government funding organizations and B)Non-Government funding such as Nonprofit or philanthropic organizations,including private and public foundations and charitable organizations.7 Whil

84、e grants from government and philanthropic organizations have unique advantages and disadvantages,they face distinct challenges.Increased funding applications and regulatory requirements have placed significant administrative pressure on public funders(United States Government Accountability Office,

85、2018).Although private grant organizations have fewer compliance requirements,the administrative burden remains high due to less expenditure on operational support(McRay J,2012).Adopting grant management software(GMS)has somewhat alleviated the administrative burden for application processing.Howeve

86、r,many organizations still rely on emails or hard copies for accepting applications and progress reports(REI Systems,National Grants Management Association&The George Washington University,2021),and those who use GMS encounter difficulties in checking the originality of research and monitoring the c

87、ompleteness of submissions and research results(Perbangsa et al.,2016).Several studies have been conducted to understand the review process of grantmaking organizations and the associated challenges.Still,these studies were often limited to a particular geography or agency or were not comprehensive

88、in including both public and philanthropic agencies(van den Besselaar et al.,2018;Demicheli&Di Pietrantonj,2007).The study by Bresselaar et al.(2018)sought to comprehend the grant selection process through a linguistic analysis of review reports,albeit their analysis was confined to European Union G

89、rants.Similarly,Demicheli et al.(1996)conducted a systematic review of the impact of the review process on the quality of funded projects,encompassing both philanthropic and public organizations.Still,their scope was limited to health-related studies.Meadmore et al.(2019)surveyed international organ

90、izations to understand their past and future approaches to decision-making,but their focus was also restricted to health-related organizations.Studies by Reckling et al.(2010),Mow(2009),and Wieczorkowska et al.(2021)evaluated the decision-making process of regional grant-making agencies and suggeste

91、d potential modifications to the process.Several studies have concentrated on understanding elements of the review process,such as external review or peer review(Schroter et al.,2010),review criterion(Abdoul et al.,2012;Falk-Krzesinski et al.,2015;Gallo et al.,2018),or review panels(Coveney et al.,2

92、017;Gallo et al.,2020;Obrecht et al.,2007;Porter,2005).A significant body of literature exists on external review,identifying challenges and recommending alternate processes(Bollen et al.,2014;Roumbanis,2019;Bentley,2009;Mayo et al.,2006;Dodek et al.,2012;Herbert et al.,2015).This includes challenge

93、s with reviewer matching and recommending algorithms for accurate matching(Zhao&Zhang,2022;Cechlrov et al.,2014;Xu et al.,2010)and methods for calculating reviewer scores(Bayindir et al.,2019).However,most of these studies are regional and lack a comprehensive comparison of global decision-making pr

94、ocesses used by non-profit and public agencies.A holistic view of the 8 decision-making process and the challenges faced could support different organizations in efficient grant management and decision-making.This chapter aims to provide a comprehensive global perspective on the decision-making proc

95、esses followed by public and private nonprofit grant-making agencies,the challenges they face,and to identify best practices from other industries for potential applicability.We analyzed a representative sample of 25 public and nonprofit foundations from countries with high R&D expenditures and eval

96、uated their grantmaking processes.The study also examined case studies of these agencies and others to understand grant-making challenges.In essence,this chapter seeks to answer the question:What are the decision-making processes in public and philanthropic grantmaking agencies,and what challenges d

97、o they face?It progresses through a methodical examination of these processes,the associated challenges,and the exploration of common interventions from other industries,ultimately concluding with insights into the integration of AI within these frameworks.Methodology and Data Collection In this sec

98、tion,we aimed to identify regions with a blend of public and nonprofit grantmaking agencies.We assumed nations with the highest R&D expenditure would likely harbor such a mix.To test this hypothesis,we collected data on Gross Domestic Expenditure on R&D(GERD)as a percentage of Gross Domestic Product

99、(OECD,2020)and GERD per capita(OECD,2020)from the OECD database.We then selected countries ranked in the top six for both criteria,ensuring representation from major continents-Asia,North America,and Europe.This selection process led us to target the following geographies:Israel,South Korea,China,th

100、e US,Belgium,and Sweden.We then embarked on a digital expedition,shortlisting 460 funding agencies located in in these countries.We applied an inclusion and exclusion criterion to ensure consistency and accuracy in our dataset.We included entities that fund science&technology-related research,innova

101、tion,or policy-making activities and manage their grant process internally.We excluded agencies that solely provide funding through contracts,those that fund other nonprofit agencies to manage their grant management process,and agencies that fund only education scholarships without any research proj

102、ect involvement.9 Figure 2.1.Methodology for data collection The first phase of the shortlisting process resulted in 30 funding agencies from a dataset of 460,ensuring diversity on parameters such as the dimension of the funding portfolio,geographical location,type of funder,target technology area,a

103、nd whether the agency primarily funds research,innovation,or both.Out of the original 460 agencies,376 met the inclusion criteria.However,identifying agencies for South Korea,China,and Israel proved challenging due to limited English-language search results and website language barriers.Future studi

104、es may benefit from using local search engines and translation tools for improved data collection.10 We collected parametric information for each agency using the following methodology:Dimensions of Funding Agencies Grant Portfolios:For assessing the dimensions of each agencys funding portfolio,we c

105、onsidered multiple indicators.Primarily,we evaluated the number of awards granted annually by the agency for the fiscal years 2019 to 2022.This data was sourced from public disclosure mechanisms such as annual reports,website disclosures,and IRS reports.Additionally,to capture a broader perspective,

106、various financial metrics were considered.These included terms like Obligations for the Year,Average Amount Invested in Research,Settlement Amount,Amount Available for Distribution,and Gifts,Contributions,and Grants Paid.For U.S.public agencies,specific data on obligations were extracted from a gove

107、rnment database.This approach allowed for a multifaceted understanding of each agencys funding activities,encompassing both the quantity of grants and the financial scope of their investment in research.Type of Funder:In the categorization of funders,we differentiated between two primary types:Priva

108、te Non-Profit Funders,encompassing all entities established as nonprofits,such as private foundations and charitable organizations,and Public Funders,which are entities receiving government funding.Additionally,private organizations not formally established as nonprofit foundations but engaging in g

109、rant-making activities under the ambit of Environmental,Social,and Governance(ESG)activities were also included.ESG activities refer to a set of criteria that demonstrate an organizations commitment and impact in three key areas:Environmental(impact on the planet),Social(impact on people including s

110、taff,customers,and the broader community),and Governance(how the organization is governed and conducts itself).These activities reflect the organizations values and ethical practices that go beyond mere financial contributions.Incorporating private organizations engaged in grant-making as part of En

111、vironmental,Social,and Governance(ESG)activities,despite not being formal nonprofits,is significant as it reflects the evolving dynamics of the funding landscape.This inclusion acknowledges the growing impact of corporate social responsibility and ESG initiatives,highlighting how these entities are

112、increasingly contributing to societal and environmental goals,thus broadening the scope and diversity of funding sources in contemporary research and development efforts.Target Technological Focus:In our analysis of funding agencies,we concentrated on the technological sectors they target,acknowledg

113、ing that their decision-making processes may differ based on the specific area of technological research.This focus was crucial for evaluating the current and potential use of AI tools in proposal evaluation processes.While our research covered agencies funding a broad range of research from basic t

114、o applied,we specifically highlighted those funding technological research.We collected this information from the agencies websites,reviewing their declared funding priorities,mission areas,or areas of technological interest.In instances where agencies did not explicitly focus on a technological dom

115、ain,their area of research interest was noted as nonspecific.This categorization was instrumental in understanding the alignment and applicability of automated/AI tools within their decision-making frameworks.Nature of Recipient Organizations:In categorizing funding agencies,a critical parameter was

116、 the nature of the organizations they primarily support.This categorization helps 11 distinguish between agencies based on the type of research and innovation activities they fund.Agencies primarily supporting industrial innovation,often involving product development and technological advancements,w

117、ere classified as Innovation Agencies.In contrast,those focusing on funding academic research,encompassing institutes and individual researchers typically engaged in theoretical or basic research,were termed Research Agencies.This distinction is crucial as it reflects the varying objectives and outc

118、omes expected by these agencies,shaping their decision-making processes and criteria for funding allocations.In our initial phase of research,we scrutinized a comprehensive pool of 397 funding agencies that satisfied our inclusion criteria.To delve deeper into their financial characteristics,we stra

119、tified these agencies into five distinct financial categories based on their funding portfolios:under 1 million,1-10 million,10-100 million,100 million-1 billion,and above 1 billion dollars.This stratification allowed for a detailed analysis across various types and sizes of funding entities.Within

120、this broad spectrum,a significant subset of 235 agencies,mainly comprising nonprofit organizations,had funding portfolios below 100 million dollars.This finding was crucial for understanding the distribution and focus areas within the larger group.Notably,in the financial range of 100 million to 1 b

121、illion dollars,we observed a mix of both nonprofit and public agencies,whereas public funding agencies predominantly dominated the category with funding portfolios above 1 billion dollars.Further refinement led us to a more concentrated group of 89 agencies,which had publicly available data across a

122、ll the study parameters.This final subset of 89 agencies,drawn from the initial pool of 397,provided a representative sample for our in-depth analysis,capturing diverse operational patterns and decision-making processes among a wide array of funding bodies.Table 2.1.The 30 Shortlisted Funding Agenci

123、es Country Funding Agency Public/Private Israel Israel Science Foundation Public Israel Israel Innovation Authority Public Israel National Institute for Psychobiology Private Non-Profit Israel The German Israeli Foundation Public Sweden Vinnova Public Sweden Swedish Foundation for Strategic Research

124、(SSF)Public Sweden The Swedish Foundation for Strategic Environmental Research(Mistra)Private Non-Profit Sweden Alfred sterlunds stiftelse Private Non-Profit Sweden The Arosenius Foundation Private Non-Profit 12 Country Funding Agency Public/Private Sweden Diabetesfonden Private Non-Profit Sweden FO

125、LKSAM Forskningstiftelse Private Non-Profit Belgium/Wallonia FNRS(Fund for Scientific Research)Public Belgium/Brussels Innoviris Public Belgium/Wallonia Research Wallonia Ministry for Public Service(SPW)Public Belgium King Baudouin Foundation Private Non-Profit Belgium Queen Elisabeth Medical Founda

126、tion for Neurosciences Private Non-Profit Belgium Belgisch Werk Tegen Kanker Oeuvre Belge du Cancer Private Non-Profit China National Natural Science foundation of China Public China Tecent Foundation Private For Profit South Korea National Natural Research Foundation of Korea Public South Korea Kor

127、ea Insitute of Marine Science and Technology Promotion Public South Korea Right Fund Pub-Private Partnership USA Alfred P.Sloan Foundation Private Non-Profit USA American Chemical Society Private Non-Profit USA Andrew W.Mellon Foundation Private Non-Profit USA Bill and Melinda Gates Foundation Priva

128、te Non-Profit USA Defense Advanced Research Projects Agency(DARPA)Public USA Little Giraffe Foundation Private Non-Profit USA National Institutes of Health(NIH)Public USA National Science Foundation(NSF)Public We selected six entities from each financial category seeking to enhance geographical,tech

129、nological,and agency-type diversity.The final list of the 30 agencies included Israel Science Foundation(Israel),Israel Innovation Authority(Israel),National Institute for Psychobiology(Israel),The German Israeli Foundation(Israel),Vinnova(Sweden),Swedish Foundation for Strategic Research(SSF)(Swede

130、n),The Swedish Foundation for Strategic Environmental Research(Mistra)(Sweden),Alfred sterlunds stiftelse(Sweden),The Arosenius Foundation(Sweden),Diabetesfonden(Sweden),FOLKSAM Forskningstiftelse(Sweden),Fund for Scientific Research(FNRS)(Belgium),Innoviris(Belgium),Research Wallonia Ministry for P

131、ublic Service(SPW)(Belgium),King Baudouin Foundation(Belgium),Queen Elisabeth 13 Medical Foundation for Neurosciences(Belgium),Belgisch Werk Tegen Kanker Oeuvre Belge du Cancer(Belgium),National Natural Science Foundation of China(China),Tencent Foundation(China),National Research Foundation of Kore

132、a(South Korea),Korea Institute of Marine Science and Technology Promotion(South Korea),Right Fund(South Korea),Alfred P.Sloan Foundation(USA),American Chemical Society(USA),Andrew W.Mellon Foundation(USA),Bill and Melinda Gates Foundation(USA),Defense Advanced Research Projects Agency(DARPA)(USA),Li

133、ttle Giraffe Foundation(USA),National Institutes of Health(NIH)(USA),and National Science Foundation(NSF)(USA).In Phase II,we collected data for these 30 agencies from their websites,public announcements,and interviews to answer the following research questions:What is the agencys decision-making pr

134、ocess?What kind of challenges do they face?What solutions do they use to resolve those challenges?We found variances in the decision-making process depending on the type of program being considered.While some agencies had a standardized approach across all programs,others varied their decision-makin

135、g process based on the specific programs.Five of the 30 agencies analyzed did not provide sufficient information on their respective websites regarding the decision-making process used for grantmaking(Alfred sterlunds stiftelse(Sweden),The Arosenius Foundation(Sweden),FOLKSAM Forskningstiftelse(Swed

136、en),Queen Elisabeth Medical Foundation for Neurosciences(Belgium),Korea Institute of Marine Science and Technology Promotion(South Korea).Therefore,our data is representative of 25 agencies in the set.Building on the foundation laid in the earlier phase II,we recognized the need for a deeper,more nu

137、anced understanding of these elements.The variances observed in the decision-making processes,coupled with the limited information available from some agencies,highlighted the complexity of the landscape we were navigating.This complexity,inherent in the diverse approaches to decision-making and the

138、 challenges faced by funding agencies,prompted us to extend our exploration beyond the surface-level analysis.To enrich our understanding and validate the preliminary insights gleaned from the landscape analysis,we transitioned into a more focused,direct engagement with the practitioners themselves.

139、This move was not merely a shift in methodology but an essential step towards achieving a comprehensive,grounded perspective on the potential integration of AI within funding agency decision-making processes.Our journey into the realm of primary research was guided by the ethical standards of schola

140、rship and policy analysis.With the approval of the Human Subjects Protection Committee(HSPC)at the RAND Corporation,we embarked on a series of interviews,17 interviews,with strategically selected participants from within the funding agency ecosystem.This selection was informed by the initial landsca

141、pe analysis,ensuring that our interviewees were well-placed to 14 provide insightful,firsthand accounts of the decision-making processes within their respective agencies.The questions posed during these interviews were not arbitrary;they were the culmination of an extensive literature review and the

142、 insights from our initial landscape analysis.These questions were designed to delve deeper into the intricacies of current decision-making processes,to uncover the prevailing challenges more clearly,and to explore the avenues for AI integration in a more focused manner.Moreover,these discussions ai

143、med to anticipate the potential hurdles that funding agencies might face in implementing AI-enhanced decision-making frameworks.Through this extended exploration,we sought to bridge the gap between the theoretical potential of AI in enhancing decision-making processes and the practical realities fac

144、ed by funding agencies.The thematic analysis presented in this chapter synthesizes the findings from both the literature and the interviews,offering a comprehensive understanding of the subject matter that builds cohesively on the landscape analysis conducted in the previous phases of our research.T

145、o provide a clear and transparent overview of our engagement with the field,we present a table of interviewees below.To respect the confidentiality and privacy of our participants,we have chosen to use generic designations for both the interviewees and their respective organizations.This approach en

146、sures the integrity of our research while safeguarding the identities of those who contributed their valuable insights to our study.Table 2.2.,The List of Interviewees Interviewee Designation Organization/Institution Director International University,Managing Internal Funds Vice President of Researc

147、h International University Program Manager Non-Governmental Organization Program Manager Federal Funding Agency(North America)Manager Small Funding Agency(North America)Manager International University Manager International University Manager Second Small Funding Agency(North America)Manager Third F

148、unding Agency(North America)15 Interviewee Designation Organization/Institution Director International Funding Agency(UK)Vice President International Funding Agency(Turkey)Director Second International Funding Agency(Sweden)Program Officer Federal Funding Agency Industry Expert and Startup Co-Founde

149、r Startup Scientist Startup Computer Scientist International Graduate University Fund Recipient Think Tank Comprehensive Analysis of Organizational Decision-Making Processes in Grantmaking Figure 2.2 delineates the multifaceted decision-making process employed by the organizations within the dataset

150、,which can be broadly categorized into several stages:(1)proposal submission and processing,(2)external or peer review,(3)internal or panel review,and(4)approval and result processing.Figure 2.2.Decision Making Process Employed by Funding Agencies 16 The grantmaking process commences with establishi

151、ng funding programs and issuing solicitations or Requests for Proposals(RFPs),which are predicated on the mission and objectives of the funding organization.RFPs can be structured based on scientific disciplines(one or multiple scientific disciplines can be solicited in a single RFP),applicant type(

152、scholars,junior researchers,experienced researchers),and the scope of technical challenges(Broad challenge,specific challenge).RFPs can be open to a large or small group of institutions and could be time-bound or open throughout the year.Distinctively,private RFPs and direct solicitations are approa

153、ches often employed by private organizations,like the Gates Foundation,to target specific applicants aligned with their unique missions and objectives.These private RFPs,unlike public ones,are not broadly advertised and are used to reach a predetermined set of potential applicants.Private RFPs and d

154、irect solicitations limit competition and target specific applicants.While private RFPs might accept proposals in narrow technology areas,a few organizations prefer a more passive approach by issuing RFPs in a broader technology sector to accommodate a range of novel ideas.Such RFPs provide a high o

155、verview of technologies of interest and include a broad set of guidelines for submission.In certain instances,the guidelines for Requests for Proposals(RFPs)do not specify a fixed budget,leaving room for flexibility based on the project type.This approach indicates that,unlike scenarios where budget

156、 constraints are a primary deciding factor,here the proposed budget is not the critical determinant of a proposals acceptance.Consequently,a proposed budget,tailored to the specific needs and scope of the project,may be accepted.For example,the Gates Foundation,as highlighted by OECD data from 2018-

157、2019,emerged as the most significant philanthropic donor among private funders(OECD,2018),follows three approaches to identify applicants:1)issues RFP publicly or privately,2)accepts Letters of Inquiry for exploring ideas,and 3)engages in direct solicitation by program managers after identifying qua

158、lified applicants.To ensure data sufficiency for our analysis,we considered entities to have sufficient data if the information was available from the proposal submission as several entities did not publicly disclose information on how they established their programs.Decision Making Process The deci

159、sion-making process in funding organizations is profoundly influenced by their scope and the types of research they fund,as revealed through our interviews.A diverse range of approaches is evident,from organizations with a broad,interdisciplinary focus to those with more targeted,discipline-specific

160、 agendas.For instance,a vice president of an international university highlighted their institutions comprehensive engagement across disciplines through their internal funding,stating:Our university doesnt limit itself to specific disciplines.We strive to push the boundaries in every field,from huma

161、nities to technology.17 This broad approach contrasts with a program manager from a North American federal funding agency who emphasized a more focused mission:Our mandate is clear,to fund research that addresses national priorities within science and technology.Each RFP is crafted with these priori

162、ties in mind,ensuring a focused and strategic allocation of resources.Similarly,the variety of types of research funded further underscores the diversity in funding strategies.An international funding agency director explained their preference for cutting-edge,interdisciplinary research that can add

163、ress global challenges,highlighting a drive to fund projects that transcend traditional disciplinary boundaries.Conversely,a manager from a small North American funding agency delineated their focus on public health initiatives,particularly those that can make a tangible difference at the community

164、level,showcasing a targeted approach to funding.This diversity in scope and research focus naturally extends to the stages of the decision-making process,which we will examine in the section below.The steps undertaken by an agencys internal staff reflect the organizations overarching goals and the t

165、ypes of research they prioritize.Proposal Submission&Processing At the core of these decision-making processes is the issuance of a call for proposals,a critical step that delineates the research themes and priorities for the funding cycle.A program officer from a federal funding agency in the US hi

166、ghlighted the importance of this stage,stating,The call for proposals sets the stage,outlining the research themes and priorities for the funding cycle.Its the starting point from which all subsequent review processes unfold.This foundational stage is pivotal in shaping the trajectory of the entire

167、funding process,guiding the direction and focus of the research proposals submitted.After the applicants submission,the internal staff of the agencies in the set conducted a combination of the following processes:1.acceptance of a proposal 2.document verification&eligibility check 3.screening 4.sele

168、cting peer reviewers.An agencys internal administrative staff primarily conducts these activities.Acceptance of Proposal Entities utilized various modalities to accept proposals and designed programs to accept proposals in stages.18 Submission Modalities:The dataset revealed three main modalities to

169、 accept proposals:online software,email,or surface mail.Most entities(80%)employ online software to accept proposals which can either be off-the-shelf or developed in-house.Some entities use email and software,depending on the program.A few public entities such as the Israel Science Foundation,Natio

170、nal Science Foundation(NSF),National Research Foundation of Korea(NRF),National Institutes of Health(NIH),and the Defense Advanced Research Projects Agency(DARPA)have made significant investments in developing software solutions.These solutions are actively being utilized across multiple agencies,de

171、monstrating their practical application and utility in enhancing the efficiency and effectiveness of grantmaking processes.However,smaller organizations with more limited funding portfolios tend to rely on off-the-shelf software solutions,which offer greater flexibility and are easier to adapt to th

172、e specific decision-making processes of an organization.Several philanthropic organizations such as Right Fund,Alfred Sloan Foundation,American Chemical Society,Oeuvre Belge du Cancer,King Baudouin Foundation,and Andrew W.Mellon Foundation have reported leveraging off-the-shelf software for acceptin

173、g applications.Submission Stages:Depending on program design,entities in the set used a one-stage or a two-stage process to accept proposals.In the latter,applicants must submit a short abstract or letter of intent(LOI)outlining the teams idea and composition.The submission is subject to a technical

174、 or eligibility review,and selected applications are asked to submit the full proposals.In contrast,the one-stage process accepts full proposals directly.The two-stage process is often seen to filter the quality of proposals at an early stage,resulting in a lower administrative burden later.However,

175、in a one-stage process,organizations such as the Israel Science Foundation and the National Institute for Psychobiology in Israel conducted technical screening to screen out low-quality proposals before sending them to peer reviewers.Document Verification&Eligibility Check Document verification ensu

176、res the applicant submits documents in accordance with program requirements.When using online submission software,this document acceptance check is partially fulfilled by automation,and applications will only be accepted when all required documents are submitted,leaving program managers to validate

177、the content of the document.However,when applications are accepted manually or by email,an additional burden of document acceptance is added to the content validation,adding to the agencys administrative burden.We examined the eligibility criteria of representative programs offered by various entiti

178、es that program managers assessed for content validation.Our analysis included collecting eligibility requirements for 1-2 representative programs per entity,which allowed us to identify a range of eligibility criteria and proposal evaluation parameters,which are as follows:Applicant Eligibility-Age

179、 of Researcher:Certain programs impose age restrictions,targeting either young or senior researchers.19-Educational Qualifications:Many programs necessitate a specific level of education,such as a degree or a particular field of study.-Experience:Prior research experience,experience as a principal i

180、nvestigator,and project leadership experience are common prerequisites.-Application History:Some programs consider whether the applicant has previously applied to the program and the outcomes of those applications.-Affiliation and Employment Status:Some programs mandate institutional affiliation.If

181、the applicant is institutionally affiliated,their employment status(part-time,full-time,consulting)may also be considered.In cases where the applicant is associated with multiple institutions,the percentage of salary collected or the association with the institution that qualifies for funding could

182、also be considered as part of the eligibility criteria.-Status of Previous Grants:Certain programs consider the status of previous grants(closed/open)received from the funding agency as a preliminary criterion for accepting a new application.-Number of Applications:A few programs limit the number of

183、 simultaneous applications an applicant can submit as a principal investigator.Institutional Eligibility-Eligibility to Receive Funds:The eligibility criteria for entities to obtain funding are highly contingent on the program and the funding institution.Innovation-based programs often delineate eli

184、gibility through a diverse range of factors,including organizational type(e.g.,small-to-medium enterprises,multinational corporations,universities,research institutes,and hospitals),legal status,performer type(e.g.,manufacturer,supplier,user),and geographical location.-Number of Applications per Ins

185、titution/Entity:Some programs limit the number of applications submitted by a single institution or entity.-Geographical Location of Institutions/Entities:International collaboration funding programs have specific eligibility requirements regarding the location of participating institutions.Some pro

186、grams permit international applicants to participate but restrict the primary applicant to be from a local institution.-Team Composition:This criterion refers to the make-up of the research team and could include a combination of factors such as the type of entity leading the project,a minimum numbe

187、r of project parties,and collaboration requirements by entity type.This assessment encompasses considerations like:Type of entity leading the project:A few programs put requirements for the type of entity(Small-to-Medium Enterprises,Multinational Corporations,Hospital,Educational Institutions,etc.)l

188、eading the project.number of allowable project parties:Vinnova,for example,in its program for bio innovation,allows a maximum of 3 project parties,with at least two of them being companies based in Sweden.Collaboration requirements by entity type:Programs restrict the entities that can participate i

189、n a research project by mandating a particular type of collaboration structure.This could encompass specifications such as the minimum number of universities,research institutions,or companies involved in the project.20-Technical Eligibility:This criterion delineates the type of research to be under

190、taken and the desired research outcome.Innovation programs frequently specify the nature and outcomes of research required by establishing particular criteria related to the technology advancement required in the project.-Adequacy of Budget and Estimated Time Frame:While some programs do not specify

191、 budget requirements,others define the acceptable budget and timeframe as a qualification for the acceptance of the proposal.Screening The screening stage is a critical juncture in the review process,often perceived as a triage mechanism(Bornmann,Mutz&Daniel,2008)aimed at reducing the administrative

192、 burden associated with identifying peer reviewers,thereby enabling both internal and external reviewers to concentrate their efforts on applications of high merit.The present analysis reveals that entities within the set employed two types of screenings to alleviate the load of application review w

193、ith the specific mode of submission determining the approach adopted.These included(1)screening of complete applications before peer review and(2)screening of Letters of Intent(LOIs)before accepting a full application.Such screening procedures are instrumental in facilitating the efficient and effec

194、tive allocation of resources,particularly in light of the substantial volume of applications received by funding entities.Notably,screening complete applications before peer review emerged as an effective strategy for reducing the administrative workload associated with identifying suitable peer rev

195、iewers for each application.Of the 25 entities in our dataset,16 had programs that accepted complete applications,and only three of these,namely the National Institute for Psychobiology,Israel Science Foundation(ISF),and SSF reported conducting technical evaluations before forwarding applications fo

196、r peer review.The criteria for screening applications were similar to the evaluation criteria used by peer reviewers,such as researchers suitability for conducting research,scope of project,strategic relevance,potential impact,and intellectual merit.The screening of LOIs before accepting the full ap

197、plication emerged as another mechanism that reduced the administrative effort of reviewing detailed applications.Eleven of the 25 entities in the dataset designed programs to accept applications in two stages,accepting an LOI at the initial stage and accepting a full application only if the LOI pass

198、ed an initial merit review.LOIs included a brief description of the proposed project,its methodology,key implementors,budget,and timeframe,and were often evaluated through eligibility criterion checks or scientifically through a merit criterion.Examples include Right Fund and Israel Innovation Autho

199、rity which invited applicants for a full proposal if the LOI met the eligibility criterion.Many other organizations evaluated LOI technically through a merit criterion.Five out of 11 entities that evaluated LOIs published criteria publicly.The merit criteria were observed to be similar to those used

200、 for the full proposal and include scientific quality,excellence,and methodology;feasibility to achieve goals;21 originality,novelty,and innovative nature of the project;intellectual merit;impact on society and sustainable development;project team and partner;and relevance of the project to program

201、priorities.Selection of Peer Reviewers The process of selecting peer reviewers is pivotal in ensuring the quality and impartiality of the external review process.Different funding agencies utilize various methods to identify and match experts with relevant expertise to evaluate research proposals.Wh

202、ile not all agencies disclose their methods,eight of those entities mentioned the methods utilized for selecting the reviewers,while nine agencies did not publicly disclose their methods.Our findings note the following methods used by agencies to identify peer reviewers:Bibliography:A bibliography o

203、f the application often serves as a good resource for identifying peer reviewers.The authors of the publications referred to could act as potential reviewers.Israel Science Foundation(ISF)mentions using a bibliography as a method to find peer reviewers.Reviewers matching:Many agencies,such as ISF an

204、d the German Israeli Foundation,conducted database searches based on the abstract or keywords in the research proposal to identify qualified and relevant reviewers.Recommendations:Applicants are often requested to provide names of potential reviewers or to indicate their preference against certain i

205、ndividuals as reviewers.The German Israeli foundation employs a committee of advisors who screen the LOI and recommend three reviewers.Additionally,the NIH solicits recommendations from scientific societies regarding suitable scientists who may serve as reviewers.Experts Database:Organizations maint

206、ain an expert database that is updated periodically based on the organizations needs.FNRS updates its expert database through its analysis,evaluation,and foresight department,considering the level of expertise needed and ensuring the load of application review with existing reviewers.Similarly,the N

207、ational Research Foundation of Korea employs its own database called NRF researcher information database and other national databases such as the Korean Researcher Base(KRB),which enlists all researchers in South Korea and other existing review pools to identify relevant reviewers.Artificial Intelli

208、gence:The implementation of Artificial Intelligence in selecting peer reviewers for grant proposals presents a vast potential in reducing administrative workload and increasing matching relevancy of reviewers.Within our dataset,the National Natural Science Foundation of China(NNSFC)is the only organ

209、ization that has reported using artificial intelligence to match peer reviewers to the proposals.The AI tool utilized by NNSFC crawls through online scientific literature databases and scientists webpages utilizing natural language processing to conduct semantic analysis and compare this information

210、 with the grant application to recommend a peer reviewer.Past Awardees:Successful grant applicants within a given area of scientific expertise are often seen as potential reviewers.22 Most often,either an internal team of the agency or a committee selected the reviewers.11 entities in the dataset me

211、ntioned that the selection of peers was performed by internal administration,while in 2 entities(ISF,Oeuvre Belge du Cancer),the selection of the peer reviewers was done by a committee.External Review Upon successful navigation of initial steps,including document verification and eligibility checks,

212、proposals enter a merit review stage.The first component of this merit review often involves an external review,where experts external to the funding organization rigorously evaluate each proposal.This peer-driven assessment focuses on the scientific and technical merits of the proposal based on pre

213、defined criteria established by the funding agency.Despite being regarded as the gold standard method by the scientific community,peer review has been criticized for its challenges in supporting radical projects and encountering bias(Bollen et al.,2014).In response to these challenges,some funding o

214、rganizations have adopted innovative strategies to enhance the fairness and objectivity of the review process.For instance,an international funding agency has implemented an online submission system combined with blinded evaluations.The vice president of this agency explains,After issuing a call for

215、 proposals,we ensure a blind review by external referees,focusing solely on the merit and potential impact of the research.This approach aims to safeguard against biases by concealing the identities of both the applicants and the reviewers,thus emphasizing the content and quality of the proposals ov

216、er the reputation or affiliations of the proposers.Furthermore,the issue of bias in reviewer selection,particularly the difficulty in finding suitable national referees for specialized topics,has been acknowledged.The Vice President of an International Funding Agency points out the complexities asso

217、ciated with this task,noting,Finding appropriate national referees for narrow subjects can be challenging,and theres a risk of personal conflicts influencing the process.This recognition underscores the importance of adopting more inclusive and diverse strategies in reviewer recruitment,ensuring a b

218、road and balanced perspective in the evaluation process.By addressing these challenges head-on,funding agencies strive to maintain the integrity and effectiveness of the peer review process,ensuring that it continues to serve as a cornerstone in the allocation of research funds based on merit and po

219、tential impact.23 Table 2.3.Review processes followed by organizations.Organization Name Accept Full proposals or LOI.Merit review process Program Considered Preliminary merit Screening*on Full Proposal/LOI Peer review Panel review*Israel Science Foundation Full Proposal Yes(Panel members screen ful

220、l proposals for peer review)Yes Yes Israel Science Foundation has four modes for review and review process changes according to programs.Full Proposal No No Yes Programs that follow a Joint review process,e.g.,The Joint Canada-Israel Health Research Program Full Proposal No No Yes Programs that foll

221、ow review by technical committee only,e.g.,New-faculty equipment grants Israel Innovation Authority LOI No No Yes Followed by a Joint innovation program,e.g.,the Uruguay-Israel R&D Cooperation Program Full Proposal Yes (Internal Administration does examination&writes an opinion on the full prop)No Y

222、es Followed by Local Innovation Program,e.g.,Government-tech track:digital innovation in the public sector National Institute for Psychobiology Full proposal Yes(Full Prop reviewed by the scientific advisory board for peer review)Yes Yes General Process The German Israeli Foundation LOI Yes (LOI rev

223、iewed by the panel members)Yes Yes Program:GIF Nexus-Solo and Collaborative Track:Vinnova Full proposal No Yes Yes Program:Bio-innovation:Enabling Technologies and processes for bio-based products SSF Full Proposal Yes Yes Yes General process for framework programs MISTRA Full Proposal No Yes Yes Pr

224、ogram:A Sustainable Blue Economy for Sweden 24 Organization Name Accept Full proposals or LOI.Merit review process Program Considered Preliminary merit Screening*on Full Proposal/LOI Peer review Panel review*Experts who perform external reviews are on the panel Diabetesfonden Full Proposal No No Yes

225、 General process for all programs FNRS Full Proposal No Yes Yes General process for all programs Innoviris Full Proposal No Occasional Yes Program:Prospective research,Proof of concept,R&D projects Research Wallonia Ministry for Public Service(SPW)LOI Yes(LOIs are reviewed by administration)Occasion

226、al Yes King Baudouin Foundation Full proposal No No Yes Program:Fund Alphonse&Jean Forton and the Belgian CF Association muco.be Oeuvre Belge du Cancer Full proposal No Yes Yes Program:Fundamental,Translational&Clinical Research 2022 LOI Yes(LOIs are screened by a scientific committee)Yes Yes Progra

227、m:Organ saving treatment and improvement of quality-of-life Grants 2022 NSFC Full proposal No Yes Yes General process for all programs Tecent Foundation Full proposal No Yes Yes New Corner Stone Investigator Program National Research Foundation of Korea Full proposal No Yes Yes General process for a

228、ll programs Right Fund Full proposal No Yes Yes Alfred P.Sloan Foundation LOI Unclear if the review is done on merit criterion Occasional No General process for all programs American Chemical Society Full Proposal No Yes Yes Program:Petroleum Research fund Andrew W.Mellon Foundation LOI Yes(Administ

229、rative Staff reviews LOIs)No No General process for all programs.The internal staff reviews LOIs and invites applicants to submit full proposals.Program staff works with the applicant to develop a full proposal.Bill and Melinda Gates Foundation LOI Yes(Administrative Staff reviews LOIs)Occasional Un

230、clear Program:Global HPV Burden Study LOI Yes(Administrative Staff reviews LOIs)Occasional Yes Program:Solutions learning implementation grant 25 In our dataset,21 entities explicitly mentioned using external review in their decision-making process,including 12 public and nine nonprofit private enti

231、ties.It was observed that organizations utilizing a scientific committee or jury for review either omitted external review or considered it occasionally,depending on the need.Entities such as the Israel Science Foundation(ISF),Israel Innovation Authority,Diabetesfonden,Innoviris,King Baudouin Founda

232、tion,and Little Giraffe Foundation did not send proposals for external review,instead predominantly conducting panel reviews by scientific committees.Philanthropic agencies such as the Gates Foundation and the Sloan Foundation and public entities like DARPA and SPW allowed program managers to evalua

233、te the need for peer review depending on the budget and complexity of the project.The external review criteria were similar to those adopted for the screening of Letters of Intent(LOIs)before accepting full proposals.Many entities in our dataset did not define the review criteria and used a combinat

234、ion of the criteria below for the merit review.Consistent with the systematic review by Hug et al.(2020),our dataset confirms the utilization of core peer review criteria such as originality,academic relevance,and plausibility/soundness across grant proposals.These criteria predominantly evaluate th

235、e projects objectives and the research methodologys robustness.However,our analysis further extends these dimensions to include criteria specific to innovation programs.This encompasses assessing the potential for commercialization,scalability,or technological exploitation of research outcomes,refle

236、cting a broader scope of evaluation that captures both the intellectual merit and the practical impact of proposed research endeavors.This extended set of criteria underlines the multifaceted nature of grant evaluations,accommodating the diverse aims of research and innovation funding agencies.Organ

237、ization Name Accept Full proposals or LOI.Merit review process Program Considered Preliminary merit Screening*on Full Proposal/LOI Peer review Panel review*Little Giraffe Foundation LOI(no full proposals are accepted)No No Yes Program:Neonatal Research Initiative DARPA LOI Yes(Program manager review

238、s LOIs)Occasional No General process NIH Full Proposal No Yes Yes General process NSF LOI/Full Proposal Yes(for screening LOIs in some programs)Depending on program Depending on program General process 26 Scientific Quality and Excellence All entities in the dataset considered the evaluation of the

239、ideas scientific merit to be a primary criterion,evaluated in terms of its originality and approach.The originality of the research idea,often referred to as novelty of research idea or originality and innovativeness of research,refers to the value addition of proposed research over prior knowledge.

240、The intellectual merit of the proposed research includes the evaluation of a scientific approach to solving a research problem.The entities in the dataset define the criterion in various forms.For example,the National Science Foundation(NSF)defines this criterion as evaluating the research projects

241、potential to advance knowledge and understanding in a field of study.ISF mentions evaluating proposals on the suitability of the systematic approach and chances of success,while the American Chemical Association evaluates if the research is testable and hypothesis-driven.Additional criteria,such as

242、the degree of interdisciplinarity,were also noted by organizations such as the Swedish Foundation for Strategic Research(SSF).Feasibility to Achieve Goals The feasibility of achieving expected outcomes is a holistic evaluation of the project plan,budget,and experience of the project team.The overall

243、 project plan is evaluated for its clear articulation and planned activities or workflows.Reviewers evaluate the approach to see if the planned activities are relevant to achieving the anticipated outcome and if it is realistic with respect to budget and planned activities.For example,Vinnova evalua

244、tes the proposals on how technology readiness levels(TRLs),market readiness levels(MRLs),and sustainability readiness levels(SRLs)are described along with their movement of these parameters during the project and whether the planned activities contribute to the achievement of expected results.DARPA

245、and ISF evaluate if the technical risks in the approach have been identified and potential mitigation efforts are defined and feasible.Project Team and Collaborators:Most entities within the dataset indicated that evaluating a teams suitability for conducting research was a significant aspect of the

246、ir review process.This evaluation encompasses an analysis of the teams composition,the individual experience of team members,their historical accomplishments(both scientific and entrepreneurial),international exposure,and the ability of the project lead and the organization to manage the projects pl

247、an,budget,and timelines.The assessment determines whether the team possesses the complementary expertise necessary to implement the projects planned activities.The evaluation of team members experience is conducted through a review of their publications,academic records,resumes,proven track records

248、supporting similar work,and other documents provided with the application.In the context of innovation programs,the involvement of the user or owner of the technology in the project team is deemed favorable.The technology owner is distinct from the user,who would utilize the technology post-commerci

249、alization or licensing.The involvement of the technology owner is critical,as they have the in-depth knowledge and vested interest in the technologys development and potential applications.Certain agencies also anticipate a plan to manage gender diversity within the team.For instance,Vinnova mandate

250、s a balanced gender distribution within the team 27 and an equitable distribution of power and influence in terms of work tasks and roles within a 40/60 percentage range.Moreover,they evaluate the extent to which the technology owner is integrally involved in the project,ensuring that the research t

251、rajectory is likely to lead to impactful commercialization or practical application of the technology.Budget and Timeframe:The project is also evaluated based on the rationale and feasibility of executing the proposed plan within the stipulated budget.DARPA incorporates cost realism as one of the ev

252、aluation factors,assessing whether the proposed costs align with the technical and management approach and are consistent with the applicants scope of work.The costs provided must be detailed,and the budget should ideally reflect the realistic effort required to complete the project.Entities such as

253、 the National Natural Science Foundation of China(NNSFC),Right Fund,American Chemical Society,and German Israeli Foundation have mentioned evaluating the rationality of the plan for the adequacy and utilization of funds.Reviewers also assess the approach concerning the timelines for project realizat

254、ion,determining whether the milestones are appropriate or aggressive and whether a rationale supports them in the case of aggressive timelines.Societal Impact and Potential for Commercialization,Scale-up,or Exploitation Researchers are often asked to provide a commercialization or exploitation plan

255、for the proposed research.The proposed project is expected to have the potential to move up in the value chain.For applied research projects,their potential for commercialization is evaluated,while for innovation projects,depending on their expected TRL level,the potential for further scale-up is ev

256、aluated.Vinnova,in its innovation programs,evaluates the proposal for its plan to make a clear shift in TRL,MRL,and SRL levels.SSF retains 4%of the grant for exploitation efforts and evaluates the proposal on its vision to utilize the research results in Sweden in the medium to long term.Strategic R

257、elevance Agencies evaluate research projects on their ability to drive their program missions and priorities.Many agencies issue programs for technological priorities,so evaluation ensures the proposed research falls within its priorities.Vinnova and SSF evaluate the research projects for their stra

258、tegic relevance,where proposed research should provide solutions to important application problems or enable future applications,products,or services.Panel Review Following the external review,or in some cases as an independent assessment,a panel review is conducted.The panel typically consists of 3

259、 10 members possessing diverse backgrounds,who rate the applications in comparison to each other based on a predetermined criterion.The members rank the proposals according to merit relevance to the agencys mission and recommend the decision to the higher management.They may consider the external ev

260、aluation,program managers evaluation,and their professional evaluation of the application.28 The program manager assists the panel by providing a summary of applications and peer review ratings,managing conflicts of interest among the panel members,highlighting any red flags,training the panel membe

261、rs on the review process,and ensuring that each application is provided a thorough review.The program manager can provide a recommendation but will not provide the rating.The board of directors chairs the panel in the case of many foundations or is led by a program director or center director in the

262、 case of public funding organizations.Twenty-one entities in the dataset mentioned using panel review in their decision-making processes.In comparison,three entities(Alfred P.Sloan Foundation,Andrew Melon Foundation,and DARPA)did not mention using panel review.The program managers in such entities i

263、nstead worked with the potential applicant to develop a full proposal.They had more discretion in making a recommendation based on the external review results.In entities such as the Gates Foundation,the utilization of panel reviews is not a fixed protocol but is adapted to suit the nature and requi

264、rements of each specific funding program.Entities in which a panel review was followed by an external review evaluated and ranked proposals based on comments from the external review and aligned the proposal with the organizations funding strategy and budget.NIH conducts a special review during this

265、 stage for the proposals that have requested$1 million or more in direct costs and considers other factors such as portfolio balance,programmatic priorities,IC priorities,and availability of funds.Meanwhile,for entities that opt not to use an external review,proposals are evaluated by an internal re

266、view panel.This panel uses a set of criteria similar to those that would be employed in an external review to ensure a consistent and thorough assessment of each submission.Agencies followed varied guidelines and processes for running the panels.In MISTRA,experts who conducted the external review we

267、re later invited to do the panel review.In others,an external review was conducted by a set of experts different from that of a panel review.Israel Science Foundation,National Institute of Psychobiology-Israel,German Israeli Foundation,Swedish Foundation for Strategic Research,and Oeuvre Belge du Ca

268、ncer involved panel members at an early stage.The panelists conducted a preliminary review for screening Letters of Intent(LOIs)for peer review.They convened a second time to review the evaluation from peer review and recommend the final decision.Few Several entities,such as FNRS,NNSFC,and King Baud

269、ouin Foundation,maintained panels in various fields,with members appointed for fixed terms.The applicant chooses the panel or scientific committee to evaluate the application based on the field.Panel reviewers from NIH and NSF reported receiving 20 to 100 proposals before the meeting and are assigne

270、d as primary or secondary reviewers on 6-8 proposals.Reviewers took 15 60 hours to read and write reviews(Porter,2005).Entities in the set were observed to follow different types of reviews depending on the program requirements.External review:Several programs within NSF,Alfred Sloan Foundation,Bill

271、 and Melinda Gates Foundation,and DARPA utilized external review as the primary 29 mechanism for evaluating the applications merit.Program managers in such programs make recommendations based on the ratings provided by external reviewers.Panel review:Certain programs of ISF,Israel Innovation Authori

272、ty,Mistra,Diabetesfonden,Innoviris,King Baudouin Foundation,Little Giraffe Foundation,and NSF solely relied on panel review to evaluate the proposals merit.NSF reported that multidisciplinary proposals may undergo multiple panel reviews.The panel in such programs recommends proposals to be awarded.C

273、ombination of external and panel review:Several programs in the entities in our dataset followed an approach that utilized external review followed by panel review for decision making.Internal review:In a few programs of NSF and Andrew Melon Foundation,the evaluation of application merit is done sol

274、ely by program managers without the involvement of external and panel reviewers.Amidst these varied methodologies,achieving consensus among reviewers,particularly in panel settings with diverse expertise,poses a significant challenge.The director of an international funding agency encapsulates this

275、challenge by stating,We aim for consensus in the group,but the diversity of backgrounds can make this challenging.This statement underscores the inherent complexities of balancing varied perspectives within the decision-making process.The diversity of reviewer backgrounds enriches the evaluation wit

276、h a broad spectrum of insights.Still,it complicates the attainment of a unanimous decision,reflecting the nuanced and multifaceted nature of assessing research proposals.Table 2.4.Overview of Grant Evaluation Processes and Criteria Across Entities Category Details Entities Using External Review 21 e

277、ntities,including 12 public and 9 private nonprofit entities Entities Omitting External Review ISF,Israel Innovation Authority,Diabetesfonden,Innoviris,King Baudouin Foundation,Little Giraffe Foundation Entities Allowing Managerial Evaluation Gates Foundation,Sloan Foundation,DARPA,SPW Common Review

278、 Criteria Originality,academic relevance,plausibility/soundness,potential for commercialization,scalability,technological exploitation Scientific Quality&Excellence Primary criterion across entities.Variability in definitions,e.g.,NSF focuses on advancing knowledge,ISF on systematic approach and suc

279、cess likelihood,American Chemical Association on testability and hypothesis-driven research.Feasibility to Achieve Goals Assessed via project plan clarity,team suitability,and budget/timeframe feasibility.Specifics like Vinnovas evaluation on TRLs,MRLs,SRLs,DARPAs and ISFs technical risk and mitigat

280、ion assessment.Societal Impact&Commercialization Evaluated based on the projects commercialization plan and its potential for value chain movement.Entities like Vinnova assess shift in TRL,MRL,and SRL levels.SSF focuses on research utilization in Sweden.Strategic Relevance Projects assessed for alig

281、nment with agency missions and priorities,with entities like Vinnova and SSF evaluating strategic relevance in terms of solving application problems or enabling future applications.30 Category Details Panel Review 21 entities mentioned using panel review,with procedures varying.Some entities like Al

282、fred P.Sloan Foundation,Andrew Melon Foundation,and DARPA might not use panel review but prefer managerial discretion.NIH has special considerations for large funding requests.Review Types and Processes Varied across entities:NSF,Alfred Sloan Foundation,and others use external review;ISF,Israel Inno

283、vation Authority,etc.,rely on panel reviews;combinations or internal reviews are also present.The process complexity reflects the diverse nature of grant evaluations,balancing intellectual merit and practical impact in decision-making.Challenges in Consensus Highlighted by the diversity of backgroun

284、ds in panel settings,affecting unanimous decision-making.This complexity underscores the nuanced nature of assessing research proposals.Approval and Result Processing The multifaceted review process,encompassing both external/peer review and internal/panel evaluations,concludes with the formulation

285、of recommendations for funding.These recommendations then advance through one or two tiers of approval within the funding organization before applicants are informed of the decisions.The authority accountable for endorsing the decisions of the review group varied among the entities in the dataset.It

286、 encompassed a board,council,jury,committee,director general,Chief Executive Officer,or government representative.The board serves as the final approving body in the National Institute for Psychobiology,German Israeli Foundation,Swedish Foundation for Strategic Research(SSF),Mistra,Diabetesfonden,Qu

287、een Elisabeth Medical Foundation for Neurosciences,Oeuvre Belge du Cancer,American Chemical Society,and Andrew Melon Foundation.A few organizations were observed to have multiple levels of approval.For instance,in the Israel Science Foundation(ISF),the review committee forwards the recommendation to

288、 the academic board,which subsequently presents it to the academic council.Similarly,in Vinnova,the program manager communicates the decision from the panel to the director,who further conveys it to the director general.In the case of Innoviris and Service Public de Wallonie(SPW),a minister or secre

289、tary of state responsible for scientific research serves as the final approving body.The emphasis on collective decision-making is echoed by a project manager from an NGO who underscores the collaborative nature of these decisions:No one person is making a unilateral decision;these are really based

290、on a breadth of reviews and perspectives to really capture the best thinking.This sentiment reaffirms the value of incorporating a wide range of insights and evaluations,ensuring that funding decisions reflect comprehensive expert opinions and a collective commitment to supporting the most promising

291、 and impactful research initiatives.31 Challenges with the current process A comprehensive literature review and web-based exploration were undertaken for each entity within the dataset to identify any reported challenges within the grant management process.It was discerned that most entities within

292、 the dataset did not publicly disclose any challenges.However,a few challenges were identified by the King Baudouin Foundation in Belgium and the National Natural Science Foundation of China(NNSFC),elaborated upon below.Evidence of such challenges noted in the literature referred to:Database Managem

293、ent The King Baudouin Foundation reported a challenge of lack of integration across its programs in 2006(King Baudouin Foundation,2006).Each program operated independently,creating over 2300 nonstandard databases by individual staff members using different formats.This led to a high volume of duplic

294、ate organizations and contacts.The organization required an online system tailored to its needs,which could further integrate the requirements of all programs.This challenge was resolved by creating a customized solution in partnership with MicroEdge(King Baudouin Foundation,2006).The increase in fu

295、nding applications over the years has also resulted in an increase in outputs such as proposals,outcomes,patents,publications,etc.,which has further posed a challenge for funding agencies to gain a higher view of their funding portfolio and assess impact.In an interview with decision-makers and prog

296、ram staff in US federal agencies,program officers expressed the need for tools to view the funding portfolio in broader terms and allow for a drill-down capacity(Vorvoreanu et al.,2015).Review System The NNSFC noted several issues with the peer review system(Chen et al.,2021).Increased administrativ

297、e burden of finding reviewers due to high volume of applications:The number of proposals received yearly at NNSFC has been increasing,with more than 280K applications in 2020,which increased the load on administrators and reviewers(Chen et al.,2021).This challenge is observed worldwide in many agenc

298、ies.For instance,in 2019,the National Science Foundation(NSF)reviewed 41,024 competitive full proposals(NSF,2019).A study on stakeholder views of peer reviews of the National Institute of Health Research(NHIR)UK reported the time and effort required to identify reviewers as its biggest challenge(Tur

299、ner et al.,2018).Another survey of 57 international public and private organizations in the field of biomedical research reported declined review requests,late reports,administrative burden,difficulty finding new reviewers,and reviewers not following guidelines as frequent or very frequent challenge

300、s(Schroter et al.,2010).The administrative burden of the process was reported to have increased over the past five years(Schroter et al.,2010).According to a grant-in-review report based on a survey of 4,700 researchers worldwide,funders spend 2-6 hours finding peer reviewers for an application(Publ

301、ons,2019).A program manager would invite at least three reviewers to secure one reviewer.It was also found that 50%of the reviewers 32 declined a proposal for review because they were too busy with other commitments(Publons,2019).Funders often maintain a pool of peer reviewers,which makes finding re

302、viewers easy,as the funders trust the reviewers with quality reviews,ensure avoiding conflicts of interest,and reduce the need to train the reviewers.However,with an increasing number of applications yearly,funders find it difficult to identify reviewers without a conflict of interest and with relev

303、ant expertise to evaluate proposals.In Publons survey,it was found that 40%of the researchers rejected the request for review because the proposal did not fit with their expertise(Publons,2019).Due to the need to find reviewers,program managers often allocate proposals to peer reviewers who might no

304、t be qualified experts to comment on the proposals.Peer reviewers in such scenarios may reject the proposal or could accept the proposal and provide a bad-quality review.Finding the right expert might require looking out of the internal pool of peer reviewers,which adds to the administrative burden

305、on the funders.Our discussions with key stakeholders from funding agencies provided a more nuanced view of the challenges associated with the reviewer selection process.A director from an International Funding Agency highlighted the significant effort involved in recruiting reviewers who are both wi

306、lling and capable of delivering high-quality assessments,stating,Identifying and ensuring the commitment of quality reviewers is a time-intensive and often imprecise part of our process.This challenge is compounded by the need for these reviews to be not just completed but of high quality,underlinin

307、g the complexity of the issue at hand.Adding another layer to this complexity,an NGO Program Manager brought attention to the lack of incentives for reviewers,which is a notable gap in the current ecosystem.The reliance on the goodwill of reviewers to contribute their expertise without tangible rewa

308、rds was underscored by their observation:Currently,we dont offer incentives for reviewing,relying on reviewers willingness to contribute to the scientific community.Reviewers Fatigue:The increase in applications results in a burden not only on administrators but on reviewers too.At NNSFC,a decrease

309、in review quality was observed due to an increase in the number of applications per reviewer,with a few experts reviewing more than 50 applications(Chen et al.,2021).This issue has been noted with other funding agencies worldwide.According to a study by Aczel et al.,it was found that the total time

310、reviewers globally worked on peer reviews was over 100 million hours to publication peer reviews in 2020 alone,equivalent to over 15 thousand years(Aczel et al.,2021).The estimated monetary value of the time US-based reviewers spent on reviews was over 1.5 billion USD in 2020.For China-based reviewe

311、rs,the estimate is over 600 million USD,and for UK-based,close to 400 million USD.This has increased“reviewers fatigue”and decreased the quality of reviews provided by reviewers(Aczel et al.,2021).In our exploration of the challenges facing funding agencies during the peer review process,a recurring

312、 theme emerged from both our literature review and interviews:the lack of clear norms and guidelines for review.The NNSFC report found that the 33 evaluation received from reviewers lacked norms and guidelines for review(Chen et al.,2021).Often,the criteria for review are not clearly defined to the

313、reviewers;hence,it was left at the reviewers discretion to select their definition of the review criterion.In an interview with panel members of the European Research Council,the members pointed out the lack of specification,applying criteria,and choosing indicators as a challenge.This ambiguity som

314、etimes caused proposals that met the unstated criteria to be perceived unfavorably,even though reviewer comments did not necessarily justify such negative assessments.(van den Besselaar et al.,2018).Further compounding these challenges are the nuanced biases and operational inefficiencies identified

315、 through our interviews.The influence of institutional prestige was acknowledged by representatives across the spectrum of funding agencies,with a small funding agency manager noting the bias towards well-known universities:Seeing that someone works at X or Y or Z,you know,immediately kind of brings

316、 them to the top of the pile.This institutional bias risks overshadowing meritorious proposals from less renowned entities,thereby limiting the diversity of research that receives funding.Another concern raised by interviewees was the impact of existing relationships,often referred to as the Old Boy

317、s Network.A small funding agency manager and an NGO project manager highlighted this dynamic,where ongoing interactions with funders can advantage previous recipients.The former described a cycle where initial funding facilitates regular contact with funders,potentially biasing future funding decisi

318、ons towards these established connections.Lastly,the lack of comprehensive evaluation tools was a significant barrier to effective decision-making.A small funding agency manager lamented the reliance on basic proposal reviews and previous relationships rather than on more sophisticated evaluation me

319、chanisms:Its mostly based on evaluating the proposal and any former relationship with the researcher,rather than any major tool that could help them.This deficiency underscores the urgent need for enhanced tools and methodologies to ensure that funding decisions are made based on the merit and poten

320、tial impact of the proposed research.The review system does not support evaluating innovative and disruptive projects that China aims to fund:As noted by NNSFC,literature has also documented this concern as a significant challenge to the review process.Although peer review is considered a critical c

321、omponent of the process,it does not support the risk-taking element of novel and unconventional ideas,thereby discouraging scientific progress(Hug&Aeschbach,2020).Other potential challenges Portfolio Monitoring and Impact Assessment:Program managers necessitate sophisticated knowledge management sys

322、tems that effectively monitor ongoing projects and support comprehensive impact assessment.While current systems provide basic functionalities for searching and filtering projects,they lack advanced monitoring and analytics features.These advanced features are crucial for understanding the broader i

323、mpacts of funded projects,including 34 their scientific,societal,and economic contributions.The ability to visualize data trends,project progress,and outcomes in real time would significantly enhance the capacity for informed decision-making and strategic planning,thereby addressing a critical gap i

324、n current grant management practices(Vorvoreanu et al.,2015).Evolution and Adaptations in Response to the Challenges In response to the challenges identified within their operational frameworks,funding agencies have embarked on a journey of evolution and adaptation.This dynamic response is character

325、ized by a shift from traditional methods to the adoption of advanced technological solutions and innovative practices.A significant step in this evolution has been the modernization of grant management systems.As articulated by a director of an international funding agency,the move away from outdate

326、d technology has been pivotal in addressing the inefficiencies and inflexibility that hampered older systems,marking a crucial step towards enhancing operational efficiency.The journey towards digital transformation is exemplified by adopting platforms like Salesforce,which has revolutionized how ag

327、encies operate.A manager from a small funding agency recounted the initial hurdles of this transition,noting,The transition was challenging,but the long-term benefits in operational streamlining and funding diversity tracking were undeniable.This reflects a broader trend towards digitalization,aimin

328、g to streamline operations and improve the management of diverse funding portfolios.Leadership changes within these organizations have often served as a catalyst for embracing technological advancements.Reflecting on the transformative impact of new leadership,a program manager highlighted,The new p

329、residency marked the beginning of a more technology-embraced era in our processes.This underscores the significant influence that visionary leadership can have in steering organizations towards more technologically integrated solutions to overcome traditional challenges.Some organizations have opted

330、 to maintain a degree of consistency in their grant management processes while acknowledging the need for updates.A director from one such agency admitted,Our methods have served us well since 1994,but the recognition that we must update our approaches to stay relevant is clear.This highlights the d

331、elicate balance between preserving what has been effective and adapting to remain competitive and relevant in a rapidly evolving landscape.35 Even incremental adjustments within these frameworks have been shown to impact the effectiveness of grant management processes significantly.A vice president

332、of an international funding agency observed,Subtle but strategic adjustments,especially in our dealings with commercial partners,have significantly smoothed our operations.This illustrates how minor modifications,thoughtfully implemented,can facilitate substantial improvements,underscoring the ongoi

333、ng commitment of funding agencies to refine and enhance their operations in response to both internal challenges and external pressures.Insights and Solutions from Grant Management and Scholarly Publishing Having delineated the challenges encountered in the current grant management process,including issues related to database management,the review system,and portfolio monitoring,it becomes imperat