The Effect and Feasibility Analysis of ChatGPT Used in Paper Innovativeness Evaluation

  • Wang Yaqi ,
  • Cao Shujin
Expand
  • 1 School of Information Management, SunYat-sen University, Guangzhou, 510006; 2 Information Management Research Institute, Shandong University of Technology, Zibo, 255000)

Online published: 2023-09-21

Abstract

[Purpose/significance] This study aims to explore the effectiveness and feasibility of using ChatGPT for evaluating the innovativeness of academic papers. If generative artificial intelligence, represented by ChatGPT, can be employed for paper innovativeness assessment, it has the potential to significantly improve the efficiency of innovative? ness evaluation. [Method/process] By simulating real-world demands and application scenarios for evaluating paper innovativeness, we conducted over 24 rounds of innovativeness evaluation experiments across six research domains: life sciences, cellular immunology, marketing, economics, artificial intelligence, and information resource management. Content analysis was employed to thoroughly examine the text generated by ChatGPT during the process of evaluating paper innovativeness, enabling us to study its approach, criteria, and feasibility. [Result/conclusion] The study re? vealed that ChatGPT demonstrated excellent performance in the task of evaluating paper innovativeness, displaying high comprehensiveness and scientific rigor in its evaluation methods and criteria. It effectively accomplished paper in? novativeness evaluations across these research fields. At present, while ChatGPT's stability, accuracy, and authenticity may have some minor limitations, it can still be considered an efficient auxiliary tool for evaluating paper innovative? ness in relevant tasks. The identified shortcomings of using ChatGPT for paper innovativeness evaluation are likely to be gradually addressed with the advancement of prompt engineering, opening up of Internet access, and increased avail? ability of academic databases.

Cite this article

Wang Yaqi , Cao Shujin . The Effect and Feasibility Analysis of ChatGPT Used in Paper Innovativeness Evaluation[J]. Information and Documentation Services, 2023 , 44(5) : 28 -38 . DOI: 10.12154/j.qbzlgz.2023.05.003

Outlines

/