专题研究

国内外生成式AI大模型执行情报领域典型任务的测试分析

  • 赵 浜 ,
  • 曹树金
展开
  • 1 中山大学信息管理学院 广州 510006; 2 山东理工大学信息管理研究院 淄博 255000)
赵 浜,男,1989年生,中山大学信息管理学院博士研究生。 曹树金,1962年生,山东理工大学特聘教授,中山大学信息管理学院教授、博士生导师(通讯作者)。

网络出版日期: 2023-09-21

Test Analysis of Typical Tasks in the Information Field Performed by Generative AI Large Models at Home and Abroad

  • Zhao Bang ,
  • Cao Shujin
Expand
  • 1 School of Information Management, Sun Yat-sen University, Guangzhou, 510006; 2 Information Management Research Institute,Shandong University of Technology, Zibo, 255000)

Online published: 2023-09-21

摘要

[目的/意义]面对AI进步与ChatGPT类大模型发展为情报领域带来的前所未有之变革,文章多方面分析 国内外生成式AI大模型执行情报领域典型任务的有效性与可靠性,为高效利用其能力助力情报工作提供参考。 [方法/过程]文章结合情报领域多方面工作内容设计问题,分别在Gpt-3.5-Turbo与ChatGLM-6B大语言模型上进 行测试、对照与提示工程探索,采用专家打分法进行测评,详细分析评价大语言模型的9项能力水平。[结果/结论] 大语言模型在执行多数典型情报任务上都展现出了较强能力,运用好提示工程能大幅提升大语言模型的优势能 力,结合外部工具可以有效弥补大语言模型自身的短板,充分挖掘大语言模型潜力将带来情报工作效率的巨大 提升。

本文引用格式

赵 浜 , 曹树金 . 国内外生成式AI大模型执行情报领域典型任务的测试分析[J]. 情报资料工作, 2023 , 44(5) : 6 -17 . DOI: 10.12154/j.qbzlgz.2023.05.001

Abstract

[Purpose/significance] In the face of the unprecedented changes brought about by the advancement of AI and the development of large language models like ChatGPT in the field of information science, the effectiveness and re? liability of generative AI large language models in performing typical tasks in the field of information science are ana? lyzed in many ways, in order to provide advice for efficiently utilizing their abilities to assist scientific research and practical work. [Method/process] Questions were formed by combining the research and practical work content of vari? ous topics in the information field. Tests, comparisons and prompt engineering explorations were carried out on Gpt3.5-Turbo and ChatGLM-6B respectively, and experts-based rating method is used to score, then analyze and evalu? ate the nine abilities of large language models in detail. [Result/conclusion] The large language model has shown strong ability in performing most typical information tasks. Using the prompt engineering can greatly improve the supe? riority of the large language model. Combining external tools can effectively make up for the shortcoming of the large language model itself. Fully exploiting the potential of a large language model will bring about a huge improvement in information work efficiency.
文章导航

/