[Purpose/significance] The rapid development of generative artificial intelligence has propelled changes in user interaction behaviors and cognitive patterns. This study focuses on the differences in users' interactive behaviors with generative artificial intelligence and their evaluations of system performance under tasks with different levels of cognitive complexity. [Method/process] This study employs the user experimentation method and uses the Kruskal-Wallis test to explore the impact of task cognitive levels on user interaction behaviors and system evaluations. It also employs thematic analysis to induce new metrics for user evaluations of generative AI. [Result/conclusion] In terms of interaction behavior, during the completion of tasks at different cognitive levels, the duration of information usage by us⁃ers remains almost unchanged, while it is always the duration of information acquisition that varies. Tasks of evaluation and creation take the most time, involve the most question-and-answer rounds, and have the shortest average response
length. In the answers to application and analysis tasks, the total copy ratio is lower than that of memory, understanding,evaluation and creation tasks. In terms of system evaluation, tasks of evaluation and creation receive the poorest overall performance. Regarding new evaluation metrics, the study has induced 6 major categories and 17 minor categories of new metrics. This paper supplements user-centered evaluation metrics for generative AI at the theoretical level and helps intelligent systems understand user behavioral characteristics under different task types at the practical level, pro⁃viding more targeted information services.
Li Yujia
,
Ran Xiaoya
,
Liu Chang
. Research on the Impact of Task Cognitive Level on User-Generative Artificial Intelligence Interaction Behavior and System Evaluation[J]. Information and Documentation Services, 2025
, 46(2)
: 51
-60
.
DOI: 10.12154/j.qbzlgz.2025.02.006