🎖️Operationalizing Knowledge Gain: Implementing and Testing the DKG Metric in Search Environments

Searching the Web is increasingly recognized as a process of knowledge construction rather than simple information retrieval, a perspective framed by the paradigm of Searching as Learning (SaL). A central challenge in this domain lies in evaluating the extent to which users actually acquire knowledge during search. Traditional approaches either rely on behavioral proxies, scalable but limited in capturing conceptual change, or structured assessments, which provide direct evidence but are often intrusive. The Degree of Knowledge Gain (DKG) metric addresses this gap by modeling reductions in uncertainty through Shannon’s entropy and integrating semantic similarity between queries and clicked documents. This paper reports on the operationalization of DKG within the CNPq project 3C-BPA: Comportamento de busca, Complexidade da informação e pensamento Crítico na Busca como um Processo de Aprendizagem. Two artifacts were developed: an initial search engine prototype embedding DKG computation, and a Chrome extension that estimates DKG in real time while users employ their preferred search engines. The latter artifact overcame earlier limitations by improving ecological validity, reducing costs, and enabling more natural experimentation. An experiment combined pre- and post-tests, the Concurrent Think-Aloud (CTA) protocol, and the plug-in’s automated logging. Preliminary findings show that DKG values are sensitive to differences in search strategies, with systematic reformulation and evaluation aligning with greater knowledge gains, while disorientation behaviors corresponded to more modest outcomes. A distinctive feature of this study was the active role of an undergraduate researcher, who contributed to artifact development, experiment setup, participant support, transcription, and ongoing content analysis.