TIBAU, Marcelo; SILVA, Rafael Tavares da; SIQUEIRA, Sean Wolfgand Matsui; NUNES, Bernardo Pereira. Evaluating Knowledge Gain in Search Environments: An Exploratory Study of Learning Measurement. In: SIMPÓSIO BRASILEIRO DE SISTEMAS DE INFORMAÇÃO (SBSI), 22. , 2026, Vitória/ES. Anais […]. Porto Alegre: Sociedade Brasileira de Computação, 2026 . p. 478-496. ISSN 3086-4836. DOI: https://doi.org/10.5753/sbsi.2026.248557.
Evaluating Knowledge Gain in Search Environments: An Exploratory Study of Learning Measurement
Authors
Marcelo Tibau (UNIRIO)
Rafael Tavares da Silva (UNIRIO)
Sean Wolfgand Matsui Siqueira (UNIRIO)
Bernardo Pereira Nunes (ANU-Austrália)
Abstract
Research Context: Searching as Learning (SaL) frames web search as a process where users construct and refine knowledge. However, measuring knowledge gain in natural search environments remains a methodological challenge. Scientific and/or Practical Problem: Traditional behavioral proxies (e.g., dwell time, clicks) scale well but fail to capture conceptual change, while pre/post-tests provide richer insights but are intrusive. This gap limits the development of search systems that can evaluate and promote learning. Proposed Solution and/or Analysis: This study advances a computational measure based on entropy reduction and semantic similarity, and novelly operationalizes it through a browser plug-in that enables real-time measurement in natural search environments, extending prior formalizations and prototype-based validations of the DKG metric. Related IS Theory: The study draws on Shannon’s Information Theory and Information Processing Theory in IS to conceptualize knowledge gain as uncertainty reduction supported by socio-technical processes. Research Method: An experiment combined three structured search tasks, pre/post-tests, and Concurrent Think-Aloud protocols. Quantitative measures (Transfer of Learning scores, also known as ToL, and values from the proposed metric) were triangulated with qualitative coding using OISS and ESKiP frameworks. Summary of Results: Statistical analysis showed a moderate positive correlation between ToL and the proposed metric (r = 0.62, p < 0.01). Bland–Altman analysis revealed systematic differences in scale, with ToL showing higher values, yet relative patterns were consistent. Transcripts emphasized how strategies such as query specialization, evaluation of sources, and persistence in reformulation aligned with higher values. Contributions and Impact to IS area: The study contributes a validated computational metric and artifacts for measuring knowledge gain in real search environments. It reinforces the sociotechnical view of IS by linking human strategies, processes, and technological advantages, and points to adaptive search systems that could measure and promote learning.
DOI: 10.5753/sbsi.2026.248557
URL: https://sol.sbc.org.br/index.php/sbsi/article/view/41338
