Pitfalls in the assessment of higher education learning gains in Brazil through multilevel regression: handling the ceiling effect

¹ Department of Production Engineering, Federal University of the State of Rio de Janeiro, Rio de Janeiro, Brazil
² Program of Production Engineering/COPPE, Federal University of the State of Rio de Janeiro, Rio de Janeiro, Brazil
³ Department of Information Systems, Federal Center for Technological Education Celso Suckow da Fonseca, Rio de Janeiro, Brazil
⁴ Department of Production Engineering, Federal Center for Technological Education Celso Suckow da Fonseca, Rio de Janeiro, Brazil

^* Corresponding author: ana.souza@cefet-rj.br

Received: 3 May 2024
Accepted: 23 October 2024

Abstract

This work assesses the validity of the Brazilian Indicator of the Difference between Observed and Expected Performances (IDD), which applies a multilevel regression model to rank university courses. It proposes a methodology to test for inconsistencies and research for the causes. With this objective, we develop an experimental methodology, applied to a database with 4432 production engineering students distributed over 329 undergraduate courses, from the government’s National Institute for Educational Studies and Research. We display a graphical analysis of the expected and achieved grades in final graduation exams, the gap in which determines the learning gains for every course. Then we apply the multilevel linear regression model of the IDD in a particular scenario, to test for consistency in value-added learning measurements using the technique reductio ad absurdum. The result reveals that the present methodology for the calculation of the IDD is inadequate and inconsistent, and that this can be explained by the ceiling effect. It needs to be radically revised to prevent unfair judgements. Finally, we propose and implement a production frontier technique to correct for the ensuing ceiling effect, namely a variable returns to scale data envelopment analysis model for adjusting the measurements using evidence based benchmarks.