The impact of assessment time limits on accounting students’ integration of technical accounting knowledge with higher-order thinking skills

Abstract

This study explores the impact of assessment time limits on students’ ability to demonstrate technical competence in accounting – which is defined as integrating technical accounting knowledge with higher-order thinking skills (HOTS), underpinned by the foundational role of lower-order thinking skills (LOTS). The intention of an accounting assessment may be to measure technical competence in accounting. However, the results may indicate competence in accounting under speeded time conditions. If performing a task accurately under speeded conditions (fluency) is not an intended construct to be measured, a construct-irrelevance threat to validity arises. The study was conducted in three phases. Phase 1 involved designing a case study-based assessment instrument with 30 multiple-choice questions (MCQs) aligned with Bloom’s Taxonomy assessment instrument to measure technical competence in accounting. In Phase 2, the instrument was subject to a validation process – internally using Item Response Theory (IRT) and externally through the Delphi method. In Phase 3, the instrument was used in a randomised post-test-only experimental design with 426 undergraduate accounting students, manipulating time limits across three conditions: Timeminus, Timestandard, and Timeplus. In addition to exploring the impact of time limits on LOTS and HOTS, the experiment design allowed for investigating the impact of question order and students’ behavioural response strategies in time-restricted assessments. Findings showed that stricter time limits negatively impact students’ ability to integrate technical knowledge with both LOTS and HOTS. Performance improved significantly from the Timeminus to Timestandard conditions. Although performance improved between the Timestandard and Timeplus conditions, the improvement was not statistically significant, indicating that additional time beyond a certain threshold has minimal impact on performance. The sequencing of questions also influenced performance: students performed better on the question type presented first, with performance when LOTS questions were placed second found to be particularly vulnerable under speeded conditions. Students’ behavioural strategies under time pressure suggest a shift toward surface-level engagement, particularly on HOTS tasks. Various stakeholders use assessment outcomes for high-stakes decision-making. The assessment results should be reliable and valid to ensure informed and fair decisions. The findings highlight the need for educators and policymakers to carefully consider time allocation and question sequencing in assessment design to ensure valid and reliable measurement of technical competence.