Quality Assurance of Arabic School Examination Development: A Discrepancy Evaluation in Muhammadiyah Elementary Schools

Abstract

The present study intended to evaluate the development of Grade VI Arabic School Examination items in Muhammadiyah elementary schools within the Special Region of Yogyakarta using the Discrepancy Evaluation Model. The focus of this evaluation was on assessing the degree of conformance to the set of criteria of the phases in developing test items, namely, design, installation, process, and product, and the discrepancies between what is done and what is supposed to be in order to generate recommendations on how to improve the quality assurance of the system of Arabic language assessment. The methodology of the study included conducting an evaluation research based on Malcolm Provus's Discrepancy Evaluation Model (DEM). The respondents in this research were Arabic test item writers, school principals, and the Vice Chair for the Al-Islam, Muhammadiyah Ideology, and Arabic (ISMUBA) Curriculum of the Primary and Nonformal Education Council of the Muhammadiyah Regional Board of the Special Region of Yogyakarta. They were selected purposively. Data collection methods were questionnaires, interviews, and reviewing documents. The content validity of the instruments was assessed using Content Validity Ratio (CVR) with the involvement of five expert validators, and all instruments were considered appropriate for use following their revision. Instrument reliability was calculated using Cronbach's Alpha. Quantitative data were analyzed descriptively on the basis of percentage achievement scores for every indicator of the evaluation, and qualitative data were analyzed by triangulating interview data and document review in order to detect discrepancies between the real condition and the standard for every stage of the evaluation. According to the results of the evaluation, the design phase conformed to all established standards, and there were no discrepancies between the real situation and the standards. Most components of the installation phase conformed to the set standards, but discrepancies appeared in relation to the readiness of resources in terms of the time devoted to the test development (57.14%), facilities (74.29%), and test development procedures understanding. As for the process phase, it was implemented in line with the set standards due to the alignment of test items with the test blueprint (93.99%) and the Standard Operating Procedures (76.79%), although there were still some shortcomings in regard to coordination, communication, and documentation of activities. At the product stage, most quality standards were achieved, including content appropriateness (95.77%), reviewer involvement (90.52%), and the completeness of archived test items and answer keys (100.00%). Nevertheless, technical documentation still requires improvement. Overall, the discrepancy analysis indicates that the implementation of Arabic School Examination item development at Muhammadiyah elementary schools in the Special Region of Yogyakarta has met the program standards and is feasible to continue, provided that improvements are made in resource management, coordination, documentation, and test quality control to further strengthen the quality assurance of Arabic language assessment.