Instruments

An initial survey using a questionnaire (Appendix C) was administered to all participants in order to gain insights into the students’ perceptions of assessment and measure the use of technology in prior assessment practices, both in Physics and other subjects. A norm-referenced, pencil and paper test measuring academic achievement, out of a total of 50 marks (Appendix D), with questions adapted from past standardised high-stakes examinations from 2008 to 2014, was taken by all participants in class as a post-intervention test. Questions were based on and categorised using Bloom’s taxonomy of educational objectives for the cognitive domain (Bloom 1956) as illustrated in Table 2. Although along the years, the taxonomy has been criticised (Furst 1981, Solman and Rosen 1986) and revised (Anderson and Krathwohl 2001), the original structure for classifying cognitive behaviour presented by Bloom (1956), remains the most widely accepted and extensively used version of the taxonomy (Armstrong 2003).

A traditional objectively scored achievement test, rather than a game-informed assessment was devised to act as a post-test for both groups, as the presented research does not aim to challenge or change the notion of traditional academic attainment as de facto defined in the research question. In fact, the study seeks to evaluate the effect of a game-informed approach to everyday assessment, which is not beyond the control of the classroom teacher (as most standardised exams unfortunately are, both in terms of mode of delivery and content), on students’ traditional academic achievement as measured by high-stakes standardised examinations. Furthermore, students’ prior academic performance which was used as a covariate and acted as a pre-test was based on the score obtained in past high-stakes examinations.

Both control and experimental groups were subject to innovation in the use of digital technologies for assessment purposes as shown by the questionnaire administered to both groups. All participants were taught the same subject content, which included Electricity in the Home, Magnetism and Electromagnetism (Matriculation and Secondary Education Certificate Examinations Board 2012) by me as their teacher. During a 10-week intervention period (each week having 4 lessons of 40 minutes each of contact time), the control group was assessed using traditional summative assessment practices while a game-informed approach to assessment was followed by the experimental group through a website, which I have specifically designed for the purpose of the study at www.in2fiziks.com. Table 3 illustrates the intervention period weekly assessment activities for both groups. At the end of the intervention period, both groups took a norm-referenced test measuring their academic achievement. Appendix E gives direct links (including username and passwords) to the weekly assessment activities shown below.

Potential threats to validity were identified during the research design and proposal stage. The following table illustrates the mitigation measures taken in order to limit the effect of possible threats to the research design.

Furthermore, both data collection instruments were piloted for face and content validity. Two independent raters categorised the post-test questions, based on Bloom’s taxonomy for the cognitive domain (Bloom 1956). Inter-rater reliability for the different cognitive levels, shown by the intra-class correlation coefficient (ICC), was good (knowledge: ICC= 0.80; comprehension: ICC = 0.93; application: ICC = 0.83; analysis: ICC = 0.91; synthesis: ICC = 0.87 and evaluation: ICC = 0.95). Internal consistency was checked using Cronbach’s alpha (Cronbach’s α = 0.71). Answers to the test were independently marked by 2 Physics teachers, other than the researcher. All written answers to open-ended questions were scored independently. Inter-rater agreement was high (ICC = 0.94).

All data was analysed in SPSS version19 (IBM 2010). Crosstabulations and Chi-Square tests for independence were performed on the data resulting from the initial questionnaire, in order to summarise the frequency of use of digital technologies for assessment purposes and establish any statistically significant differences in responses across the control and experimental groups (Field 2009). On the other hand, establishing the difference between the post-intervention means test scores attributed to the educational intervention using an independent samples t-test, would have introduced a source of selection bias, as randomisation of the individual participants to the groups was not possible; hence the groups could not be assumed to be ‘probabilistically equivalent’ (Trochim 2001, p 184). Thus, any statistically significant difference and respective effect size (Cohen 1988) between the post-intervention means test scores attributed to the educational intervention and adjusted for prior academic performance were determined using an analysis of covariance (Field 2009).