Master's Dissertation First Full Draft

5.2. Overview of test results

As described in Section 4.7., each of the four tests used in these experiments contained 5 question types or categories, with each test comprising a total of 10 items, with two items per question category. The 5 question categories are: Factual, Seductive Detail, Conceptual, Inferential, and Application. However, given the relatively small number of participants in some of the samples, that each question category is served by only two items, and that there are meaningful similarities between the question categories, it was decided to collapse these five categories into two, as detailed in the next sentence. For this analysis, these 5 question categories were collapsed into 2 groups, namely ‘Factual’ questions (comprising the Factual and Seductive Detail categories) which involve recall of information without a significant degree of application or conceptual understanding and ‘Conceptual’ questions (comprising the Conceptual, Inferential, and Application categories) which involve a significant degree of application and conceptual understanding. The scores for the Factual and Conceptual groups were calculated by averaging the standardised scores of the constituent question categories for each group.

5.3. Research question 1

(1) Are there differences in learning (measured by the ability to correctly answer questions from each of Butler’s (2010) categories of question, with a delay of approximately one week between reading task and test) when text is read (only) on paper compared to when this is done using a tablet PC?

(a) Between-subjects (one-week delay)

Due to the small number of participants in the tablet condition (n = 12) and the imbalance between tablet and paper, non-parametric, independent-samples Mann-Whitney U tests were conducted on the scores obtained by the 45 participants who completed the reading only condition using either paper (n = 33) or tablet (n = 12). Separate tests were conducted for the Factual and Conceptual question groups. Table 9 below details the results obtained.

Table 9. Between-subjects (one-week) reading only condition

Question Group	Device	N	Mean	SD	Mean Rank	Sum of Ranks	Mann-Whitney U	Exact Sig.	Effect Size
Factual	Paper	33	.271	.68	25.79	851	106	.017	.69
	Tablet	12	-.204	.71	15.33	184
Conceptual	Paper	33	-.139	.70	23.53	776.5	180.5	.658	.04
	Tablet	12	-.163	.47	21.54	258.5

For factual questions, participants who used paper performed significantly better than those who used tablets (sig. = .017), similarly to the pattern seen in participants in the corresponding paired-samples section. For conceptual questions, however, the means are almost identical and there is therefore no statistically significant difference between the two distributions.

(b) Within-subjects

The relevant parametric assumptions were met and a matched-pairs or paired-samples t-test was conducted on the scores obtained by the 27 participants completed the reading only condition using both paper and tablet. Separate paired samples t-tests were conducted on both the Factual and Conceptual question groups. A positive mean indicates that paper-device scores are higher than those for tablets. The obtained results are listed in Table 10 below.

Table 10. Within-subjects reading only condition

Question Group	Means (Paper/ Tablet)	Mean Diff.*	Stand. Dev.	Std Error Mean	95% CI of Difference		t	df	Sig. (2-tailed)	Effect Size (Cohen’s d)
					Lower	Upper
Factual	.31	.51	.96	.18	.132	.89	2.78	26	.010	0.65
	–.20
Conceptual	.01	.24	.63	.12	-.004	.49	2.02	26	.054	0.33
	–.23

* Positive mean indicates that paper-device scores are higher than those for tablets; N = 27

For the Factual group of questions, with a statistical significance of 0.01 and moderate-to-large effect size of 0.65, participants performed significantly better when they used paper compared to when they used tablets. For the Conceptual group of questions, the mean difference is also positive, indicating that participants tended to perform better when using paper compared to when using tablets for reading. The statistical significance of this difference is 0.054, which is very marginally above the widely-used level of significance of 0.05, while the effect size is small-to-moderate at 0.334. This suggests the possibility of weak but real effects on performance for higher order conceptual questions, but the evidence is inconclusive given the failure to achieve significance.