Master's Dissertation First Full Draft

6.3. Research question 3

Research Question 3: What differences exist, if any, between the results of research questions 1 and 2 and what is the nature of these differences?

As discussed in the previous two subsections (6.1. and 6.2.), the differences in results observed for research questions 1 and 2 are consistent with some existing empirical evidence and do lend weight to some theorised claims around the digital word built upon this existing evidence and several theories. Research question 1 provides further support for the notion of the digital word subtly discouraging deep reading particularly for factual points and details, but provides less insight into the dynamics of differences for conceptual questions. Research question 2’s results are effectively opposite to those of research question 1, but do provide very tentative support for ideas around the digital word and the impact of the ontological disconnectedness between marking and substrate in relation to the visuo-spatial nature of literacy. However, the potential existence of a range of confounding variables mean that this evidence only hints at theorised aspects of the digital word and that a number of carefully-designed and -controlled experiments are required to address these potential confounding variables and provide stronger evidence for these claims around the digital word.

6.4. Research question 4

Although enforced by logistical requirements rather than being a planned experiment, research question 4’s deviation from the one-week delay procedure used in the rest of the data collection process has provided a useful insight to an important question for future research. In practical, logistical terms, the one-week delay (required to create a more ecologically valid measure of learning) lends itself to high rates of attrition, as participants must attend both sessions, a week apart, in order for any useable data to be obtained by the researcher. If shorter delay times were possible, this would improve researchers’ ability to conduct further experiments which aim to measure learning rather than simply measuring comprehension. This finding, although admittedly limited by small sample sizes, provides introductory evidence to suggest that a shorter delay between reading task and testing may provide an approximately equally ecologically valid measure of learning as the original one week delay for participants for conceptual questions on both paper and tablet and factual questions on paper. However, participants in the shorter delay period who used tablets did perform statistically significantly better on factual questions. The very large effect size and its implication that the mean difference exceeds one standard deviation and highly significant p-value (0.02) suggest that this result represents a real effect. However, given the very small sample size for students in this cell (N = 9), it is not possible to say with any degree of confidence whether this effect is in fact real – further research with larger sample sizes is required.

6.5. Supplementary analysis

The supplementary analysis was necessitated by (1) the highly multilingual nature of the participants involved and the fact that all experiments were conducted entirely in English and (2) the gender imbalances in some cells which were identified after completing these research experiments. Both variables have the potential to act as confounds and as such, these analyses provide an opportunity to account for two potentially confounding variables.

The obtained results for language exposure indicate no statistically significant correlations between self-report exposure to the English language and test scores for all within-subjects cells as well as for participants in the reading-only condition of the between-subject sample group. For reading and note-taking within the latter group, however, strongly significant correlations are observed for both factual and conceptual type questions. Both the between-subjects and within-subjects sample groups show a similar pattern – more highly insignificant correlations for reading-only and closer to significant for reading and note-taking. It must be noted that the measure used here – estimated English proficiency based on self-report percentage exposure to English, most dominant language and order of language acquisition – is not a bona fide measure of language proficiency and is simply the best of the available items of the Language Experience and Proficiency Questionnaire (LEAP-Q) to provide a very rough estimate of the impact of language proficiency on test scores. The LEAP-Q itself is a widely-used and well-validated instrument (Marian et al., 2007) and even this individual item does provide some degree of insight – for example, all participants save for one across the entire study who listed English as their most dominant language reported being exposed to the English language more than 50% of the time. Future research will need to make use of a more robust instrument to assess proficiency in the language in which the experiments are being conducted.

The eight comparison of means tests conducted yield only one statistically significant result – males performed higher than females on conceptual questions in the reading-only condition from the statistically more powerful within-subjects group. This lack of statistically significant differences for gender suggests that the imbalance in these cells is potentially less problematic than might otherwise be the case. While this is not sufficient evidence to disregard the imbalance in its entirety, it does strengthen the data’s results.

As mentioned in previous subsections and discussed in detail in Chapter 2.3.5., empirical evidence has demonstrated a visuospatial component of conceptual processing. However, it must be noted that conceptual processing being demonstrated to have a visuospatial component does not necessarily mean that conceptual understanding in textual engagement is a good proxy measure for visuospatial processing and that the obtained sample sizes are small and imbalanced – N = 16 (males) and N = 38 (females). Further research is necessary.