Theoretical Framework

3. Assessment

In this chapter we look at the role of assessment in learning..
A. Problem statement

How can we efficiently make use of assessment for learning, and in particular for learning algebra?.

B. Theoretical overview
Introduction

Black and William (1998) have argued for many years now that assessment should be a part of learning and apart from learning. They describe how attention goes out to formal methods of assessment, but that there are informal ways of looking at assessment. To make sure that the whole learning process is served well more attention should be given to this aspect of learning. Just measuring a student's existing state of knowledge then, is not enough.

 

The Second Handbook of Research on Mathematics Teaching and Learning (Lester jr., 2007) specifically deals with assessment as a " bridge"  between teaching and learning. In my research -apart from this aspect- I would also like to stress the use of assessment activities to promote learning based assessment.

Purpose of assessment

Black and Williams (2004) broadly see three functions:

• 
  supporting learning (formative)
  
• 
  certifying the achievements or potential of individuals (summative)
  
• 
  evaluating the quality of educational programs or institutions (evaluative)
  

Summative assessment is also characterised as assessment of learning and is contrasted with formative assessment, which is assessment for learning.

Summative assessment

In most curricula summative assessment is used. Summative assessment is mostly aimed at grading and scoring. Some researchers argue that, instead of providing a certain grade which seems to say what "level of knowledge" a student has, formative tests give the student an insight into the nature of -for example- their misconceptions.

Formative assessment

Black and William have made a case for more formative assessment. In their article from 1998 they state that "improving formative assessment raises standards". Actively involving the student, implementing formative assessment as an essential part of the curriculum and motivating students through self-assessment are key benefits of formative assessment. Means to do this are feedback, self-assessment, reflection and interaction. This makes a good case for tools that aid these factors.

Formative assessment is a process in which self-reflection should result in more insight (Crooks, 2001). Cowie and Bell (1999) define it as the process between teacher and student to “enhance, recognise and respond to the learning”. Black and Wiliam (1998) only define assessment as being ‘formative’ when the feedback from learning activities is actually used to modify teaching to meet the learner's needs.
Formative versus summative

Recently a shift from more summative towards formative assessment has taken place. This shift has also stressed the apparent disadvantages of formative assessment.

Firstly the fact that less teachers available means more work as formative assessment is more time-consuming. Secondly modularization means that students learn about their mistakes when finished with a subject. This promotes a grade-culture, for assessment -albeit formative- is used as a means to test whether a student has learned enough.

Here we clearly see tension between summative and formative assessment. As Black & Wiliam (2004) put it:

“teachers seemed to be trapped between their new commitment to formative assessment and the different, often contradictory demands of the external test system” (p. 45).
A positive connection between formative and summative assessment is sorely needed (Broadfoot & Black, 2004) In this sense one could argue that summative and formative assessment are potentially complementary (e.g.(Biggs, 1998); (Harlen, 2005)) and should be integrated more (Shavelson, Black, Wiliam, & Coffey, 2002). This opposed to the viewpoint that they should generally be kept apart (e.g. (Knight & Yorke, 2003))
Pre-emptive formative assessment

In reality education seems to tend to a balance of formative and summative assessment. They have one element in common: they both are mostly conducted after learning. To be able to actually learn during the process of learning means assessing throughout the course of a module. Black and William call this pre-emptive formative assessment. It builds on constructivist learning principles (Black & Wiliam, 2003) learning starts from the learner's existing knowledge and learning entails actively incorporating new knowledge into an existing knowledge framework. Using pre-emptive formative assessment means using feedback as a central element in the learning process (Hattie, Biggs, & Purdie, 1996). Instead of serving up feedback too little too late, feedback is used "pre-emptively", to make sure whether a student is on the right track or not. The role of feedback in formative evaluation was already stated in the 60s by Bloom (1969):

"Quite in contrast in the use of 'formative evaluation' to provide feedback and correctives at each stage in the teaching-learning process. By formative evaluation we mean evaluation by brief tests used by teachers and students as aids in the learning process. While such tests may be graded and used as part of the judging and classificatory function of evaluation, we see much more effective use of formative evaluation if it is separated from the grading process and used primarily as an aid to teaching".
I contend that both a score-based approach, combined with the strength of formative assessment is a good approach to learn more. Scoring has an inherent motivational aspect. Also, adding several " modes"  from practice to " exam" and in-between will facilitate different uses of assessment, both formative and summative.
Carless (2007) suggests that a suitable timing for pre-emptive formative assessment is the class, classes or a longer period preceding high stakes assessment. (see (Lester jr., 2007))
Framework for Classroom Assessment in Mathematics

In De Langes “Framework for Classroom Assessment in Mathematics” several principles for good assessment in mathematics classrooms. This framework connects well to the OECD framework that was designed for the PISA program.

“Principles for Classroom Assessment

1. 
   The main purpose of classroom assessment is to improve learning (Gronlund, 1968; de Lange, 1987; Black & Wiliam, 1998; and many others).
   
2. 
   The mathematics is embedded in worthwhile (engaging, educative, authentic) problems that are part of the students’ real world.
   
3. 
   Methods of assessment should be such that they enable students to reveal what they know, rather than what they do not know (Cockroft, 1982).
   
4. 
   A balanced assessment plan should include multiple and varied opportunities (formats) for students to display and document their achievements (Wiggins, 1992).
   
5. 
   Tasks should operationalize all the goals of the curricula (not just the “lower” ones). Helpful tools to achieve this are performance standards, including indications of the different levels of mathematical thinking (de Lange, 1987).
   
6. 
   Grading criteria should be public and consistently applied; and should include examples of earlier grading showing exemplary work and work that is less than exemplary.
   
7. 
   The assessment process, including scoring and and grading, should be open to students.
   
8. 
   Students should have opportunities to receive genuine feedback on their work.
   
9. 
   The quality of a task is not defined by its accessibility to objective scoring, reliability, or validity in the traditional sense but by its authenticity, fairness, and the extent to which it meets the above principles (de Lange, 1987).”
   

In operationalizing these principles, especially principle 5, it is important to distinguish different mathematical levels of thinking. In chapter 6 –when we choose our content- we will see that De Langes “Assesment pyramid” has many similarities with other categorizations of mathematical skill level.

The role of feedback

Using feedback after learning does not provide an incentive to actually use it to adjust knowledge and/or beliefs. This means that feedback is often very ineffective (D. Carless, 2006), caused by the lack of iterative cycles of feedback and revision. In line with this criticism Gibbs and Simpson (2004) have stated that timeliness and potential for student action are key components of good feedback. This means that timing is important and that it should be possible to act on the basis of the feedback provided, for example by enabling the possibility to correct an answer. As some research suggests written feedback is less effective than oral feedback (Boulet, 1990), as students pay little attention to teachers' written comments (Zellermayer, 1989) or find them difficult to understand and act upon (Clarke, 2000).  These two criteria form a basis for a pre-emptive approach of assessment. On the teacher side a teacher should have insight into the misconceptions a student has.

Feedback is routinely applied to any information that a student is given about their performance. Historically the term was used in systems engineering by Norbert Wiener in 1940 working on guns. In education feedback tends to be used differently. (Ramaprasad, 1983) says: " Feedback is information about the gap between the actual level and the reference level of a system parameter which is used to alter the gap in some way" .

 

An interesting aspect, as Sadler (1989) noted, is that feedback is used to influence a certain gap. So if we don't use recorded information for this purpose, it is not considered feedback. This means that formative assessments -Ramaprasads feedback- are closely linked to instructional consequences. The function of assessment becomes formative if student's learning is served, through feeding feedback back into the system.

 

An example clarifies this: telling someone to work harder is not formative, as it doesn't involve any feedback as how to work harder. Telling someone to use more steps when solving an equation is formative.

 

Perhaps the distinction becomes even clearer if we see monitoring assessments provide information of whether a student is learning or not, diagnostic assessments provide information on what is going wrong and formative assessment provides information on what to do about it.

Principles of feedback

In the learning process adapting instruction to meet students learning needs showed substantial benefits, for example in studies by (Carpenter, Fennema, Peterson, Chiang, & Loef, 1989) and (Black, Harrison, Lee, Marshall, & Wiliam, 2003). As the role of feedback had to be taken into account, drill-and-practice use of the computer made formative assessment difficult. An interesting question is whether the use of "more intelligent" new technology makes a difference in this respect. (Bangert-Drowns, Kulik, Kulik, & Morgan, 1991) found that not being able to see the answer before trying a question is better. Also giving details of the right answer, instead of just wrong or right, seemed more effective, as other research has also confirmed ((Elshout-Mohr, 1994), (Dempster, 1991, 1992).

How effective is feedback?

Reviews conducted by (Natriello, 1987), (Crooks, 1988), (Bangert-Drowns et al., 1991) and (P. Black & D. Wiliam, 1998) showed that that not all kinds of feedback to students about their work are equally effective.

Mason and Bruning distinguish eight types of feedback based on available research {Mason, 2001 #157. Nyquist (2003) reviewed 185 studies in higher education, developing a typology of different kinds of formative assessment:

 

Weaker feedback only: students are given only the knowledge of their own score or grade, often described as " knowledge of results";

Feedback only: students are given their own score or grade, together with either clear goals to work towards or feedback on the correct answers to the questions they attempt, often described as "knowledge of correct results";
Weak formative assessment: students are given information about the correct results, together with some explanation.
Moderate formative assessment: students are given information about the correct results, some explanation, and some specific suggestions for improvement.
Strong formative assessment: students are given information about correct results, some explanation, and specific activities to undertake in order to improve.

 

In these levels typology I do miss a category concerning feedback on the process of problem solving. This also involves feedback on sub-parts of the solution. Also allowing for several representations can provide insight in a given problem. In my research I will take into account this process and representational feedback.

Several studies -including Nyquists- seem to show more effectiveness in assessment when using feedback. The stronger the feedback the larger the effect seems to be. ((Elawar & Corno, 1985); see (Greer, 2000)) for many more references).

 

Another issue is the use of scoring: scoring seems to have a negative impact (Butler & Nisan, 1986). Butler (1988) even concluded that the effects of diagnostic remarks completely disappeared when grades were added. There also are indication that games (reference) enhance motivation. So here we have a dilemma: do we use grades/marks and feedback together, hoping motivation will overcome the disadvantages or not. We have indications that games do motivate students, but how does this correspond with less motivation through grading.

Felix (2003) asserts humanizing feedback is important and proposes this can be done by providing structural hints, personalized hints, structural graphics, personalized graphics and games.
It should also be noted that some research indicated that (Simmons & Cope, 1993) that the possibility of using ICT tools, by providing room for the strategy "trial and improvement", requires less mental effort than the "harder" way with paper-and-pencil. More is not always better, scaffolding an exercise seems to be more effective than providing a complete solution with a new assignment. It remains unclear how effective feedback actually is, because the effectiveness seems to be determined by variables not yet well understood. It is also clear that learning intentions play a significant role.

 

Of course, students themselves also play a large role: metacognition (more metacognition from students enhance learning), motivation and learning. This is important to acknowledge as we defined feedback as "influencing learning". But then we have to take into account those factors that influence learning as well.

Nicol and MacFarlane (Nicol & MacFarlane-Dick, 2006) reinterpreted existing literature on formative assessment and the use of feedback to their respective roles in self-regulated learning. These “seven principles of good feedback practice” are provided in table 1.

Good feedback: 1. helps clarify what good performance is (goals, criteria, standards) 2. facilitates the development of self-assessment and reflection in learning 3. delivers high quality information to students about their learning 4. encourages teacher and peer dialogue around learning 5. encourages positive motivational beliefs and self esteem 6. provides opportunities to close the gap between current and desired performance 7. provides information to teachers that can be used to help shape teaching.

Table 1: Seven principles of good feedback practice
(Nicol & MacFarlane-Dick, 2006)
Gibbs and Simpson (Gibbs & Simpson, 2004) were able to extract 11 conditions under which assessment might support student learning and improve chances of success. Table 1 gives these 11 conditions.

Assessment tasks [conditions 1-4] · Capture sufficient study time and effort (in and out of class) · Are spread evenly across topics and weeks · Lead to productive learning activity (deep rather than surface learning) · Communicate clear and high expectations. Feedback [conditions 5-11] · Is sufficient (in frequency, detail) · Is provided quickly enough to be useful · Focuses on learning rather than marks · Is linked to assessment criteria/expected learning outcomes · Makes sense to students · Is received by students and attended to · Is acted upon to improve work and learning

Table 2: Gibbs and Simpson’s (2004) 11 conditions

Computer Aided Assessment

In computer aided assessment (CAA) there has also been a shift from more summative use towards formative. This of course corresponds with the qualities and limitations of computers. As computers become more powerful and "smarter" more innovative use of feedback is called for.

Harlen and Deakon Crick {Harlen, 2003 #155} conducted a review of studies to find out whether there was evidence of the impact of the use of ICT for assessment of creative and critical thinking skills on students and teachers. Evidence from two high-weight studies showed that:

• 
  Teachers are helped in understanding students in their understanding of new material, when information is stored an recorded.
  
• 
  Taking over some roles of assessing and providing feedback frees up time for the teacher to focus on other ways to support learning.
  
• 
  Computer feedback during a test improves student performance when the same test material is used again later on.
  
  
  

Sangwin's  (2006) research on Computer Algebra Systems (CAS) provides an indication of these new possibilities. For formative evaluation more open questions would be helpful, contrary to multiple choice questions with distracters. Also, multi-step exercises, and feedback on these steps, could provide formative evaluation for the student and teacher alike.

Types of questions
For CAA in mathematics Provided Response Questions (PRQ) are most common. This category includes Multiple choice and matching questions. There are many problems with this type of question: question distortion, only lower order thinking is stimulated and strategic thinking. As Sangwin states this especially holds true for reversible mathematical skills where one direction is significantly more difficult than the other, e.g. factoring and expanding. Providing PRQ questions then probably means testing something the teacher doesn't intend to test.

These types of questions correspond a lot with the well-known behavioural paradigm or “drill instruction”. Also, because the questions and the answers are simple to construct and simple to evaluate large-scale assessment, providing many numbers and figures (“16% of students scored correctly on questions concerning integrals”) For management purposes this perhaps would suffice, but for math educators a more qualitative approach would be better: what type of mistakes are made, so I can address this issue in the next lesson. The learned lessons from this, in fact a grouped list of misconceptions, could not only provide input for classroom practice, but in the future also for improvements on software. This also holds for feedback on the process of solving equations.

In another paper (Sangwin & Grove, 2006) also point out that ease of authoring tests for teachers should not be neglected.

 

Implementing feedback in tools

Implementing more elaborate feedback in mathematics software is still is at an amateur level. As Jeuring (2007) points out: much can be improved, especially on the level of providing strategies. Approaches that can be used can be Sangwin's approach, but also based on rewrite rules and state transitions (Goguadze, Gonzalez Palomo, & Melis, 2005). Unfortunately this also means that interoperability issues (Goguadze, Mavrikis, & Palomo, 2006) exist.
Pyramid
C. Assessment in this research 
We argue that a shift from summative assessment towards pre-emptive formative assessment could be beneficial for learning. Feedback plays a large role in this type of assessment. We will focus on using a tool that can pre-emptively perform formative assessment for learning algebraic skills. Here we will use the eleven conditions under which assessment might support student learning and improve chances of success (Gibbs & Simpson, 2004), as well as the seven principles of good feedback practice (Nicol & MacFarlane-Dick, 2006). The ability to cater for these conditions will be used –along with other criteria- to choose a tool for assessment.