Understanding what works in oral reading assessments

by Professors Christine Merrell and Peter Tymms

Last week saw the publication of the new online and open-access report, Understanding What Works in Oral Reading Assessments. Produced collaboratively by the UNESCO Institute of Statistics, the Global Partnership for Education and the William and Flora Hewlett Foundation, the report contains an important chapter, ‘Assessing Young Children: Problems and Solutions’, co-authored by CEM’s Director of Research, Professor Christine Merrell and iPIPS Director, Professor Peter Tymms.

The report offers a range of experiences from a collaborative and technically rigorous perspective focusing on the importance and benefits of oral reading assessments in measuring a child’s cognitive and non-cognitive development. It explores considerations of culture and language differences in children, and the importance of using data to raise awareness and develop an assessment plan for comprehensive reform.

Merrell and Tymms’ chapter, ‘Assessing Young Children: Problems and Solutions’, describes the challenges of designing reliable assessments of children age 4 to 7. These include a range of issues associated with measuring children’s development which have been observed in their previous research, such as taking into consideration the limited concentration spans and short-term memory of young children, and their limited capacity for self-management. Additional difficulties in obtaining reliable information include the diverse developmental levels within each cohort being assessed, and the possibility that children fail to display their full potential during assessment.

Merrell and Tymms have gathered extensive data from the Performance Indicators in Primary Schools (PIPS) and the international iPIPS assessments, and have identified critical elements that provide a fair analysis of a child’s proximal development. Assessments for young children must be limited to around 15 to 30 minutes and be conducted on a one-to-one basis. They must provide an indicator of a child’s later success or difficulties, and provide reliable and fair results. Their content should also be appropriate for children with a range of abilities.

The only way to fulfil these requirements is to use an adaptive approach. Children are asked reasonably easy questions for a child of their age to begin with. The child’s response then determines which question they are presented with next. If questions are answered incorrectly, the assessment switches to a new topic with easy items to start with. If answered correctly the following questions become more difficult until the child falters or the section is completed.

Merrell and Tymms have found that it is possible to gain a picture of children’s development with the combined use of a booklet and a relatively inexpensive, widely-used intelligent device, such as a smartphone or tablet with an adaptive assessment app installed.

Child and assessor look at the booklet together and the adaptive app is controlled by the assessor. The app selects items and the assessor records the child’s responses electronically. The assessment is divided into sections such as vocabulary, concepts about print, letter recognition, simple arithmetic and so on, and within each section the questions are organised in order of increasing difficulty.
Merrell and Tymms have successfully used this approach in an international context and have found that it generates high quality data about children’s early language and mathematical development in just 15 to 20 minutes.

Internationally comparable assessments have provided their own challenges however, as some areas of development, for example simple arithmetic, can more easily be applied across cultures. Other areas, such as rhyming and certain behaviours and gestures cannot be so easily compared and some are unique to particular situations.

A recent report by the Organisation for Economic Cooperation and Development (OECD) emphasizes the importance of the non-cognitive skills for positive outcomes in life. However, there is much debate surrounding the development of a method to make comparable observations of non-cognitive development and children’s behaviours from different contexts and cultures, as ratings of non-cognitive skills are likely to be influenced by prevailing norms of behaviour and by individual perceptions (Merrell et al., 2013. “Cross-country Comparisons of Inattentive, Hyperactive and Impulsive Behaviour”)

Merrell and Tymms have been developing a method using short video clips of children exhibiting different levels of particular behaviours. Teachers’ evaluations of behaviours are moderated by asking them to rate the videoed children and then rate the children in their own class.

Merrell and Tymms have developed an assessment system that works in different cultures. The PIPS and iPIPS assessments provide reliable and valid data measuring children’s varying developmental stages which is useful not only for schools, but can also provide analysis for policymakers.

These methods are currently being used within the iPIPS project and data is currently being collected in South Africa, Russia, China, Brazil, England and Scotland.

Download Understanding What Works in Oral Reading Assessments