Time to question simplistic interpretation of research

Some policy makers and school leaders, including our Prime Minister, have been tempted by simplistic and over-generalised research that adds authenticity to the claim that student achievement is low because teacher standards and quality have dropped.

With the work of John Hattie being used to legitimise the badgering of teachers to focus on "using data more effectively" (code for raising the school's NAPLAN results), we need to apply our critical literacy skills and ask – who benefits from this text?

Those of you fortunate enough to have attended the QTU's 2009 Professional Issues Conference would have heard Professor Martin Thrupp and Professor Bob Lingard explain how the "politics of blame" and the "evaluation message system" place unhelpful pressures on teachers.

As Lingard said: "The top-down, one way gaze upon teachers as the sole source and solution to all school problems" is doing nothing to earn the trust of teachers. Thrupp said: "Yes, of course we can improve our practice and make a difference to the achievement of young people and to their lives, but don't look to us to solve all the problems of student underachievement, because it's not fair on us and you are going to be sorely disappointed too."

In 2011, it seems that John Hattie has become the preferred authority to add weight to calls for more focus on the teacher as the fault in the system.

School leaders or officers hoping to add weight to their next job application are encouraged to show how they have improved NAPLAN scores or other easily measured scoring devices. Some may be tempted to look to teachers to make that a reality.

If you wish to "analyse the text" at the next staff meeting or district and regional leaders meeting at which Hattie's supposed message is espoused, you may be interested in critiques of his work, and, in fact, Hattie's own descriptions of the limitations of his research.

Hattie himself emphasises that his book, Visible Learning: A Synthesis of over 800 Meta-Analyses Relating to Achievement, isn't about "what cannot be influenced in schools – thus critical discussions about class, poverty, resources in families, health in families, and nutrition are not included – this is NOT because they are unimportant, indeed they may be more important than many of the issues discussed in this book. It is just that I have not included these topics in my orbit".

Ivan Snook and his colleagues from Massey University noted that Hattie's book does much to explore what makes successful teaching, but raised concerns about how much damage may be done by policy makers unaware of the significant limitations to the study, which include:

  • social effects/background/context effects are ruled out
  • the various studies have not been appraised for their validity
  • the research is limited to one dimension of schooling i.e. achievement that is amenable to quantitative measurement
  • the research may not be applicable to regular classrooms. In at least one study the class size was kept artificially low
  • low quality studies are included in the meta-analyses.

Hattie may be surprised to find his work being used to place downward pressure on teachers, eroding their morale. In fact, as noted in the South Australian AEU Branch's submission to the Senate inquiry on NAPLAN, Hattie holds teachers in high esteem. His advice on how to "esteem and grow those who have powerful influences on student learning" is "not by...promoting different topics for these teachers to teach, or by bringing in more sticks to ensure they are following policy".

Questioning an over devotion to and uncritical use of John Hattie's research – especially when it is used to justify micro management of teachers – is just another way to be professionally responsible, or for those of you enamoured of good literacy pedagogy, modelling the role of text analyst.

Catherine Day
Acting Research Officer

References:

Australian Education Union (South Australian Branch) NAPLAN submission http://www.aph.gov.au/senate/committee/eet_ctte/naplan/submissions.htm

Hattie, J (2003) Teachers Make a Difference: What is the research evidence? Australian Council for Educational Research,
http://www.acer.edu.au/documents/Hattie_TeachersMakeADifference.pdf

Lingard, B Testing Times: The need for new intelligent accountabilities for schooling, QTU Professional Magazine, vo24_lingard.pdf

Masters, G. N. (2009) A Shared Challenge: Improving Literacy, Numeracy and Science
Learning in Queensland Primary Schools. Camberwell, Vic: Australian Council for Educational Research.

Misa, T (2009) League tables are an unhelpful sideshow, New Zealand Herald, 6 July http://www.nzherald.co.nz/maori/news/article.cfm?c_id=252&objectid=10582708.

Snook, I., Clark, J., Harker, R., & O’Neill, J. (2009) Invisible Learnings? A commentary on John Hattie’s “Visible Learning: a synthesis of over 800 meta-analyses relating to achievement” http://www.nzei.org.nz/site/nzeite/files/misc documents/Invisible_Learnings.pdf

Thrupp, M Teachers, social contexts and the politics of blame, QTU Professional Magazine, 2009, vo24_thrupp.pdf

University of Queensland (2009) Review of the Masters Report

Zyngier, D Teachers under the pump - and over a barrel - and up the creek: Reframing the current debates about 'quality' of education Curriculum Perspectives 2009, 29(1) http://monash.academia.edu/DavidZyngier/Papers/109779/Teachers_under_the_pump_-_and_over_a_barrel_-_and_up_the_creek_Reframing_the_current_debates_about_quality_of_education

Queensland Teachers' Journal, Volume 34 Number 2, 25 March 2011, p23