We have a rogue organisation of state on our hands (as though you didn’t know)

What follows in this posting is some solid analysis about how the review office undertakes its reports in general and undertook the 2009 one in particular. Some of the analysis by the professors may be complex, but I urge you to at least get a feel for the message they communicate.

In the previous posting (‘Stoop must go’) I accused the review office of producing the 2009 report on literacy and numeracy for it to be available to the minister as a source of information/propaganda. In that posting I said the motivation for the report was highly suspicious. The 2007 report, unlike the 2009 one, was undertaken when national standards were only a twinkle in the eye of Key, English, and Hattie. It seems likely, I said, that the exceptionally high quality of performance in literacy and numeracy expressed in the 2007 report was something of an embarrassment to the review office and the minister. The government, for instance, in its central argument for national standards’ legislation in the Explanatory note, ignored the literacy and numeracy figures to use some other figures. A behaviour I described as akin to lying.

In commenting on the 2009 report, I was especially critical of the ‘teacher expectation’ parts, an aspect not covered in the previous report. The subjectivity of the question and the diffuse way the review office handled it looks outlandish even by review office standards. The review office was searching, it seems, for a new angle to support the minister; and to justify the undertaking of the report.

I applied under the Official Information Act to find out the motivation and origins of the report. None was forthcoming. I then applied for any communications between the review office and the ministry on the matter. A significant piece of information was unearthed: a highly placed review office person said to her ministry counterpart, ‘… I think you will pleased and dismayed at the same time with the expectations section’. I described this as a significant piece of information, an example of the ‘gotcha’ culture that exists in both organisations, but more particularly in that of the review office. The ‘pleased’ overwhelms the ‘dismayed’ in emotion and power.

It needs to be recognised that the review office has a vested interest in national standards because the implementation of the policy will greatly increase its powers in schools. National standards are dream come true for the review office with its measurement-based pedagogy.

How conscious is the review office of the bias it displays? The review office talks of being impartial and unbiased, but it can easily feign that because it is its pedagogy that carries much of the partiality and bias. The pedagogy is based on a measurement-based system that in its full expression exists nowhere, and will never exist – yet such a system is put forward as not only desirable but attainable.

There was certainly, however, a conscious bias in the way Graham Stoop, head of the review office, allowed a quote from the 2007 report, a quote I describe as akin to a lie, to proceed in association with the national standards’ legislation.

As someone who worked in an education government department for 15 years, I am aware of the way expectations held by senior management can filter into the culture of an organisation and influence behaviour without direct instructions being issued. That is what I suggest has happened in this case, and has happened throughout the history of the review office’s existence.

Some other things happen too: things less easily explained away; things that happen in the setting up of the reports; things that happen to the shaping of information received; things that happen in communication to the media; things that I think are disgraceful.

Let us look at the analyses of two professors: Ivan Snook and John O’Neill.

John O’Neill first. He directs his analysis at the minister’s claim that the review office report provides the ‘facts’

According to the Minister's response, her ERO report ‘clearly states ...’ I beg to disagree.  Some thoughts follow on the ERO report as a basis of advice on which the Minister relies for her policy decisions and public pronouncements:

1. Note 25, p. 48 of the report states that the representativeness of the
sample was established on various criteria (school type, roll size, locality and decile) using a Chi square test, to measure the significance of any
difference between expected and actual distribution frequencies. Table 4 is incorrect. On this basis (decile of schools in sample versus schools
nationally), the sample is not nationally representative of school deciles
(Decile 8-10 schools are statistically very significantly overrepresented in the sample). Therefore the report cannot be used as the basis of policy as the Minister does not have any factual basis for making comments about what occurs in schools as a whole, nor in year 1 and 2 classrooms as a whole.

2. The ERO report is a review of 212 schools’ ‘systems and practices’
practices (p.4) in terms 1 and 2, 2009, not those of individual teachers.
The sample characteristics describe the 212 participating schools, not the participating teachers. There are no data about the demographic
characteristics of the sample of teachers whose classroom practices were reviewed, so it is impossible to determine whether the sample of teachers reviewed is nationally representative. It is therefore misleading for ERO or the Minister to make any claims about the perceived quality of practice of 30% of teachers in the report as if these were claims about the quality of practice of 30% of teachers nationally.

3. The evaluative approach and sources of data are described in detail (p. 6) and the specific overarching and investigative questions are listed (Appendix 2). The overarching questions require reviewers to judge teachers’ use of instructional strategies or assessment in reading and writing (which may be observed). Conversely, the investigative questions require reviewers to judge teachers’ understanding or decision-making, or their capacity to engage students in learning (which may only be inferred). Neither ERO nor the Minister has clarified whether and to what extent claims about the quality of teachers’ practices are based on observation or inference.

4. The reviewers are asked to judge the school’s ‘expectations of
achievement’ but not those of the classroom teachers. In contrast, p.5 of the ERO report refers to research which demonstrates the importance of teacher expectations; there is no specific mention of the importance of school expectations. One may reasonably assume, however, that because ‘the reviewers’ were required to gather school level data on expectations of achievement no claims may be made by ERO or the Minister about the expectations of teachers in the sample, or nationally, with regard to reading or writing.

5. The investigative questions require the reviewer to review ‘how well’
teachers, schools or leaders variously instruct, use, engage, understand or monitor reading, writing and achievement. It is acknowledged that the reviewers exercised judgement (p. 6). However, no information is given on how these subjective judgments were made or how they were
inter-subjectively validated or moderated. It is therefore impossible to
determine whether and to what extent equivalent phenomena were reported across the sample. Given the range and complexity of the judgments that reviewers were asked to make, the diversity of school and classroom contexts in which these would be made, and the number of reviewers involved, this omission is inexplicable. In the absence of such information, one cannot have any confidence that the reviewers judged the same phenomena.

6. The reviewers’ aggregated findings are summarised in various graphs as percentages of teachers or schools. Four distinct descriptors are used in each graph and the visual representation implies discrete categories of judgement. No information is given in the report on the criteria reviewers used to allocate observed or inferred practice to one of the four categories, nor how these were validated or moderated to ensure
inter-reviewer reliability of allocation. Given this lack of information,
one cannot have any confidence that the reviewers as a whole allocated their judgements to the four categories consistently. This means that it is
impossible for ERO or the Minister to draw categorical distinctions between percentages of teachers or percentages of schools.

7. The report writers demonstrate considerable confusion about what they are reporting. For example, Figure 3 shows that 33 percent of teachers were judged to have ‘very good use’ of instructional reading strategies. The commentary on the Figure, however, states that Figure 3 shows that ‘high quality teaching of reading was observed in a third of schools’ (p. 14). These are quite simply not the same thing. The confusion is compounded by the qualifying statement which follows: ‘The quality and range of instructional strategies varied within and between schools’ (p. 14).

8. Similar confusion is evident on the first page of the report, quoted by
the Minister. It states (p. 1) that about 70 percent of teachers ‘made good use of a range of effective reading and writing teaching practices’ and that ‘the remaining 30 percent had little or no sense of how critical important it was for children to develop confidence ...’ Again, these are not one and the same thing. The 70 percent figure refers to ‘use’; the 30 percent figure to ‘sense’ or understanding. The reviewers were asked to investigate use and understanding separately in the 212 schools. It therefore makes no sense to conflate them in the report.

When the Minister states that ‘these are the facts’, she may not be aware how few facts there are in the ERO report.

John O’Neill a QPED vice-president and Massey University Professor of Education.

Ivan Snook points to further serious faults in the 2009 report and in the way it was reported in the Dominion.

The Dominion Post headline said ‘Schools set Sights too Low.’ (15^th December, 2009). Nothing in the ERO report supports this damning generalisation. Rather the report finds that the standard of goal setting, use of assessment data and methods of teaching are adequate or better.

1. According to the Dominion Post ‘some teachers and principals are ignoring achievement data for year 1 and 2 pupils that do [sic] not show positive results. In some cases the information had not been given to the boards of trustees and parents.’ This suggests that schools typically hide relevant data but this is not substantiated.

2. ‘These teachers [one in three teachers who, allegedly, ‘had little sense of how critical it was for year 1 and 2 pupils  to develop confidence in reading and writing’] had minimal understanding of effective reading and writing teaching and set inappropriately low expectations.’ I could find no evidence of this in the report.

3. ‘Teachers stated that, if expectations were raised, too many children would be seen to be failing. Consequently, they set lower goals which they thought were more attainable.’ This is an accurate quotation from the ‘executive summary’ but the relevant section of the report is rather different.  It found that ‘In some high decile schools, [my emphasis] teachers described how they preferred to give children time to consolidate new learnings and, accordingly, set expectations lower or just approaching, nationally referenced expectations.’ These teachers may be criticised on some educational grounds but they are certainly not obviously setting standards that are too low and no generalisations should be made about ‘teachers’ doing so. Apart from this reference, there is nothing in the report relating to ‘setting sights too low.’

4. ‘In some schools, the crucially important role of monitoring the progress of year 1 and 2 pupils was left to teachers.’ This is presented as a damning criticism of school leaders but the report itself found that these principals ‘trusted their junior school teachers or leaders who knew the students well. While this position might be criticised from a gross ‘managerialist’ perspective, it is certainly not unreasonable and, in a collegial, system, might be seen as perfectly appropriate: obviously not all principals are experts in beginning reading and writing. Indeed the report noted that ‘School leaders were generally more confident discussing and sharing assessment results for the middle or upper primary school than for Years 1 and 2.’  It has long been recognised that the supervision and monitoring of progress in the first years of schooling is a specialised activity. Principals and leaders may have sometimes been over optimistic in relying on the teachers but their ‘inaction’ may well have been quite reasonable. It certainly does not deserve a generalised critique of school leaders.

5. In the ERO report itself the basic argument is viciously circular. The basic thrust is that there are successful (or ‘effective’) teachers and unsuccessful (or ‘ineffective’) teachers and that, teacher in these discrete categories exhibit patterns which the reviewers either approve or   disapprove of. But there is NO separate definition of ‘effective’ (or ‘ineffective’) teachers and certainly no linking of their work to student learning. Indeed the ‘ERO reviewers recognised that they were not able to see the eventual outcomes resulting from the lessons.’ (p 8).  Thus they were unable to show if the students of the ‘effective’ teachers achieved better than the students of the ‘ineffective’ teachers. The so-called argument is quite circular.

6. Furthermore the criteria used to distinguish ‘effective from ‘ineffective’ teachers are not rooted in research on teaching but instead simply reflect the subjective judgments of the reviewers.  Thus, for example, in ‘bad’ schools, the topics for writing are set centrally, there is a lot of diary writing and the lessons are often repetitive. At other times teachers are criticised for engaging in ‘whole class’ instruction. It is far from obvious (and no research is cited to prove) that such ways of teaching are inferior to other kinds. It might be argued that they are less desirable but this would constitute a legitimate debate among teachers. There is no reason to suppose that in matters like this the judgment of external reviewers is any better than that of practising teachers.

7. It should also be remembered that the reviewers do not have robust methodologies for distinguishing ‘high, ‘good, ‘adequate’ or ‘inadequate’ teaching. The final results, expressed in what look like scientifically derived tables are nothing more than an aggregated set of the subjective judgments of individual reviewers.

8. Having derived these tables, they summarise them in ways which support a particular point of view. For example, under ‘Qualities of Teaching of Reading’ we find: High: 26%, Good 43%, Adequate 21%, and Limited 10%. (p 8). In commenting on these the reviewers contrast the first two categories and the second two categories to ‘show’ that a third of the teachers are ‘failing’.  But surely, if the teaching is ‘adequate’ it is ‘adequate’ (not excellent or outstanding but adequate) and a more honest reporting would place the unsatisfactory teachers at 10% rather than 31%.

Ivan Snook is an Emeritus Professor of Education at Massey University

Confirmation for my concerns about the ‘teacher expectation’ parts of the 2009 report was provided by information gained under the Official Information Act; there is also John O’Neill’s expression of similar concerns in his analysis; and Ivan Snook’s expression of his similar concerns in his. While this is a more outlandish example of the review office’s behaviour, as far as schools are concerned, it won’t come as a surprise to them.

My conclusion is that we have on our hands a rogue organisation of state. My recommendation is that the teacher organisations, and political parties with a genuine concern fairness and truth in education, advocate, as part of their policy, an inquiry into the practices, culture, philosophy, and leadership of the education review office.

I leave the final words about the parlous state of the education review office and the media handling of education to Ivan Snook:

To sum up, the Dominion Post headline ‘Schools set sights too low’ is a fiction engineered by a submissive state agency, politically motivated press releases and media that do not understand either educational research or the complexities of classrooms and are too passive to go to the original sources to check the soundness of the releases handed to them.