Ministry gathering ethnicity statistics using a method that is invalid and unethical

This posting is of huge significance. Please read it. If your head starts to spin, don’t worry, read on, regular summings-up will clarify.

The argument is that the government is using an invalid method to gather education ethnicity statistics, a method that is unethical, inconsistent with international best practice and, when explained to teachers and parents would be considered objectionable, stupid, insulting.

The posting begins with a brief extract from yesterday’s posting – a posting that opened the matter up.

Then an exchange of e-mails between a principal and the ministry that serves to present the ministry’s reasoning for a statistical process called ‘prioritising’.

Finally, an analysis of the soundness and validity of that reasoning. (In this analysis I was helped by Professor John O’Neill of Massey University.)

This axis-shifting statistical debacle began with a school communicating with me about a situation it found utterly bewildering.

The following was part of that posting.

This amazing clause has just become part of school reporting requirements:

Determining and reporting ethnicity

‘Ethnicity should be reported in the same way that it is in your school roll returns.  This reporting uses prioritised ethnicity where learners who report themselves in one or more ethnicity are assigned to one ethnic group.  This means that if a leaner identifies themselves as Mâori in any of the ethnic groups, then they are recorded as Mâori.  If any of the ethnic groups the learner identifies with is Pasifika, and they haven’t also identified themselves as Mâori, then they are recorded as Pasifika.  The prioritised ethnicity of each student is stored within your Student Management System.’

That is, whenever Maori is placed in a sequence, that categorisation overrides all others.

.

The school communicating with me said they had grave concerns about this, believing that parents have the right to determine how their child should be classified.

It said it had a number of families who have chosen to classify their children as NZ European or Pasifika first and Maori second, and two have placed Maori third. Under this requirement, the choice in all these cases, without consultation, or the right to challenge, has been taken from the families: the child is categorised as Maori.

Maori trumps Pasifika and, of course, NZ European, no matter what!

Brent Godfrey of Pembroke School (a complete hero in the national standards’ fight, by the way) read my posting and sent me an exchange of e-mails that had taken place with the ministry. He suggests that it was this exchange that prompted the ministry to produce the clause.

The eerie thing about the exchange of e-mails is the serenity and good manners with which it proceeds, and then, suddenly, both sides realise that something is very wrong, and indeed, there is, they are in a minefield.

Brent's final e-mail changes tone and he responds powerfully.

This final e-mail exposes with a sense of wonder the gulfing disjunction between ministry and school practice, the huge illogic, the infringement of human rights, and the devastating implications for education statistics.

Steve Allen from the ministry makes a speedy departure. 

The following is Brent’s letter to me.

Hi Kelvin

I fear I am the one who may have raised the ethnicity categorisation issue with the ministry and, as a result of that, they have then included it in their guidelines.

I was challenging them on the difference between my numbers and theirs in ethnicity.

I have cut and pasted the replies I had from Steve Allen from the ministry to give you some context, and their reasoning, as no one else may have this information.

I have challenged the NZPF to seek legal advice on whether or not the NAGS stipulate this method of classification, and whether the guidelines, being guidelines not mandatory, can be ignored to a certain point.

What worry me are these clauses in the NAGS as they are as clear as mud.

‘(d) report the NAG 2A (c) information in the format prescribed by the Secretary for Education from time to time.’

Copies of e-mails below in chronological order. The more I look the more I think I am responsible for the issue becoming highlighted by the MOE.

Brent

(15/10/2012)

The following is the exchange of e-mails with the ministry.

Hi Mary

Can you please enlighten me as to what ethnicity columns are used in our student management systems for the creation of the National Standards tables? There are multiple ethnicity columns and it would be good to know how many are used by the MOE to identify when a student's ethnicity is flagged to appear in the reporting.

That is, if a student is identified with as NZ European in column one and then Maori in column two, or Pasifika in column one and Maori in column two, which set of statistics will they show up in?

I need this information for our Maori whanau hui later this month (31^st October), and for our Pasifika fono next month and would like to make sure my information for the parents is correct.

It will also help our board to get our statistical figures right across the school. Is there a rule for this anywhere?

Your support is appreciated.

Brent Godfery

From: Steve Allen [mailto:Steve.Allen@minedu.govt.nz]
Sent: Tuesday, 16 October 2012 11:59 a.m.
To: Brent Godfery | Pembroke School
Cc: Mary Geary; Pauline Cleaver
Subject: SMS and ethnicity
Importance: Low

Kia ora Brent

Your question to Mary has been passed on to me to clarify for you.  Firstly, what Student Management System are you using?

This may have a bearing on the answer.

My understanding is that generally all SMSs use a priority ranking.  On your SMS this would be Table J6 from the roll return print out and the table that shows your school’s ethnic count for MoE purposes.

What goes into the system re ethnicity data in the first place is that which is nominated by the parent or caregiver enrolling the learner.

Ethnicity should be reported in the same way that it is in your school roll returns.  This reporting uses prioritised ethnicity where each learner is assigned to one ethnic group. The prioritised ethnicity of each student is stored within your Student Management System.  

Using prioritised ethnicity, if any of the ethnic groups the learner identifies with is Mâori then they are recorded as Mâori.  If any of the ethnic groups the learner identifies with is Pasifika and they haven’t also identified themselves as Mâori then they are recorded as Pasifika.  (If a learner identifies as both Mâori and Pasifika then they are recorded as Mâori).

For the purpose of reporting to the Ministry under the new NAG2A all Pasifika ethnicities are grouped together under Pasifika.  Therefore the Ministry will have the following cohorts to report National Standards under the NAG2A requirements.

All students

Mâori

Pasifika

Male

Female

By year level e.g. After 1 year at school ……. End of Yr 4, ……….End of Yr 8

Of course depending on individual school make ups they may wish to also track and report to the board and their communities other cohorts as well e.g. if they had a large Asian cohort or ESOL learners it might be useful to disaggregate this from “All students” and track and report.  This is school specific.

Please get back to me re the SMS you are using and I can confirm or otherwise that the information I have given above is accurate for your system.

Ngâ mihi

Steve

Steve Allen

Senior Programme Advisor

(16/10/2012)

Thanks for the prompt reply Steve

We are using MUSAC.

So what you are saying is if they are listed with us as NZ European/Maori then they will show up in the all students’ category and the Maori statistics.

Pasifika/Maori then all students’ category and Maori? Not Pasifika?

If they are a Fijian Indian are they Pasifika/Asian/Indian? At times parents aren’t sure what to put on the enrolment form.

Thanks for clarification

Cheers Brent

(16/10/2012)

Hi Steve

Have had a look at the J6 table. There seems to be an anomaly in the way your system has been set up.

As there are multiple ethnicity columns parents, are choosing the order that they record student ethnicity.

Your J6 table is not selecting from the first ethnicity column it seems to be picking out the word Maori regardless of which ethnicity column it is in.

This is wrong from my point of view. If I have to filter students I always use the first column. This is giving much different figures than the J6 table.

Thus my questioning of the process.

Please elaborate as to why and where the Ministry has obtained the mandate to confer/prioritise a main ethnicity on students regardless of parental preference.

Your support in establishing this is appreciated.

Cheers Brent

From: Steve Allen [mailto:Steve.Allen@minedu.govt.nz]
Sent: Tuesday, 23 October 2012 8:40 a.m.
To: Brent Godfery | Pembroke School
Cc: Pauline Cleaver; Mary Geary; Lynn McKinney
Subject: SMS and ethnicity

Kia ora Brent

I have sought clarification for you on your query as below.

Please elaborate as to why and where the Ministry has obtained the mandate to confer/prioritise a main ethnicity on students regardless of parental preference.

The Group Manager of Education, Information & Analysis (EIA) a division of the Ministry of Education responds thus:

The Ministry’s approach is to follow Statistics NZ standards/guidelines with respect to our statistical collections.  So far as possible for ethnicity recording we collect up to three different ethnicities for each individual.  And where possible/appropriate we do ‘total response reporting’ rather than ‘prioritised ethnicity’; this move from ‘prioritised ethnicity’ to ‘total response’ mirrors changes in Statistics NZ standards over the last decade.  Where prioritised ethnicity is used we use the Statistics NZ standard to assign – this standard prioritises Mâori over other ethnicities; we understand the point that you are making but there is nothing in the standard that takes account of the order (in which different ethnicities are signalled by a parent) in doing the prioritisation.

Prioritisation is a classification which assigns the ethnicity of a person who has given multiple responses to just one ethnicity. This process ensures that the total number of responses equals the total population. It was a common approach used when accurate individual counts were wanted, for example, for funding schools.  It was also a much easier approach when collecting information from schools using paper-based returns.

Prioritisation conceals diversity within and overlap between ethnic groups by eliminating multiple ethnicities from data.  With the increased use of computers and the development of SMSs this form of expressing ethnic data is progressively being discontinued from Ministry of Education data collections.  For example, data collections such as the Ministry of Education’s school leaver’s data, which is a predominantly electronic collection, use total response reporting for ethnic groups.

However, where prioritisation is still occurring it is important that it is carried out in a technically appropriate way.  Prioritisation is not about people’s preference or the order ethnicity is given, captured and/or stored.  It is about a data rule which ensures all people in the same situation are treated the same.  The Ministry of Education, as noted, follows the historical Statistics New Zealand standard.  The logic for the J6 table referred to follows this rule correctly.

I trust this clarifies the situation for you.

Ngâ mihi

Steve

Steve Allen

(23/10/2012)

Thanks Steve

I appreciate the time taken to reply. This is interesting as it raises questions around how schools are supposed to report ethnicity numbers and achievement data to their own communities when our systems may not be set to do the same filtering. Thus our data and the ministry data may well be different.

Has this been an issue raised at any level within the ministry? If so what is being done about it?

This brings me to achievement reporting again. Thus the data in the SMS is set to be prioritised for responses to the ministry for roll count, ethnicity, and I assume National Standards data.

I am quite sure that most schools do not know this.

I am sure some schools will collate the data themselves re national standards. It will not show the same as the SMS reporting as the prioritising will not have been done.

I am sure most parents are unaware their child is counted as Maori at school by the ministry if they have put Maori as a second or third ethnicity.

Our SMS system is set by me so that when I am reporting I report on the first ethnicity column. I have the option for the SMS system to search the other ethnicity columns if I so choose. I then get data for all students with that ethnicity in any column. This means my total response data is wrong as the students appear in more than one ethnicity column thus adding up to more students than I have at school. Using the extra columns does not give prioritised data it gives all data.

For this reason I do most of my school based reporting using only the first column.

There will be many schools out there that are unaware of this.

The prioritised ranking of ethnicity is not explained well to schools.

Parents are also not made aware that when they sign our enrolment forms which statistics their child is going to be showing up in. They should be informed of this.

Is the ministry aware that by using this prioritised ethnicity ranking they may well be dragging down the statistical underachievement of Maori by including all the under achieving NZ Europeans in the data block?

They may well be lifting the achievement of NZ Europeans by the reverse assumption.

Does the ministry know if this is happening? If not then reporting these statistics could be considered unethical.

How is the ministry going to ensure that all schools have the same understanding of how the statistics are gathered?

Are all schools using SMS systems? If not then your data will be faulty as those paper based schools will not be prioritising ethnicity. Is this the case?

Your response has posed for me more questions than I thought it would.

I look forward to your replies.

Regards

Brent Godfery

(6/11/2012)

Kia ora Brent

Again, thanks for your response and the need for further clarification.

Schools are able to access information regarding the prioritising of ethnicity that is included in support material called “School Planning – National Standards template and guidelines” which is now on the MoE web site. 

Further material, including ethnicity count information, will also be added to additional resources being developed at the moment and will be loaded on the web site when completed.

Nga mihi

Steve Allen

Steve Allen

Analysis of the soundness and validity of the ministry reasoning.

I have studied the ministry e-mails carefully. 

It seems to me that the ministry is obfuscating about the problems in the way it reports ethnicity data but it is consistently using its problematic method.

It seems more cock-up than conspiracy, but that won’t be determined until statisticians and academics undertake more analysis.

The 23 October e-mail from the ministry states:

1. The ministry follows Statistics NZ standards and guidelines.

2. The ministry collects up to three ethnicities per individual.

3. Where possible/appropriate the ministry uses total response rather than prioritised ethnicity.

4. Where prioritised ethnicity is practised, the ministry uses ‘the Statistics NZ standard to assign - this standard prioritises Maori over other ethnicities’.

5. ‘Prioritisation conceals diversity within and overlap between ethnic groups by eliminating multiple ethnicities from data.'

6. The ministry is progressively discontinuing prioritisation reporting.

7. Nevertheless, where it is still used, the NZ Statistics standard applies.

But:

On the NZ Statistics site there is a document entitled: ‘The Impact of Prioritisation on the Interpretation of Ethnicity Data’.

It explains why prioritisation is problematic, compared with total responses and shows that prioritisation can lead to significant under-reporting of all ethnic groups, except Maori (who are prioritised). 

‘While prioritisation may have been a useful analytical tool in the past, it is no longer considered viable in reflecting the changing face of ethnic diversity in New Zealand. Valid analysis of a group depends on the consideration of all its members, taking into account differences within and between groups.’

In other words, the ministry’s requirement to use prioritisation of Maori in reporting ethnicity is not considered valid because it doesn't take into account differences within and between groups.

The Statistics New Zealand website explains further:

Why are prioritised ethnic data not recommended?

Prioritisation re-classifies a person with multiple ethnic responses to just one ethnicity or ethnic group. There are several problems with choosing one ethnicity for a person who has said that they belong to more than one ethnicity:

The ethnic question asks a person to choose which ethnic group or groups they belong to.

The question accepts more than one answer and prioritisation goes against the principle of self-choice.

People are not asked to specify one main ethnic group and do not know that prioritisation would make this choice for them.

Prioritisation misrepresents the number of people who identify with an ethnic group.

See The Impact of Prioritisation on the Interpretation of Ethnicity Data.

Other countries which ask a similar ethnic question allow multiple responses and do not use prioritisation (e.g. United States, Canada, Australia).

Statistics New Zealand no longer prioritises ethnic responses but uses total response where people are included in each ethnic group that they identify with. This approach applies to all official statistics and is consistent with the findings from the 2004 Review of the Measurement of Ethnicity. Statistics New Zealand can provide prioritised ethnic data as a customised request. For more information email info@stats.govt.nz or telephone 0508 525 525 toll free.

In other words, not only is it an invalid method, it is unethical and inconsistent with best practice internationally.

Despite this, on the ministry website:

The School planning national standards data Excel template only has the categories: All students, Maori, Pasifika, male, female. In other words, it is not consistent with NZ Statistic’s approach to ‘all official statistics’ which ‘is consistent with the findings from the 2004 Review of the Measurement of Ethnicity’.

Moreover, in the ministry document ‘School Planning National Standards templates and guidelines (p. 5), schools are instructed to use the prioritisation method and are not told that they are, in effect, reporting invalid ethnicity data and are required to act unethically in so doing.

However, in one sense there is nothing new or sinister in this e-mail exchange between Steve Allen and Brent Godfrey. The ministry regularly reports prioritised data despite the fact that the method is now not recommended by the government’s own statistical experts, and is being discontinued. But in another sense it is new and sinister, because virtually no-one would be aware of the implications of the method used.

It needs to be noted that this requirement potentially could have the most disproportionate and distorting effects on the national standards data in lower decile school communities. For example, in the 1 July 2012 returns, using prioritisation reporting of ethnicity, there were 76,415 Maori students in decile 1-3 schools and only 28,560 in decile 8-10 schools. In decile 1 schools, Maori were 50 percent of students compared with 5 percent European/Pakeha. In decile 10 schools, Maori were 6 percent of students, compared with 72 percent European/Pakeha. However, given the way prioritisation skews the reporting of ethnicity, it would be interesting to see what differences, if any, the NZ statistics recommended methods of reporting ethnicity data would make.

The statistical method used, because it invalid and inconsistent with best practice internationally, and because very few people would know the method was being used, let alone understand it – has highly significant implications for the education system.

Parents will be confused and enraged that bureaucrats have infringed their human right to determine the ethnicity of their child.

If schools reported to their community on the basis of prioritisation there would be an uproar. One school I spoke to of mainly Pasifika children said a third of their children also had Maori in their ethnic sequence which, if prioritising was used, would lead to absurd community reporting. No school would do this, meaning there is a huge discrepancy between school and ministry practice. And if it is absurd at the school level, it is exponentially more absurd at the national level.

Finally, to use an invalid method which is also unethical, inconsistent with best practice internationally, and done without being understood by most people has fantastical implications for all education statistics.

At this stage, I’m not suggesting that there has been a conspiracy to distort education statistics in certain directions, but the validity of a considerable range of education statistics is now blowing in the wind.

The implications are axis shifting.

This has not been a rip, tear, and stomp posting. The matter in hand has been too serious for that. Anyway, my first concern was to get my own head around what it was all about. Nor do I know how long prioritising has been going on. Has it been brought in by this government or been established practice? If it has been brought in by National all hell will let loose. I sense, however, that it has been established practice. Let us wait and see.

However, how long has it been not best practice? Don't our teachers and children deserve best practice? Other comparable countries don't use the practice. Don't our teachers and children deserve comparable practice?

And why, in area as sensitive as children's school characteristics and performances, is an invalid method of gathering statistics persisted with, when the ministry in other areas is moving to a valid one? There are questions to be asked and agendas to be scrutinised.

There urgently needs to be debate.