IATEFL ELT Journal Debate: Language testing does more harm than good

Nothing like a good debate! Besides, I couldn’t pass up the opportunity to see Richard Smith present, given how heavily I’ve drawn on his work in my learner autonomy projects…

The ELT Debate is an annual fixture in the IATEFL conference. It is a symbolic event in some ways, representing the relationship between ELTJ and IATEFL.

Language testing is a hot topic this year, with 46 other talks on testing and evaluation. Ways of testing have been described elsewhere in the conference as an uncertain and approximate business, there has been talk of fairness too.

The motion for today is that Language testing does more harm than good. We have Richard Smith to propose, from University of Warwick. To oppose, we have Anthony Green from the University of Bedford, a certified language testing enthusiast!

The rules of engagement:

For 15 minutes, Richard will propose. Then, over to Tony for 15 minutes of opposing the motion. After this, the floor is thrown open to comments for 20minutes. These will be addressed by the speakers all together at the end of the debate. At the end, there will be a vote!

Richard Smith proposes:

Richard isn’t a testing specialist but has an interest in the history and future of the profession and perhaps is the only person foolish enough to accept the challenge (his words!). He hopes to give voice to concerns of teachers, his students at Warwick, from many countries and start a debate which will continue beyond the conference. We need to be critical of the dominance of testing in our profession at this moment. The growing dominance of testing in ELT is reflected in the number of talks at the conference. Debating and being critical enables us to make changes. As we have done in materials, use thereof, content, with regards to native speaker models etc but we haven’t yet had this kind of debate about testing.

Looking at the pages of the newspaper in the conference pack, there is a large number of advertisements for tests and for preparation for these tests. It would be interesting to analyse the change in this over time.

Richard defines the proposal as referring to proficiency tests and their use, as well as institutional tests (achievement tests) that students must undergo and university entrance tests. Large-scale testing, institutional testing rather than assessment more broadly e.g. classroom assessment. So we are not talking about portfolio assessment, continuous assessment etc. (Which Richard thinks are good things, as are diagnostic tests etc.) It is achievement tests and proficiency tests that are in question.

Why has there been so little criticism until now? Because it is a big business and many of us are involved in some way. We feel in some way removed from the business. The specialism of testing is engaged with at specialist conferences. As teachers, we feel we lack the expertise to criticise, to change things.

Richard has done some research and is pleasantly surprised that there are critical debates going on amongst testers. However these haven’t filtered down to us as professionals here. He recommends The Power of Tests to follow up on the subject. He also went back to Language Testing for Teachers by Hughes, who says that too often language tests have a damaging influence on learning. I.e. backwash, or now, wash-back. Too often they fail to measure what they are intended to measure. It is an uncertain and approximate business. Tests don’t measure everything that is worth testing, only what is measurable.

What should we try to measure that we aren’t measuring?

Education itself. (He is quoting from a book from 1961 by Robert Lathan)
Insight into one’s language.
Insight into other cultures
Insight into one’s own culture
Attitude to minority groups.

We only measure utilitarian things.

Can we assess intercultural competence? learner autonomy? Things we think are valuable? These tests are not doing that. “Any assessment of language impoverishes one’s view of the nature of language as communication” – 2001. What about the role of native speaker norms in testing? Tests are still predicated on these. We also can’t capture the dynamism of language. We are expecting students that we test to conform to a norm to which they do not belong. Tests are not yet reflecting the realities of trans-languaging. They act as a barrier to change in our profession. They are a conservative force.

Areas of harm:

teaching to test/backwash.
increasing dominance of tests leaves less and less time for higher values (that we value as educators)

There is a hegemony in our field of proficiency as all-important. We see around the world the dominance of proficiency tests. But do we really want a system where only utilitarian goals are important?

What about the impact in a broader sense on the psychological well-being of learners? What is the psychological and social impact?

Anthony Green opposes:

Starts with a health-warning – those of a nervous disposition, with a weak heart, should leave the room now. AG is a language tester. Be afraid. Be very afraid. He is going to give us a test. If we fail, a preprogrammed testing noose will tighten around your neck…

First question:

MCQ can choose more than one answer

Are you or have you ever been a teacher?
Are you now or have you ever been a trainer?
Are you now or have you ever been a publisher?
Are you now or have you ever been a psychopath?
Are you deliberately setting out to harm your students?

T/F

You have made a test and given it to your students.

MCQ

Why did you do this?

a) to kill time because you were bored and wanted to keep students busy
b) you don’t much like your students, you want to punish them
c) because you wanted to see how successfully they performed on the content of the course

Is it harmful? No. Tests are jolly good things because we know that regularly testing students is a way to improve performance and memory. It encourages practice, enhances the practice done, it guides teaching and learning. We know what students have learnt, we know what we need to cover in more detail. Tests are an absolute necessity. Without tests, there can be no effective teaching. We only know if we have successful learners if we do some kind of testing of our students.

AG recognises that RS wants to talk about standardised tests (2% of testing that gets done) and that’s fine. So let’s imagine we have a world where there are none of these tests. What might it be like? We don’t have to think that hard, history can inform us. Testing has only been with us for a relatively short time. Not true to say there has been no debate over it. In 1889, Herbert Spencer brought together essays called The sacrifice of education to examination. So it has been talked about for a long time.

What are tests accused of?

causing the wrong sort of learning, they narrow down the scope of it.
narrowing the curriculum, only what is on the test is studied
encouraging students to think about competition rather than self-improvement
being biased in favour of one group or another

All of this may be true. But the motion is that language TESTING that does more harm than good. But is it the tests or the ways in which people are using and abusing them? Is it the test or the teacher’s idea of what test preparation involves? Is it the fault of the test or the fault of the teachers or the teacher trainers or the education system or the publishers? Who drives this? The content of the test, or use/misuse?

Let’s go back to this idea of throwing away formal tests. No more IELTS, TOEFL, Pearson etc. We can go back to pre-1800s when it was so. What happened? Say you wanted to get a job where you had to use a language, what did you do? If you had a rich, influential relative, you would be handed it; if not, your option would be to buy one.

Think about your pilot who will fly you home? Would you prefer they had done a carefully constructed test that shows they could communicate with air traffic control? Or one who had paid someone for the job?

Tests are a better alternative than the bribery, corruption and old-boy networks that we used to have. We depend, sometimes for our lives, on effective language tests. That is not to say it’s perfect. There is a lot wrong with tests in widespread use today. So we need to all get involved and engaged with language tests. For this, we need to get involved with language testing – studying how tests are made, why they are made that way, whose interests they serve and what social purposes they carry out? We shouldn’t bury our heads in the ground and leave it to Cambridge etc. but study how these things are done, understand the thinking that goes into them, and engage with it. Take responsibility. Identify problems and deal with them, if there are problems.

AG would urge us to engage more with language testing, identify where its doing good and support that good, and identify where harms are happening as well as what is causing those harms. Even in our own classrooms, we can do better. The way we do better is by not pushing language assessment/testing to the end of the CELTA or an afterthought in an MA. It should be central to our practice. We have to engage with the way tests are made in order to improve them and maximise the real good that they do for our students.

Comments from the floor

Now that the scene has been set, we have 20 minutes for audience comments.

From Turkey: Agrees with AG – it would be unfair for teacher and students if there were no language tests. From students’ perspective, they are spending a long time learning and if there were no tests, at the end of the time you wouldn’t know where you are or where you are going. The critical point here is the data. What we do with the result of the test. If the data is used for assessment for learning, it is beneficial. If not, it is loss of time.

There is a difference between assessment and testing: the nature of the organisation providing the test has a big influence. What is their agenda? Responsibility is with the test-makers and organisations to make sure they are testing what they say they are testing, to produce reliable results.

From Middlesex University: sympathetic with Richard – frequently has nightmares that she has to take her maths O’ Level again, or that she has to do her university Spanish exam again. Speaking for the students who undergo the psychological terror of testing and exams. But..works in testing now. A lot of people would be out of a job if tests were scrapped. Not a good reason but the biggest issue at the moment is the visa issue. On pre-sessional courses, IELTS score or equivalents are required. There are politics and politicians deciding.

Mexico city: works with a publisher, schools are constantly asking for testing/standardisation tools.

Alan Maley: the consequences of failing a test, on the student. When he was 11, he took the 11+, the test that at that time decided whether you went to grammar school or a rubbish dump. Your whole career was decided by that test. He failed it. Then he got a scholarship to a Cambridge college later on.

Salford: language testing is not isolated from broader educational trends. Those trends are test, test, test, test. We are driven by tests and test scores. Language testing is part of that narrative. We need to be cautious about when and why we test.

The problem isn’t with the tests themselves but the overuse of them and the publishing/testing industry coming together with ever more endless tests. And again and again, you are preparing students for tests. Proficiency tests have their place but maybe there are too many moments where students have to take tests. The testing industry is part of this narrative.

Jeremy Harmer: quoting someone, “the whole point about testing is that it is a money delivery system, delivering money from governments to private companies”. The current government is trying to put in place tests for children when they enter primary school. The whole point is, we are all involved in testing but if you look at the way testing is used, which can’t be divorced from the test itself, it frequently does more harm than good.

The point is, whose responsibility is it? AG put responsibility with teachers, trainers etc but that is not the only responsibility around. But it also lies with testers. Do testers consider the social and political consequences of tests? Gatekeeping? Tests have got consequences and it’s up to the testers to think about these. “It is not permissible that the authors of devastation should also be innocent, it is the innocence which constitutes the crime”

What is education for? Do testers realise how much power they have? Power in the wrong hands does a lot of harm. What about test-taking strategies and skills? How do they affect the validity of a test? The fundamental question is, what do we want our students to come to school for? What about growth and development?

We live in a world of scarce resources. So we have to have a method to decide who gets access to resources. If we didn’t have tests, there would be another way. When teachers teach to the test, they undermine the value of the test. If you want a better test, it’s going to be a lot more expensive and take a lot longer to assess people? Tests serve a very limited purpose, so we shouldn’t try to over-interpret what they can do. We all are in this together and unless we are all talking about it, and taking responsibility, then you can’t lay the blame at someone else’s door.

Denmark: supports Richard. Really concerned with all the national tests that exist in Europe from the age of 3. Everything said in favour of tests is teacher-directed. What about the learners?

Time for comments ran out

Our two speakers have 4 or 5 minutes to summarise their case

Richard Smith:

Testing has been exploited as a method of control and power, as a way to select, motivate and punish. Richard did a mock debate with this motion, with his students within the language testing option, a popular option, and most of them were in favour of the motion but by the end, they were in against it, as it is “a necessary evil”. Do we want evil in our profession? If there is harm being done, which we have established there is. Not just wash-back but psychological damage etc.

So what to do about this problem? “Tests and their demands are forced on test takers and teachers from above and they are forced into a position where they have no choice but to comply with demands.”

So let’s do something about it! RS wants to extend an olive branch to AG and thank him for engaging with teachers, with the profession. RS wants critical assessment literacy not just testing literacy. The testing industry should be giving back to the profession not just taking from it e.g. scholarships, financial support, planting forests for reducing carbon footprint etc.

Anthony Green

99% of testing that happens is done by teachers in the classroom for the benefit of student learning. There is no way that we can teach our students unless we test them and train them to assess themselves too.

Back to large-scale testing. The tobacco industry caused a lot of damage. Did it reform itself? No. It reformed because consumers took control and pushed reform onto it. The motor industry. 30 years ago, cars were death traps with no safety features. As educators, you have a duty to students to take back control of language education from the language testing industry and give it back to learners where it belongs. Make language testing and assessment work for learners, not the language testing and assessment industry. But they are not going to reform themselves. If you want to improve the practice of language assessment, YOU need to take control. AG is delighted that there is this debate, THIS is being critical of language testing. Always ask the question, in whose interest is this being done, and how might it be done more in the interest of the learners. It will only be done better if you engage with testing. Let’s develop our assessment literacy and improve testing, not leave it to the evil Goliaths!

Conclusion:

Now the final moments – it’s time to vote. We are going to vote by popular acclaim. Graham has a device to monitor how loud the “yay” cheer and the “nay” cheer is… He will compare the data and we may have a winner…

Ooops…the device broke down… make your own decision! 🙂