I had no idea this was being done, but erased answers are now analyzed on standardized tests. Schools with a high number of wrong-to-right changes across multiple tests are presumed to have cheated: teachers changing the answers after the students are done.
vwm • February 16, 2010 6:52 AM
That’s one of the reasons why non-erasable pens should be used for exams.
Anon • February 16, 2010 6:59 AM
An excellent teacher in my kids’ district was fired for this last year. It turns out that there were an unusually large number of answers changed from incorrect to correct responses on the standardized test. The superintendant said the proof of cheating by the teacher was that the variance was two standard deviations from the mean.
Apparently is never occurred to them that the gifted program requires an IQ score three standard deviations from the norm and students qualify for it every year.
I can’t tell you how frustrated and sad I was to see an inspiring teacher’s career ruined by an administration that failed to understand elementary statictics.
Clive Robinson • February 16, 2010 7:13 AM
The root cause of this is Politico’s and their inability to come to terms with the fact that the majority are actually incapable of behaving rationaly.
There is an increasing body of evidence that “testing” is bad for children and esspecialy “test craming” which is a consiquence of poiticians trying to prove their latest mind numbingly wrong idea on education is working.
Oh the way they ensure that the tests show what they want, they fire teachers who don’t sing from their song sheet.
Yes there are bad teachers especialy in mathmatics and sciences but not as many children would fail by them as are failing by “exam cram”.
In the UK it is interesting to note that the Education Secretary and his deputy will not take the simple test they require of teachers (possibly as sugested by various members of the press that they cannot get a grip on addition let alone division and fractions).
Brandon Thomson • February 16, 2010 7:17 AM
Once on a standardized test in primary school I realized I had been marking the answers in the wrong section on the test. I told the admins and during our lunch break someone erased my incorrectly-placed answers for me and moved them to the correct spot.
I suppose they can’t do that sort of thing on a test that looks at erasures since it would register as HUGE CHEATING DETECTED!
The USofA seems to suffer from the “Chinese Exams Syndrome”. The ultimate book on this is “The Scholars” by Wu Jingzi.
It does not matter what you learned, or what you are good at. If only you pass the test better than others.
“The classical Chinese imperial examinations were the ultimate in pointless testing and learning for the test. Since the Han dynasty (200 BC), applicants to government positions were required to do an Imperial exam (http://www.newworldencyclopedia.org/entry/Imperial_Examinations_%28Keju%29)
The contents of the exams were utterly disconnected from the work the civil servants were supposed to do and equally disconnected from any other form of reality of the time. But students would commit most of their waking time to study for these exams.”
Looks fitting here.
R
AC2 • February 16, 2010 7:40 AM
@R re ‘Chinese Exam Syndrome’
Err this is how pretty much every single important exam in India works as well… OK barring a few honorable exceptions.
I should know having sat through SEVERAL of them…
peri • February 16, 2010 7:44 AM
You can find the actual report here:
Brian • February 16, 2010 7:58 AM
Anon: “Apparently is never occurred to them that the gifted program requires an IQ score three standard deviations from the norm and students qualify for it every year.”
Really?
Winter • February 16, 2010 8:33 AM
Anon: “Apparently is never occurred to them that the gifted program requires an IQ score three standard deviations from the norm and students qualify for it every year.”
That in itself is not a problems. It becomes a problem if >25% of the students are 3 SD from the norm. 😉
Winter
peri • February 16, 2010 8:35 AM
I also wanted to suggest reading this:
http://www.gaosa.org/news.aspx?mode=detail&obj=1919
Of all the schools in Georgia, 20% will have to alter their testing regime and 10% will be investigated by their superintendent and file a report with OSA.
Scott • February 16, 2010 8:36 AM
This is inevitable outcome. When the education industry is judged by a single, standardized test, the test will inevitably be fudged to keep the players in the game. “No Child Left Behind” translates to “Teach To The Test” – ask anyone who’s involved in K-12 education. Predictably, effort and funding will now go to protecting the test rather than educating students.
peri • February 16, 2010 8:43 AM
@Winter: “>25% of the students are 3 SD from the norm”
Actually “Severe Concern” is when more than 25% of the tests are flagged. They think anything above 6% of the tests being flagged is cause for concern.
Carlton • February 16, 2010 8:47 AM
.
.
The falsification of data & reports in bureaucratic organizations has been a standard problem for over two centuries.
It’s inherent in bureaucratic systems.
People are rewarded or punished based on what their bureaucratic managers & senior managers perceive they are doing. But in large organizations those managers must usually rely upon written reports/data (..often just summaries) from the lower levels– which are difficult for the distant managers to verify.
Therefore, there’s a strong incentive for lower level personnel to present their reports in the most favorable light… and often to “adjust” the data to make themselves look good, or conceal problems.
In the U.S. military… the age-old slang term for this is “Pencil-Whipping” — you whip out your pencil and adjust the data/report to whatever will make the Colonel’s & General’s happy… and keep them off your back.
That’s how bureaucratic organizations work.
Wise managers (and Generals) know they must dig deep and personally visit the front-lines to see what’s really going on in their organization.
kashmarek • February 16, 2010 9:02 AM
Easy workaround: mark all answers then erase all and remark the correct answer.
kashmarek • February 16, 2010 9:03 AM
Correction –
Easy workaround: EVERYBODY mark all answers then erase all and remark the correct answer.
aristos_achaion • February 16, 2010 9:04 AM
Of course, inherent in this is the teacher’s interest in the test scores. Maybe if we didn’t judge teachers by standardized tests, we wouldn’t be in this mess?
Michael Ash • February 16, 2010 9:12 AM
Hate your teacher? All your classmates hate him too? Easy solution! Conspire to have as many students as possible erase fake answers on your tests.
aikimark • February 16, 2010 9:15 AM
Maybe if we replaced the mark-sense paper with butterfly ballots for these tests, we wouldn’t have this problem. Everyone knows how reliable they are and how easy it is to change your choices.
Oh…Wait…never mind. 😉
=================
Would it eliminate cheating if tests were administered at different locations and randomly ‘graded’ by different teachers?
Tangerine Blue • February 16, 2010 9:22 AM
“presumed to have cheated: teachers changing the answers”
Why is it presumed the teacher cheated? Why couldn’t cheating be driven by the Principal, or Superintendent of Schools?
Anyway, from the article, the statistic methodology used is unclear. Were alterations from right to wrong, or wrong to wrong, also factored in?
By definition, not every sample will conform to the norm; there have to be deviations. It’s unclear to me why teachers should be presumed to cheat when their class deviates from the norm.
If they want to use abnormal scores as a trigger to investigate teachers, I’m ok with that. But before calling them guilty they should catch them red-handed, or at least have evidence stronger than abnormality.
Mace Moneta • February 16, 2010 9:34 AM
It always bothered me that school is so different from ‘training’.
In a business environment, we frequently brought in trainers for week to two-week training sessions – it’s more cost effective than sending 20 people out. In those courses, the emphasis was on problem solving and creativity, rather than testing. For example, writing software or configuring routers.
It’s not only more effective – learning by doing – but more enjoyable as well. I always wondered why schools are not oriented the same way: learn one thing, demonstrate proficiency, move on to the next thing. It seems to be the way real life proceeds as well, outside of a school environment.
Jason • February 16, 2010 10:02 AM
I had a 10th grade science teacher (10 years ago …) that for one test, one random time, decided to ‘seal’ the test score by placeing a difficult-to-remove clear tape over her score.
Applying clear packing tape over the scores of a bubble-in standardized test can’t be that difficult (esp those on narrow cards). THe student should do this as the final step in his exam to prevent alterations by supervisors …
RT • February 16, 2010 10:10 AM
Wouldn’t it be better to separate the questions from the teachers, and have each test randomized so that all of the answer sheets would look different?
Guilty Prof • February 16, 2010 10:11 AM
Unfortunately, this idiotic notion of judging teachers based on what percentage of their students pass is creeping into higher education as well. I’m a math professor at a public university in the U.S., and we have recently been told that in our basic calculus classes, at least 85% of students need to “pass” with a grade of at least C-.
Now, here’s the kicker… I routinely have calculus classes in which fewer than 70% of the students can add fractions. Clearly, the powers that be think that “graduation rates” somehow reflect actual learning.
Of course, I’m part of the problem. I don’t have tenure yet, so I’m not going to buck the system. I take all those “D” students and half the “F” students and bump them up to a “C-“. Makes me sick, but it beats unemployment.
RT • February 16, 2010 10:15 AM
Jason: Or have them feed the answer card into a machine that laminates it automatically, or treats it with some other type of film that would detect tampering.
jeff • February 16, 2010 10:28 AM
So, a smart teacher would also create erasures on wrong answers that they haven’t changed to defeat the wrong->right/right->wrong statistic. Could the analyst infer that the teacher was cheating just because an increase in erasures where there is no discernable bias in the erasures themselves?
This is quickly becoming a counter-intelligence exercise.
jeff
M • February 16, 2010 10:34 AM
I’m curious: why is ‘teaching to the test’ bad, assuming the test is relevant to the curriculum?
I realize my assumption is not reality in many cases, but I’d like to focus argument based on this assumption. How else are we to measure the performance of teachers, and of students, if testing for mastery of the prescribed curriculum isn’t the answer?
I’ve heard the argument that students could game the system to get a teacher in trouble, but as long as there are negative consequences for the student (i.e. fail the test, repeat the course), I don’t see why this couldn’t be mitigated.
M • February 16, 2010 10:37 AM
@guilty prof: That is indeed a sad state of affairs. I see no reason that testing for mastery of curriculum, in and of itself, is bad, but mandating that a certain percentage of students “pass” pretty much defeats the whole purpose.
Daniel • February 16, 2010 10:40 AM
“2. Rejection of H0 does not necessarily imply cheating. Alternative explanations are possible.”
Exactly. Yet somehow this got lost in the news article.
I’m HIGHLY skeptical of the ability of scanner to determine whether or not an answer was changed. If you look at the numbers in the report closely, you’ll see that according to the scanner almost all changes were wrong to right; there were very few wrong to wrong answers recorded. That alone strikes me as wildly improbable. One big flaw of this study is that they is no evidence that took a random sample of the recorded changes and visually inspected those documents to determine if what the scanner was recording was in fact accurate.
Don’t misunderstand. I am sure there are teachers who cheat. I’m just skeptical that this study is anything other than a witch hunt.
HJohn • February 16, 2010 10:42 AM
@ Guilty Prof at February 16, 2010 10:11 AM
I agree with you. Pass rates are a poor measurement of a teacher’s quality. A lot of low pass rates are subject to other factors, like lax admission requirements or just bad luck on getting a lot of lazy students.
To be sure, a teacher’s ability to teach no doubt has influence, but it is not the only one.
My main problem with things like this is that it creates a whole new agenda for teachers. (Much like how using fines as a revenue stream creates a whole new agenda for law enforcement, government, etc.)
lazlo • February 16, 2010 10:44 AM
@Mace Moneta:
The problem inherent in applying business-like training methods to a school setting lies in the motivation of the students. At my work I’m highly motivated to get the most I can out of my training, because I can easily see that the training impacts my ability to do my job, and thus keep my job. Some college students may believe that actual learning will be more beneficial in the long run than a passing grade (and they may even be correct), but when you get into high school or even more so primary school, there’s very seldom any student motivation to learn, so you have to fall back to the next best thing, which is to try and motivate (sometimes successfully) a student to achieve the short-term goal of passing a test. In theory, some actual learning may occur as a side effect.
HJohn • February 16, 2010 10:45 AM
@HJohn at February 16, 2010 10:42 AM
We do have this nasty little tendency to create backwards incentives.
Give a student the grade they deserve and get fired and have a dark mark on your record (maybe preventing future hiring), or bump them up to a 70% and keep your job. Not the way to go.
Of course, then the students are promoted through classes due to teachers doing CYA and they go into the real world with a diploma or degree they might not be able to spell.
jgreco • February 16, 2010 10:56 AM
@M
I don’t hold strong opinions on this matter as I’m not an educator and managed to mostly avoid these types of situations during my public schooling. The main argument against “teaching the test”, as I understand it, is that it prevents good teachers from teaching their students in a more effective, taloring the curriculum to fit the interests and pace of the students. Opponents of the system and the laws that encourage it generally suggest that the power to judge the competancy of teachers should be left with local administration, who would be free to make judgement calls of their own. Legislation such as No Child Left Behind is a relatively new development, so arguably the system that was in place before was a workable one.
“but as long as there are negative consequences for the student (i.e. fail the test, repeat the course), I don’t see why this couldn’t be mitigated”
The trouble (or beauty, depending on perspective) with these situations is that the students can potentially game the system to get teachers in trouble, while at the same time doing very well on their tests. In fact, it works best if the students do very well on their tests, the better they do, the more convinced the administration will become that the teachers cheated.
Guilty Prof • February 16, 2010 11:22 AM
@M
I have no problem at all with well-designed, relevant, standardized tests. However, I haven’t encountered many of these. It is extremely difficult to design a multiple-choice test which can rigorously test problem solving ability, yet multiple-choice tests are the most common testing method because they are easy to score in mass quantity.
A good test must give the students hard problems which require multiple logical steps. In a timed setting, even the best students are bound to make simple careless errors which give them the “wrong” answer which, on a multiple-choice test, is indistinguishable from a random guess by a poor student.
So, a good test needs to be graded by hand with the instructor reading through the complete solution written by each student. That takes time and money.
Instead, test like the calculus Advanced Placement exam have a bunch of easy multiple choice problems. These sorts of problems test skills like pattern-matching and rote memorization, but they certainly don’t test critical thinking. (And now that the students all use calculators, they don’t even need strong arithmetic skills.)
Matt from CT • February 16, 2010 11:38 AM
I always wondered why schools are >not oriented the same way: learn one
thing, demonstrate proficiency, move
on to the next thing.
Because they want to have their cake and eat it, too.
They want lots of money to come from bureaucracies several layers removed from the school — the Feds & State, as well as mandates of how local tax dollars are spent.
That means the local School Board isn’t able to provide much meaningful oversight since they have little discretionary funding — it’s either mandated expenditures or comes with strings attached.
They want strong unions with lifetime tenure for teachers. Hard to fire a teacher who loses interest in their career.
They don’t want to be held accountable by market based reforms to let people chose between options for their children. Vouchers are an evil word.
They don’t want to be held accountable by standardized tests. NCLB is almost as evil as Vouchers.
What it seems is they want no accountability to a local school board, no accountability to any definitive standard…just to have a happy sit around the campfire singing Kumbya and have a group hug with fellow government bureaucrats about how well they’re doing.
Your training classes? If the employees don’t provide positive feedback and/or management doesn’t see tangible improvement, they hire someone else to teach the next class.
Unfortunately, this idiotic notion of
judging teachers based on what
percentage of their students pass is
creeping into higher education as well.
Creeping?
In 1991 UConn suspended one of their Chemistry professors for having too high of a failure rate. She normally would’ve taught my class. The ironic thing is she wrote the text book we used.
Of course, with her being tenure protected, how do you punish someone if they spent more effort writing a text book then teaching their students?
While it other factors should be involved in decisions, tests represent one of the few objective data points we can to evaluate the effectiveness of the education of individuals or small groups.
BobW • February 16, 2010 11:43 AM
M:
“I’m curious: why is ‘teaching to the test’ bad, assuming the test is relevant to the curriculum?”
It’s because a test is supposed to be a sample, not an inventory. It is not supposed to drag every last bit of knowlege out of the student.
If the students study the answers to the questions, or the likely questions, and only that, then the students end up knowing much less than they need to know.
HJohn • February 16, 2010 11:44 AM
@Guilty Prof: “It is extremely difficult to design a multiple-choice test which can rigorously test problem solving ability, yet multiple-choice tests are the most common testing method because they are easy to score in mass quantity.”
In 1996, when I was studying my masters, a professor was asked about using multiple choice instead of essay. She said that she prefered multiple choice because essay was biased against students when english wasn’t their first language.
Tangerine Blue • February 16, 2010 12:11 PM
@Guilty Prof,
I guess I understand that student performance is not tightly coupled to teacher performance. But isn’t there some correlation between teaching and learning?
On what criteria should we judge the quality of a teacher?
[Ask me how to judge the quality of my own peers, software engineers, and no standardizable measurements seem adequate. But it is reasonable to rate the product quality, and that seems like a fairly good starting place for evaluating the workers.]
LeSpocky • February 16, 2010 12:42 PM
Isn’t this a problem, which only occurs with multiple-choice tests? This would be nearly impossible with exams you have to pass in Germany, because the students are forced to write text and give explanation or even outline the way they took to get to the answer. I assume it’s harder to cheat here, because you don’t only have to make some crosses.
HJohn • February 16, 2010 1:39 PM
@LeSpocky: “Isn’t this a problem, which only occurs with multiple-choice tests? This would be nearly impossible with exams you have to pass in Germany, because the students are forced to write text and give explanation or even outline the way they took to get to the answer. I assume it’s harder to cheat here, because you don’t only have to make some crosses.”
On raw “right or wrong”, yes. However, in essay and written answers professors have much more discretion in assigning points.
On essay’s in college, I would always write an answer on essays even if i made it up. I also noticed that some teachers gave more credit for a response than others. So, teachers can still influence grades by giving someone partial credit for an answer that isn’t even partially correct.
Part of the appeal of multiple choice, aside from the ease of grading, is teachers are much less likely to be accused of discrimination because its on an exact plane for everyone. On essays, a student can claim s/he was discriminated against because they got 5 out of 10 points for a made up answer and someone else got 6.
No easy solution.
Clive Robinson • February 16, 2010 1:51 PM
@ LeSpocky,
“… because the students are forced to write text and give explanation or even outline the way they took to get to the answer. I assume it’s harder to cheat here, because you don’t only have to make some crosses.”
You have to ask who is cheating the system and why.
The usual assumption is it’s the pupil thus the test security is based around this assumption.
Now if you assume the person doing the cheating is the tutor then a quick look at exam security shows it does not protect against this.
A simple example would be handing out “mock papers” with “sample answers”. It’s a standard practice with most exams.
Now if the teacher sets the test, or the teacher has control of the exam papers prior to the test then it is possible for questions very very similar or the same as the exam questions to be given to the students as a last revision excercise.
And I know for a fact this happens somtimes covertly sometimes blatantly.
The problem then becomes if you assume the teacher is also going to cheat what changes do you make…
I used to go to an educational establishment where the teachers where fairly smart. They could from fairly simple analysis tell you what questions where going to come up on an exam paper set by an external test agency long before the test papers arived.
How did they do it well it’s a fault of the exam setting process.
Few people realise just how much effort goes into selecting good exam questions for written papers is. What most exam companies did was something like 80% of the questions would have been on a previous paper in the past five years (the values might change but the question would be almost identical).
Thus it was possible to train students on previous papers and have a better than 50% pass rate without even teaching the subject. The teachers concerned where quite open and honest about this if you as a student asked this.
One of the reasons given for switching to “multiple guess” papers was that there would be considerably more questions than a tutor could guess at or a student remember. Time has shown this argument to be incorrect.
JimFive • February 16, 2010 1:55 PM
@Mace
There are many reasons this type of modular learning is not the norm at public schools. Modular learning raises many issues with respect to grade promotions, grading in general, the balance of the curriculum, and more.
When I was in elementary in 1980 my school switched to a modular language curriculum. In order to correctly place students within the curriculum we were all given a series of tests to see what level we were at. 10% of my class tested all the way through the curriculum. What do you do with those people. (I note that these 10% would have tested out of a modular curriculum in any elementary school subject, not just language)
JimFive
HJohn • February 16, 2010 1:58 PM
@ LeSpocky at February 16, 2010 12:42 PM
Also….
Consider a corporation that hands out big bonuses and raises to managers based on how highly their subordinates are evaluated, and lays off managers based on how poorly their subordinates are rated.
Considering managers evaluate their own staff, we can clearly see how this would compromise the performance evaluations.
I see little difference in test scores for teachers. With multiple choice they can cheat outright by changing answers, and on written they can cheat indirectly (as in the evaluations example) by falsely inflating the scores.
kashmarek • February 16, 2010 2:01 PM
The reason the teachers were presumed to be guilty is to protect the school administrators from being caught.
JimFive • February 16, 2010 2:02 PM
@M
Why is teaching to the test bad? Well, because the test is inherently limited. Even long tests like the SAT or GRE are limited in the amount and type of questions they can ask. Due to these limitations it is possible to teach someone how to answer specific types of questions that appear on the test WITHOUT teaching them the problem solving principles that the test is attempting to gauge.
JimFive
JimFive • February 16, 2010 2:09 PM
As someone earlier commented this was in the book “Freakonomics” over 10 years ago. One thing the book mentioned looking at wasn’t just erasures, but strings of correct answers on the more difficult questions for students that were getting the easy questions wrong. In the case examinied in the book a string of around 20 questions near the end of the test were looked at and it was pretty clear that there was a statistical anomoly — highlighted by the fact that everyone got the last question wrong (the teacher had the wrong answer)
It is also noted that by retesting using proctors instead of the teachers, the scores for those students went back to where they had been previously.
JimFive
Joe A • February 16, 2010 2:15 PM
I haven’t seen much discussion in the comments on the basic premise of this report: that the percentage of answers that are changed from wrong to right is a good indicator of cheating. A few questions:
And of course, I agree with the commenters who wondered how accurately the scanners can pick up erased answers.
HJohn • February 16, 2010 2:15 PM
@JimFive: “determining actual cheating is a bit more complex.”
Agreed.
Another wild card is, as you alluded to, mistakes on the test by the teacher. and I don’t just mean when a teacher gets the answer wrong.
Sometimes, seemingly difficult questions have the answer telegraphed because the correct answer is the only one with grammer, gender, or numbers, that match the question. Just one example.
Tom • February 16, 2010 2:34 PM
Here is a possible workaround for cheaters.
Assuming that correct answers and passing rates are well above 50%, you repeatedly find differences between two tests and switch the answers. This should tend to reduce the good scores and increase the bad scores while not favoring suspicious erasures. Now don’t do too many, or else you’ll have too many erasures 🙂
Roger • February 16, 2010 2:39 PM
@Mace Moneta:
I think you’ve had better than average results from these business courses. I’ve also experienced them with a lot of perverse incentives interacting to make the whole thing a waste of time.
It goes like this: the employer is required, for reasons of tax relief or union rules, to spend a certain number of hours on training for each employee each year.
* Most of the time the employer doesn’t actually care what people learn, so long as they are happy with the results, and the hourly rate is cheap.
* Most of the employees don’t care either, so long as they have fun and don’t get bad feedback from the instructor to their employer
* The training agency wants to please the employer (by being cheap), the tax auditors (by bearing some semblance to real training), and the students (so they don’t rate down the course to their employer.)
The nett result of all this is a soft-skills course of “learning by doing” without testing, where a few highly motivated students might pick up a few pointers, but most of the class might as well have spent the day at the beach. (Except that if they actually spent the day at the beach, that would be tax fraud.)
They give you a little certificate to hang on your cube; nobody bothers. We call them “certificates of attendance” since the only way to “fail” the course is to go AWOL. Recently a co-worker “tested the envelope” by attending the first and last days of a five day course and going shopping for the rest. She told the instructor where she would be, gave a long explanation as to why she absolutely had to go shopping on those days, and left a cell number “in case I am needed”. She still got the certificate.
Joe • February 16, 2010 2:55 PM
Joe. your problem is that you are trying to make educational policy by common sense. The way we make educational policy in America is in perfect evidence by the last two major news stories.
(1) Look, I’m pretty and I have numbers.
(2) Look, I’m ugly and I have a gun.
That’s how we make education policy.
peri • February 16, 2010 3:06 PM
@Joe A:
“I haven’t seen much discussion in the comments on the basic premise of this report: that the percentage of answers that are changed from wrong to right is a good indicator of cheating. A few questions.”
Because that wasn’t the premise. The premise was to flag each class whose mean was higher than 3 standard deviations of the state distribution, which should only happen with a probability of around 0.15% (~1/667).
“Were all students given identical instructions in all classrooms?”
The CRCT is a statewide test so I would guess yes; you can find out yourself here:
http://public.doe.k12.ga.us/ci_testing.aspx?PageReq=CI_TESTING_CRCT
Clive Robinson • February 16, 2010 3:34 PM
@ Mace Moneta,
“It always bothered me that school is so different from ‘training’.”
There are various reasons for this.
First off is that schools are “seen to be” teaching the basis for subjects. What they should be teaching is “how to learn”, but this is difficult at the best of times.
However what has happened is that “technology” has been introduced from above to reduce costs.
For instance it always amazes me that Politicos rob the education budget to give to the Prison budget….
Thus out of necessity schools have become “sausage machines” pumping out a homoganised and mainly usless product.
In training you have (supposadly) got past this initial educational road block (more by chance than anything else) thus the course can concentrate on a depth not bredth information provision.
Secondly is motivation, children can be difficult to motivate. Learning by “1&2 is 2, 2&2 is 4…” whilst essential has to be positivly re-inforced by reward, not negativly reinforced by punishment.
Large class sizes (ie above 15-20 children) don’t alow for the inter action required for positive reinforcment the teacher is actually spending a significant period of time preventing disruption getting out of hand and this can egender a negative re-inforcment reaction in children.
In the workplace training is often seen as desirable by those being educated as it at the very least puts a check mark in their HR record that removes a potential promotion bar. the courses are generaly short and of interest to the participants. Thus the courses can concentrate on gettting across content and concepts to a willing audience.
With regards,
“In those courses, the emphasis was on problem solving and creativity, rather than testing. For example, writing software or configuring routers.”
This is the real essence of why modern education fails those being educated.
In the UK if you where brought up and had your education in the 1960’s and 70’s the teaching proffession in the UK was probably at it’s best. The political constraint was minimal, the proffession officialy recognised that “education by punishment” was bad and the class sizes where moderate. This alowed curiosity to be developed by directed play and each child was assured of a few minutes one-to-one time with the teacher. The teacher quickly became aware of the student’s strengths and weaknesses and thus could (if they where so motivated) tailor to a certain extent the teaching style to each student.
But importantly there was a lot of teaching that could not be tested. That is “how to learn”.
We have a saying/saw in the UK which is “You can take a horse to water but you can not make it drink”. This is especialy true of education each student is different, treating them like a hurd of automata does not work they have to “want to learn” and our basic biology sugests we do this through play.
As you say,
“It’s not only more effective – learning by doing – but more enjoyable as well.”
Children and adults respond well to positive education.
However there is one major “hold back” which is “social skills” children with low social skills (ASD etc) find engaging in play difficult. There appears to be a corelation to left handed people and social communications issues.
Recent studies have shown that people with low social skills rarely if ever become as successful as their “tested education” would indicate.
Arguably large class sizes and “sausage machine” education is making society fail big time.
But the Politicos stick their fingers in their ears and go “nah nah nah nah” till you either go away or they get you taken away…
Of course there are other problems with learning appart from people learn in different ways.
There is a problem of certain subjects being “ladder subjects” others are “garden subjects” (or whatever termanology is in use in your area).
Maths for instance is a ladder subject. If you don’t get a firm foothold on each rung in turn you don’t get to climb the ladder to success you slip and fall and not just cannot progress you actually regress.
Subjects to do with human social aspects (History for instance) tend to be “garden subjects” you can quite usually mis out large blocks of the subject and still progress.
That is you get taught events that have the same basic story over and over again just with different players. what you should be learning is the story, what you get tested on is the recorded details and dates. It is why we have the expression “those that don’t learn from history are condemed to re-live it”, “history moves in circles”, “The past teaches us about the future”,….
The real subject in history is the “human social behaviour” not the actual fairly dull events.
Thus you indirectly get taught the same “social thing” over and over again just with different players.
That is you are learning the “ways of man” by repeated example (repatition ;), the difference being the parameters change slightly. Thus you could grasp the majority of the social lesson from just one or two good examples and sleep through the rest and cram up on the details shortly before the test and still get very good marks.
In essence it’s the same way you can learn to grow flowers of one type for one flower bed, a different flower for a different bed and the same with vegtables etc…
As a very broad statment you could say that teaching the ladder subjects (maths / science / engineering) properly is more important than the garden subjects.
Because the ladder subjects are not “repetative” the garden subjects are. So catching up on garden subjects is always possible, not so ladder subjects.
dob • February 16, 2010 3:43 PM
Matt from CT:
Schools aren’t funded just through local property taxes because that privileges the schools, and the kids, in the rich area of town at the expense of the poorer areas of town. Equal access to educational opportunity is not only a matter of justice, it’s long been understood to be necessary for a strong middle class.
Xavier • February 16, 2010 5:53 PM
This is why I tell all students to fill in all possible answers and then erase the incorrect ones.
This way no trends can be found. Kids and teachers are free to cheat.
db Cooper • February 16, 2010 6:47 PM
@Mace Moneta:
“It always bothered me that school is so different from ‘training’.”
I consider it a good thing for schools to educate, not train. Critical thinking and all that as elaborated by other posts here.
When I was 14 my mom had to sign a note, giving me permission to attend sex education classes. She likely would not have signed if it were instead sex training class.
Matt from CT • February 16, 2010 6:58 PM
@dob
Schools aren’t funded just through
local property taxes because that
privileges the schools, and the kids,
in the rich area of town at the
expense of the poorer areas of town.
Huh?
You’re saying we need state and federal funding to equalize funding within the same town?
Everything I ever understood about the history of equalization of education in regards to state constitutional guarantees of equal access to education (there is no such federal requirement) and the reliance on the property tax was in regards to disparities between towns.
I think you’re confusing several different issues in the history of U.S. school funding.
Schools don’t exist for some “justice” under a theory of positive rights; they exist as part of Government’s role in creating the conditions for a vibrant market economy.
My major concern with our current schools is that we spend an outsized amount of resources on “special education” to the detriment of the majority in the middle.
The best students will take care of themselves; in spending several times as much per pupil on the poorest students as we do on the average students are we really achieving our biggest economic bang for the buck in the future?
Tests are needed to measure performance of small groups over short periods; we can measure the overall effectiveness nationwide over decades. Is our current emphasis on special ed spending going to pay dividends there? Or would the money be better spent to boost the average students up higher?
Lyle • February 16, 2010 7:03 PM
@Anon: gifted students don’t change their answers from wrong to right, they answer the questions correctly the first time.
Stephanie • February 16, 2010 8:17 PM
In terms of teaching to the test, I think its important to differentiate between schools where people have higher SES so the home environment has more literacy rich resources, etc.
At the higher SES (socioeconomic status), kids have better test taking skills. If you don’t know how to take a test you can’t jump through the hoops to get into college, to get good jobs, etc. Who didn’t know that the well to do have advantages?
Test taking skills are strategies. Teaching to the test, as others point out, isn’t bad if the test reflects the curriculum. Everyone wants to know what it takes to succeed, in a job,or in a classroom. What are your expectations so I can get a promotion? An A?
The pressure put on teachers in low and middle SES schools is as high as that on upper SES schools. You can’t take an inner city program and magically turn it into Grosse Pointe test scores. Yet NCLB demands this.
Cheating is reprehensible, and a bad role model. Teachers have a lot of pressure but its no excuse to cheat.
This problem is much bigger that test scores. Inner city kids have to deal with getting mugged on the way to school, not enough decent food, poor housing, eviction, etc. Their schools aren’t doing great either. In a school I taught in the lead teacher said “face it we are ghetto, if you are cold, wear a sweater they aren’t going to fix the heat this winter.” The kids laughed. Teachers do their best overall, these tests do not reflect the big picture and its ridiculous to blame the schools.
Stephanie • February 16, 2010 8:19 PM
Great post Bruce.
In terms of teaching to the test, I think its important to differentiate between schools where people have higher SES so the home environment has more literacy rich resources, etc.
At the higher SES (socioeconomic status), kids have better test taking skills. If you don’t know how to take a test you can’t jump through the hoops to get into college, to get good jobs, etc. Who didn’t know that the well to do have advantages?
Test taking skills are strategies. Teaching to the test, as others point out, isn’t bad if the test reflects the curriculum. Everyone wants to know what it takes to succeed, in a job,or in a classroom. What are your expectations so I can get a promotion? An A?
The pressure put on teachers in low and middle SES schools is as high as that on upper SES schools. You can’t take an inner city program and magically turn it into Grosse Pointe test scores. Yet NCLB demands this.
Cheating is reprehensible, and a bad role model. Teachers have a lot of pressure but its no excuse to cheat
.
This problem is much bigger than test scores. Inner city kids have to deal with getting mugged on the way to school, not enough decent food, poor housing, eviction, etc. Their schools aren’t doing great either. In a school I taught in the lead teacher said “face it we are ghetto, if you are cold, wear a sweater they aren’t going to fix the heat this winter.” The kids laughed. Teachers do their best overall, these tests do not reflect the big picture and its ridiculous to blame the schools.
jgreco • February 16, 2010 10:25 PM
@Lyle
“@Anon: gifted students don’t change their answers from wrong to right, they answer the questions correctly the first time.”
Apparently not. If they answered the questions correctly the first time then their teacher would have had no reason to go back and modify the answers.
That is a pretty strange comment to make without any citations anyways. I know I certainly do not always mark the correct answer first on all of my tests. In fact, I would wager that gifted children tend to have more unusual testing behaviours than regular students. I would not be surprised if things like “going back to recheck answers” were more common in that type of group.
Clive Robinson • February 17, 2010 2:30 AM
@ Lyle,
“… gifted students don’t change their answers from wrong to right, they answer the questions correctly the first time.”
No I don’t think so.
I used to get very high scores in some subjects and in those tests I made quite a few corrections.
The reason is I went through the test quickly and went back to re-read / check at a slower pace. And often found mistakes sometimes with a question.
Such as,
Q1 – 10 grams of a salt are disolved in 1 liter of water and 10ml of the solution is drawn up by a student into a pipet. How much of the salt is in the pipet?
A:1g, B:0.1g, C:10mg, D:1mg.
Q2 – From the solution in Q1, another student draws up a fifth as much Again, how much of the salt is in the pipet?
A:0.12g, B:80mg, C:40mg, D:20mg.
I answered A the first time through for Q2 and D the second time, on realising the full stop was missing between the two sentances.
I complained to the teacher who helped me write a letter to the exam board. It was only in later life, I found out that this sort of typo was rife in exam papers, that teachers had a suspicion that it was a deliberate ploy by the exam board, in that any letter of complaint got a student upgraded…
Which makes me wonder if you could Excel by complaint.
Oh and did you ever have one of those anoying teachers who would pick up on pipet / pipette as a personal preffrence?
Niels • February 17, 2010 5:23 AM
“An excellent teacher in my kids’ district was fired for this last year. It turns out that there were an unusually large number of answers changed from incorrect to correct responses on the standardized test. The superintendant said the proof of cheating by the teacher was that the variance was two standard deviations from the mean.”
The superintendant doesn’t understand the concept of proof. Statistics can show probability, but do not show proof.
From that point of view, I doubt whether the termination of the contract of the teacher was justifiable from a legal point of view (regardless whether or not the teacher was indeed ‘cheating’).
brainfart • February 17, 2010 6:40 AM
Can someone please explain this topic to us non-Americans? I don’t understand what this is about. Back when I was in school the students cheated, not teachers,. Why would they do that? Makes no sense.
Winter • February 17, 2010 8:20 AM
@brainfart:
“Can someone please explain this topic to us non-Americans?”
As a fellow non-USAian, I can try to help.
Most countries have some kind of standardized test (mostly multiple choice tests) which all children must perform at some time. Say, to move to enter the next level in the schooling system or to pass an exam.
Teachers and schools are generally evaluated based on different criteria that include, among others, the way their students perform on such tests.
As I understand it, in (some parts of) the USA, the performance on these standardized test are the only criteria on which teachers and schools are evaluated.
If a teacher or school lacks adequate resources to teach and cannot unload underperforming students, both the teachers and school will suffer badly.
Manipulating the test results is then the only recourse to stay in business.
It has often been written that in the USA, all human judgement is distrusted. Therefore, rules are designed to preclude human judgement with really dire consequences (read “THE DEATH OF COMMON SENSE How Law Is Suffocating America” By Philip K. Howard Random House, 1994).
Winter
HJohn • February 17, 2010 8:37 AM
@Winter: “It has often been written that in the USA, all human judgement is distrusted. Therefore, rules are designed to preclude human judgement with really dire consequences (read “THE DEATH OF COMMON SENSE How Law Is Suffocating America” By Philip K. Howard Random House, 1994). ”
I can’t argue with that. I see less and less common sense as I age.
Someone was speaking to a group and played the game “finish the sentence.” He said “If you can’t afford a house, then you ______.” Almost everyone finished the sentence with “you shouln’t buy one.”
He went on the say that when a few people do it anyway, it makes for personal tragedies. But when a million do so, it is a national crisis. I add that it could have been averted altogether if common sense was more common.
Though the story is about the financial sector rather than the educational sector, the premise is the same. Poor incentives begets poor actions that breeds poor results.
The death of common sense is definitely suffocating many people on many levels, including academia, technology, family, and economy, to name but just four of the planes.
Good post.
Jeff H • February 17, 2010 8:52 AM
Way back when I was still in university, I took an intro psych class. This was a very large class (~200 students) and all tests were multiple choice. Our prof told us that after completing a test we should review our answers, and if we felt we’d made a mistake, to DEFINITELY change it. Studies had shown that you are more likely to change a wrong answer to a correct one than vice-versa. I did a quick google on “changing answers on multiple-choice tests” and this was my top hit:
http://www.biomedcentral.com/1472-6920/7/28
This is a study performed in 2007 that seems to back up what I was told. The counter-point to this would be that the test administration documents instruct students to work in a test booklet and only transfer answers to the answer sheet when they are correct. But if that was being done, there should be few or no erasures on the answer sheets.
David • February 17, 2010 9:30 AM
@Winter: Correct: there is a prejudice here against leaving things up to human judgment, which has come about at least partly because of abuses of that judgment. This causes a strong push to come up with objective measures, even in cases where that’s difficult or unfair.
@HJohn: How about, “If the person applying for a mortgage clearly can’t pay it, you should ________________.”? After all, people who can’t afford a mortgage are often those who don’t have good financial judgment, and a society that relies on everybody to do the right thing won’t last long.
The correct answer, in many cases, was “find pretexts to approve the loan, and try to convince the applicant to sign”, because that was what was financially rewarded. Other sentence-completion questions include “If you’re a senior banker, and a mortgage broker offers to sell you a batch of loans that mostly won’t be paid, _______.”
Society isn’t seriously harmed by a small minority who are incompetent to run their own finances. It is by people who bet large banking firms on making and acquiring bad loans.
Seth J • February 17, 2010 9:34 AM
One possible way to track teacher performance that wouldn’t be subject to a lot of the flaws people bring up here:
Track the performance of the students in a teachers class as they take follow-up courses in later terms. I’m not a statistician, but I’d assume one could differentiate trends for: a given student over time, members of a current class, and members of a past class. This should let one identify which teachers’ students have trouble in classes that are supposed to build on material they taught.
One obvious limitation is that you have less data to analyse for later classes than former ones, but I think it would still be worth doing.
HJohn • February 17, 2010 9:39 AM
@David at February 17, 2010 9:30 AM
No arrgument there. It could just as easily be said that “if someone can’t afford to repay a mortgage, the business should not loan them the money.” It could also go further and be said “if omeone can’t afford to repay the mortage and the financial insitution does not want to loan them the money, the government has no business pressuring them to do so.” (see Sept 29, 1999 NY Times). However, I could only beat that drum so much without going off topic.
The point was about common sense, for which our points are not mutually exclusive. In fact, they are both monumentally true.
The same holds true for education. Poor measurements and policies yield poor results.
Best,
HJohn
HJohn • February 17, 2010 9:46 AM
@David at February 17, 2010 9:30 AM
@HJohn at February 17, 2010 9:39 AM
For discussions sake, I’ll reel the analogy into the academic relm:
“If a student does not earn a passing grade, we should not provide incentive for a teacher to change their grade for financial or career benefit.”
vix • February 17, 2010 11:50 AM
@Jeff H at February 17, 2010 8:52 AM
“Studies had shown that you are more likely to change a wrong answer to a correct one than vice-versa.”
This is interesting as I had long been taught (back to elementary school) that you should trust your first instinct and that you’d inevitably be changing a correct guess to a wrong one.
On another note, this all makes me feel ancient as I come from the old school of “If I don’t know the answer, I don’t cheat to get the right one, because I don’t deserve the good grade if I don’t know the material.” Sad that so many in higher echelons were not taught similarly. (Also, as I grew older, I realized even if I were tempted to cheat on any test, chances are my guesses were better than my neighbors’ anyway.)
kog999 • February 17, 2010 2:29 PM
“If the person applying for a mortgage clearly can’t pay it, you should ________________.”?
make the loan then pass the liability on the fannie may, or the FHA, or put it in a pool of mortgages and sell it to investors?
moo • February 17, 2010 5:09 PM
@Jeff: Interesting post.
“The counter-point to this would be that the test administration documents instruct students to work in a test booklet and only transfer answers to the answer sheet when they are correct.”
The tests are timed, and some students have difficulty completing them within the alloted time; or at least, many of them feel some time pressure as they do the test. So it would be prudent of them to ignore such a stupid instruction, because the extra step of copying their answers from the test booklet to the answer sheet will take up some valuable time that they could have spent reading and answering questions.
Because the students are incented to ignore that instruction (because of the time pressure), I think the existence of instructions like that does not make an erased-and-changed answer much more likely to be evidence of cheating.
I don’t doubt the idea that some teachers (or more likely, some stuffy school administration bureaucrats) change test answers to improve their school’s results. These kind of statistical analyses can’t prove wrongdoing though, they can only point out suspicious results which deserve to have more investigative effort focused on them. To fire teachers (or even administrators) without some more concrete evidence of wrongdoing, seems like a bad idea.
moo • February 17, 2010 5:11 PM
“If the person applying for a mortgage clearly can’t pay it, you should ________________.”
Haul them out back and shoot them.
It used to work until about a hundred years ago. Funny how much individual responsibility has declined over the last few decades…
School Employee • February 17, 2010 7:51 PM
I work at a flagged inner-city school (NOT APS). I have no doubt that tampering occurred. Our flags occurred ONLY on the math portion of every test. Two SpEd students who cannot read miraculously passed the CRCT. It is just as clear from our school board’s “investigation” that the administrator’s “explanation” that minority children erase more than other children will be accepted.
Blaufish • February 18, 2010 2:24 AM
I employ the mark-quick-finish-early and then spend 50-70% of the time on the few I was uncertain of. take time to figure out the right answer. Therefor you would see a close to 100% wrong-to-right transition on my tests dating many years back.
Given that all students trained in this methology will incorrectly recognized as cheaters, I have concern that the findings may be less accurate than percieved.
GreenSquirrel • February 18, 2010 3:57 AM
@vix
Same here. On multiple choice exams I strongly feel that my first, instinctive, answer is often going to be my best effort.
Sometimes checking over the question (at the end for example) shows I misread something first time round, but on the whole I find the longer I spend deliberating something I dont know, the less likely I am to get it right.
Looking back on the various tests (certification, promotion, further eduction etc) I have taken over the years and I am confident that the the majority of my initial answers were right.
I find I make changes when I really have no idea of the answer and doubt creeps in. I have probably made as many wrong – wrong changes as wrong – right or right – wrong.
GreenSquirrel • February 18, 2010 4:02 AM
COMMON SENSE – does it really exist? (did it ever?)
Surely it is nothing more than the sum total of your own life experience, education and cultural signals?
Most “common sense” things are only apprent when the options are explained.
For example, the sequence of actions you carry out to remove a belt of ammunition from a GPMG seems like “common sense” to me, but few untrained people seem to know it. Likewise, if you want to observe through a window, “common sense” says you stay back inside the room’s shadow and look out, but lots of people dont realise this.
It gets worse when we have a situation were most people do something different than the claimed “common sense” route. Surely by definition that means the “common” sense answer is not the best one…
Robert in San Diego • February 18, 2010 8:01 AM
Let me get this straight — the kids might be innocent, but the TEACHER might be doing the cheating? Is this what happens when we use the most easily tested of student metrics (multiple guess) for teacher compensation?
It could be worse — I had a problem at one former job when the managers asked me to double check an inventory item from location X because their monthly inventory gave a negative number. Yup, I was told, positively, by the person there in charge of counting stuff, the negative value was, in fact, correct. Uhm, isn’t the reason why we do physical counts on a routine basis to make sure our processes for ordering and allocating resources are followed, and to make sure we can verify the rationale behind those orders and allocations?
Yeah, we had a history of high-end parts getting allocated without following procedures. Almost all the time, it was for perfectly legitimate business goals, though, and it was just a paperwork thing. ALMOST all the time.
Bring back fountain pens! Go ahead, try to erase that stuff! Yeah, felt pens and some of the rollerball and gel pens are also tough to erase.
John • February 18, 2010 8:15 AM
We have crappy schools and crappy tests that churn out a crappy uneducated workforce for a very simple reason: Mexicans.
Or rather, our aversion to Mexicans.
Most directly, our country, our people, our politicians, and the sheep all don’t want Mexicans working every single position in every McDonalds, Wendy’s, Burger King, every super market register and stock boy and bagger, stock boys in K-Mart, Wal-Mart, freight unloading in the freight yard, every menial position in every factory, every janitorial job…
Think about it.
All those jobs.
All those jobs you figure an uneducated illegal immigrant could do cheaper. If only the cops wouldn’t bust businesses for hiring illegal immigrants.
If we didn’t have a large, stupid, uneducated, vulgar workforce; then we’d have a huge saturated market of educated professionals to do all the $100k/yr jobs for McDonalds salaries, and cheap Mexican immigrants making minimum wage plus half a quarter working at McDonalds.
An educated society is a very, very poor society.
Ross • February 19, 2010 11:54 AM
1) I was quite good at standardized tests, but would occasionally miss skip a line while filling out the bubbles. I’d have to go back and erase maybe 20 in a row when I finally caught on, shifting everything up one line — also changing a whole mess of answers from wrong to right. That would probably trigger a flag now, for innocent behavior.
2) @HJohn: “If you can’t afford a house, then you ______.” Almost everyone finished the sentence with “you shouln’t buy one.”
It’s easy when you beg the question like that. If you already KNOW you can’t afford a house, sure, it’s a simple answer. The can of worms is not in that logic — the issue is with understanding whether or not you can afford a house. And as every house is different, it’s just just a general pass/fail on affording A house, but “can I afford THIS house at THIS price?” which is a much more challenging question. Particularly when the actual cost of a house isn’t nearly as clear up front as you’d expect.
This situation is definitely made worse when the experts who are supposed to be assisting you in making decisions have incentives to convince you that you can indeed afford something that’s borderline or perhaps really out of your range.
HJohn • February 19, 2010 12:26 PM
@Ross at February 19, 2010 11:54 AM
The analogy is certainly over simplified. The point was not to come up with something all encompassing. The point was about the logic.
It is of course true that people buy things they, and the creditor, think they can afford. Or things they actually can afford today but cannot in the future due to a change in circumstance.
All that aside, however, the financial sector, at the prodding of the government I might add, loaned to individuals with a very high probability of not being able to pay it back.
That’s all I was saying.
The example of what happened financially was intended to illustrate what could happen academically if we create situations where teachers, for their own well being, would have an incentive to overrate students and promote them through the educational ranks unprepared. Granted, it would not have the same impact, but poor policy tends to lead to poor results.
Have a nice weekend.
Laurie • February 23, 2010 11:06 AM
“If you look at the numbers in the report closely, you’ll see that according to the scanner almost all changes were wrong to right; there were very few wrong to wrong answers recorded. That alone strikes me as wildly improbable.”
Can someone tell me: does the report (or any other publicly available data) show how many wrong-to-right versus right-to-wrong (other even just total) erasures there were in each classroom? I can’t find the data that seems to be referred to in the post I just quoted. Thanks.
Subscribe to comments on this entry
Sidebar photo of Bruce Schneier by Joe MacInnis.
yt • February 16, 2010 6:45 AM
This was also explored in Freakonimics ( http://www.amazon.com/Freakonomics-Economist-Explores-Hidden-Everything/dp/0060731338/ref=sr_1_1?ie=UTF8&s=books&qid=1266323953&sr=8-1 ). By following the test scores of the same group of students over time, it was possible to spot which teachers had been cheating. Whenever a group of students showed dramatic improvement one year, only to show an equally dramatic decline the next year, it was a pretty good indication that teachers were cheating.
One of the reasons this happens at all in the first place is that there is so much emphasis on standardized testing in US schools, and so much pressure on teachers to improve their students’ scores. Even in classrooms with honest teachers, the teachers spend a lot of time “teaching to the test”, which can leave relatively little time for students to actually learn anything.