Schneier on Security

Clive Robinson • November 11, 2020 4:18 PM

@ David Leppik,

you’re confusing proctoring with grading. AI doesn’t work for either of them.

Proctoring originally ment to supervise the collection of tithes for the Church or Monarchy. Over time in England the term has migrated via the legal system to one who ensures discipline is enforced. It only gets used as a title/job in the functioning of state, church and the Universities that used to be strongly related to Monarch or Church (ie not the “red bricks”).

Over in the US however it appears the scope had narrowed from enforcing all discipline in a University, to just that surrounding examinations. Where it has expanded scope to include nearly all things related to the administration of exams including the marking of them. Initially just multichoice questions that were “marked” by what are effectively tabulating machines but now all computer related issues to do with examinations. That is these companies are selling a “nose to tail” process over the entire exam process.

In England there has in times past been a distinction between “marking” and “grading”.

Where marking is what you do to homework and exams as a process to produce results. It’s the results amongst other things that go towards “grading a students” abilities.

With that nose to tail service ethos the proctoring companies are getting involved in not just ensuring cheating is minimized to some criteria but maintaining students grades, as well as marking exams electronically where they can be. It’s this expantionist “can do” mentality backed up by the notion of “not leaving money on the table” that is driving forward the “extras’.

Clive Robinson • November 12, 2020 11:16 PM

@ metaschima,

Just make it open book and I suppose put a reasonable time limit on it.

Would be my favoured path if exams are to be used, as it would produce a more “life like” valuation of the student (going down the submitted project / thesis route without examinations would be better yet and the next logical step).

However, it is far from immune to “colusion” or “assistance” issues. Worse it definitely needs a high level of comprehension from the system[1] to meet most university “honour codes”[2].

What the pure closed book exam proctoring / invigilation aspect is trying to do is actually directly related to the impossiblity of stopping end to end encryption as both can only stand a chance of working in a fully controled environment. Something that we know has not worked where security is in a “cost no object” National Security environment.

As I noted before such closed book exams realy do not test the qualities of a student other than their ability to remember a whole load of statments usually of little worth (like the order of monarchs and the dates they came to the throne etc which is fairly pointless). It’s only in a very small minority of “pure” science subjects where the student can be tested in closed book exams for anything other than memorized information, and often that only tests the ability to use maths or rule sets as tools. Which generally is not that usefull in the modern working environment.

But even with projects there are colussion and assistance issues… A few years back the UK school qualifications in computer science got thrown into a tail spin. Someone put up the exam project questions on the internet and it quickly spread around to various websites where people give others help. In effect the world answered the question in so many different ways that checking a students work had not in effect been downloaded and tweaked a little was not realistically possible… So the major part of the qualification got suspended.

Now we have COVID-19 giving rise to similar problems, and those who believe themselves in charge are “stumped”. So like all politicians who have no clue as to what to do, they fall back on pointless “reorganisation”[3] and “magic pixie dust thinking”.

The current pixie dust being “AI” sold preserved in the snake oil of these proctoring / invigilating systems that even when they massively invade a student’s privacy fail to control the environment the student is in sufficiently to stop collusion etc. But do throw up sufficient false positives to ruin many students efforts.

Thus when the situation is considered by a “rational actor”, in the students position it actually makes more sense to be dishonest than honest… By the same consideration in the invigilator / proctors position it makes rather more sense not to use such systems, and infact fundamentally reconsider the examination / testing methods of students…

But that is not the message politicians want to hear because it goes against their “cherished notions” thus cognative bias…

[1] Traditionaly proctoring falls in to two parts, on the day “issolation” techniques and subsequent “content appraisal” which is a value judgement based on experience. Modern coursework apprasing systems that look for “plagiarism” are now being applied to exams to replace the “content appraisal”. Such systems are more notable for their failings than their successes which is hardly surprising.

[2] The Internet and legal profession have turned honour codes into fairly intricate and lengthy documents. Some however still remain readable, but length is another matter. This page is just the subsection for open book exams,

https://www.ox.ac.uk/students/academic/exams/open-book/honour-code

Back a few years ago such seperate signed statments did not exist, nor needed to be.

[3] One the subject of reorganisation there are pithy quotes that are claimed to have come from Roman statesmen of two millennia ago. However,

https://quoteinvestigator.com/2013/11/12/reorganizing/

But for people that can reason probably the most appt military quote about incompetent managment is the one that says “If it moves salute it, If it does not move paint it”.