A depressing ritual

Suffered through the School’s Examiners Meeting yesterday, in what is becoming a rather depressing bi-annual ritual.

In this meeting, we all get together to look at the marks that are going to be sent to Faculty for ratification by the Faculty’s Board of Examiners. As is frequently the case, the Faculty’s contentious “scaling policy” played more of a role in the final marks for many units than any effort or achievement on the part of the class. For my second-year unit, the required scaling was so extreme that, while I have frequently been  uncomfortable in the past with this requirement, this year I feel for the first time that the practice has crossed the rubicon from “marking on the curve” to outright academic fraud.

The scaling policy was introduced many years ago because the various flavours of Engineering (Civil, Electrical etc.) discovered that giving high marks for units in their sub-discipline was a highly effective marketing tool in enticing students from the other sub-disciplines, and so grade inflation became a problem. To counter this, a policy was introduced whereby all units with enough students (to avoid statistical anomalies) were required to return an overall mean mark within a fairly narrow range. The argument at the time was that the students in each unit were statistically similar and so any deviation from this range meant that the unit was too easy, too hard, too well taught or too poorly taught or the lecturer was simply not calibrated, and it would be unfair for the students to either suffer or benefit from any of those factors. In fact, these all make reasonable sense in an Engineering context where the cohorts genuinely are rather similar, all the students in the units are enrolled in a Bachelor of Engineering degree, and the units are required units for the Engineering degree.

However, the scaling policy has various other rather less desirable consequences when it is rigidly applied “across the board” to every unit offered within the Faculty, particularly to the units offered by Maths. Here, the net effect is not to prevent grade inflation, but to cause grade inflation, and of the worst possible kind. Much lower level Maths is taken by a whole range of students who are not intending to major in Maths, but are required to take it as part of their degree, either Engineering, Physics, Science/Engineering etc. and these students are all mixed in together studying a common unit. Many of these students simply don’t understand why they have to study Maths at all, don’t care about it, don’t see why a proof is important, and have no tolerance for the difficult, time-consuming persistence and  intellectual effort required to work out for themselves how to solve a problem. And who can blame them? At school, everything is a computation, and you just learn how to do the computation and then get evaluated by solving an isomorphic instance of the same computation!

So, come the end of each semester, we are faced with a sizeable minority of students who have done absolutely nothing, but (for some mysterious reason) turn up to the final examination and score a negligible number of marks. Enter the scaling policy: as soon as a student sits the final exam, they are counted in the statistics as having “attempted the unit” and their mark counts towards the averages. Of course they still fail the unit, but in order to accommodate a large number of 15% marks, yet still meet the required averages, everyone’s mark has to be scaled up, sometimes dramatically. This has the obvious consequence that a large number of students who should fail – in the objective sense that they cannot perform the Calculus and Linear Algebra tasks described in the syllabus – end up passing the unit. Of course, the mean is such a crude measure that it is possible to return bizarre distributions (e.g. every mark is either 79 or 39) that meet the requirement, without passing too many people who shouldn’t, but most people find this equally unpalatable.

What happens to the students who can’t actually do Calculus or Linear Algebra, but are nevertheless given a pass? Well, the sensible ones with a bit of self-knowledge realise that a “gifty fifty” in a first-year unit is not a solid foundation on which to build a technical or scientific career and act accordingly by changing the focus of their degrees and studying things for which they have genuine interest and aptitude. However, all-too-many simply carry on to the next year  of Engineering or Science/Engineering where –  their study habits having been vindicated by their “success” in 1st year – it is business as usual.  Of course, the poor old 2nd-year lecturer is faced with a class containing a substantial bloc of students who have very limited interest, have not actually mastered the prerequisite material,  and who have no concept that a unit is meant to involve a disciplined 15-week period of study, as opposed to a 3-day cramming period. This bloc of students then of course performs extremely badly, but the scaling policy is rigidly applied once more, the marks are forced upwards and another group of students is pushed onwards.

My own second-year unit is at the end of a chain of three pre-requisite units. According to the unit outline, we are teaching abstract vector spaces and multivariate calculus (line integrals, surface integrals, Green’s Theorem etc). But it is impossible. More than 1/3rd of the class cannot write down a spanning set for an explicitly-defined plane in 3-space, which was material (not) covered three units previously, and so expecting that that group will be able to write down a basis for the subspace of n \times n symmetric matrices over the complex numbers, and other more abstractly defined subspaces is just not realistic. But guess what. The scaling policy must now be rigidly applied again. So, gritting my teeth and holding my nose, I have to pass students who cannot perform more than a fraction of the most fundamental tasks in Linear Algebra and Multivariate Calculus. Occasionally someone rebels, refuses to change the marks, and tries to mount a case based on the fecklessness of a significant proportion of their class. The Faculty listens politely, then tells the lecturer to change them anyway or it will be done for him/her.

But perhaps I’m just a grumpy old lecturer dreaming of the good ol’ days when integral calculus was done in primary school and other myths, and I should be more “realistic”. Well, I’m actually quite happy to be realistic. I’ve taught in several universities, and there is no objective “first-year standard”. You just have to teach the students who turn up as much as you can, based on their background. What gets under my skin is the divergence between what the unit claims to cover and what it actually does cover. If the unit description was an accurate reflection of its content and level, then I wouldn’t mind in the least; in fact it would be a huge relief.  Unfortunately, much of what the lower-level units are meant to cover is material that is deemed to be necessary for Engineering students (who make up more than 80% of these units). Of course, when these students reach the Engineering units where they need to actually do the mathematical tasks required to study statics, dynamics etc, the Engineering lecturers are horrified that so many of the students can’t, and immediately, loudly and publicly blame us in Maths for “poor teaching” (to the extent that a working party on “improving Faculty culture” had to be abandoned after two senior academics got close to chest-bumping and fisticuffs!).

It’s all a bit ironic actually – the Engineering Schools introduce a scheme to stop them from cheating on each other, force us to adopt it without the slightest regard for its validity or applicability and then complain bitterly when they reap the entirely predictable consequences.  Just wait till next year when we’ve been asked (by the Engineers) to claim to teach even more stuff in the core units for our “new courses”!

Bah, humbug and Merry Christmas to you!

9 thoughts on “A depressing ritual”

  1. Well said Gordon – a great read! It’s funny how the knowledge of scaling can have an effect on the modern student. I myself admit to saying things during my undergraduate degree such as “Well, I’m on 75% at the moment. I just need to get about the same in the exam and scaling will ensure that I get a HD”. I suppose this is somewhat more ‘honourable’ than the student who says ‘45%’ and ‘P’ instead of ‘75%’ and ‘HD’.

    I would think one solution to increase the mean is just to drastically scale up students who received maybe 65 or more. Though isn’t it also true that in addition to having a range for the mean to fall in, there is also a minimum pass rate i.e. at least 80% of students need to pass?

  2. There’s no minimum pass rate, but there used to be. So there is more potential to “play” with the figures below 45 to get a mean in the right place (e.g., scale 10 up to 20 etc). The methods for scaling vary quite a lot: translations, dilations and piecewise affine transformations! This makes it difficult for the student to predict or surmise what will or what has happened. Most of the time, if all is well and the aptitude of the students is consistent with previous years, we simply write an exam as best as possible to achieve a mean in the right place. But there are many situations where the cohort changes from year to year, particularly when the class size gets smaller. My subject this year didn’t need much scaling at all (a small translation), and the exam seemed to be pitched at the right level. The students who failed were at least 10 marks away from a pass. This was a good situation for a lecturer to be in. But for many, like in the situation Gordon descrives, it can be a disheartening experience, particularly when it feels like you are being pushed in the back by bureaucracy to achieve unethical outcomes. Scaling encourages perverse outcomes, and I’m hoping that New Courses will bring an end to it.

    1. I fear that New Courses will make it worse… as I mentioned, we are now claiming to cover even more material than previously – in fact, now more than any other G08 university – so unless there is some way of encouraging (or enforcing) consistent study, we’ll have an even worse scaling issue at the end of the semester.

      There’s all this talk of having an “external examiner” looking over the exam papers etc. to make sure they are sufficiently high quality – how can an external examiner make any judgement about such a thing, when the marks are subsequently manipulated to such an extreme degree?

  3. Nice little rant gordon.

    I suppose this is the motivation behind your push for the component stuff at Jenny’s meeting? Some method by which marks below a certain threshold are wiped off the table seems the best solution for at least the part of the problem you describe here.

    I will say that at times I think maths units are marked too lightly, but that might be my bias.

  4. The problematic situation you describe is not only a very good argument against scaling, but it is also a good argument against grading itself. A grade of, say 70, means absolutely nothing. Does it certify that a student can successfully solve a certain type of differential equation? Diagonalize a matrix? And we are only talking about purely technical skills … we are far from asking whether a student can usefully apply mathematics to some field of interest.

    Why not make the time needed to complete a unit flexible, and require a high level of mastery before certifying that a student has successfully completed the unit? Knowing that one will not receive credit until one actually understands the unit would change the atmosphere in a school immediately. The engineering professor would be happy to know that students entering his unit were actually well-prepared and have every chance of success. Students would realize that they are not going to get a free pass, so they had better get to work. And wouldn’t mathematics professors feel much more satisfied knowing that they had successfully shepherded a certain number of students to *mastery* of their units, rather than (this is from John Bamberg’s comment) “we simply write an exam as best as possible to achieve a mean in the right place”?

    This would require quite a change in the structure of undergraduate education; how would students pay, what about the workload of professors, how would the whole thing be administered with students working at their own pace, how could we even run units when students would become ready for a unit at many different times of the year, etc., etc., etc. But this is the 21st century, and surely we can do better than the ineffective system we have in place now.

  5. Yes, I quite agree with you.

    When a significant purpose of the unit is to prepare students to apply specific technical skills in a different discipline, then it would be far better to clearly identify the required skills and then to only allow students to proceed to those units once they had acquired those skills.

    Imagine if the School of Surgery (I don’t think there actually IS such a thing; this is just a gedankenexperiment) had a scaling policy like ours and you had one of the students who couldn’t actually locate the appendix, but was scaled up to pass the “abdominal surgery” unit (again, I don’t think there IS such a unit), then doing their first appendectomy on you!

  6. The surgery analogy is apt! The same kind of effective training goes on in other fields, too; pilot training comes to mind. The flight instructor takes you up to 8000 ft, asks you to put the plane into a spiral spin, then you get yourself out of it and return to level flight, then you climb back up to 8000 ft and do it again. If you should ever find yourself in the unfortunate situation for real, then you know what to do and do it automatically, because it’s in your body. It’s like tying your shoes or driving a car.

    There is an aspect of mathematics that is also like this … making certain basic skills automatic is a great aid to a student, both in applying her knowledge in engineering, biology, or whatever, and also in learning higher mathematics. Yes, I know that there are lots of higher-level thinking skills that also need to be developed and assessed (modelling, problem-solving, conceptual understanding, understanding the history of the subject, etc.), but it’s depressing that in the very simplest aspect of mathematics education (basic skills) our system of training and assessment is so ineffective. In the last university at which I taught, I woke up to the level of suffering that this induces in students to a degree that I had not experienced previously. Maybe I just reached the “enough is enough” stage.

  7. I’m surprised Gordon managed to wait this long to deliver this piece – it seemed to be on the verge of giving itself away on at least a few occasions in lectures of the unit in question.

    As a student, I don’t like being scaled. The point of assessment is to judge whether or not I know the material, regardless of how poorly prepared the rest of the cohort may have been.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s