AI introduces competing priorities for assessment. AI literacy is part of academic literacy skills but we also assess language profiency: AI may need to be restricted to allow authentic asessment of language skills.

At Edinburgh, they wanted to re-design foundation EAP assessment for Arts, Humanities and Social sciences. They decided it could only be done by looking at their assessment as a whole not only individual parts.

Two lanes approach:

Lane 1 – AI Exclusion – Designed to make GenAI impractical or irrelevant. E.g. live discussions, closed book exams, practice-based tasks.

Lane 2 – AI Collaboration – Lower-order tasks offloaded to AI, higher order and critical thinking promoted. Students interpret, select, adapta and critically engage with materials.

What they did:

Semester 1:

• Take-home essay still there: (AI collaboration – Lane 2) Student unpack, braintosrm, develop and iterate and essay throughout the course. Can’t use AI to translate from L1 (40%)

• Seminar discussion: (AI exclusion – Lane 1) Students discuss self-selected questions on a key text (40%)

• Exam, summary of source text: (AI exclusion – Lane 1) Students summarise a text, unaware of which text it will be until the exam, they do it under exam conditions. (20%).

They wanted to start with doing more Lane 1 in Semester 1 and then do more Lane 2 in Semester 2.

Semester 2:

• Student led project: (AI collaboration – Lane 2) Students brainstorm, investigate and research a topic related to their destination subject. (40%)

• Project presentation: (AI collaboration – Lane 2) Students present an aspect of their project. This is largely lane 2 but the actual delivery is Lane 1. (40%)

• Exam: Project Reflection: (Preparation for AI exclusion Lane 2; Exam Lane 1) Students do a written reflection about their learning experience from the project. (20%)

Positive impacts: academic misconduct referrals way down; students demonstrate awareness of academic integrity in relation to AI and also its limitations.

Factors to consider: AI is always changing, so asessments will have to be agile too. Misuse of AI, when AI use is permitted (e.g. Lane 2), it is hard to identify/judge. It affects teachers’ willingness to report misconduct: uncertainty = less willing to report it.

We did a lot of group discussion in this session. Things we discussed:

• what our institutions have done in response to AI so far
• our experience of student use of AI so far
• what issues there are with student use of AI so far
• ideas for how to improve things going forward

One of the members of my group, a history tutor, was saying that AI literacy is certainly very possible to integrate, as are assessments like those in Lane 2, and students are very ready to use AI in that way, but then if they are using AI in that way, farming out the lower order skills, how do they then develop the skills needed to succeed in the Lane 1 AI exclusion assessments where they have to do it all themselves without AI?

Another, a Journalism tutor, talked about a student who used AI and it was noticeable how their voice was lost from the writing, so this tutor had a talk with the student about the importance of not losing their voice, and is hoping the student won’t use AI in the same way next time. So we talked about the importance of having these conversations with students.

In the wider discussion, I think, there was mention of different levels of AI that students can access, and that in the future there will be more payment barriers than currently. And therefore, if we are building AI use into assessment, then are we designing assessments that students can’t all access equally and biasing assessment and achievement towards those with more money.

My thoughts

My notes for this session were somewhat scant because it was very discussion-based and I struggled to combine note-taking and participating in the discussion. (Which is interesting – it is something we often expect from students, that there will be a note-taker in a discussion who will subsequently summarise the discussion for the whole class! I suppose if there were multiple questions to discuss, students could take it in turns to be the note-taker to spread the load. I ended up writing down what I could remember subsequently – made more difficult as there were other sessions to attend after this one, and then travelling back to Sheffield at the end of the day, so I couldn’t get round to getting down what I could remember til the following day. I could remember most of what the two people either side of me said, but not the person sitting furthest from me in the group.)

The approach to AI (with its Lane 1 AI Exclusion and Lane 2 AI collaboration) is similar yet slightly different to the Sheffield University approach. Here, we have AI-free (corresponds to Lane 1) and AI-required (2 x summative assessments required annually), then for any assessments that don’t fall into either of these categories (and all assessments should be clearly labelled for which category they belong to), fair and appropriate use of AI should be possible (as long as it doesn’t constitute “false [automated] authorship”) in a variety of ways. We should include a statement on AI and academic misconduct on all assessment briefs. Alongside this, we need to teach students what good use of AI is in the context of a given assessment. However, this should not just be a list of what is and isn’t approved, as that is not something we can enforce in terms of the things on the “what isn’t” list. So, it was interesting to see the similarities and differences in the two approaches. Of course, in my foundation context, there isn’t an official approach as such, and we are exempt from following University policy to the letter because of the nature and length of the courses, but we have been working to integrate some AI literacy into our course and there have been changes to the coursework essay assessment criteria to try and make it so that poor/unethical use no longer attracts high scores. Work on the criteria is ongoing so we will see what direction they take next.

Was also interesting to see that the assessment overview included an assessed reflection task in Semester 2 – issues around that were explored in detail in “Reflective Practice or Reflective Performance?: Rethinking reflection in foundation.” I wonder if this will ever be introduced in my context. I’m inclined to hope not as there is already so much assessment and so little time, I can’t imagine having to fit in teaching yet something else meaningfully! I’m not sure how the preparatory phase works at Edinburgh compared to in the session linked to above or what is assessed and how. What do they reward, what do they penalise? I wonder if grammar/vocabulary are given a criterion in the assessment criteria for it. Do they then teach specific language for reflection that they expect to see? Do they teach a particular reflective framework that students are expected to use, like the Gibbs model in the session linked to above? Does it become a hoop-jumping exercise for students rather than a genuine reflection opportunity? So many questions (which I hadn’t thought of yet during the session so wasn’t able to ask!). 20% of Semester 2 marks is not insignificant, more than just a throwaway bolt-on. Our assessments are not percentages of a semester score but instead a skill. So for writing, it is 50% coursework essay and 50% exam. Listening is 100% exam. And all of the skills are worth 25% of the overall progression score. The 50-50 split for writing used to be 60-40 with the coursework essay at 60%, which change is a result of technological developments making at-home writing potentially less reflective of student ability. Will be interesting to see how things continue to develop, both in my context and beyond!