This was an internal workshop which was aimed the whole university not only us ELTC folk. Being the end of marking week and my work week (final hour thereof!) it was a good chance to grab the opportunity to do some development (woohoo!)!

The plan was to reflect on the impact of gen AI on assessment, hear about the university’s new common approach to gen AI in the curriculum and then focus in on what fair and appropriate use of gen AI in the context of an assessment might look like and what an “AI-required” assessment might look like.

Here are my notes from the session:

Currently, students can gain a passing grade or even a high pass, using AI in a written assessment. This use is not something that we can detect accurately and fairly, and therefore we can’t ban it as the ban would not be enforceable. However, that does give us the responsibility of ensuring that students use AI well and effectively, and develop appropriate knowledge/skills. We also have to prepare students for workplaces where AI will be used (though as yet what exactly this looks like is unclear!). So, rather than focusing on the negative impact on assessment, we should use these developments as an opportunity to focus and reflect on what, how and why we assess, with a view to improving the process for all concerned.

By Sept 2028, all programmes at the university will include 2 summative AI-Required assessments annually. Well, not *all* – this applies to Year 1 and 2 of undergraduate programmes. “AI free” and “AI-required” assessments are to be explicitly categorised. In AI-free assessments = it should not be possible to use AI, and skills must be demonstrated in an environment where AI assistance is not possible. The choice to do this type of assessment should be grounded in the learning outcomes e.g. communication and interpersonal skill evaluation, development of oral defence and articulation skills, verification of minimum competence, preparation for high-stakes professional contexts, simulation of professional practice conditions. In AI – required assessments, students would have to use AI in some way in order to complete the assessment and develop AI literacy in the context of their subject.

For all assessments that are not labelled either “AI-free” or “AI-required”, fair and appropriate use of AI should be possible (as long as it doesn’t constitute “false [automated] authorship”) in a variety of ways. We should include a statement on AI and academic misconduct (see below) on all assessment briefs. Alongside this, we need to teach students what good use of AI is in the context of a given assessment. However, this should not just be a list of what is and isn’t approved, as that is not something we can enforce in terms of the things on the “what isn’t” list.

There is still quite a lot of grey area – how to define “mostly” or “entirely” for example – where is the line? It does require unpacking for us to understand it fully and communicate to students effectively what it means.

Next, we looked at a generic task and what would be fair/unfair use of AI. Using AI to support elements of the task that the student is doing themselves and getting feedback of what they have done, are examples of appropriate use. A question that arose from a participant: Does getting AI involved in this way minimise student development in terms of the editorial process? Or, does it help them develop the ability by showing them a model of how to do it? It was suggested that perhaps it depends on whether the student lets the AI have the last word. That is, if the student simply adopts the AI comments wholesale without critique, then possibly the overall effect on the development of editorial process is negative but if the student approaches it critically, and learns from it, perhaps it can be helpful?

How could we communicate effective use to students? How could we adapt the task to discourage inappropriate use? Need to be careful of “traffic light” systems because they are not so suitable as we can’t actually enforce the “red” area rules. We need to talk to students about it, come to a shared agreement with them. There is also an issue that students have access to widly different AIs, of varying power. For AI-required assessments here, Gemini should be used – but again, how do we enforce that? Answers on a postcard! AI isn’t limited to LLMs, a participant suggested we need to consider what other platforms/programmes that exist and might support students. The response was that in large part, the focus on LLMs is because they are applicable to everybody, while more specific AI have more specific subject applications. To consider others, you need that more specific knowledge and skill set. (Which is an issue for departments and department-specific training I suppose!!)

Currently, there are some things that students can do better than AI but how long will it be until AI does do everything well in relation to the task? Then we won’t have that option anymore. Probably we are not that far from it. Is it worth putting in for major changes to an assessment when it takes 2 years for that to go through by which time the changes may be obsolete? E.g. hallucinations are getting rarer and it is less likely for it to use invented sources. A more current problem is it won’t go for the best journals or do a great job of assessing what is good or not. But then, you can get round this by specifying in a prompt what you want to be included/excluded. Assessing students in real time, using pen and paper, rather than digitally is an option, as is making the task more personal, having to draw on student experience rather than a generic case.

What about AI-required assessments? AI use might just be a component of an assessment, not a whole assessment that is about use of AI. It doesn’t even have to include student use of AI tools. They could look at pre-created content, talk about why they have decided NOT to use AI, analyse use of AI in a specific context, create a plan for how AI could be used ethically etc Students can ethically object to using AI but should still be able to learn about it.

So what might an AI-required assessment look like? What we need to start from is what skill do we want them to develop by using AI in this assessment? The university are putting together an AI literacy bank, broken down into Awareness, Competence and Ethics, which we can use to help inform task design/adaptation. In our context, students are in the very early stages of their academic journey so we would need to pick out what these students would most benefit from at this stage in their academic journey. [This is useful information as we were wondering if and how this would apply to us. Looking at the framework, it should be possible to identify which element(s) are possible for us to integrate and assess, building on what we have done already in this direction.]

We were asked to suggest modifications to the example task, and responses were more muted – possibly because it was quite complicated and there wasn’t much time!

It was noted that an AI-free assessment should be AI-free for pedagogical reasons, we should have a pedagogical justification. It is a descriptor rather than a label to adopt because we are scared of AI and student use of AI. As we start adapting assessments, need to think about what sort of changes – administrative, minor or major changes – are required. The reality is, the majority of changes relating to AI would hopefully fall under administrative changes: changing how the task is written, changing the information in the assessment brief provided to students = relatively quick to make. Changing an assessment type completely would then be a minor adjustment and require a longer process. A major change is to the programme level learning outcomes, so is unlikely to apply here. So anything around assessments would be classed as at most a minor change. We’ve got two years lead time – January 2028 ready for September 2028 intake would be when to put through minor changes to assessments. [Obviously for us on the AES (Academic English Skills) programme, it is a bit different as Studygroup timelines and processes are involved!]

It was an interesting session. Currently, our coursework assessments are neither AI-free or AI-required, and we have tried to encourage students to use it ethically to support their learning in the context of a given assessment rather than to do it for them. We have adapted our criteria for the writing coursework (extended essay) so that students need to do the skills we teach (critical evaluation, synthesis etc) well in order to score well, and conversely can submit a polished piece of work grammar/vocabulary-wise yet still score poorly. As mentioned above, how long the new criteria will hold effective for is anybody’s guess.

Based on today, and on the trajectory of travel for our coursework speaking assessment (presentation), which is moving towards greater emphasis (in terms of time and marks awarded) on the q&a part than there is currently, I wonder if that is where more personalisation could be incorporated and/or where we could get students to elaborate on how they have used AI, or any decisions they made around their AI use? Anyway, we shall see. In terms of how we teach students how to use AI effectively, going beyond checklists, that is one of our development aims for the coming semester – integrating that teaching more fully into lessons rather than it being more of a bolt-on of do’s and don’ts.

There will be more of these kind of sessions in the nearish future, so I will be interested to attend them!