Recently we held a breakfast of leading early careers recruiters and university careers services, to understand the impact of ChatGPT on employers and students.  Here is our overview of that fascinating session.  Thank you to everyone who came along.

Within just two months of launch, ChatGPT had 100m active users – the fastest growth of any application known to man. In many ways, it has been a revelation – helping people to do a whole host of things including writing songs, getting relationship advice, avoiding parking charges and of course, securing that dream job.

And there’s the rub.

Through our own research, more than 70% of students openly admit to using ChatGPT – and chances are, that figure is much higher.  And a significant proportion are already using it to help them secure a job at a graduate employer.  Maybe for background research; maybe for cover letters; and perhaps even to help them get across the line at the assessment stage.

For many assessment providers, the arrival of ChatGPT has opened up a lot of questions about how “ChatGPT-resilient” their processes and systems are.  And it’s not simply what ChatGPT can do today; it’s only going to get smarter as more data is thrown at it – and more features are added.

There is no doubt that ChatGPT has compromised many aspects of recruitment.  It can write you a cover letter, be used to complete application forms and help with answers in pre-recorded video interviews (it’s amazing what it can do in 2 minutes!), and in virtual assessments.

As one employer put it, “online application forms are completely irrelevant now because of ChatGPT”.

Jamie from Neurosight believes that the major problem with many of the current assessment products is that they only assess the output and not the information that has been the input.

Of course, Covid drove the majority of employers to find a virtual solution to recruitment; and now many haven’t changed.  Partly because of cost; partly because of ease.  So it was very much weighted in favour of employers, and we talked about addressing this and finding a better balance between what’s best for employers and what is candidate-centric.

But also remember students.  At unis, they are also contending with ChatGPT as students increasingly use it as part search engine, part research assistant – and sometimes even essay writer.  Universities are working with students to warn them of the danger of using it in the recruitment process (i.e. what happens if they get caught??), and at the same time how they can use it successfully for their studies without compromising their work.

It’s here to stay – and is only going to be more prominent.

So what can employers do?

Put their head in the sand.  Speaking to one employer (away from this breakfast event), they were hoping that students would do the right thing and be ethical in their approach.

Sadly the truth is that ChatGPT is a bit like drugs in cycling.  You know everyone else is using them, so there’s pressure on you to use them as well.  After all, if it gives you a competitive advantage, in what is a very competitive jobs market, who wouldn’t?!

AI Detection is one option – in particular proctoring.  A few assessment providers have come out fighting with AI Detection modules.  We can’t comment on the efficacy of these solutions, although it seems to be an invasive solution – and not necessarily as accurate as you might want.  We know that online exam providers do this routinely but for students in student digs this might seem too much to handle, and certainly adds to stress.

Interestingly, re-testing participants seems to be a way of countering ChatGPT – and seems widely accepted by candidates.  Maybe someone had ‘super-human’ scoring in the online test; there’s no harm in retesting them when they come in for the assessment centre, or to find another occasion to run this in-person test.

A couple of employers at our breakfast certainly do this regularly; one employer re-testing some 25% of successful applicants.  But perhaps many employers don’t have the time or budget to do this!!

Possibly the biggest piece of advice is to stress-test your current recruitment process – not against the free version of ChatGPT, but using the paid-for version of ChatGPT4 (or the latest version available). The paid-for version is far more advanced than the fremium version – and testing the results against the free version may lure you into believing that you’re safe.

We’d also suggest using a good ‘prompt engineer’ to test your process on ChatGPT – as we know that students are becoming better and better at the way they use the tool.

Jamie from Neurosight claims that his tool is resilient to these Generative AI tools because it looks at the way people work, rather than just output.  He also endorses gamification as being resilient to such tools as well.  So Arctic Shores and others may have something to celebrate.

But as many recruiters agreed on the day, perhaps we should be going back to a more hybrid model of assessment.  Many students are likely to welcome coming to an in-person assessment centre, and we should be thinking about the best possible candidate experience – and one which is both effective and inspiring.

Looking forward

Of course, there will be more tech solutions in the future that will be developed to be resilient to some form of AI – but perhaps this is a recalibration in the market and a reminder for recruiters to think about what might be best for candidates as much as what is best for themselves.

Also, perhaps we need to think not of AI as a tool to be afraid of, but one to embrace.  We even talked about rewarding student use of AI and even giving lower socio-economic students the paid-for version (to level the playing field) – as it might well be a tool that they use at work in the future – and moving the focus to behaviours and how they respond to questions rather than simply assessing their answers.

The prevalence of online testing means that students are aware of the game they are playing – and see it not as a work exercise per se, but as a problem to solve; and when we’re set a problem, we find ways to solve them using any tools that we can find.  Who can blame students?

So, employers, take a breath.  Audit your process.  Find ways to strengthen your process.  But never forget about the candidate.  ChatGPT has reminded us that maybe online only is not the only or best option.

A ChatGPT audit

If you want an independent team to stress-test your assessment questions on ChatGPT paid-for version, let us know.  We’ve got a prompt engineer ready and waiting.