Large language models for careers provision in higher education

While a third of students are open to using artificial intelligence for careers advice, there are concerns about trust, data privacy and equal access to AI tools, according to new research

This report by Arden University and CareerChat (UK), funded by the Jisc careers research grant, explores the possibility of using a careers chatbot powered by the latest advances in large language models (LLMs) to enhance student support within higher education.

Key findings 

• AI and LLMs applied in the higher education careers ecosystem are currently under-developed compared to other sectors and compared with early adopter students, perhaps due to limited awareness of the technology and how to access its potential with appropriate safeguards.
• Current experience with LLMs varies widely. Over 20% of careers professionals and 50% of students have either not heard of or did not understand LLMs.
• Even prior to support from an adviser, around half of careers professionals are content for students to use LLM technologies for labour market information type data, job descriptions, drafting support, and general advice.
• Once students have had a discussion with an adviser, about three quarters of careers professionals are supportive of students using LLMs for such tasks. However, the vast majority continue to oppose it for personalised advice or long-term guidance/mentoring.
• There was cautious support for a specific implementation proposal regarding a university-managed careers advice chatbot. About a third would be supportive of it being directly available to students if done carefully and about a further half would probably support it or would need more information to decide.
• There was also support for further restricting such an LLM tool, particularly suggesting specific data sources to double-check specific facts or requiring the bot to provide sources for its claims, with modest support for requiring users to have had some training in how to use it first - there was little support for restricting the bot from attempting to provide facts.

About the report 

 This project investigated the possibility of building a careers chatbot for students using the latest advancements in generative AI, specifically focusing on large language models. The main objectives were to:

• Identify the specific needs and expectations of students in the design and development of an inclusive and effective careers chatbot.
• Agree the parameters of the effective use of generative AI data observing what can and cannot be integrated successfully within a careers chatbot.
• Develop an alpha version with LLMs that can be stress-tested with a specific cohort of students and careers professionals.

The findings are followed by a series of recommendations for the higher education and careers/ employability sector.

This research was funded by the Jisc careers research grant. If you're a careers professional planning to undertake research, you may be eligible for funding of up to £5,000.