Morley College London, a premier institution specialising in arts, culture, and applied sciences, recognised the need to streamline its policy compliance process across its diverse staff and student body spanning three campuses. Addressing this need, the college innovatively introduced an AI Policy Assistant, a cutting-edge tool to simplify and enhance policy understanding and compliance.
The AI Policy Assistant, developed as a web-based chatbot, employs a private Large Language Model (LLM) meticulously trained on the college's comprehensive collection of 64 policy documents. This advanced AI tool is designed to provide staff with instant and contextual answers to their policy-related inquiries, effectively reducing the time and effort previously required to navigate through extensive policy literature.
The chatbot's development focused on creating a user-friendly interface accessible via any standard web browser. This approach ensures ease of use for all staff members, regardless of their technical expertise. When a staff member queries the AI Policy Assistant, it presents the answers in a conversational manner, enhancing the user experience. Moreover, the chatbot's ability to translate responses into various languages is a testament to the college's commitment to its multicultural environment.
One of the AI Policy Assistant's notable features is its ability to provide direct links to the source policy documents corresponding to each response. This ensures staff can validate the information and explore the policies in greater depth if needed. Furthermore, the AI's design to handle follow-up questions enables a more refined and specific information delivery, tailored to the user's needs. The chatbot's capability to draft follow-up communications like emails and letters marks a significant stride in administrative efficiency, saving considerable time for staff.
The implementation of the AI Policy Assistant began with a pilot phase involving college managers in the autumn term of 2023. This phase aimed to evaluate various aspects of the chatbot's performance, including its functionality, accuracy, and user-friendliness. A crucial component of this evaluation was to measure the chatbot's impact on staff performance and satisfaction, ensuring that it effectively contributes to a more efficient and informed workplace.
The pilot also emphasised testing for potential issues like AI biases and hallucinations. Given that the chatbot operates solely on approved college policies, reducing or eliminating these issues is paramount to maintain the integrity and reliability of the provided information. In cases where the AI cannot find an answer within the documents, it is programmed to acknowledge this, ensuring transparent and honest communication with the user.
As the evaluation of the pilot concludes at the end of the autumn term 2023, the focus will be on gathering both quantitative and qualitative data. This includes surveys, interviews, focus groups, and analytics to assess various dimensions of the AI Policy Assistant's effectiveness, efficiency, and overall impact on the staff's daily operations and policy compliance.
The scenario in this case study is genuine and based upon real events and data, however its narration has been crafted by AI to uphold a standardised and clear format for readers.
Key Learning
Time savings: staff can get quick and accurate answers totheir policy queries without having to search through multiple documents orcontact other departments.
Accessibility and awareness:staff can access the AI Policy Assistant anytime, and can also learn more about the policies and their implications through the chatbot’s responses.
Compliance: staff can follow the policies more consistently and confidently, as the AI Policy Assistant references its answers back to the relevant policy documents.
Reduced workload: staff can focus on their core tasks and responsibilities, rather than spending time on policy-related issues.
More human and joined-up responses: staff can provide more empathetic and holistic solutions to students and colleagues, as the AI Policy Assistant can combine multiple policy documents to provide a more useful answer.
Risks
Over-reliance on AI: Dependence on AI for policy interpretation could potentially miss the nuanced understanding that human judgment offers.
Data Security and Privacy: Ensuring robust data security measures to protect sensitive policy information.
Continuous Updating: The necessity of regular updates to the AI system to reflect current and accurate policy documents.
Bias and Accuracy: Monitoring and mitigating AI biases and ensuring the accuracy of the information provided.
Change Management: The challenge of integrating AI tools into existing workflows and ensuring staff adaptation.
