Optimizing Retrieval-Augmented Generation through Agentic RAG Ecosystem Based on Fine-Tuned BERT Cross Encoder and GPT-4 Model
An Agentic Retrieval-Augmented Generation (RAG) system that enhances chatbot-based academic advising by integrating a BERT-based agent to filter and validate retrieved information, ensuring contextually relevant and factually accurate responses.
Abstract
Education plays a fundamental role in personal and professional growth, yet many students struggle with selecting the right major due to insufficient guidance, leading to dissatisfaction in their academic and career paths. To address this, we propose an Agentic Retrieval-Augmented Generation (RAG) system that enhances chatbot-based academic advising by integrating a BERT-based agent to filter and validate retrieved information, ensuring contextually relevant and factually accurate responses. Additionally, GPT-4 is employed as the Natural Language Generation (NLG) component to produce fluent, structured answers. Experimental results show that incorporating the agent significantly enhances response accuracy and relevance, where from 11 majors the METEOR Score resulted at $74.33 \%$, Jaccard similarity at $58.77 \%$, and Cosine similarity at $94.13 \%$, improving by 6.54%, 5.57%, and 3.57%, respectively. The BERT Relevancy score remains consistently high at $96.91 \%$. Deployment using Django is also implemented to allow real-use scenarios. Although promising, it is suggested that the next research involved a larger dataset consisting of tens of thousands of rows if possible to reduce bias and enable a more fine-tuned agent.