The Journey to A Knowledgeable Assistant with Retrieval-Augmented Generation (RAG)
This talk describes the journey in building a knowledgeable AI assistant by harnessing LLM techniques, and constructed a federated Retrieval-Augmented Generation (RAG) system that integrates external information from both the web and knowledge graphs in text generation.
Abstract
Large Language Models (LLMs) have demonstrated strong capabilities in comprehending and generating human language, as well as emerging abilities like reasoning and using tools. These advancements have been revolutionizing techniques in every front, including the development of personal assistants. However, their inherent limitations such as lack of factuality and hallucinations make LLMs less suitable for creating knowledgeable and trustworthy assistants. In this talk, we describe our journey in building a knowledgeable AI assistant by harnessing LLM techniques. We start with a comprehensive set of experiments designed to answer the questions of \em how reliable are LLMs on answering factual questions and \em how the performance differs across different types of factual knowledge. Subsequently, we constructed a \em federated Retrieval-Augmented Generation (RAG) system that integrates external information from both the web and knowledge graphs in text generation. This system supports conversation functionality for the Ray-ban Meta smart glasses, providing trustworthy information on real-time topics like stocks and sports, and information on torso-to-tail entities such as local restaurants. Additionally, we are exploring the potential of external knowledge to facilitate multi-modal Q&A. We will share our techniques, our findings, and the path forward in this talk.