Top Research Papers on RAG

This talk describes the journey in building a knowledgeable AI assistant by harnessing LLM techniques, and describes a federated Retrieval-Augmented Generation system that integrates external information from both the web and knowledge graphs for trustworthy text generation on real-time topics like stocks and sports, as well as on torso-to-tail entities like local restaurants.

Compared to model fine-tuning, Class-RAG demonstrates flexibility and transparency in decision-making, outperforms on classification and is more robust against adversarial attack, as evidenced by empirical studies.

This survey provides a comprehensive exploration of Agentic RAG, beginning with its foundational principles and the evolution of RAG paradigms, and presents a detailed taxonomy of Agentic RAG architectures, highlights key applications in industries such as healthcare, finance, and education, and examines practical implementation strategies.

This talk describes the journey in building a knowledgeable AI assistant by harnessing LLM techniques, and constructed a federated Retrieval-Augmented Generation (RAG) system that integrates external information from both the web and knowledge graphs in text generation.

This study emphasizes that the RAG architecture’s ability to retrieve information by dynamically using the learnings obtained from large datasets of LLMs strengthens applications in the field of NLP.

A novel knowledge graph-based RAG framework with a refined retrieval pipeline, robust chunking mechanism, and source traceability for enhanced diabetes-focused LLM is proposed, demonstrating effective performance in diabetes-focused LLM.

Speculative RAG is introduced - a framework that leverages a larger generalist LM to efficiently verify multiple RAG drafts produced in parallel by a smaller, distilled specialist LM, which accelerates RAG by delegating drafting to the smaller specialist LM, with the larger generalist LM performing a single verification pass over the drafts.

GFM-RAG is a novel graph foundation model (GFM) for retrieval augmented generation powered by an innovative graph neural network that reasons over graph structure to capture complex query-knowledge relationships, making it the first graph foundation model applicable to unseen datasets for retrieval without any fine-tuning required.

This paper proposes learning to Refine Query for Retrieval Augmented Generation (RQ-RAG), endeavoring to enhance the model by equipping it with capabilities for explicit rewriting, decomposition, and disambiguation, and demonstrates enhanced performance in handling complex, multi-hop QA datasets.

The study demonstrates the effectiveness of the RAG system in generating relevant suggestions with a consistent accuracy of 93% and highlights the value of identifying and understanding knowledge gaps to guide future endeavours.

Graphical Eigen Memories For Retrieval Augmented Generation (GEM-RAG), motivated by human memory encoding and retrieval, is introduced, which aims to improve over standard RAG methods by generating and encoding higher-level information and tagging the chunks by their utility to answer questions.

Multi-Agent Filtering Retrieval-Augmented Generation (MAIN-RAG) is proposed, a training-free RAG framework that leverages multiple LLM agents to collaboratively filter and score retrieved documents and introduces an adaptive filtering mechanism that dynamically adjusts the relevance filtering threshold based on score distributions.

This paper evaluates various advanced solutions proposed in recent literature, comparing their efficacy and discussing the trade-offs involved, and delves into the central architecture of RAG systems, encompassing retrieval components, generative components, and knowledge bases.

A novel framework to evaluate the reliability of multi-modal RAG using two performance measures: the relevancy score (RS), assessing the relevance of retrieved entries to the query, and the correctness score (CS), evaluating the accuracy of the generated response.

RAGGED, a framework for analyzing RAG configurations across various DBQA tasks, discovers distinct LM behaviors in response to varying context quantities, context qualities, and retrievers and provides a deeper analysis of these differences.

A critical analysis of failure points in existing KG-based RAG methods is presented, identifying eight key areas of concern, including misinterpretation of question context, incorrect relation mapping, and ineffective ambiguity resolution, which aim to improve the reliability and effectiveness of KG-RAG systems.

Plan*RAG is introduced, a novel framework that enables structured multi-hop reasoning in retrieval-augmented generation (RAG) through test-time reasoning plan generation by isolating the reasoning plan as a directed acyclic graph outside the LM's working memory.

Experimental results indicate that Pistis-RAG improves alignment with human preferences relative to the baseline RAG system, showing a 6.06% increase in MMLU (English) and a 7.08% increase in C-EVAL (Chinese) accuracy metrics, highlighting Pistis-RAG's effectiveness in overcoming the limitations associated with traditional RAG approaches.

The RAG system utilizes a 2-step vector search using the vector search with cosine similarity metric on an HNSW index on the paper’s abstracts and the papers itself to pass only relevant information to LLM; this enables enhanced data retrieval and contextually aware text generation.

It is shown that retrieval-augmented fine-tuning of NMT mod-012 els under the RAGMT framework results in an average improvement of 2.03 BLEU scores over simple fine-tuning approaches on English 015 to German domain-specific translation.

RAG Foundry is introduced, an open-source framework for augmenting large language models for RAG use cases and demonstrates the framework effectiveness by augmenting and fine-tuning Llama-3 and Phi-3 models with diverse RAG configurations, showcasing consistent improvements across three knowledge-intensive datasets.

This paper proposes the ‘Blended RAG’ method of leveraging semantic search techniques, such as Dense Vector indexes and Sparse Encoder indexes, blended with hybrid query strategies, which achieves better retrieval results and sets new benchmarks for IR datasets like NQ and TREC-COVID datasets.

This project presents a novel system designed to enhance code generation by leveraging Retrieval-Augmented Generation, Grounding techniques, and Prompt Parameters, which represents a significant advancement in automating code generation from natural language descriptions.

It is found that RAG-Fusion was able to provide accurate and comprehensive answers due to the generated queries contextualizing the original query from various perspectives, however, some answers strayed off topic when the generated queries' relevance to the original query is insufficient.

Auto-RAG is introduced, an autonomous iterative retrieval model centered on the LLM's powerful decision-making capabilities that can autonomously adjust the number of iterations based on the difficulty of the questions and the utility of the retrieved knowledge, without requiring any human intervention.

A novel dataset, MultiHop-RAG, which consists of a knowledge base, a large collection of multi-hop queries, their ground-truth answers, and the associated supporting evidence, and it is hoped MultiHop-RAG will be a valuable resource for the community in developing effective RAG systems, thereby facilitating greater adoption of LLMs in practice.

The paper addresses the enhancement of AI chatbots through the integration of Retrieval-Augmented Generation with the Large Language Model (LLM) and focuses on the development of a restaurant chatbot that not only engages in natural-language conversations but also addresses context optimization and LLM optimization for restaurant context learning.

This research explores the application of Self-Reflective Retrieval-Au’gmented Generation within analytical systems, specifically focusing on data distribution systems, and discusses the potential benefits and limitations of implementing Self-RAG in such systems.

C-RAG is proposed, the first framework to certify generation risks for RAG models and it is proved that RAG achieves a lower conformal generation risk than that of a single LLM when the quality of the retrieval model and transformer is non-trivial.

This paper presents the first comprehensive study of RAG systems that incorporate fairness-aware rankings, focusing on both ranking fairness and attribution fairness - ensuring equitable exposure of sources cited in the final text.

Retrieval-Augmented Generation (RAG) is a promising solution that can enhance the capabilities of large language model (LLM) applications in critical domains, including legal technology, by retrieving knowledge from external databases. Implementing RAG pipelines requires careful attention to the techniques and methods implemented in the different stages of the RAG process. However, robust RAG can enhance LLM generation with faithfulness and few hallucinations in responses. In this paper, we discuss the application of RAG in the legal domain. First, we present an overview of the main RAG method...

Common RAG benchmark evaluation techniques have not been useful for evaluating responses to novel user questions, so a flexible, human in the lead"approach is required.

This paper discusses how the Einstein Trust Layer facilitates the safe practical application of RAG in enterprise systems, including its general architecture, functionality, and the precise processes that demonstrate why the Einstein Trust Layer is a reliable means of incorporating LLMs into commercial processes.

: This survey explores the various vector-indexed and graph-based approaches employed in Retrieval-Augmented Generation (RAG) systems, a paradigm that enhances large language models (LLMs) by incorporating external knowledge through retrieval mechanisms. RAG systems enable LLMs to generate more accurate, contextually aware, and informative responses by integrating external documents or structured data during the generation process. We delve into two main techniques: vector-indexed retrieval, which leverages high-dimensional vector embeddings and methods like Hierarchical Navigable Small World ...

. Large Language Models have captured the imagination of the public and the technical community. As powerful as they are they have problems that prohibit their use for highly skilled users. These issues are hallucinations, bias, black-box reasoning, and lack of domain depth. One of the most popular architectures to alleviate these problems is Retrieval Augmented Generation (RAG). In a RAG architecture the LLM is utilized to generate vectors and to parse and generate natural language. The knowledge base for a RAG architecture is typically a set of documents focused on a particular type of verti...

This paper introduces OHRBench, the first benchmark for understanding the cascading impact of OCR on RAG systems, and conducts a comprehensive evaluation of current OCR solutions and reveals that none is competent for constructing high-quality knowledge bases for RAG systems.

This work proposes a novel approach called Progressive Retrieval Augmented Generation (P-RAG), which not only effectively leverages the powerful language processing capabilities of LLMs but also progressively accumulates task-specific knowledge without ground-truth.

We argue that the alignment issue between LLMs and external knowledge ranking methods is tied to the model-centric paradigm dominant in RAG systems. We propose a content-centric approach, emphasizing seamless integration between LLMs and external information sources to optimize content transformation for specific tasks. InGreek mythology, Pistis symbolized good faith, trust, and reliability. Drawing inspiration from these principles, Pistis-RAG is a scalable multi-stage framework designed to address the challenges of large-scale retrieval-augmented generation (RAG) systems. This framework cons...

OG-RAG applies to domains where fact-based reasoning is essential, particularly in tasks that require workflows or decision-making steps to follow predefined rules and procedures, as well as knowledge-driven tasks such as news journalism, investigative research, consulting and more.

Recent research highlights the challenges retrieval models face in retrieving useful contexts and the limitations of generation models in effectively utilizing those contexts in retrieval-augmented generation (RAG) settings. To address these challenges, we introduce RAG-RL, the first reasoning language model (RLM) specifically trained for RAG. RAG-RL demonstrates that stronger answer generation models can identify relevant contexts within larger sets of retrieved information -- thereby alleviating the burden on retrievers -- while also being able to utilize those contexts more effectively. Mor...

The FutureDial-RAG challenge at SLT 2024 is launched, which aims at promoting the study of RAG for dialog systems, and baseline results show that it is very challenging to perform well on the two tasks, which encourages the participating teams and the community to study how to make better use of RAG for real-life dialog systems.

LegalBench-RAG is introduced, the first benchmark specifically designed to evaluate the retrieval step of RAG pipelines within the legal space, and serves as a critical tool for companies and researchers focused on enhancing the accuracy and performance of RAG systems in the legal domain.

Document processing and query generation tasks in customs declaration scenarios face key challenges such as the complexity of multimodal data, adaptability to dynamic regulations, and ambiguity in query semantics. This study proposes a Retrieval-Augmented Generation system (ICCA-RAG) that addresses the core issues of processing complex customs documents and dynamically generating queries through multimodal document parsing, sparse-dense hybrid storage, and context-driven large language model generation. In terms of multimodal document parsing, the system supports comprehensive parsing of PDFs,...

It is demonstrated that LaB-RAG achieves better results across natural language and radiology language metrics compared with other retrieval-based RRG methods, while attaining competitive results compared to other fine-tuned vision-language RRG models.

The TREC 2024 RAG Track is proposed, which introduces a web-based user interface for an interactive arena allowing benchmarking pairwise RAG systems by crowdsourcing and opens-source the Ragnar\"ok framework and baselines to achieve a unified standard for future RAG systems.

RagVerus is introduced, a framework that synergizes retrieval-augmented generation with context-aware prompting to automate proof synthesis for multi-module repositories, achieving a 27% relative improvement on the novel RepoVBench benchmark.

Experiments show that UniMS-RAG achieves state-of-the-art performance on the knowledge source selection and response generation task with itself as a retriever in a unified manner.

This paper endeavors to establish a straightforward yet potent benchmark for semantic search in Arabic within the framework of retrieval augmented generation (RAG) and examines the effectiveness of metrics and the dataset.

A fairness evaluation framework tailored to RAG methods is proposed, using scenario-based questions and analyzing disparities across demographic attributes, indicating that fairness issues persist in both the retrieval and generation stages, highlighting the need for more targeted fairness interventions within RAG pipelines.

This work argues that the alignment issue between LLMs and external knowledge ranking methods is tied to the model-centric paradigm dominant in RAG systems, and proposes a content-centric approach, emphasizing seamless integration between LLMs and external information sources to optimize content transformation for specific tasks.