Top Research Papers on Retrieval Augmented Generation

This talk describes the journey in building a knowledgeable AI assistant by harnessing LLM techniques, and describes a federated Retrieval-Augmented Generation system that integrates external information from both the web and knowledge graphs for trustworthy text generation on real-time topics like stocks and sports, as well as on torso-to-tail entities like local restaurants.

This work proposes a Classification approach employing Retrieval-Augmented Generation (Class-RAG), and suggests that increasing the library size is a viable and low-cost approach to improve content moderation.

This talk describes the journey in building a knowledgeable AI assistant by harnessing LLM techniques, and constructed a federated Retrieval-Augmented Generation (RAG) system that integrates external information from both the web and knowledge graphs in text generation.

Speculative RAG is introduced - a framework that leverages a larger generalist LM to efficiently verify multiple RAG drafts produced in parallel by a smaller, distilled specialist LM, which accelerates RAG by delegating drafting to the smaller specialist LM, with the larger generalist LM performing a single verification pass over the drafts.

The study demonstrates the effectiveness of the RAG system in generating relevant suggestions with a consistent accuracy of 93% and highlights the value of identifying and understanding knowledge gaps to guide future endeavours.

Graphical Eigen Memories For Retrieval Augmented Generation (GEM-RAG), motivated by human memory encoding and retrieval, is introduced, which aims to improve over standard RAG methods by generating and encoding higher-level information and tagging the chunks by their utility to answer questions.

This paper proposes learning to Refine Query for Retrieval Augmented Generation (RQ-RAG), endeavoring to enhance the model by equipping it with capabilities for explicit rewriting, decomposition, and disambiguation, and demonstrates enhanced performance in handling complex, multi-hop QA datasets.

The Mindful-RAG approach is proposed, a framework designed for intent-based and contextually aligned knowledge retrieval that explicitly targets the identified failures and offers improvements in the correctness and relevance of responses provided by LLMs, representing a significant step forward from existing methods.

Experimental results indicate that Pistis-RAG improves alignment with human preferences relative to the baseline RAG system, showing a 6.06% increase in MMLU (English) and a 7.08% increase in C-EVAL (Chinese) accuracy metrics, highlighting Pistis-RAG's effectiveness in overcoming the limitations associated with traditional RAG approaches.

The RAG system utilizes a 2-step vector search using the vector search with cosine similarity metric on an HNSW index on the paper’s abstracts and the papers itself to pass only relevant information to LLM; this enables enhanced data retrieval and contextually aware text generation.

RAG Foundry is introduced, an open-source framework for augmenting large language models for RAG use cases and demonstrates the framework effectiveness by augmenting and fine-tuning Llama-3 and Phi-3 models with diverse RAG configurations, showcasing consistent improvements across three knowledge-intensive datasets.

RAGGED, a framework for analyzing RAG configurations across various DBQA tasks, discovers distinct LM behaviors in response to varying context quantities, context qualities, and retrievers and provides a deeper analysis of these differences.

Plan-guided Retrieval Augmented Generation offers a new perspective on integrating external knowledge in LMs while ensuring attribution by design, contributing towards more reliable LM-based systems.

This paper proposes the ‘Blended RAG’ method of leveraging semantic search techniques, such as Dense Vector indexes and Sparse Encoder indexes, blended with hybrid query strategies, which achieves better retrieval results and sets new benchmarks for IR datasets like NQ and TREC-COVID datasets.

It is found that RAG-Fusion was able to provide accurate and comprehensive answers due to the generated queries contextualizing the original query from various perspectives, however, some answers strayed off topic when the generated queries' relevance to the original query is insufficient.

A novel dataset, MultiHop-RAG, which consists of a knowledge base, a large collection of multi-hop queries, their ground-truth answers, and the associated supporting evidence, and it is hoped MultiHop-RAG will be a valuable resource for the community in developing effective RAG systems, thereby facilitating greater adoption of LLMs in practice.

The paper addresses the enhancement of AI chatbots through the integration of Retrieval-Augmented Generation with the Large Language Model (LLM) and focuses on the development of a restaurant chatbot that not only engages in natural-language conversations but also addresses context optimization and LLM optimization for restaurant context learning.

C-RAG is proposed, the first framework to certify generation risks for RAG models and it is proved that RAG achieves a lower conformal generation risk than that of a single LLM when the quality of the retrieval model and transformer is non-trivial.

The first systematic evaluation of RAG systems integrated with fair rankings is presented, focusing specifically on measuring the fair exposure of each relevant item across the rankings utilized by RAG systems (i.e., item-side fairness), aiming to promote equitable growth for relevant item providers.

Common RAG benchmark evaluation techniques have not been useful for evaluating responses to novel user questions, so a flexible, human in the lead"approach is required.

This paper discusses how the Einstein Trust Layer facilitates the safe practical application of RAG in enterprise systems, including its general architecture, functionality, and the precise processes that demonstrate why the Einstein Trust Layer is a reliable means of incorporating LLMs into commercial processes.

: This survey explores the various vector-indexed and graph-based approaches employed in Retrieval-Augmented Generation (RAG) systems, a paradigm that enhances large language models (LLMs) by incorporating external knowledge through retrieval mechanisms. RAG systems enable LLMs to generate more accurate, contextually aware, and informative responses by integrating external documents or structured data during the generation process. We delve into two main techniques: vector-indexed retrieval, which leverages high-dimensional vector embeddings and methods like Hierarchical Navigable Small World ...

The FutureDial-RAG challenge at SLT 2024 is launched, which aims at promoting the study of RAG for dialog systems, and baseline results show that it is very challenging to perform well on the two tasks, which encourages the participating teams and the community to study how to make better use of RAG for real-life dialog systems.

This work proposes a novel approach called Progressive Retrieval Augmented Generation (P-RAG), which not only effectively leverages the powerful language processing capabilities of LLMs but also progressively accumulates task-specific knowledge without ground-truth.

We argue that the alignment issue between LLMs and external knowledge ranking methods is tied to the model-centric paradigm dominant in RAG systems. We propose a content-centric approach, emphasizing seamless integration between LLMs and external information sources to optimize content transformation for specific tasks. InGreek mythology, Pistis symbolized good faith, trust, and reliability. Drawing inspiration from these principles, Pistis-RAG is a scalable multi-stage framework designed to address the challenges of large-scale retrieval-augmented generation (RAG) systems. This framework cons...

LegalBench-RAG is introduced, the first benchmark specifically designed to evaluate the retrieval step of RAG pipelines within the legal space, and serves as a critical tool for companies and researchers focused on enhancing the accuracy and performance of RAG systems in the legal domain.

The TREC 2024 RAG Track is proposed, which introduces a web-based user interface for an interactive arena allowing benchmarking pairwise RAG systems by crowdsourcing and opens-source the Ragnar\"ok framework and baselines to achieve a unified standard for future RAG systems.

Experiments show that UniMS-RAG achieves state-of-the-art performance on the knowledge source selection and response generation task with itself as a retriever in a unified manner.

This paper endeavors to establish a straightforward yet potent benchmark for semantic search in Arabic within the framework of retrieval augmented generation (RAG) and examines the effectiveness of metrics and the dataset.

A fairness evaluation framework tailored to RAG methods is proposed, using scenario-based questions and analyzing disparities across demographic attributes, indicating that fairness issues persist in both the retrieval and generation stages, highlighting the need for more targeted fairness interventions within RAG pipelines.

This work argues that the alignment issue between LLMs and external knowledge ranking methods is tied to the model-centric paradigm dominant in RAG systems, and proposes a content-centric approach, emphasizing seamless integration between LLMs and external information sources to optimize content transformation for specific tasks.

The study explores the basic architecture of RAG, focusing on how retrieval and generation are integrated to handle knowledge-intensive tasks, and examines ongoing challenges such as scalability, bias, and ethical concerns in deployment.

The experimental results on multi-hop QA datasets show that DR-RAG can significantly improve the accuracy of the answers and achieve new progress in QA systems.

ARM-RAG (Auxiliary Rationale Memory for Retrieval Augmented Generation), a system that learns from its successes without incurring high training costs is proposed, and it is demonstrated that the storage and subsequent retrieval of reasoning chains have a positive influence on performance in grade-school math problems.

This work proposes SAGE, a novel two-stage synthetic data generation paradigm that employs an attribute-based extraction and generation approach to preserve key contextual information from the original data and enhances the privacy properties of the synthetic data through an agent-based iterative refinement process.

This work introduces RAGAs (Retrieval Augmented Generation Assessment), a framework for reference-free evaluation of Retrieval Augmented Generation pipelines, and posit that such a framework can contribute crucially to faster evaluation cycles of RAG architectures, which is especially important given the fast adoption of LLMs.

This work conducts extensive empirical studies with novel attack methods, which demonstrate the vulnerability of RAG systems on leaking the private retrieval database and reveals that RAG can mitigate the leakage of the LLMs' training data.

It is demonstrated that RAG systems that operate on databases with untrusted content are vulnerable to a new class of denial-of-service attacks the authors call jamming, and a new method based on black-box optimization is described and measured that does not rely on instruction injection.

This paper presents good practices to implement, optimize, and evaluate RAG for the Brazilian Portuguese language, focusing on the establishment of a simple pipeline for inference and experiments, and presents the complete architecture of the RAG.

This work proposes a new approach called WeKnow-RAG, which integrates Web search and Knowledge Graphs into a "Retrieval-Augmented Generation (RAG)" system, which effectively balances the efficiency and accuracy of information retrieval, thus improving the overall retrieval process.

This paper proposes RAG-end2end, an extension to RAG that can adapt to a domain- specific knowledge base by updating all components of the external knowledge base during training and introduces an auxiliary training signal to inject more domain-specific knowledge.

This work introduces CBR-RAG, where CBR cycle's initial retrieval stage, its indexing vocabulary, and similarity knowledge containers are used to enhance LLM queries with contextually relevant cases, and presents an evaluation of CBR-RAG.

An experience report on the development of Retrieval Augmented Generation systems using PDF documents as the primary data source and aims to offer insights to researchers and practitioners developing similar systems using two distinct approaches: OpenAI's Assistant API with GPT Series and Llama's open-source models.

This work proposes a lightweight unlearning framework based on Retrieval-Augmented Generation (RAG) technology that is particularly effective for closed-source LLMs, where existing unlearning methods often fail.

A Differentiable Data Rewards (DDR) method, which end-to-end trains RAG systems by aligning data preferences between different RAG modules, which makes generation module more effective in extracting key information from documents and mitigating conflicts between parametric memory and external knowledge.

Retrieval-augmented generation (RAG) has emerged as a significant advancement in the field of large language models (LLMs). By integrating up-to-date information not available during their initial training, RAG greatly enhances the practical utility of LLMs in real-world applications. However, even with RAG, LLMs can still produce inaccurate outputs, such as distorting or misinterpreting source content, posing risks in high-trust scenarios. To address these issues, we introduce a novel approach called Hallucination Aware Tuning (HAT). This method involves training hallucination detection model...

Experimental results across multiple datasets demonstrate that SMART significantly enhances QA performance and surpasses previous unsupervised context selection methods, showing a promising strategy for RAG.

A Hybrid Parameter-Adaptive RAG (HyPA-RAG) system tailored for AI legal and policy, exemplified by NYC Local Law 144, shows enhanced correctness, faithfulness, and contextual precision, addressing the need for adaptable NLP systems in complex, high-stakes AI legal and policy applications.

Current leading Text-To-Audio (TTA) generation models suffer from degraded performance on zero-shot and few-shot settings. It is often challenging to generate high-quality audio for audio events that are unseen or uncommon in the training set. Inspired by the success of Retrieval-Augmented Generation (RAG) in Large Language Model (LLM)-based knowledge-intensive tasks, we extend the TTA process with additional conditioning contexts. We propose Audiobox TTA-RAG, a novel retrieval-augmented TTA approach based on Audiobox, a conditional flow-matching audio generation model. Unlike the vanilla Audi...

A Comprehensive Full-chain Evaluation (CoFE-RAG) framework to facilitate thorough evaluation across the entire RAG pipeline, including chunking, retrieval, reranking, and generation, and a holistic benchmark dataset tailored for diverse data scenarios covering a wide range of document formats and query types is released.