Reducing Hallucination in Structured Outputs via RAG

Researchers at ServiceNow shared a new paper where they discuss how to deploy an efficient RAG system for structured output tasks.

The RAG system combines a small language model with a very small retriever. It shows that RAG can enable deploying powerful LLM-powered systems in limited-resource settings while mitigating issues like hallucination and increasing the reliability of outputs.

The paper covers the very useful enterprise application of translating natural language requirements to workflows (formatted in JSON). So much productivity can come from this task but there is a lot of optimization that can be further achieved (eg., using speculative decoding or using YAML instead of JSON).

The paper provides some great insights and practical tips on how to effectively develop RAG systems for the real world.

Best Practices and Lessons Learned on Synthetic Data for Language Models

This paper provides an overview of best practices and lessons learned on synthetic data for language models ans was published by Google DeepMind and other collaborators.

It focuses on synthetic data and covers applications, challenges, and future directions. This is an important paper given the significant advancements we are seeing from the use of synthetic data in the field of AI.

We know for sure that the more high-quality data we give these models, the better the performance. Creating synthetic data is not hard but ensuring its quality is really the challenge.

The paper also discusses important topics when working with synthetic data such as ensuring quality, factuality, fidelity, unbiasedness, trustworthiness, privacy, and more.

There are a lot of great references mentioned in the related work section as well.

Reasoning with Intermediate Revision and Search for LLMs

This work by Chi et al. (2024) presents an approach for general reasoning and search on tasks that can be decomposed into components.

The proposed graph-based framework, THOUGHTSCULPT, incorporates iterative self-revision capabilities and allows an LLM to build an interwoven network of thoughts.

Unlike other approaches such as Tree-of-thoughts that shape the reasoning process using a tree, this new approach incorporates Monte Carlo Tree Search (MCTS) to efficiently navigate the search space.

This new method uses an LLM-powered thought evaluator to provide feedback on candidate partial outputs. Then a thought generator component produces potential solutions. The thought evaluator and thought generator are considered the expansion phase which helps with refining the current solution.

Finally, the decision simulator (which acts as part of the MCTS process) simulates consecutive lines of thought to evaluate the potential value of a path.

Due to its ability for continuous thought iteration, THOUGHTSCULPT is particularly suitable for tasks such as open-ended generation, multip-step reasoning, and creative ideation.

We might be seeing more advanced approaches that use similar concepts and search algorithms to elevate the reasoning capabilities of LLMs and the ability to tackle problems that require complex reason and planning. Great paper to keep track of this research trend.