Refine LLM Agents without the necessity of fine-tuning LLMs, thanks to Memento!
In the ever-evolving world of artificial intelligence (AI), a new framework named Memento is making waves. This memory-based AI system, a significant step towards Artificial General Intelligence (AGI), is not just designed to act intelligently, but to learn and adapt in a way that feels more human-like.
Memento operates on a two-stage framework: Case-Based Planning and Tool-Based Execution. In the initial stage, an Language Learning Model (LLM) acts as the Planner. It breaks down user queries into sub-tasks and retrieves past experiences from the Case Memory to inform the current plan.
During the Tool-Based Execution stage, the Planner hands off sub-tasks to the Executor, an LLM enhanced with external tools like web search, code interpreters, and file processors. This allows Memento to effectively tackle complex tasks that require the use of various tools and autonomous planning.
One of the key components of Memento is the Case Bank. This repository contains past experiences, including both successful and failed attempts. The Planner uses the Case Bank to avoid previous mistakes and apply proven strategies, a practice that has been confirmed as critical through ablation studies. These studies revealed that the Case Bank can boost accuracy on out-of-distribution tasks by as much as 9.6%.
Memento's effectiveness has been demonstrated on the DeepResearcher dataset, where it achieved an impressive 66.6% F1 score and 80.4% PM. This outperforms state-of-the-art training-based systems, securing Memento the #1 spot on the GAIA leaderboard, a benchmark for complex, long-horizon tasks requiring tool use and autonomous planning.
Powered by models like GPT-4.1 and o4-mini, Memento represents a shift in how AI agents are built. It offers a scalable and efficient pathway toward building generalist LLM agents, paving the way for a new era in AI development.
It's important to note that the development of the memory-based AI framework Memento remains shrouded in mystery, with the search results not explicitly identifying the person or institution behind its creation.
In conclusion, Memento, with its innovative two-stage framework and emphasis on learning from past experiences, is set to redefine the landscape of AI. Its impressive performance on various benchmarks underscores its potential to revolutionize the field and bring us closer to achieving AGI.
