Preprint · Under review

RHM-Agent: Teaching VLMs to Reinforce
Hierarchical Memory

Anonymous Authors

Author information withheld for double-blind review

Abstract

Long-form video understanding remains challenging for vision-language models (VLMs) due to the cost of processing extensive frame sequences and the limited ability of flat or heuristic memory structures to support efficient retrieval and long-horizon reasoning. To address these challenges, we introduce Reinforced Hierarchical Memory Agent (RHM-Agent), which organizes visual evidence into hierarchical memory to selectively retrieve relevant cues and construct explainable, intelligent, and non-linear reasoning paths. Equipped with dynamic memory, the RHM-Agent is optimized via a hierarchical reward mechanism to balance local details with global context, enabling it to navigate long videos in a human-like manner. To evaluate long-horizon temporal reasoning and memory efficiency beyond standard accuracy, we propose HMBench-2K, a long-form egocentric video benchmark spanning four domains: medical, industrial, marine, and autonomous driving. Additionally, we introduce two complementary diagnostic metrics, frame brevity and node precision, to assess evidence efficiency and temporal precision, respectively. Extensive experiments show that our RHM-Agent surpasses baselines with 4× the parameters or 2× the frames by up to 6.63% on LVBench and 5.60% on Ego4D, proving strategic memory optimization yields greater improvements than the brute-force scaling of model size or frame count. Evaluated directly on our proposed HMBench-2K, RHM-Agent outperforms SOTA VLMs and long-video agents by 2.00% and 7.15%, respectively, demonstrating effective long-form egocentric video understanding. Further analysis shows that our method not only improves accuracy in a model-agnostic design but also achieves better temporal alignment than current dynamic memory-augmented agents.

Overview

Five-minute walkthrough

The memory loop end to end: what the agent observes, how it decides to consolidate, and how the reward shapes what survives.

Video poster: RHM-Agent hierarchy of working, episodic, and semantic memory.

Teaser

One policy, three timescales of memory

Working memory holds the live episode; episodic memory stores compressed trajectories; semantic memory distils reusable facts and skills. A single policy decides which tier to touch at every step.

Figure 1. RHM-Agent alternates between acting in the environment and managing its own memory. Memory actions are sampled from the same policy as task actions, and the hierarchical reward credits both the task return and the utility of what was retained.

Method

Memory as an action space

Instead of a fixed context window, the agent owns a structured store and a learned policy over memory operations.

01

Three-tier store

Working memory keeps the last k observations verbatim. Episodic memory holds per-episode summaries keyed by goal and outcome. Semantic memory contains distilled facts, tool affordances, and reusable sub-skills with embedding indices.

02

Memory policy

Alongside task actions, the model emits WRITE, COMPRESS, RETRIEVE, and FORGET calls. Each call is a tool with typed arguments, so memory behaviour is auditable and inspectable.

03

Reinforced consolidation

Group-relative policy optimisation trades task return against retrieval utility and memory cost. The agent is rewarded for memory that later proves useful, not for writing more of it.

Figure 2. Training loop. Rollouts are interleaved with memory operations; the consolidation reward is computed from downstream retrieval success, so credit assignment spans episodes rather than steps.

Results

Stronger over long horizons

Success rate (%) and average context tokens per episode. Higher is better for success; lower is better for tokens.

+12.4 points success (long-horizon split)
2.1× fewer context tokens
3 benchmarks, one policy
Table 1. Main results. Placeholder values, pending final runs.
Method VisualNav GUI-Manip DocQA Tokens / ep.
VLM baseline (zero-shot) 41.2 38.7 55.0 18.4k
+ full-history context 46.8 44.1 58.3 31.7k
+ flat summary memory 49.5 47.9 61.2 12.6k
RHM-Agent (ours) 58.9 59.6 68.4 8.9k
Figure 3. Success as the horizon grows. Flat summaries degrade once episodes exceed the summary budget; RHM-Agent keeps improving because semantic memory accumulates across tasks.

Demo

A memory trace you can step through

One synthetic episode, replayed. Watch where the agent chooses to write, compress, and retrieve — and what that buys it later.

Idle — 8 steps queued.

    Citation

    Cite this work

    @inproceedings{rhmagent2026,
      title     = {RHM-Agent: Teaching VLMs to Reinforce Hierarchical Memory},
      author    = {Anonymous Authors},
      booktitle = {Under review},
      year      = {2026}
    }