Abstract
Long-form video understanding remains challenging for vision-language models (VLMs) due
to the cost of processing extensive frame sequences and the limited ability of flat or
heuristic memory structures to support efficient retrieval and long-horizon reasoning.
To address these challenges, we introduce Reinforced Hierarchical Memory Agent
(RHM-Agent), which organizes visual evidence into hierarchical memory to
selectively retrieve relevant cues and construct explainable, intelligent, and non-linear
reasoning paths. Equipped with dynamic memory, the RHM-Agent is optimized via a
hierarchical reward mechanism to balance local details with global context, enabling it
to navigate long videos in a human-like manner. To evaluate long-horizon temporal
reasoning and memory efficiency beyond standard accuracy, we propose
HMBench-2K, a long-form egocentric video benchmark spanning four domains:
medical, industrial, marine, and autonomous driving. Additionally, we introduce two
complementary diagnostic metrics, frame brevity and node precision, to
assess evidence efficiency and temporal precision, respectively. Extensive experiments
show that our RHM-Agent surpasses baselines with 4× the parameters or 2× the frames by up
to 6.63% on LVBench and 5.60% on Ego4D, proving strategic memory optimization yields
greater improvements than the brute-force scaling of model size or frame count. Evaluated
directly on our proposed HMBench-2K, RHM-Agent outperforms SOTA VLMs and long-video
agents by 2.00% and 7.15%, respectively, demonstrating effective long-form egocentric
video understanding. Further analysis shows that our method not only improves accuracy in
a model-agnostic design but also achieves better temporal alignment than current dynamic
memory-augmented agents.