1
Iteration 1
364,055 frames
Takeover 81.2%
- Human 81.2%
- BC policy 18.8%
ALOE: Action-Level Off-Policy Evaluation for Vision-Language-Action Model Post-Training
Human-in-the-loop RL for VLA with current-policy off-policy value estimation
1 AgiBot 2 The Hong Kong University of Science and Technology 3 Fudan University 4 Nanjing University 5 Independent Researcher * Equal contribution † Corresponding author
Abstract
We study how to improve large foundation vision-language-action (VLA) systems through human-in-the-loop reinforcement learning (RL) in real-world environments. A key challenge is learning reliable value functions from heterogeneous real-world experience, as value estimation provides the primary learning signal for VLA training. In practice, replay buffers contain trajectories collected from historical policies, online rollouts, demonstrations, and intermittent human interventions. Because replay buffers mix trajectories generated by different behaviors, the observed returns can be mismatched with the quality of the current policy. Prior VLA post-training methods often rely on progress-style value signals, which reflect the average quality of historical behaviors, leading to mismatched learning signals for the current policy. In this paper, we propose ALOE, an off-policy evaluation framework whose value function directly evaluates current-policy behavior for each iteration. Specifically, ALOE combines chunked temporal-difference bootstrapping and conservative value aggregation to perform stable current-policy evaluation, then uses these estimates for advantage-weighted policy improvement. This design improves credit assignment to critical action chunks under sparse rewards and supports stable policy improvement. We evaluate ALOE on four real-world manipulation tasks encompassing long-horizon and high-precision scenarios: smartphone packing, laundry folding, multi-object sorting, and phone assembly. Across all tasks, ALOE outperforms other VLA post-training methods, highlighting the benefit of off-policy value estimates for real-world VLA post-training.
ALOE post-trains a flow-based VLA with human-in-the-loop replay, off-policy critic estimation, and advantage-weighted policy improvement.
Method
ALOE starts from a simple observation: in iterative real-world VLA RL, the replay buffer is useful but not policy-clean. As the policy is repeatedly updated while collecting new real-world data, replay mixes expert demonstrations, human interventions, autonomous rollouts, and trajectories from previous checkpoints. For example, on the Phone Assembly task, the four donuts below show the real cumulative replay distribution across policy-improvement iterations, with each takeover badge reporting the human intervention frame ratio in that iteration.
2
Iteration 2
752,184 frames
Takeover 61.4%
- Human 71.0%
- BC policy 9.1%
- π1 19.9%
3
Iteration 3
1,083,178 frames
Takeover 26.1%
- Human 57.3%
- BC policy 6.3%
- π1 13.8%
- π2 22.6%
4
Iteration 4
1,385,537 frames
Takeover 6.0%
- Human 46.1%
- BC policy 4.9%
- π1 10.8%
- π2 17.7%
- π3 20.5%
Phone Assembly mixed replay distribution over four policy-improvement iterations. Each donut is cumulative: old checkpoints remain in the buffer while new policy rollouts and human-corrected segments are added.
This mixed replay is exactly where Monte Carlo return regression becomes misaligned: the target describes the implicit data policy πD, not the current executable policy π. ALOE therefore changes the critic from behavior-policy progress estimation, VπD, to current-policy action-value estimation.
From Behavior-Policy Value to Current-Policy Action Value
-
Current-Policy Evaluation
Evaluate what the VLA would do now
ALOE keeps the replay transition but bootstraps from the next action sampled by the current actor, aligning the critic with the policy being optimized.
-
Action-Level Credit
Score the action chunk, not just the state
The critic learns Q(s, achunk), so recovery motions and failure-inducing actions receive distinct learning signals under sparse rewards.
Practical Stabilizers
ALOE focuses on making off-policy critic learning stable enough for real-world VLA post-training. It combines Q-chunking, pessimistic ensembles, and AWAC-style policy extraction.
Q-chunking TD
Instead of propagating sparse reward one step at a time, ALOE backs up over executed action chunks and bootstraps from the next chunk sampled by the current policy.
Pessimistic ensemble
An ensemble of Q-heads estimates uncertainty, and policy extraction uses the minimum value to reduce overestimation on weakly covered or OOD actions.
Advantage-weighted extraction
The pessimistic Q estimate is converted into a clipped advantage weight, then used to fine-tune the flow-based VLA on replay actions in an AWAC/AWR style.
Experiments
Task Demos
We evaluate ALOE on four representative tasks designed to stress long-horizon reasoning, precise action selection, robustness, and learning from human-in-the-loop replay: Pack Smart Phone (place the phone into a rigid container and close the lid), Folding Laundry (retrieve, flatten, and fold deformable garments), Product Sorting (empty two material bins onto a conveyor belt), and Phone Assembly (place phone accessories into their target locations).
Final Results
Average success rates on four manipulation tasks under real-world evaluation. Each final policy is evaluated over 60 trials per task; ALOE achieves the highest final success rate across all tasks.
Policy Improvement
Across online RL iterations, ALOE continues to improve from the behavior-cloning warm-start policy as new real-world interaction data is collected.
Efficiency, Generalization & Robustness
ALOE improves phone-packing throughput, zero-shot generalization on unseen sorting objects, and recovery under laundry-folding disturbances.
Success Rate
Pack Smart Phone is precision-critical: the phone case (17.8cm by 8.8cm) has minimal clearance inside the rigid container (17.5cm by 8.6cm). Under position control, slight pose errors can cause insertion failure, tipping, or damage. The video shows 21 consecutive trials with 19 successes.
Robustness
We inject random perturbations during execution (e.g., to the garment in Folding Laundry or to the assembly in Pack Smart Phone) and measure whether the policy can re-adjust and still complete the task — evaluating recovery from unexpected disturbances.
Pack Smart Phone — recovery under disturbance.
Folding Laundry — recovery under disturbance.
Zero-Shot Generalization
We evaluate generalization on the Product Sorting task by replacing objects with ones never seen during training (different shapes and colors). The policy is evaluated without fine-tuning, assessing the VLA's ability to generalize to unseen object appearances and geometries. Below: in-distribution (seen objects) for comparison, then OOD (unseen objects) in parallel.
In-distribution (seen objects).
OOD (unseen objects).
Objects used during training (left) and unseen objects used for zero-shot evaluation (right).
Critic Analysis
Beyond final success rates, ALOE directly tests whether the critic is a useful learning signal. The evidence points to three properties: it tracks current-policy returns, separates good and bad actions under similar states, and becomes more robust when Q-chunking and pessimistic aggregation are enabled.
Where Do the Gains Come From?
Removing action-level evaluation or off-policy TD bootstrapping steadily weakens the policy, showing that the critic design is central rather than incidental.
| Variant | Phone | Laundry | Sorting | Overall |
|---|---|---|---|---|
| ALOE full | 80.0 | 58.7 | 87.0 | 75.2 |
| w/o action-level | 65.0 | 50.3 | 80.0 | 65.1 |
| w/o action-level, w/o off-policy | 50.0 | 45.0 | 80.0 | 58.3 |
Action-Level Credit Assignment
ALOE identifies failure modes and successful recovery: the Q-value drops when the robot fails a critical action and rises when the policy recovers, providing finer credit assignment than trajectory-level estimation.
Action Ranking & OOD Conservatism
The critic assigns higher scores to actions that lead to successful trajectories and lower scores to OOD or meaningless actions, giving the actor a more faithful preference signal.
Advantage Localization
On Phone Assembly, ALOE assigns high advantage to task-relevant approach-and-grasp actions, while state-value baselines can overvalue empty-grasp states.
Critic Component Analysis
Larger Q-chunks improve phone-task throughput and success rate, while pessimistic ensemble aggregation improves unseen-object sorting under distribution shift.
Citation
@article{yang2026aloe,
title = {ALOE: Action-Level Off-Policy Evaluation for Vision-Language-Action Model Post-Training},
author = {Yang, Rushuai and Wang, Hecheng and Wu, Zhichao and Liu, Chiming and Yan, Xiaohan and Du, Xuan and Yue, Shuoyu and Zhang, Chuheng and Wang, Yunlong and Liu, Yongcheng and Qi, Lizhe and Chen, Yi and Shan, Wei and Yao, Maoqing},
year = {2026},
eprint = {2602.12691},
archivePrefix = {arXiv},
primaryClass = {cs.RO}
}