ResMerge: Residual Policy Learning and Merging for Continual Adaptation of Pre-trained Robot Policies

1. We propose ResMerge, a two-stage framework for continual adaptation of pre-trained robot policies. It learns mergeable residual corrections on top of a frozen base policy with no additional expert demonstrations during adaptation. Base-policy initialization uses demonstrations, and residual learning uses online RL interaction.
2. ResMerge operates via a Learn-and-Merge loop: in Learn, a LoRA-based residual policy is trained per task with residual RL on sparse rewards while the base policy is frozen; in Merge, RFC (Residual-Field Consistency)—a single scalar measuring directional and magnitudinal alignment of two residual action fields—decides whether the new residual is merged with an existing expert or kept as a new one.
3. RFC assesses compatibility in residual action-field space, comparing correction direction and magnitude. Experiments on LIBERO across two backbones and on a real-world Franka robot show improved continual learning performance over the evaluated PECFT baselines. Selective merging trades some performance for lower residual storage in settings such as the real-robot study.

Consider a generalist robot policy deployed in a home or factory. The robot must continuously learn new tasks as they arrive sequentially, adapting to each one without degrading performance on tasks it already knows. Demonstration-trained, adapter-per-task designs for Parameter-Efficient Continual Fine-Tuning (PECFT) isolate updates, but they (1) require substantial per-task demonstrations and (2) add one adapter per task, so both data and parameter cost grow linearly with the task stream.

ResMerge learns residual corrections without collecting additional expert demonstrations during adaptation and uses RFC-guided parameter-space merging to reduce storage. A similar task may need an opposing correction, while a different task may need an aligned one. RFC evaluates this compatibility directly in residual action space. When a candidate fails the RFC gate, the expert bank grows; its size is not guaranteed to remain constant.

• Simulation: LIBERO-Spatial and LIBERO-Object. ViT-T uses all 10 tasks per suite; OpenVLA-OFT uses five tasks per suite on which the initial base has nonzero success, with the same subsets for all baselines. Object task IDs are (1, 6, 7, 8, 9), and Spatial task IDs are (0, 2, 3, 6, 7).
• Base policies and initialization: the 32M ViT-T BC policy is pre-trained on LIBERO-90 and initialized with two demonstrations per target task. The 8B OpenVLA-OFT uses the released SimpleVLA-RL checkpoint, initialized by supervised fine-tuning on one trajectory from the target suite. The real-robot IMLE BC base uses demonstrations from all four tasks.
• Residual observations: simulation pairs a frozen visual base policy with state-based residuals using proprioception and privileged object-to-end-effector pose features. Real-robot residuals use frozen ViT-S RGB features and proprioception, without object poses or privileged simulator state.
• Baselines: Sequential SFT, Experience Replay (ER), PackNet, TAIL, DMPEL, and a Multi-Task oracle. ViT-T results for Sequential SFT, ER, PackNet, and Multi-Task are adopted from the LIBERO paper; the residual-RL setup additionally uses state-based observations and online interaction.
• Metrics: Forward Transfer (FWT) is average success on future tasks without subtracting initial base success. Negative Backward Transfer (NBT) measures forgetting in percentage points (pp); negative values indicate improvement. AUC averages success on learned tasks across training stages. Average final-stage success rate (SR) is a separate metric.
• Demonstration counts: tables count additional expert demonstrations during continual adaptation, excluding base-policy initialization and online RL interaction.

On OpenVLA-OFT, ResMerge reaches FWT 77%/53% (Spatial/Object), NBT 2/1 pp, and AUC 95%/87%, compared with the Multi-Task oracle's 95%/94% AUC. Seq. SFT (LoRA) has NBT 98/70 pp and AUC 47%/62%. On ViT-T, ResMerge attains AUC 83%/79% and NBT −3/−8 pp. These adaptation results require no additional expert demonstrations, while using online RL interaction. Merging does not improve every metric: on OpenVLA-OFT/Object, no-merge achieves FWT 58% and AUC 90%, compared with ResMerge's 53% and 87%.

Table I: mean ± standard deviation over five seeds. FWT and AUC are percentages; NBT is in pp. MPE is the ratio of ResMerge's cumulative added-parameter overhead to the method's overhead, capped at 1.00. Lower MPE means more added parameters; 1.00 does not imply a constant-size expert bank. Dashes in the MPE column denote non-comparable per-task footprints.

Merging strategy (LIBERO-Object, ViT-T): RFC-gated merging achieves the highest post-refinement SR and FWT and the lowest NBT among the three evaluated strategies. Residual architecture: MLP and LoRA residuals both reach 36% Learn-SR; after merging and RL refinement, their success rates are 14% and 78%, respectively. These comparisons evaluate the complete merging-and-refinement procedure.

We evaluate RFC along the real-robot four-task sequence. Statistics use 2,000 replay observations sampled with a fixed seed and shared by both experts, using the full 836-dimensional observation representation and 22-dimensional residual actions. RFC compares both direction and magnitude in the original action space; the paper's PCA projection is used only for visualization. (a) R₁ and R₂: RFC 0.81, 1.25% dangerous states ⇒ merge accepted into R_AB. (b) R_AB and R₃: RFC 0.01, 96.6% dangerous states ⇒ merge rejected; R₃ remains separate. (c) R₃ and R₄: RFC 0.50, 6.6% dangerous states ⇒ merge accepted into R_CD. The final pool has two experts for four tasks. These field diagnostics do not by themselves establish closed-loop task success.

Sequential RFC diagnostics reported in Section V-C. A score meeting or exceeding the compatibility threshold accepts a merge.

The real-robot merge-all comparison is an offline field diagnostic; no closed-loop merge-all success rate is reported.

We deploy ResMerge on a Franka Emika Panda with a 16-DoF Leap dexterous hand and a single front-mounted RealSense D435 camera, with a frozen IMLE BC base policy trained on demonstrations from all four tasks. The robot sequentially adapts to four tabletop tasks in two paired groups: (T₁) Pinch-Cube and (T₂) Pinch-Apple pinch-grasp a small object into the box; (T₃) Power-Steel and (T₄) Power-Plastic power-grasp a cup onto the plate. Within each group tasks share grasp primitive and container but differ in object appearance, geometry, and friction. This grouping helps interpret RFC decisions; shared grasp primitives do not establish that policy parameters can be merged successfully.

ResMerge attains FWT 50%, NBT 7 pp, and AUC 75%, compared with DMPEL's 24%, 23 pp, and 54%. RFC merges (T₁,T₂) into R_AB and (T₃,T₄) into R_CD, leaving two consolidated experts rather than four. Average final-stage success is 72.5%, compared with 67.0% for TAIL and 78.75% without merging. Consolidation retains 92.1% of the no-merge success rate while halving the expert count, at a cost of 6.25 percentage points in average success.

Success Across the Task Sequence

Table VII: success rates (%) over 20 trials per task at each stage. Rows are training stages; columns are evaluated tasks. The final row gives 58 successes in 80 trials (72.5%).

Representative rollouts of the four real-world tasks executed by ResMerge. For each task we show a success rollout alongside a failure case, illustrating both the learned behaviors and the remaining challenges (slippery objects, fine pinch grasps, and precise placement).