PRIMT: Preference-based Reinforcement Learning with Multimodal Feedback and Trajectory Synthesis from Foundation Models

Ruiqi Wang 1,†, Dezhong Zhao1,2,†, Ziqin Yuan1,†, Tianyu Shao1,
Guohua Chen2, Dominic  Kao3, Sungeun  Hong4, Byung-Cheol Min 1,5
1Purdue University; 2BUCT; 3UIUC; 4Sungkyunkwan University; 5Indiana University Bloomington; †Equal Contribution
🎤 To be presented at NeurIPS 2025 as an Oral
Paper Poster Presentation Code

AI Generated Audio Summary

Synthesized by NotebookLM to help you understand the PRIMT paper.

Abstract

Preference-based reinforcement learning (PbRL) has emerged as a promising paradigm for teaching robots complex behaviors without reward engineering. However, its effectiveness is often limited by two critical challenges: the reliance on extensive human input and the inherent difficulties in resolving query ambiguity and credit assignment during reward learning. In this paper, we introduce PRIMT, a PbRL framework designed to overcome these challenges by leveraging foundation models (FMs) for multimodal synthetic feedback and trajectory synthesis. Unlike prior approaches that rely on single-modality FM evaluations, PRIMT employs a hierarchical neuro-symbolic fusion strategy, integrating the complementary strengths of large language models and vision-language models in evaluating robot behaviors for more reliable and comprehensive feedback. PRIMT also incorporates foresight trajectory generation, which reduces early-stage query ambiguity by warm-starting the trajectory buffer with bootstrapped samples, and hindsight trajectory augmentation, which enables counterfactual reasoning with a causal auxiliary loss to improve credit assignment. We evaluate PRIMT on 2 locomotion and 6 manipulation tasks on various benchmarks, demonstrating superior performance over FM-based and scripted baselines.

Motivation

Preference-based RL offers an alternative to hand-designed rewards by learning from comparative feedback; yet its reliance on human labels prevents scalability. Our goal is to enable scalable, zero-shot PbRL with foundation models (FMs). We identify three critical challenges in existing preference-based RL frameworks:


1. Limitations of Single-Modality Synthetic Feedback

VLM Limitation

Vision-based reasoning → reliable spatial grounding and goal-state assessment, but limited ability to interpret temporal progression or subtle motion dynamics.

LLM Limitation

Text-centric analysis → good temporal and logical reasoning, but often hallucinate or miss fine-grained spatial interactions and key events.

2. Query Ambiguity in Early Training

Query Ambiguity

Early trajectories from random policies are uniformly low quality, lacking meaningful task variations → cannot provide informative comparisons.

3. Preference Credit Assignment Uncertainty

Credit Assignment Uncertainty

Preferences are given at trajectory level, but reward models operate at state-action level → hard to determine which steps caused the preference.

Framework Overview

Animation of the PRIMT framework

Overview of the PRIMT, comprising of two synergistic modules: 1) Hierarchical neuro-symbolic preference fusion improves the quality and reliability of synthetic feedback by leveraging the complementary collective intelligence of VLMs and LLMs for multimodal evaluation of robot behaviors; and 2) Bidirectional trajectory synthesis consists of foresight trajectory generation, which bootstraps the trajectory buffer to mitigate early-stage query ambiguity, and hindsight trajectory augmentation, which applies SCM-based counterfactual reasoning to improve reward learning with a causal auxiliary loss that enables fine-grained credit assignment.

1 Hierarchical Neuro-symbolic Preference Fusion

Click to expand

2 Foresight Trajectory Generation

Click to expand

3 Hindsight Trajectory Generation

Click to expand

Experimental Results

- Better Task Performance; - Improved Synthetic Feedback Quality; - Enhanced Credit Assignment; - Better Cost-Performance Efficiency


Result 1

Learning curves of PRIMT and baseline methods across all tasks


Result 2

Ablation study on FM backbone selection.


Result 3

Distribution of preference labels, showing the proportion of correct, incorrect, and indecisive labels across different methods.


Result 4

Reward alignment analysis, comparing the learned reward outputs of PRIMT, ablations, and baselines against ground-truth rewards.


Result 5

Cost-performance trade-off comparison of PRIMT against baseline methods

Experimental Demos


PRIMT can lead to more efficient robot behaviors across diverse tasks in zero-shot.

MetaWorld

Button Press

Door Open

Sweep Into

ManiSkill

PegInsertionSide

PickSingleYCB

StackCube

DMControl

Hopper Stand

Walker Walk

Real-world Deployment


⚠️ To ensure safety during deployment, we imposed a soft constraint: if the turning angle or acceleration exceeded a threshold, the corresponding action was discarded.

Block Lifting

Baseline

Baseline

PRIMT

PRIMT

Block Stacking

Baseline

Baseline

PRIMT

PRIMT