User-Conditioned Reward Learning from Conversational Data

Our broader research goal is to build systems that adapt to individual users rather than to an averaged representation of humans. Concretely, we aim to train user-conditioned reward models that can shape model behavior at inference time or during post-training.

However, training such reward models requires access to large and diverse sets of user-specific preference data. Explicit preference labeling is expensive, slow, and cognitively unnatural for most users. So the central challenge is:

How do we derive pairwise preference signals directly from conversational interaction, without requiring explicit feedback?

Why we think Memory is Required

To infer preferences from natural language interaction, the system needs a stable representation of:

• what the user consistently values,
• how their preferences change across time and context, and
• which parts of an interaction are meaningful versus incidental.

This requires a memory system that is selective, continuous, and user-specific. Not merely storage, preference-aware representation.

Evaluation: The Locomo Long-term Memory Benchmark

We evaluate our system on Locomo, a benchmark designed to test whether a model can retain and apply user-specific context across long-term conversations. Unlike short-context dialogue tasks, Locomo requires the model to:

• Track personal events, beliefs, preferences, and relationships across sessions
• Resolve references to past experiences and emotional states
• Reject incorrect or misleading inferences when evidence is insufficient

Locomo frames this as a question-answering task across five reasoning categories:

• Single-hop: Answerable from one conversation turn.
• Multi-hop: Requires synthesizing information across multiple sessions.
• Temporal reasoning: Requires understanding timelines and evolving circumstances.
• Open-domain knowledge: Requires combining conversation details with world knowledge.

Results

From Memory to Preference Data

Once the memory stabilizes, we use it to extract implicit pairwise preferences from conversation while conditioning on the user's memory:

• Identifying when a user endorses a style, tone, approach, or reasoning pattern
• Detecting contrastive rejection (dispreferred alternatives)
• Encoding choices as structured preference tuples (A > B)

These pairs form a user-specific preference dataset without explicit feedback:

"Given phrasing A vs. phrasing B, the user consistently prefers A."

We have also developed methods that allow us to generate counterfactual candidates with high confidence as a direct consequence of the memory system:

"Given phrasing A and memories X, Y, Z, the user prefers/does not prefer A over B̂."

This dataset is then used to train user-conditioned reward models that guide generation either:

• During post-training (stable alignment), or
• At inference (steering).

Next Steps

We are now evaluating this pipeline end-to-end on real conversational settings. The natural next step is to examine:

• Preference extraction coverage and precision
• Reward model stability across contexts
• Impact on downstream personalization quality

We are interested in collaborating with groups focused on personalization evaluation and domain-specific alignment, particularly in settings where capturing real user intent is critical.