Under double-blind review · ICLR 2027

Beyond Oracle Communication: Benchmarking Interactive Intent Alignment Under Miscommunication and Evolving User Intent

Real users miscommunicate, change their minds mid-conversation, and run out of patience. Drift-Bench++ drops the oracle-communication assumption: every task carries a verified miscommunicated request, an executable graph of alternative goals the user may silently drift to, and a finite patience budget — scored by GRIP, an eleven-metric protocol for grounding, role-realism, inquiry, and persistence.

Anonymous authors — paper under double-blind review
📄  Paper ▶  Interactive Cases ❞  BibTeX
3executable environments
WebShop · τ²-Retail · τ²-Airline
11verified communication-fault types
4typed silent intent-shift relations
5GDMS persona simulations
11GRIP evaluation metrics
2,543verified benchmark instances
Overview of the Drift-Bench++ task
One episode of interactive intent alignment.
(a) The episode. The user's opening request carries a verified communication fault; mid-conversation, the latent intent silently shifts while the patience budget drains. The agent must notice, ask well, and commit before patience runs out.
(b) The user. Simulated users follow General Decision-Making Style personas that control disclosure, patience, response style, and how readily the goal moves.
(c) The score. The GRIP protocol scores the episode along four complementary dimensions — grounding, role-realism, inquiry, persistence — instead of a single success rate.
Challenge 1

Miscommunication, verified

An LLM asked to write a "flawed request" often produces a different flaw — or none. Every Drift-Bench++ opening is generate-then-verify: two independent extractor models must agree on what the text actually miscommunicates, and the recovered fault must provably change the executable answer.

Challenge 2

Intent that evolves silently

Goals move mid-conversation over an executable intent graph — refined, relaxed, substituted, or pivoted — without announcement. Some shifts are scheduled on the patience axis; others are triggered by the agent's own clarifying question prompting the user to reconsider.

Challenge 3

Finite patience

Questions cost 2, rejected proposals cost 4, from a persona-scaled budget of 10. When the budget runs out the agent must commit. Clarification stops being free — it becomes a spending decision.

Benchmark construction

From host benchmarks to verified instances

Drift-Bench++ builds on executable host benchmarks — every latent intent, every alternative goal, and every adjudication is executable, so ground truth never depends on a judge model.

Construction pipeline
1

Intent graphs

Native tasks become executable condition sets; validated retrieval and witness-guided synthesis add typed alternatives (refine / relax / substitute / pivot), each with recomputed ground truth.

2

Free-form generation

A writer model renders the true intent into a request carrying exactly one taxonomy fault — free-form, so the flaw reads like a real person wrote it.

3

Dual extraction

Two extractor models from different families independently recover the structured edit the text expresses; the candidate survives only if they agree on the fault family and its target.

4

Executable admission

The literal reading is executed against the environment: a substitution must move the answer, a withholding must widen it, an ambiguity must split it. No consequence, no instance.

BenchmarkTasksGraphs Rejection rateSamples
WebShop50025731%1,780
τ²-Retail1149847%555
τ²-Airline504048%208
Verification has teeth. The three-step verifier rejects 31–48% of generated candidates, leaving 2,543 verified instances over 395 intent graphs. Composed-fault subsets (k = 2, 4) are built on top of the verified WebShop pool for the complexity study.
Communication faults

A verified fault taxonomy — click a family

Eleven fault types across four families, inherited from communication theory and operationalized as executable edits. Each released instance carries one verified fault (or a composed set at k = 2, 4).

Interaction protocol

Users with patience, personality, and changing minds

A state machine owns everything true — intents, disclosures, budgets, shifts, adjudication. The language model only voices replies, so simulated users sound natural but can never bend the task.

🎛 Try the patience economy

Every user-facing action spends the budget. Scheduled shift opportunities sit at persona-specific marks and fire silently with probability 0.6 when crossed. Spend the budget and see what the agent lives with.

0patience 10.0 / 10.0

Click a persona to see its configuration. Bios are inherited from the source benchmark; the behavioral parameters — disclosure, refusal, patience multiplier, hint probability, shift placements, suggestibility — are Drift-Bench++'s.

Case studies

Step through real episodes

Three unedited episodes from the released evaluation runs — every quote verbatim, every event machine-checked. Watch the fault land, the questions earn their cost, and the intent move silently.

patience
Evaluation

GRIP: eleven metrics, four dimensions, no composite

A single success rate cannot separate an agent that understood the user from one that guessed. GRIP reports each dimension with its own denominator and never collapses them — a bad score names the fault. Click a dimension for its metrics.

Findings

Interaction helps — and is nowhere near solved

Six experiment lines from the paper: the capability landscape across eleven methods, then how it holds up across model backbones, composed faults, silent intent shifts, and user personas — plus a validation that the simulated users are worth believing. Dots mark the best value per column; underlines the second best.

Asking pays for itself. On τ²-Retail every clarification method beats silence by double digits in Success — No Ask 49.9 vs 67.6 for a single well-placed question.
Recovery is the separator. Success clusters tightly while Recovery and PostShift spread methods apart — exactly the capabilities a flat success rate cannot see.
Far from oracle. Even the strongest method recovers the full hidden intent in a minority of episodes, and no method approaches oracle-communication performance.
Citation

BibTeX

@inproceedings{driftbenchpp2027,
  title     = {Beyond Oracle Communication: Benchmarking Interactive
               Intent Alignment Under Miscommunication and Evolving
               User Intent},
  author    = {Anonymous},
  booktitle = {Under review},
  year      = {2027},
  url       = {https://drift-bench-plus.github.io/}
}