Re-Centering Humans in LLM Personalization

Personalization is becoming a core promise of LLM systems: chatbots remember your job, interests, preferences, and past conversations to tailor responses. But “personalized” does not always mean helpful — it can also feel uncomfortable, offensive, or just unnecessary.

This raises a basic but surprisingly under-examined question: 𝗪𝗵𝗼 𝗱𝗲𝗰𝗶𝗱𝗲𝘀 𝘄𝗵𝗲𝘁𝗵𝗲𝗿 𝗽𝗲𝗿𝘀𝗼𝗻𝗮𝗹𝗶𝘇𝗮𝘁𝗶𝗼𝗻 𝗶𝘀 𝗮𝗰𝘁𝘂𝗮𝗹𝗹𝘆 𝗵𝗲𝗹𝗽𝗳𝘂𝗹 — 𝘁𝗵𝗲 𝗺𝗼𝗱𝗲𝗹, 𝗼𝗿 𝘁𝗵𝗲 𝗽𝗲𝗿𝘀𝗼𝗻 𝗯𝗲𝗶𝗻𝗴 𝗽𝗲𝗿𝘀𝗼𝗻𝗮𝗹𝗶𝘇𝗲𝗱 𝗳𝗼𝗿?

Most existing benchmarks rely heavily on synthetic personas, simulated conversations, and LLM judges. In this work, we put 𝗿𝗲𝗮𝗹 𝗵𝘂𝗺𝗮𝗻𝘀 back into the loop.

We study personalization as a three-stage pipeline:

🧠 𝗔𝘁𝘁𝗿𝗶𝗯𝘂𝘁𝗲 𝗲𝘅𝘁𝗿𝗮𝗰𝘁𝗶𝗼𝗻 — what should the system infer from conversation history?
🎯 𝗥𝗲𝗹𝗲𝘃𝗮𝗻𝗰𝗲 𝗺𝗮𝘁𝗰𝗵𝗶𝗻𝗴 — which attributes actually matter for the current request?
✍️ 𝗥𝗲𝘀𝗽𝗼𝗻𝘀𝗲 𝗴𝗲𝗻𝗲𝗿𝗮𝘁𝗶𝗼𝗻 — does personalization improve the user experience?

Using 550 real user conversations and nearly 19,000 human judgments, we find systematic 𝗵𝘂𝗺𝗮𝗻–𝗟𝗟𝗠 𝗴𝗮𝗽𝘀 at every stage:

• Models extract noisy and overgeneralized attributes from real conversations. Synthetic data underestimates this difficulty.
• LLMs and humans disagree on which attributes should be used in a new question(κ=0.30), but each agree well within their own group (κ=0.60 and 0.43).
• LLMs select 𝟮–𝟯× 𝗺𝗼𝗿𝗲 𝗮𝘁𝘁𝗿𝗶𝗯𝘂𝘁𝗲𝘀 as relevant than humans do, suggesting a tendency to over-personalize.
• Even with human-selected relevant attributes, 𝟱𝟰.𝟲% of personalized responses are judged 𝗻𝗼 𝗯𝗲𝘁𝘁𝗲𝗿 𝘁𝗵𝗮𝗻 𝗴𝗲𝗻𝗲𝗿𝗶𝗰 ones by humans.
• LLM judges often overestimate personalization quality, sometimes rewarding surface-level attribute mentions that humans do not find useful.

We also find that lightweight training improves attribute verification and relevance matching substantially. But response-level personalization remains much harder, likely because “good personalization” is inherently individual.

𝗢𝘂𝗿 𝘁𝗮𝗸𝗲𝗮𝘄𝗮𝘆 𝗶𝘀 𝘀𝗶𝗺𝗽𝗹𝗲: Synthetic users and LLM judges aren't enough to capture the complex nature of human preferences. We highlight the importance of 𝗵𝘂𝗺𝗮𝗻 𝗱𝗮𝘁𝗮 and call for 𝗿𝗲-𝗰𝗲𝗻𝘁𝗲𝗿𝗶𝗻𝗴 𝗵𝘂𝗺𝗮𝗻𝘀 in LLM personalization.