这篇工作的启发在于，个性化长期记忆不能把图像当作普通 caption 的附属材料，而应保留可检索、可解释、可延迟提交的结构化视觉证据。它补齐了 personal assistant 中“图像承载私人事实”的评测与系统缺口，也为 PersonalVCL、PVChat、PEARL、PersonaVLM 这类个性化理解或长期助手方法提供了更强的视觉记忆入口。需要继续解决的是真实数据规模、生成图像 benchmark 外推、记忆冲突/遗忘、pending pool 管理和隐私权限控制。
Related work: VisualMem 与 GDP 都把用户个性化建模推进到长期状态管理，但二者侧重点互补：VisualMem 负责把图像中的个人实体、所有权和隐性事实结构化保存，GDP 则进一步把用户偏好组织成带置信度和边权的动态图并用于动态偏好问答。相较 PersonalVCL，VisualMem 更关注上下文进入系统前如何被构造成长期记忆；相较 PersonaVLM、PEARL，它提供的是个人视觉证据层，可作为这些长期或流式个性化助手的记忆底座。
Abstract
Long-term memory is increasingly important for personalized AI agents, yet existing benchmarks and methods remain largely text-centric. Even when images are included, the user-specific information needed for later questions is typically recoverable from text alone, and most memory systems reduce image turns to generic captions. Yet images often carry personal information that text rarely states -- both explicit evidence, such as recurring user-associated entities, and implicit evidence, such as latent user facts inferred from visual or multimodal cues. We introduce a benchmark for personal visual memory that targets both forms of evidence, and propose VisualMem, a hybrid visual--text architecture that augments a text-memory backend with a structured personal visual memory module. Rather than collapsing images into captions, VisualMem uses conversational context to resolve identity, ownership, and durable user facts. Experiments show that VisualMem substantially outperforms prior memory systems on our benchmark while remaining competitive on standard text-memory benchmarks, indicating that personal visual memory is a distinct and important component of long-term memory for personalized AI agents.
2
2026arXivPersonalVCLPersonalized Understanding
Personal Visual Context Learning in Large Multimodal Models
Related work: PersonalVCL 与 PEARL、PersonaVLM、TAME、ATMBench 一样把个性化推进到长期或连续个人助手场景，但它更聚焦“相关视觉上下文已经给出后，LMM 是否真的会用”的 context utilization 问题。相比 Online-PVLM、SLC、Ego 这类概念注入或私有实体 grounding 方法，PersonalVCL 处理的是人物、物体、行为和 wearer 身份的多证据聚合；相比 PEARL 的流式视频检索与问答，它提供了更细的上下文利用诊断和 Agentic Context Bank 这种结构化证据访问 baseline。
Abstract
As wearable devices like smart glasses integrate Large Multimodal Models (LMMs) into the continuous first-person visual streams of individual users, the evolution of these models into true personal assistants hinges on visual personalization: the ability to reason over visual information unique to the wearer. We formalize this capability as Personal Visual Context Learning (Personal VCL), the prompt-time capability of using user-specific visual context to resolve personalized queries. To systematically evaluate this, we present Personal-VCL-Bench, a comprehensive benchmark capturing the personal visual world across persons, objects, and behaviors. Our analysis of frontier LMMs identifies a profound context utilization gap, revealing that the mechanisms for leveraging visual evidence, as well as aggregating multiple visual observations, remain critically understudied. Motivated by these findings, we propose the Agentic Context Bank, a strong inference-time baseline that structures a user's visual context into a self-refining memory bank and employs query-adaptive evidence selection. Our baseline approach consistently improves over standard context prompting regimes across tasks and evaluated backbones, demonstrating a practical path towards future personalized LMMs.
Multimodal Large Language Models (MLLMs) serve as daily assistants for millions. However, their ability to generate responses aligned with individual preferences remains limited. Prior approaches enable only static, single-turn personalization through input augmentation or output alignment, and thus fail to capture users' evolving preferences and personality over time (see Fig.1). In this paper, we introduce PersonaVLM, an innovative personalized multimodal agent framework designed for long-term personalization. It transforms a general-purpose MLLM into a personalized assistant by integrating three key capabilities: (a) Remembering: It proactively extracts and summarizes chronological multimodal memories from interactions, consolidating them into a personalized database. (b) Reasoning: It conducts multi-turn reasoning by retrieving and integrating relevant memories from the database. (c) Response Alignment: It infers the user's evolving personality throughout long-term interactions to ensure outputs remain aligned with their unique characteristics. For evaluation, we establish Persona-MME, a comprehensive benchmark comprising over 2,000 curated interaction cases, designed to assess long-term MLLM personalization across seven key aspects and 14 fine-grained tasks. Extensive experiments validate our method's effectiveness, improving the baseline by 22.4% (Persona-MME) and 9.8% (PERSONAMEM) under a 128k context, while outperforming GPT-4o by 5.2% and 2.0%, respectively. Project page: https://PersonaVLM.github.io.
4
2026arXivPEARLPersonalized Understanding
PEARL: Personalized Streaming Video Understanding Model
PEARL 的价值在于把 MLLM personalization 从图像概念识别推进到实时、连续、可交互的视频助手场景，并给出一个无需训练即可迁移到多种 VLM 的系统基线。它提示后续个性化助手需要显式维护概念记忆和时序证据，而不是只依赖长上下文或一次性 RAG。局限也很清楚：概念描述质量依赖注册阶段 VLM，动作级数据仍偏合成，长期记忆维护、冲突纠正、遗忘和开放式回答还没有充分解决。
Related work: PEARL 与 TAME、ATMBench、PersonaVLM 同属把 personalization 推向长期或连续交互场景的路线，但它把焦点放在流式视频和精确时间戳问答上：相比 TAME 的对话状态双记忆，PEARL 的双记忆更强调概念描述与视频片段证据检索；相比 ATMBench 的多来源个人记忆 QA，它提供了可插拔的在线视频系统基线。它也和 Online-PVLM、SLC、CoViP、RePIC 形成互补：这些工作更多处理概念注入、视觉触发记忆或 personalized captioning，而 PEARL 可以把这些前端能力接入连续视频流中的历史检索与实时问答。
Abstract
Human cognition of new concepts is inherently a streaming process: we continuously recognize new objects or identities and update our memories over time. However, current multimodal personalization methods are largely limited to static images or offline videos. This disconnects continuous visual input from instant real-world feedback, limiting their ability to provide the real-time, interactive personalized responses essential for future AI assistants. To bridge this gap, we first propose and formally define the novel task of Personalized Streaming Video Understanding (PSVU). To facilitate research in this new direction, we introduce PEARL-Bench, the first comprehensive benchmark designed specifically to evaluate this challenging setting. It evaluates a model's ability to respond to personalized concepts at exact timestamps under two modes: (1) Frame-level, focusing on a specific person or object in discrete frames, and (2) a novel Video-level, focusing on personalized actions unfolding across continuous frames. PEARL-Bench comprises 132 unique videos and 2,173 fine-grained annotations with precise timestamps. Concept diversity and annotation quality are strictly ensured through a combined pipeline of automated generation and human verification. To tackle this challenging new setting, we further propose PEARL, a plug-and-play, training-free strategy that serves as a strong baseline. Extensive evaluations across 8 offline and online models demonstrate that PEARL achieves state-of-the-art performance. Notably, it brings consistent PSVU improvements when applied to 3 distinct architectures, proving to be a highly effective and robust strategy. We hope this work advances vision-language model (VLM) personalization and inspires further research into streaming personalized AI assistants. Code is available at https://github.com/Yuanhong-Zheng/PEARL.
5
2026arXivEgoPersonalized Understanding
Ego: Embedding-Guided Personalization of Vision-Language Models
AI assistants that support humans in daily life are becoming increasingly feasible, driven by the rapid advancements in multimodal language models. A key challenge lies in overcoming the generic nature of these models to deliver personalized experiences. Existing approaches to personalizing large vision language models often rely on additional training stages, which limit generality and scalability, or on engineered pipelines with external pre-trained modules, which hinder deployment efficiency. In this work, we propose an efficient personalization method that leverages the model's inherent ability to capture personalized concepts. Specifically, we extract visual tokens that predominantly represent the target concept by utilizing the model's internal attention mechanisms. These tokens serve as a memory of that specific concept, enabling the model to recall and describe it when it appears in test images. We conduct a comprehensive and unified evaluation of our approach and SOTA methods across various personalization settings including single-concept, multi-concept, and video personalization, demonstrating strong performance gains with minimal personalization overhead.
6
2026arXivATMBenchPersonalized Understanding
According to Me: Long-Term Personalized Referential Memory QA
提出首个面向多模态、多来源、长期个性化指代记忆问答的基准 ATM-Bench，并用 SGM 结构化记忆表示系统性揭示现有 personal memory assistant 在困难子集上的明显短板。
Related work: 在这一页的工作谱系里，ATMBench 更像评测与问题定义底座：它补的是长期、多来源、证据级 personalization memory QA。与 CoViP、Online-PVLM、SLC 这类偏概念识别或视觉触发记忆的方法相比，它把焦点前移到“系统最终是否真的能从复杂个人记忆中找对证据并答对问题”。与 TAME 则形成自然衔接：TAME 进一步把长期对话中的状态变化建模为可管理的 memory。
Abstract
Personalized AI assistants must recall and reason over long-term user memory, which naturally spans multiple modalities and sources such as images, videos, and emails. However, existing Long-term Memory benchmarks focus primarily on dialogue history, failing to capture realistic personalized references grounded in lived experience. We introduce ATM-Bench, the first benchmark for multimodal, multi-source personalized referential Memory QA. ATM-Bench contains approximately four years of privacy-preserving personal memory data and human-annotated question-answer pairs with ground-truth memory evidence, including queries that require resolving personal references, multi-evidence reasoning from multi-source and handling conflicting evidence. We propose Schema-Guided Memory (SGM) to structurally represent memory items originated from different sources. In experiments, we implement 5 state-of-the-art memory systems along with a standard RAG baseline and evaluate variants with different memory ingestion, retrieval, and answer generation techniques. We find poor performance (under 20% accuracy) on the ATM-Bench-Hard set, and that SGM improves performance over Descriptive Memory commonly adopted in prior works. Code available at: https://github.com/JingbiaoMei/ATM-Bench.
7
2026arXivM2APersonalized Understanding
M2A: Multimodal Memory Agent with Dual-Layer Hybrid Memory for Long-Term Personalized Interactions
This work addresses the challenge of personalized question answering in long-term human-machine interactions: when conversational history spans weeks or months and exceeds the context window, existing personalization mechanisms struggle to continuously absorb and leverage users' incremental concepts, aliases, and preferences. Current personalized multimodal models are predominantly static—concepts are fixed at initialization and cannot evolve during interactions. We propose M2A, an agentic dual-layer hybrid memory system that maintains personalized multimodal information through online updates. The system employs two collaborative agents: ChatAgent manages user interactions and autonomously decides when to query or update memory, while MemoryManager breaks down memory requests from ChatAgent into detailed operations on the dual-layer memory bank, which couples a RawMessageStore (immutable conversation log) with a SemanticMemoryStore (high-level observations), providing memories at different granularities. In addition, we develop a reusable data synthesis pipeline that injects concept-grounded sessions from Yo'LLaVA and MC-LLaVA into LoCoMo long conversations while preserving temporal coherence. Experiments show that M2A significantly outperforms baselines, demonstrating that transforming personalization from one-shot configuration to a co-evolving memory mechanism provides a viable path for high-quality individualized responses in long-term multimodal interactions. The code is available at https://github.com/Little-Fridge/M2A.
8
2026arXivCoViPPersonalized Understanding
Contextualized Visual Personalization in Vision-Language Models
Related work: CoViP 处在这页工作里“视觉触发个性化记忆”的中心位置。它比 Online-PVLM 和 SLC 更强调检索并整合历史 personal context，而不只是识别概念；但它又没有像 TAME 或 ATMBench 那样进入长期状态管理与多来源证据推理。可以把它看成从概念级 personalization 走向真正 memory-grounded MLLM assistant 的中间桥梁。
Abstract
Despite recent progress in vision-language models (VLMs), existing approaches often fail to generate personalized responses based on the user's specific experiences, as they lack the ability to associate visual inputs with a user's accumulated visual-textual context. We newly formalize this challenge as contextualized visual personalization, which requires the visual recognition and textual retrieval of personalized visual experiences by VLMs when interpreting new images. To address this issue, we propose CoViP, a unified framework that treats personalized image captioning as a core task for contextualized visual personalization and improves this capability through reinforcement-learning-based post-training and caption-augmented generation. We further introduce diagnostic evaluations that explicitly rule out textual shortcut solutions and verify whether VLMs truly leverage visual context. Extensive experiments demonstrate that existing open-source and proprietary VLMs exhibit substantial limitations, while CoViP not only improves personalized image captioning but also yields holistic gains across downstream personalization tasks. These results highlight CoViP as a crucial stage for enabling robust and generalizable contextualized visual personalization.
Unified large multimodal models (LMMs) have achieved remarkable progress in general-purpose multimodal understanding and generation. However, they still operate under a "one-size-fits-all" paradigm and struggle to model user-specific concepts (e.g., generate a photo of `<maeve>`) in a consistent and controllable manner. Existing personalization methods typically rely on external retrieval, which is inefficient and poorly integrated into unified multimodal pipelines. Recent personalized unified models introduce learnable soft prompts to encode concept information, yet they either couple understanding and generation or depend on complex multi-stage training, leading to cross-task interference and ultimately to fuzzy or misaligned personalized knowledge. We present OmniPersona, an end-to-end personalization framework for unified LMMs that, for the first time, integrates personalized understanding, generation, and image editing within a single architecture. OmniPersona introduces structurally decoupled concept tokens, allocating dedicated subspaces for different tasks to minimize interference, and incorporates an explicit knowledge replay mechanism that propagates personalized attribute knowledge across tasks, enabling consistent personalized behavior. To systematically evaluate unified personalization, we propose OmniPBench, extending the public UnifyBench concept set with personalized editing tasks and cross-task evaluation protocols integrating understanding, generation, and editing. Experimental results demonstrate that OmniPersona delivers competitive and robust performance across diverse personalization tasks. We hope OmniPersona will serve as a strong baseline and spur further research on controllable, unified personalization.
10
2026KDD 2026TAMEPersonalized Understanding
TAMEing Long Contexts in Personalization: Towards Training-Free and State-Aware MLLM Personalized Assistant
TAME 试图解决的是比“识别用户私有概念”更进一步的问题：在长期图文对话中，个性化概念会不断发生状态变化，模型需要区分哪些属性是长期稳定的，哪些只是近期短期状态，并在回答时优先使用与当前问题最匹配的那部分记忆。作者认为，现有个性化方法普遍把概念理解成静态实体，因此即便引入 RAG，也很难处理随时间变化的偏好、健康状态或近期事件。
核心方法
TAME 的核心是双记忆设计与 RA2G。系统先用 Dynamic State Memory 增量记录近期变化，再把被判定为长期稳定的属性迁移到 Static Personalized Memory；当动态记忆过长时，还会执行容量控制与过期清理。面对新问题时，框架先做实体 grounding，再在双记忆中检索与该 concept 最相关的上下文，随后通过 Align 步骤把多条记忆压缩成与当前 query 真正匹配的文本上下文，最后再交给底层 MLLM 回答。整个流程不依赖重新训练 backbone。
在本页所有工作里，TAME 最接近“真正的 personalized assistant”形态。它把 personalization 从概念识别、属性替换推进到状态跟踪与长时记忆管理，并且给出了无需额外训练的系统路径。后续做多轮 assistant、episodic memory 或 user state modeling 时，双记忆分层和 retrieve-then-align 这两个设计都很值得直接沿用。
Related work: 如果把这一页的方法按能力层级排开，TAME 明显是往长时状态管理方向走得最远的一篇。它和 ATMBench 一起把 personalization 拉进长期记忆场景，但 TAME 提供了可运行的 state-aware system；相较 CoViP、Online-PVLM、SLC 这些更多停留在视觉触发概念识别或记忆恢复的工作，TAME 更强调“记忆应该如何随时间被维护和选择”。
Abstract
Multimodal Large Language Model (MLLM) Personalization is a critical research problem that facilitates personalized dialogues with MLLMs targeting specific entities (known as personalized concepts). However, existing methods and benchmarks focus on the simple, context-agnostic visual identification and textual replacement of the personalized concept (e.g., "A yellow puppy" -> "Your puppy Mochi"), overlooking the ability to support long-context conversations. An ideal personalized MLLM assistant is capable of engaging in long-context dialogues with humans and continually improving its experience quality by learning from past dialogue histories. To bridge this gap, we propose LCMP, the first Long-Context MLLM Personalization evaluation benchmark. LCMP assesses the capability of MLLMs in perceiving variations of personalized concepts and generating contextually appropriate personalized responses that reflect these variations. As a strong baseline for LCMP, we introduce a novel training-free and state-aware framework TAME. TAME endows MLLMs with double memories to manage the temporal and persistent variations of each personalized concept in a differentiated manner. In addition, TAME incorporates a new training-free Retrieve-then-Align Augmented Generation (RA2G) paradigm. RA2G introduces an alignment step to extract the contextually fitted information from the multi-memory retrieved knowledge to the current questions, enabling better interactions for complex real-world user queries. Experiments on LCMP demonstrate that TAME achieves the best performance, showcasing remarkable and evolving interaction experiences in long-context scenarios.
11
2026TMLR 2026PeKitPersonalized Understanding
Personalization Toolkit: Training Free Personalization of Large Vision Language Models
Personalization of Large Vision-Language Models (LVLMs) involves customizing models to recognize specific users or object instances and to generate contextually tailored responses. Existing approaches rely on time-consuming training for each item, making them impractical for real-world deployment, as reflected in current personalization benchmarks limited to object-centric single-concept evaluations. In this paper, we present a novel training-free approach to LVLM personalization called PeKit. We introduce a comprehensive, real-world benchmark designed to rigorously evaluate various aspects of the personalization task. PeKit leverages pre-trained vision foundation models to extract distinctive features, applies retrieval-augmented generation (RAG) techniques to identify instances within visual inputs, and employs visual prompting strategies to guide model outputs. Our model-agnostic vision toolkit enables efficient and flexible multi-concept personalization across both images and videos, without any additional training. We achieve state-of-the-art results, surpassing existing training-based methods.
Related work: GDP 与 VisualMem 都把个性化理解从单张图像识别推进到长期记忆管理，但 GDP 更强调偏好状态的实时更新和图结构检索，VisualMem 更强调图像中显性实体与隐性事实作为个人长期记忆证据的保留。与 PersonaVLM、PEARL、PersonalVCL 相比，GDP 的特色是用带置信度的动态图统一身份识别和偏好问答；它也可以吸收 VisualMem 的 pending/结构化视觉记忆思想来降低错误写入和隐性事实遗漏。
Abstract
Multimodal large language models (MLLMs) have demonstrated strong cross-modal capabilities, and recent work has begun incorporating user identity and preference modeling to enhance personalization. However, existing approaches struggle to reliably recognize users in complex visual conditions (e.g., occlusion, non-frontal views) and rely on static user profiles without dynamically updating preferences, limiting their adaptability to real-world evolving user states. To address these challenges, we propose Graph-based Dynamic Personalization (GDP), a framework that applies preference confidence as dynamic weights to model user preferences in real time. Specifically, the system employs a Personalization Enhancement (PE) Agent to extract preference cues, estimate preference confidence, and leverage graph associations to enhance identity recognition under challenging conditions. Second, the dynamic user preference representations are stored and updated through a weighted graph. Finally, the system retrieves relevant preference-aware content to generate responses aligned with both immediate intent and long-term preferences. In addition, we construct the GDP dataset, the first benchmark dedicated to dynamic multimodal personalization, along with a complete data construction pipeline. Experimental results show that GDP achieves state-of-the-art performance on personalized multimodal tasks such as identity recognition and personalized question answering.
13
2026Findings of ACL 2026LongMP-BenchPersonalized Understanding
LongMP-Bench: A Benchmark for Multimodal Persona Understanding in Long-Term Dialogues
LongMP-Bench 构建长期、多模态、动态 persona 对话基准，系统暴露当前 MLLM 在视觉身份一致性、persona 演化追踪和长程记忆利用上的短板。
Understanding multimodal user personas over long-term dialogues is essential for personalized and human-like dialogue systems. In realistic interactions, user personas evolve over time and are expressed through both language and visual cues. However, existing benchmarks provide limited support for evaluating such dynamic and multimodal persona understanding, due to shallow persona coverage, weak visual consistency, and static settings. We introduce LongMP-Bench, a benchmark for evaluating models’ ability to understand, track, and utilize evolving multimodal user personas in long-term dialogues. We propose a scalable, multi-step data construction pipeline to synthesize extended multimodal interactions, followed by human refinement. The resulting dataset contains long-term conversations from 150 distinct users, each maintaining visual identity consistency while exhibiting progressive persona evolution. Based on LongMP-Bench, we define evaluation tasks for persona tracking, multimodal reasoning, and personalized response generation. Extensive experiments show that current multimodal large language models struggle with long-term persona consistency, persona shifts, and effective multimodal integration. Our data and code are available at https://github.com/skspass/LongMP-Bench.
14
2025arXivOnline-PVLMPersonalized Understanding
Advancing Personalized VLMs with Online Concept Learning
Personalized Visual Language Models (VLMs) are gaining increasing attention for their formidable ability in user-specific concepts aligned interactions (e.g., identifying a user's bike). Existing methods typically require the learning of separate embeddings for each new concept, which fails to support real-time adaptation during testing. This limitation becomes particularly pronounced in large-scale scenarios, where efficient retrieval of concept embeddings is not achievable. To alleviate this gap, we propose Online-PVLM, a framework for online concept learning by leveraging hyperbolic representations. Our approach makes a train-free paradigm for concept embeddings generation at test time, making the use of personalized VLMs both scalable and efficient. In addition, we develop OP-Eval, a comprehensive and large-scale benchmark comprising 1,292 concepts and over 30K high-quality instances with diverse question types, designed to rigorously assess online concept learning in realistic scenarios. Extensive experiments demonstrate the state-of-the-art performance of our proposed framework. Our source code and dataset will be made available.
Visual personalization is essential in user-facing AI systems such as smart homes and healthcare, where aligning model behavior with user-centric concepts is critical. However, recent large Vision-Language Models (VLMs), despite their broad applicability, remain underexplored in their ability to adapt to individual users. In this paper, we introduce MMPB, the first extensive benchmark for evaluating VLMs on personalization. MMPB comprises 10k image-query pairs and includes 111 personalizable concepts across four categories: humans, animals, objects, and characters, with the human category enriched with preference-grounded queries. We structure personalization into three main task types, each highlighting a different key property of VLMs. Using 23 widely used VLMs including both open- and closed-source models, we evaluate personalization performance via a three-stage protocol: concept injection, multi-turn dialogue, and personalized querying. Our findings indicate that most VLMs (including some closed-source models) struggle with personalization, particularly in maintaining consistency over dialogue, handling user preferences, and adapting to visual cues. Our analysis reveals that the challenges in VLM personalization (such as refusal behaviors and long-context forgetting) highlight substantial room for improvement. By identifying these limitations and offering a scalable benchmark, MMPB offers valuable insights and a solid foundation for future research toward truly personalized multi-modal AI.
16
2025arXivSLCPersonalized Understanding
Small-Large Collaboration: Training-efficient Concept Personalization for Large VLM using a Meta Personalized Small VLM
SLC 用一个可复用的 meta personalized small VLM 生成个性化概念线索，再由冻结的大 VLM 反思校验并回答问题，以低训练成本把概念级个性化扩展到开源和闭源大模型。
这篇工作的价值在于把大模型个性化从“直接训练大模型”改写成一个可审计的协作协议：小模型低成本吸收私有概念，大模型负责校验、推理和语言生成。它天然兼容闭源大 VLM，也为产品中的概念注册、概念检测、误检过滤和回答生成提供了清晰模块边界。局限是 personalization 仍主要停留在概念级识别与问答，长期记忆、用户状态演化、推理延迟和 API 成本还需要后续系统补齐。
Related work: SLC 与 Online-PVLM、Ego 都在探索低成本概念注入，但路线不同：SLC 训练一个 meta personalized small VLM 产生可校验的结构化 cue，Online-PVLM 维护可缓存的概念 embedding，Ego 则直接抽取 LVLM 内部视觉 token 作为概念记忆。相较 CoViP、RePIC 这类更重视 personalized captioning 和多概念 grounding 的工作，SLC 更强调训练效率与闭源大模型兼容性；相较 TAME、ATMBench、PersonaVLM、PEARL，它还不是完整长期记忆助手，但可以作为这些系统的前端概念感知层。
Abstract
Personalizing Vision-Language Models (VLMs) to transform them into daily assistants has emerged as a trending research direction. However, leading companies like OpenAI continue to increase model size and develop complex designs such as the chain of thought (CoT). While large VLMs are proficient in complex multi-modal understanding, their high training costs and limited access via paid APIs restrict direct personalization. Conversely, small VLMs are easily personalized and freely available, but they lack sufficient reasoning capabilities. Inspired by this, we propose a novel collaborative framework named Small-Large Collaboration (SLC) for large VLM personalization, where the small VLM is responsible for generating personalized information, while the large model integrates this personalized information to deliver accurate responses. To effectively incorporate personalized information, we develop a test-time reflection strategy, preventing the potential hallucination of the small VLM. Since SLC only needs to train a meta personalized small VLM for the large VLMs, the overall process is training-efficient. To the best of our knowledge, this is the first training-efficient framework that supports both open-source and closed-source large VLMs, enabling broader real-world personalized applications. We conduct thorough experiments across various benchmarks and large VLMs to demonstrate the effectiveness of the proposed SLC framework. The code will be released at https://github.com/Hhankyangg/SLC.
17
2025NeurIPS 2025RePICPersonalized Understanding
RePIC: Reinforced Post-Training for Personalizing Multi-Modal Language Models
Recent multi-modal large language models (MLLMs) often struggle to generate personalized image captions, even when trained on high-quality captions. In this work, we observe that such limitations persist in existing post-training-based MLLM personalization methods. Specifically, despite being post-tuned with large-scale caption data through supervised fine-tuning (SFT), these models frequently fail to produce faithful descriptions in real-world scenarios, such as multi-concept image captioning. However, acquiring large-scale, high-quality captions for such complex settings is both costly and difficult. To address the data-centric nature of SFT, we propose a reinforcement learning (RL)-based post-training framework. To the best of our knowledge, this is the first RL-based approach to post-train MLLMs for personalized image captioning. Our method significantly enhances both visual recognition and personalized generation capabilities of MLLMs, and consistently outperforms existing SFT-based baselines, especially in the challenging multi-concept image captioning task. Project page: https://github.com/oyt9306/RePIC
18
2025ICCV 2025R2PPersonalized Understanding
Training-Free Personalization via Retrieval and Reasoning on Fingerprints
Vision Language Models (VLMs) have lead to major improvements in multimodal reasoning, yet they still struggle to understand user-specific concepts. Existing personalization methods address this limitation but heavily rely on training procedures, that can be either costly or unpleasant to individual users. We depart from existing work, and for the first time explore the training-free setting in the context of personalization. We propose a novel method, Retrieval and Reasoning for Personalization (R2P), leveraging internal knowledge of VLMs. First, we leverage VLMs to extract the concept fingerprint, i.e., key attributes uniquely defining the concept within its semantic class. When a query arrives, the most similar fingerprints are retrieved and scored via chain of thought reasoning. To reduce the risk of hallucinations, the scores are validated through cross-modal verification at the attribute level: in case of a discrepancy between the scores, R2P refines the concept association via pairwise multimodal matching, where the retrieved fingerprints and their images are directly compared with the query. We validate R2P on two publicly available benchmarks and a newly introduced dataset, Personal Concepts with Visual Ambiguity (PerVA), for concept identification highlighting challenges in visual ambiguity. R2P consistently outperforms state-of-the-art approaches on various downstream tasks across all benchmarks. Code and data are available at the project page: Training-Free Personalization.
19
2025ICCV 2025PVChatPersonalized Understanding
PVChat: Personalized Video Chat with One-Shot Learning
Video large language models (ViLLMs) excel in general video understanding, e.g., recognizing activities like talking and eating, but struggle with identity-aware comprehension, such as "Wilson is receiving chemotherapy" or "Tom is discussing with Sarah", limiting their applicability in smart healthcare and smart home environments. To address this limitation, we propose a one-shot learning framework PVChat, the first personalized ViLLM that enables subject-aware question answering (QA) from a single video for each subject. Our approach optimizes a Mixture-of-Heads (MoH) enhanced ViLLM on a synthetically augmented video-QA dataset, leveraging a progressive image-to-video learning strategy. Specifically, we introduce an automated augmentation pipeline that synthesizes identity-preserving positive samples and retrieves hard negatives from existing video corpora, generating a diverse training dataset with four QA types: existence, appearance, action, and location inquiries. To enhance subject-specific learning, we propose a ReLU Routing MoH attention mechanism, alongside two novel objectives: (1) Smooth Proximity Regularization for progressive learning through exponential distance scaling and (2) Head Activation Enhancement for balanced attention routing. Finally, we adopt a two-stage training strategy, transitioning from image pre-training to video fine-tuning, enabling a gradual learning process from static attributes to dynamic representations. We evaluate PVChat on diverse datasets covering medical scenarios, TV series, anime, and real-world footage, demonstrating its superiority in personalized feature understanding after learning from a single video, compared to state-of-the-art ViLLMs.
20
2025ICLRPRPGPersonalized Understanding
Personalized Representation from Personalized Generation
Modern vision models excel at general purpose downstream tasks. It is unclear, however, how they may be used for personalized vision tasks, which are both fine-grained and data-scarce. Recent works have successfully applied synthetic data to general-purpose representation learning, while advances in T2I diffusion models have enabled the generation of personalized images from just a few real examples. Here, we explore a potential connection between these ideas, and formalize the challenge of using personalized synthetic data to learn personalized representations, which encode knowledge about an object of interest and may be flexibly applied to any downstream task relating to the target object. We introduce an evaluation suite for this challenge, including reformulations of two existing datasets and a novel dataset explicitly constructed for this purpose, and propose a contrastive learning approach that makes creative use of image generators. We show that our method improves personalized representation learning for diverse downstream tasks, from recognition to segmentation, and analyze characteristics of image generation approaches that are key to this gain.
21
2024arXivRAPPersonalized Understanding
Remember, Retrieve and Generate: Understanding Infinite Visual Concepts as Your Personalized Assistant
Large Multimodal Models (LMMs) have demonstrated impressive capabilities in recognizing and understanding visual concepts. However, the open-world nature of visual concepts poses significant challenges for LMMs, as a model’s understanding is inherently limited to the concepts learned from its training data. Human users often wish to customize their models by introducing personal concepts through small samples. However, existing methods for learning new visual concepts typically require additional training, which is both time-consuming and labor-intensive. In this paper, we propose Retrieval-Augmented Personalization (RAP), a training-free method that enables models to understand infinite visual concepts by retrieving relevant concept information from a user-defined database. RAP consists of three stages: Remember, Retrieve, and Generate. The model first remembers user-provided concepts by storing images and descriptions in a database. Then, it retrieves relevant concepts based on the input image and query. Finally, it generates personalized responses using retrieved information. We further construct a personalized multimodal instruction dataset and instantiate RAP on LLaVA and Phi-3-Vision. Experiments on personalized captioning, question answering, recognition, and grounding show that RAP achieves strong performance with only one image per concept and supports flexible real-time updates.
22
2024NeurIPS 2024YoLLaVAPersonalized Understanding
Yo'LLaVA: Your Personalized Language and Vision Assistant
Large Multimodal Models (LMMs) have shown remarkable capabilities across a variety of tasks (e.g., image captioning, visual question answering). While broad, their knowledge remains generic (e.g., recognizing a dog), and they are unable to handle personalized subjects (e.g., recognizing a user's pet dog). Human reasoning, in contrast, typically operates within the context of specific subjects in our surroundings. For example, one might ask, "What should I buy for my dog's birthday?"; as opposed to a generic inquiry about "What should I buy for a dog's birthday?". Similarly, when looking at a friend's image, the interest lies in seeing their activities (e.g., "my friend is holding a cat"), rather than merely observing generic human actions (e.g., "a man is holding a cat"). In this paper, we introduce the novel task of personalizing LMMs, so that they can have conversations about a specific subject. We propose Yo'LLaVA, which learns to embed a personalized subject into a set of latent tokens given a handful of example images of the subject. Our qualitative and quantitative analyses reveal that Yo'LLaVA can learn the concept more efficiently using fewer tokens and more effectively encode the visual attributes compared to strong prompting baselines (e.g., LLaVA).
23
2024ECCV 2024MyVLMPersonalized Understanding
MyVLM: Personalizing VLMs for User-Specific Queries
Recent large-scale vision-language models (VLMs) have demonstrated remarkable capabilities in understanding and generating textual descriptions for visual content. However, these models lack an understanding of user-specific concepts. In this work, we take a first step toward the personalization of VLMs, enabling them to learn and reason over user-provided concepts. For example, we explore whether these models can learn to recognize you in an image and communicate what you are doing, tailoring the model to reflect your personal experiences and relationships. To effectively recognize a variety of user-specific concepts, we augment the VLM with external concept heads that function as toggles for the model, enabling the VLM to identify the presence of specific target concepts in a given image. Having recognized the concept, we learn a new concept embedding in the intermediate feature space of the VLM. This embedding is tasked with guiding the language model to naturally integrate the target concept in its generated response. We apply our technique to BLIP-2 and LLaVA for personalized image captioning and further show its applicability for personalized visual question-answering. Our experiments demonstrate our ability to generalize to unseen images of learned concepts while preserving the model behavior on unrelated inputs.
