2025 X-AGI & The 18th China-R Conference

过去十年，多模态生成经历了从 VAE/GAN ，自回归模型（AR Models），到扩散模型（Diffusion Models）的演化与放大（Scaling）；近两年多模态生成技术又与大语言模型深度耦合，形成多模态理解与多模态生成相结合的智能系统。在多模态智能迅速演进的进程中，高质量的数据和可扩展的建模方法成为了模型进步的两个重要方面。本报告以文-图多模态场景为例子，首先将从数据和建模方法的角度出发，系统梳理多模态生成模型的历史演进。我们将涵盖自回归模型，扩散模型和单步/少步生成模型等主流模型架构，并比较各个模型方案的利弊。同时我们将简要梳理介绍对生成模型友好的现有开源数据集；其次，我们将重点讨论一个研究热点问题，即多模态生成与多模态理解的关系，囊括了多模态生成理解一体化的一些近期工作以及报告者自身的一些思考，如生成理解之间的抑制与促进关系，生成理解模型的放大策略等。最后，我们将对多模态生成模型的未来演化方向做一个展望，并尝试归纳出一些重要的研究问题和应用场景。

Diffusion models, which generate new data instances by learning to reverse a Markov diffusion process from noise, have become a cornerstone in contemporary generative modeling. While their practical power has now been widely recognized, the theoretical underpinnings remain underdeveloped. Particularly, despite the recent surge of interest in accelerating sampling speed, convergence theory for these acceleration techniques remains limited. In this talk, I will first introduce an acceleration sampling scheme for stochastic samplers that provably improves the iteration complexity under minimal assumptions. The second part focuses on diffusion-based language models, whose ability to generate tokens in parallel significantly accelerates sampling relative to traditional autoregressive methods. Adopting an information-theoretic lens, we establish a sharp convergence theory for diffusion language models, thereby providing the first rigorous justification of both their efficiency and fundamental limits.

本报告系统性地呈现了我们在通用多模态大模型构建中的全链路探索。在感知基础层面，我们构建了高质量的通用图文对齐数据集ALLaVA-4V、长尾视觉知识数据集以及专业医学图像数据集（HuatuoGPT-Vision），并开发了支持长上下文的多模态模型（如LongLLaVA），能够有效处理多图与长视频（基于MileBench评测）。同时，我们积极拓展全模态感知维度，融合语音（EchoX与SoundWave）、动作理解（MotionLLM）、中医诊疗知识（时珍GPT）以及脑电信号（WaveMind），构建了更为立体的信息理解体系。在认知与推理层面，我们重点攻坚复杂推理能力，在视频理解（Video-R1）、医疗多模态推理与语音推理等任务上取得显著进展。在交互与生成前沿，我们开发了基于大模型的图像生成与编辑框架Janus-4o、医疗视频生成系统MedGen、微观世界模拟器MicroVerse，并探索VR/AR交互应用，全面提升自然交互体验。最终，我们前瞻布局物理世界孪生与智能体自进化，通过环境交互模拟，推动Agent在持续反馈中实现自主演进，为迈向具备环境适应与进化能力的通用智能奠定坚实基础。

本报告将系统介绍在昇腾平台上开展的多模态理解大模型训练与算法探索。内容涵盖昇腾亲和的视觉编码器设计、离散化语音编码器设计，以及面向大规模训练的数据格式规范与高效数据处理流程。在模型训练方面，报告将重点探讨基于数千卡集群的高效多模态对齐方法，涉及模态对齐的范式选择与具体实现。通过这些系统性的研究与实践，我们总结了多模态大模型在昇腾平台上的关键Know-how，为实现高效、稳定、可扩展的多模态理解模型训练提供参考。

现有智能体系统缺乏统一的理论框架和自我进化能力。本论文提出了一个受大脑启发的基础智能体架构，包含认知、记忆、感知、行动等核心模块，并探讨了智能体的自我优化机制和多代理协作模式。论文还分析了智能体系统面临的安全挑战，为构建智能且安全的智能体系统提供了系统性的理论指导。

现有的轨迹级强化学习方法在训练智能体时忽略了工具调用后产生的高熵token分布，无法充分探索步骤级工具使用行为。ARPO通过熵驱动的自适应采样机制，在高熵工具使用步骤中动态分支采样，结合优势归因估计更好地学习工具交互行为。实验显示ARPO在13个基准测试中超越传统方法，且仅需一半工具调用预算。

强化学习与大模型的发展演进，以及需要面对的算法挑战。AReaL通过Agent优先的接口设计，以及全异步强化学习算法，让用户改动极少的代码就可以实现3-5x的智能体强化学习训练加速。

当前的自主 AI Agent 在实际应用中面临两大核心挑战：与环境的实时交互以及从经验中学习。本文探讨这两大难题并提出相应解决方案。针对传统 Agent 在语音和图形界面（GUI）交互中的高延迟问题，我们提出了“交互式 ReAct”框架，它通过融合快慢思考模式，允许 Agent 边听边想、边想边说，从而克服传统“观察-思考-行动”循环的局限性；并通过强化学习（RL）训练 VLA 模型提升 GUI 操作的精确性和实时交互能力。为弥补 Agent 从经验中学习机制的缺失，我们探索了四种方法：利用知识库快速应用新经验；借助长上下文的内置注意力机制形成记忆；通过 RL 后训练提升任务熟练度；以及利用代码生成能力自动创建工具以加速重复性任务。本文旨在为构建能高效、自主地完成现实世界复杂任务的下一代 AI Agent 提供一套综合性解决方案。

While looped transformers (termed as Looped-Attn) often outperform standard transformers (termed as Single-Attn) on complex reasoning tasks, the theoretical basis for this advantage remains underexplored. In this paper, we explain this phenomenon through the lens of loss landscape geometry, inspired by empirical observations of their distinct dynamics at both sample and Hessian levels. To formalize this, we extend the River-Valley landscape model by distinguishing between U-shaped valleys (flat) and V-shaped valleys (steep). Based on empirical observations, we conjecture that the recursive architecture of Looped-Attn induces a landscape-level inductive bias towards River-V-Valley. Theoretical derivations based on this inductive bias guarantee a better loss convergence along the river due to valley hopping, and further encourage learning about complex patterns compared to the River-U-Valley induced by Single-Attn. Building on this insight, we propose SHIFT (Staged HIerarchical Framework for Progressive Training), a staged training framework that accelerates the training process of Looped-Attn while achieving comparable performances.

随机梯度下降算法是求解机器学习问题中的常见算法。在高维学习问题中，随机梯度下降算法的迭代次数往往低于模型参数量，算法对于模型的产生隐式正则效应是模型具有良好泛化的主要原因。本次讲座，我们将研究随机梯度下降算法在不同学习情境下求解简单模型的泛化性能，并进行定量比较。我们将分别讨论算法在不同学习尺度（即样本数与问题维度不同依赖关系）与协变量偏移条件下的学习效率，尝试理解算法对于学习问题的适应性与涌现发生的条件。理论将启发设计内存节约大模型训练算法，在GPT2等标准测试平台取得更优性能。

在当前大模型对齐研究中，一个备受关注的现象是弱到强泛化（Weak-to-Strong Generalization, W2SG），即通过弱教师模型生成伪标签，指导强学生模型进行训练，从而实现学生在目标任务中反超教师的现象。尽管这一现象已被实证观察到，但其理论机理仍未被充分揭示。本报告围绕W2SG的理论分析展开，重点在于用Bregman散度下的广义偏差-方差分解刻画学生与教师之间的风险差异，首次在不依赖假设空间凸性这一强假设的前提下，推导出基于"预测不匹配"的W2SG不等式。我们进一步理论证明对于容量足够大的学生模型，W2SG现象更有可能出现。与此同时，我们发现适度的正则化或早停对于避免学生模型过度拟合教师标签是必要的。此外，在W2SG损失函数选择方面，我们理论上比较了标准交叉熵与反向交叉熵在W2SG场景下的表现，指出后者在面对教师预测不确定性时更加稳健。最后，我们通过实证分析验证了上述理论发现，包括学生模型容量对W2SG的影响，以及来自多个教师模型的平均监督对提升学生性能的作用。

Optimizing large-scale nonconvex problems, common in machine learning, demands balancing rapid convergence with computational efficiency. First-order (FO) stochastic methods like SVRG provide fast convergence and good generalization but incur high costs due to full-batch gradients in large models. Conversely, zeroth-order (ZO) algorithms reduce this burden using estimated gradients, yet their slow convergence in high-dimensional settings limits practicality. We introduce VAMO (VAriance-reduced Mixed-gradient Optimizer), a stochastic variance-reduced method combining FO mini-batch gradients with lightweight ZO finite-difference probes under an SVRG-style framework. VAMO's hybrid design uses a two-point ZO estimator to achieve a dimension-agnostic convergence rate of O(1/T + 1/b), where T is the number of iterations and b is the batch-size, surpassing the dimension-dependent slowdown of purely ZO methods and significantly improving over SGD's O(1/√T) rate. Additionally, we propose a multi-point ZO variant that mitigates the O(1/b) error by adjusting number of estimation points to balance convergence and cost, making it ideal for a whole range of computationally constrained scenarios. Experiments including traditional neural network training and LLM finetuning show VAMO outperforms established FO and ZO methods, offering a faster, more flexible option for improved efficiency.

从 ChatBI 到 Data Agent，分析工具正从交互式报表向自主智能分析演进。Data Agent 利用 LLM 实现深度的知识整合、自主分析推理和动态决策，相比传统分析工具在效率与深度上具备明显优势。本议题将介绍 Data Agent 的工程架构与调优方法论，并结合零售等场景的归因分析实践，展示如何实现性能优化。最后，将展望 Data Agent 未来的技术发展。

拥有多个行业头部品牌大模型应用落地经验专家，为你分享在当下模型能力以及市场背景下，AI再企业内最好的应用形态以及各头部企业的成功落地案例

探索AI如何赋能教育，从智能教学到个性化学习，分享行业落地案例与实践经验，助力教育创新升级。

Predicting molecular ground-state conformation (i.e., energy-minimized conformation) is crucial for many chemical applications such as molecular docking and property prediction. Classic energy-based simulation is time-consuming when solving this problem, while existing learning-based methods have advantages in computational efficiency but sacrifice accuracy and interpretability. In this work, we propose a novel and effective method to bridge the energy-based simulation and the learning-based strategy, which designs and learns a Wasserstein gradient flow-driven SE(3)-Transformer, called WGFormer, for ground-state conformation prediction. Specifically, our method tackles this task within an auto-encoding framework, which encodes low-quality conformations by the proposed WGFormer and decodes corresponding ground-state conformations by an MLP. The architecture of WGFormer corresponds to Wasserstein gradient flows -- it optimizes conformations by minimizing an energy function defined on the latent mixture models of atoms, thereby significantly improving performance and interpretability. Extensive experiments demonstrate that our method consistently outperforms state-of-the-art competitors, providing a new and insightful paradigm to predict ground-state conformation.

在生命科学与药物发现领域，高成本、长周期、高失败率是长期存在的核心痛点。自监督学习正通过挖掘海量、未标注的生物医学大数据，为解决这些痛点带来革命性突破。本报告将深入探讨自监督学习如何通过学习基因序列、分子结构和蛋白-分子相互作用的内在规律，显著加速靶点发现、药物筛选和性质预测等关键环节。我们将通过前沿案例证明，自监督学习不仅是技术工具的升级，更是驱动从“假设驱动”向“数据驱动”科研范式转变的核心引擎。

Artificial intelligence (AI) has shown unprecedented capability as a computational approach to solving scientific tasks. Nevertheless, scientific tasks hold their unique challenge of the enormous problem space, while data generation approaches are limited in accuracy, scale, system coverage and conditions. This talk will introduce recent advances that go beyond the data limitation by using physical laws, the blessing from the science side. We demonstrate a series of effective approaches for molecular science across scales, starting from improving the accuracy and generalizability of electronic structure prediction beyond data using quantum mechanics equation, which in turn improves energy and microscopic property prediction. We then show that leveraging the energy function corrects data bias for thermodynamic ensemble modeling, which leads to accurate prediction of observable design properties.

近年来，AI基座模型驱动的分子设计在生物医药、能源化工、功能材料等等领域取得了非常多显著的进展，Uni-Mol作为首个3D分子预训练大模型，也在多领域赋能AI for Science，整体围绕以下几点进行介绍： 1. 分子基座模型的现状和发展； 2. 基于Uni-Mol的分子设计整体框架； 3. 案例介绍（工业界、学术界等等）；

物体操纵能力的习得一般源自两种途径，即从示教中模仿或从交互中历练。从示例中学习是一种能力快速增长的有效方式，但其有限的探索空间所形成的能力瓶颈却制约着对外界环境特性的深刻理解与操纵行为的有效规划；而同环境的交互能够提供可拓展的探索空间和丰富的信息反馈，但是其高昂的探索成本却阻碍着操纵能力的持续增长。在本次报告中，我将以“观察-协作-反思”的技术路径，思考如何从示教学习出发，以矫错为核心，将形成的基础能力逐步拓展到交互学习场景中，以高效支持物体操纵能力的持续提升，进而形成示教与交互协同支撑的操纵能力习得。

We present Image-GOal Representations (IGOR), aiming to learn a unified, semantically consistent action space across human and various robots. Through this unified latent action space, IGOR enables knowledge transfer among large-scale robot and human activity data. We achieve this by compressing visual changes between an initial image and its goal state into latent actions. IGOR allows us to generate latent action labels for internet-scale video data. This unified latent action space enables the training of foundation policy and world models across a wide variety of tasks performed by both robots and humans.

脑认知科学、脑机接口（BCI）与具身智能的融合，正开启人机交互的革命性前沿。其核心机制在于构建了一个“感知-行动-反馈”的闭环：脑认知科学为BCI提供了大脑功能的“地图”；BCI技术负责“读取”大脑意图（如运动指令），并将其解码以控制具身智能体（如机械臂、外骨骼）；同时，BCI还能“写回”智能体从环境中获取的感官信息（如触觉），使大脑得以学习和适应，形成高效的神经可塑性。在应用上，这一联动已帮助瘫痪患者通过意念控制智能假肢并重获触觉。尽管目前仍面临信号保真度、长期稳定性及数据隐私等伦理挑战，但三者的结合预示着“人机共生”的未来，将在医疗康复、虚拟现实和认知增强等领域产生深远影响。

随着具身智能技术向智能抓取、精密制造、医疗康复等领域推广渗透，触觉传感器在机器人领域的需求愈发增加，磁触觉传感器作为一种高灵敏度、宽量程、柔性可拉伸的触觉传感器，为机器人提供了理想解决方案，成为机器人感知领域的研究热点。

本报告介绍了磁触觉传感器的感知机理、制备方法及数据处理方法，通过结构优化与算法优化，实现了高灵敏度力感知与位置识别、磁性特征识别等任务，显著提升了机器人对不同特性物体的抓取适应性、安全性与稳定性。在此基础上，结合视触觉传感器提出了多模态触觉传感器方案，用于增强视触觉传感器感知精度和接近觉感知能力，以提高机器人感知操作能力。

电子病历包含患者诊疗过程中的丰富医学信息，通过自动化信息提取并对术语进行标准化，转化为规范的结构化数据，可以为医学研究与临床自动化提供丰富的数据支持、大幅降低人力成本、提高科研数据采集与医院管理效率。然而，真实的临床病历用语高度个性化、存在大量简称和非标准表达，电子病历的结构化、标准化对自然语言处理工具和基础设施都是巨大挑战。本报告将介绍利用大模型中的庞大背景知识，实现电子病历的准确理解，并在其术语抽取的基础上，利用大模型实现大规模数据驱动的医学本体自动化建设。

药物设计和开发是一个既漫长又昂贵的过程，涉及从分子发现到临床试验的多个复杂步骤。人工智能（AI）技术展示了巨大的潜力，可以显著加速这一过程并降低成本。在药物发现的初期阶段，目标是识别具备理想药理特性的分子。本报告将深入探讨最新的药物设计方法，包括连续空间深度生成模型和离散空间药物设计路径搜索算法。这些先进的AI工具能够高效地探索化学空间，预测新化合物的活性和安全性，并优化候选药物的设计，以满足特定的治疗需求。进一步讲，在药物开发的后期阶段，重点转向了临床试验，这是评估药物对人体安全性和有效性的重要环节。为了提高临床试验的成功率和效率，本报告将介绍一系列最新的可信赖的方法，包括可解释性、不确定性感知的临床试验设计与预测技术。这些方法不仅能够模拟真实的临床试验过程，还能帮助科学家更好地理解潜在的风险和收益，从而做出更加明智的决策。

Accelerated Magnetic Resonance Imaging (MRI) is essential for clinical practice but introduces uncertainty into brain volumetry. We propose a framework combining the semiparametric modelling with Multivariate Conformal Prediction (MCP) to generate statistically-guaranteed prediction sets for brain regions volumes.Our model-agnostic approach provides rigorous uncertainty bounds without making distributional assumptions. Integrating the method with the reconstruction and segmentation algorithms, a pipeline is developed to construct prediction sets directly from the k-space downsampled MRI measurements. The pipeline is validated on the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset, demonstrating its effectiveness. This work provides a practical tool to enhance the reliability of volumetric biomarkers from accelerated MRI scans.

The emergence of diverse pathological foundation models has empowered computational pathology tasks, including tumor classification, biomarker prediction, and gene expression prediction. However, variations in model architecture and data sources lead to inconsistent downstream performance and complicate centralized training. Specifically, the lack of data sharing makes retraining foundation models with pooled data infeasible. Alternatively, the release of model parameters enables combining multiple models during fine-tuning. Inspired by the meta-analysis method, we propose the Meta-Encoder framework, which integrates features from multiple foundation models to generate a comprehensive representation, improving downstream fine-tuning task performance. Comparative experiments demonstrate that Meta-Encoder is more effective than individual foundation models, with its strengths more pronounced in handling complex tasks. While single models may perform sufficiently well for simple tasks, Meta-Encoder can match or even surpass the best-performing single model, alleviating concerns over model selection. Moderately challenging tasks benefit from Meta-Encoder's concatenation or self-attention strategies, with the latter demonstrating superior performance in more challenging scenarios. For highly complex tasks, such as high-dimensional gene expression prediction, self-attention proves to be the most effective Meta-Encoder strategy, balancing feature integration and computational efficiency. For three patch-level spatial gene expression prediction tasks (HEST-Benchmark, CRC-inhouse, and Her2ST), the self-attentionstrategy improved the Pearson correlation by 38.58%, 26.06%, and 20.39%, respectively, compared to the average performance of three patch-level single models. Similarly, for the TCGA-BRCA, TCGA-NSCLC, and TCGA-CRC WSI-level bulk gene expression prediction tasks, the Pearson correlation increased by 14.36%, 9.27%, and 42.55%, respectively, compared to the average performance of two WSI-level single models. By leveraging multiple pathological foundation models using Meta-Encoder, it can further improve molecular characterization in pathology images to advance precision oncology.

随着AI/ML等前沿技术的快速演进，医药研发正迎来数智化转型的重要机遇。复星医药围绕AI在信息洞察、决策支持、药物发现、临床开发、上市营销等全生命周期中的应用，系统构建了以PharmAID决策智能体为代表的数智化平台，推动从数据资产管理到智能决策的深度融合。通过分享平台架构、场景落地及价值评估等方面的实践经验，助力行业共同探索AI赋能医药研发的创新路径与协同生态。

Clinical study protocols serve as foundational documents for trial execution but typically exist in non-machine-readable formats, creating barriers to system integration and data exchange. Traditional protocol development is labor-intensive, error-prone, and constrained by tight timelines. This presentation demonstrates how artificial intelligence can revolutionize protocol development through digitalization and clinical content reuse. AI-driven implementation increases quality in a tight timeline while maintaining regulatory compliance. Digital protocols enable seamless integration with electronic systems and automate document generation, reducing manual data entry and minimizing transcription errors. This transformative technology addresses critical industry challenges, and establishes foundations for accelerated drug development while improving patient outcomes through more efficient clinical trial execution.

我国肝病患者约4亿，占总人口28%，严重威胁人民健康。目前肝病的精确诊疗对医生的知识广度和深度要求极高，往往需多科室跨专业的综合性会诊。但仍面临以下挑战：（1）肝病类型众多，早期表现隐匿，其症状和体征特异性差，病变诊断严重依赖经验，缺乏结构化知识和系统量化标准，易导致误诊和漏诊；（2）肝病病程长，可同时合并全身多系统病变，诊断需多学科交叉知识的综合性判别，具有不同背景的临床专家治疗决策冲突点多；（3）不同级别医疗机构诊疗水平参差不齐，严重影响疑难复杂肝病的决策精度和效率，难以为肝病患者提供同质及时的诊疗方案。

为此，我们团队提出基于LLM的多智能体激活图推理协同模型MAGIC应对这一挑战。通过多尺度分析增强肝脏疾病认知，包括相似病例研究、异常指标识别和知识图谱分析。在模拟临床渐进式诊断过程中，该模型利用多智能体辩论结果动态调整图推理中的关键节点与关系权重，提升预诊断准确性。同时，模型通过临床指南对预诊断结论进行验证，确保其符合既定医疗标准，最终生成可靠诊断结果。我们提出的方法一举将疑难肝病的 AI 诊断准确率提升到 94.5%，多中心验证仍保持 91.6%，显著领先现有模型。本研究已经发表于人工智能顶刊Information Fusion中，IF＝20.4（2024），并作为核心辩论技术开发了一套人机融合疑难肝病会诊系统，并在8家医院实现了示范应用。

Applying artificial intelligence to the design, operation, analysis, and interpretation of clinical research will help improve efficiency. In this report, we will review the current applications of artificial intelligence, especially large language models, in the field of enhancing clinical research efficiency. We will also introduce our work in areas such as protocol review and virtual digital twins.

Clinical trial emulation has emerged as an important approach in real-world drug research, enabling investigators to replicate the design and analysis of randomized controlled trials using observational data. However, traditional emulation typically relies on expert knowledge and extensive literature review to construct a hypothetical trial, a process that is often time-consuming and constrained by limited scalability. In this work, we develop a domain-specific large language model (LLM) based AI agent trained with advanced direct preference optimization techniques to facilitate semi-automated trial emulation. Given the drugs or interventions of interest, the agent generates a complete hypothetical trial design, including detailed inclusion and exclusion criteria, treatment allocation strategies, follow-up protocols, and outcome definitions. Building on this, the system leverages LLM to align the hypothetical trial with the user’s diverse private datasets, producing tailored data extraction schemes that enable efficient retrieval of relevant patient cohorts and variables. This AI agent significantly improves the efficiency and reduces the cost of conducting emulation studies for researchers with heterogeneous, privately held data sources, expanding the accessibility and scalability of real-world evidence generation.