I am Pengcheng Huang (黄鹏程), a Ph.D. student in Computer Science at Northeastern University, advised by Prof. Zhenghao Liu. I received my M.S. degree in Computer Science from Northeastern University, where I was advised by Prof. Tong Xiao. My research focuses on large language models (LLMs), foundation models, and LLM agents. You can find my Google Scholar profile here citations 158 .
🔥 News
2026-10: 🏅 Selected for the CAST Young Talent Cultivation Program for Doctoral Students (中国科协青年科技人才培育工程博士生专项计划) by the China Association for Science and Technology (CAST)2026-08: 🎉 Two papers accepted to EMNLP 2026 (Main)2026-04: 🎉 Three papers accepted to ACL 2026 (2 Main, 1 Findings)2026-01: 🎉 One paper accepted to ICLR 20262025-09: 🎉 One paper accepted to NeurIPS 20252025-09: 🎉 Three papers accepted to EMNLP 2025, with one selected for an oral presentation2025-08: 🎉 We released UltraRAG 2.0, an low-code framework for building complex RAG systems
2025-06: 🎉 One paper received the Highlight Poster Award 🏆 at YSSNLP 20252024-05: 🎉 One paper accepted to CCL 2024
💻 Internships
- 2025.3 - now, THUNLP, Beijing. Visiting Ph.D. Student.
- 2025.3 - now, ModelBest, Beijing. Working on improving foundation model general agentic capabilities, e.g., OpenClaw, Claude Code, etc.
- 2024.10 - 2025.3, Language Technologies Institute (LTI), Carnegie Mellon University, Online.
- 2024.05 - 2024.09, Taobao-Tmall Search (Alibaba Group), Hangzhou.
- 2023.12 - 2024.05, ByteDance, Douyin E-commerce, Beijing.
📝 Publications
You can also find my articles on my Google Scholar profile. Note: * indicates equal contribution; \† indicates the corresponding author.
Empirical Analysis of Decoding Biases in Masked Diffusion Models
Pengcheng Huang, Tianming Liu, Zhenghao Liu†, Yukun Yan, Shuo Wang, Tong Xiao, Zulong Chen, Maosong Sun
ACL Anthology · arXiv · Code · Project Page
- This work reveals two systematic decoding biases in Masked Diffusion Models — rigid boundary bias and trivial token bias — and proposes Uncode (a.k.a. PC-Sampler, the name of its arXiv v1), a lightweight calibration framework that reshapes unmasking priorities via positional and semantic priors, outperforming existing decoding strategies by over 7% on average and achieving performance comparable to autoregressive models.
ParamMute: Suppressing Knowledge-Critical FFNs for Faithful Retrieval-Augmented Generation
Pengcheng Huang, Zhenghao Liu†, Yukun Yan, Haiyan Zhao, Xiaoyuan Yi, Hao Chen, Zhiyuan Liu, Maosong Sun, Tong Xiao, Ge Yu, Chenyan Xiong
NeurIPS · arXiv · Code · Project Page
- This work presents ParamMute (a.k.a. PIP-KAG, the title of its early arXiv versions), a novel framework for enhancing the faithfulness of Retrieval-Augmented Generation. By identifying and suppressing unfaithfulness-associated feed-forward networks (FFNs), and incorporating a knowledge preference adaptation module, ParamMute effectively steers language models to better leverage retrieved evidence, paving the way for more reliable and trustworthy RAG systems.
Empowering Edge Agents: A Systematic Analysis of Edge-Cloud Collaboration in LLM Agent Execution
Pengcheng Huang, Zhenghao Liu†, Shixin Ren, Shi Yu, Sijia Yao, Yukun Yan†, Chaojun Xiao†, Maosong Sun
arXiv · Code · Project Page · Dataset
- This work introduces EC²-Bench (EC2-Bench), a benchmark for edge–cloud collaboration in long-horizon LLM agents with 128 executable digital-workspace tasks (100 privacy-annotated), jointly measuring task performance, resource usage and privacy exposure. It shows that collaboration balances the three, that stronger edge models close the performance gap but not the privacy gap, and that the preferred collaboration mode shifts from external routing toward edge-controlled cloud advisory as edge capability grows.
ExpandR: Teaching Dense Retrievers Beyond Queries with LLM Guidance
Sijia Yao*, Pengcheng Huang*, Zhenghao Liu†, Yu Gu, Yukun Yan, Shi Yu, Ge Yu
- This work introduces ExpandR, a unified framework that jointly trains a large language model (LLM) and a dense retriever by having the LLM generate rich query expansions and optimizing both expansion generation (via Direct Preference Optimization with combined rewards for retrieval utility and consistency) and retriever ranking performance at the same time, resulting in >5% boost over strong baselines on retrieval benchmarks.
Chunks as Arms: Multi-Armed Bandit-Guided Sampling for Long-Context LLM Preference Optimization
Shaohua Duan*, Pengcheng Huang*, Xinze Li*, Zhenghao Liu†, Xiaoyuan Yi, Yukun Yan, Shuo Wang, Yu Gu, Ge Yu, Maosong Sun
ACL Anthology · arXiv · Code
- This work proposes LongMab, a framework that treats divided context chunks as arms of a Multi‑Armed Bandit and leverages a UCB‑guided rollout strategy to progressively identify the most informative chunk combinations for sampling high‑quality, diverse responses, which are then used to construct effective preference pairs for DPO training, achieving over 4% average improvement on long‑context reasoning benchmarks across both Llama and Qwen.
Autoencoding-free context compression for llms via contextual semantic anchors
Xin Liu, Runsong Zhao, Pengcheng Huang, Xinyu Liu, Junyi Xiao, Chunyang Xiao, Tong Xiao†, Shengxiang Gao, Zhengtao Yu, Jingbo Zhu
- This paper introduces Semantic-Anchor Compression (SAC), a novel method that achieves efficient context compression for LLMs by directly selecting and aggregating information into contextual “anchor tokens,” bypassing the limitations of traditional autoencoding-based training.
Show 13 more publications
Teaching LLMs to Learn Tool Trialing and Execution through Environment Interaction
Xingjie Gao*, Pengcheng Huang*, Zhenghao Liu†, Yukun Yan, Shuo Wang, Zulong Chen, Chen Qian, Ge Yu, Yu Gu
- This paper proposes ToolMaster, a framework that enables LLMs to master complex tool usage by shifting from passive imitation of expert trajectories to an active trial-and-execution paradigm, where models learn to explore and self-correct through direct environment interaction.
SPARK: Adaptive Parametric Knowledge Participation for Faithful Retrieval-Augmented Generation
Ruochang Li*, Pengcheng Huang*, Zhenghao Liu†, Yukun Yan, Huiyuan Xie, Yu Gu, Ge Yu, Maosong Sun
- This work presents SPARK (a.k.a. SHIFT, the title of its arXiv v1), which treats control of parametric knowledge in RAG as learning a latent participation policy: a lightweight policy trained with a faithfulness-oriented reward modulates how much FFN-stored knowledge contributes during generation while the backbone LLM stays frozen, adding only 0.01% trainable parameters and giving more faithful answers on six datasets.
Forgetting curve: A reliable method for evaluating memorization capability for long-context models
Xinyu Liu, Runsong Zhao, Pengcheng Huang, Chunyang Xiao, Bei Li, Jingang Wang, Tong Xiao†, Jingbo Zhu
- This work discovered that existing methods for measuring memorization in long‑context models (e.g. via perplexity or “needle in a haystack” tasks) are unreliable—being overly sensitive to prompts, corpora, or downstream tasks—and proposes the Forgetting Curve, a prompt‑free, robust metric based on comparing copy accuracy vs. LM accuracy over prefixes to more faithfully capture a model’s memory over long contexts, revealing that many claimed context extension techniques may not truly improve memorization for very long inputs.
Position IDs Matter: An Enhanced Position Layout for Efficient Context Compression in Large Language Models
Runsong Zhao, Xin Liu, Xinyu Liu, Pengcheng Huang, Chunyang Xiao, Tong Xiao†, Jingbo Zhu
- This work identifies that existing LLM context‑compression methods like ICAE suffer from suboptimal position identifier layouts for compressed tokens and that the auto‑encoding loss alone insufficiently promotes memorization; it then proposes spreading position identifiers uniformly across input and introducing a “compression loss” focused purely on memorization, enabling much higher compression ratios (up to ~15× vs 4×) while retaining reconstruction and downstream performance.
Knowledge Intensive Agents
Zhenghao Liu†, Pengcheng Huang, Zhipeng Xu, Xinze Li, Shuliang Liu, Chunyi Peng, Haidong Xin, Yukun Yan, Shuo Wang, Xu Han, Zhiyuan Liu, Maosong Sun, Yu Gu, Ge Yu
- This work provides a comprehensive overview of Retrieval-Augmented Generation from an agentic perspective, categorizing knowledge-intensive agents into acquisition and utilization roles, and highlighting future directions for joint optimization in multi-agent RAG systems.
ClueAnchor: Clue-Anchored Knowledge Reasoning Exploration and Optimization for Retrieval-Augmented Generation
Hao Chen, Yukun Yan, Sen Mei, Wanxiang Che, Zhenghao Liu, Qi Shi, Xinze Li, Yuchun Fan, Pengcheng Huang, Qiushi Xiong, Zhiyuan Liu, Maosong Sun
- This work discovered that RAG systems often under‐utilize retrieved documents because critical evidence (“clues”) is implicit, dispersed, or obscured by noise, and ClueAnchor addresses this by first extracting key clues from the retrieved content, generating multiple reasoning paths under different knowledge configurations (internal, external, and clue‑anchored), and then using reward‑based preference optimization to select the most effective path — yielding much better reasoning completeness and robustness.
Revealing the Attention Floating Mechanism in Masked Diffusion Models
Xin Dai, Pengcheng Huang, Zhenghao Liu†, Shuo Wang, Yukun Yan, Chaojun Xiao, Yu Gu, Ge Yu, Maosong Sun
ACL Anthology · arXiv · Code
- This paper investigates the internal attention behaviors of Masked Diffusion Models (MDMs), identifying a unique “Attention Floating” mechanism where attention anchors dynamically shift across layers and denoising steps, which provides a mechanistic explanation for their superior in-context learning performance compared to autoregressive models.
Causal Autoregressive Diffusion Language Model
Junhao Ruan, Bei Li, Yongjing Yin, Pengcheng Huang, Xin Chen, Jingang Wang, Xunliang Cai, Tong Xiao†, JingBo Zhu
- This paper introduces Causal Autoregressive Diffusion (CARD), a framework that unifies the training efficiency of autoregressive models with the high-throughput parallel inference of diffusion models by using a strictly causal attention mask and a dynamic decoding mechanism.
Translate-and-Revise: Boosting Large Language Models for Constrained Translation
Pengcheng Huang, Yongyu Mu, Yuzhang Wu, Bei Li, Chunyang Xiao, Tong Xiao†, Jingbo Zhu
arXiv · PDF · Project Page
- This work discovers that large language models (LLMs) prompted for constrained translation often ignore or violate specified lexical or structural constraints—partly because they’re overly confident and let their own fluency priorities override constraint satisfaction—and Translate‑and‑Revise remedies this by adding a revision process that prompts the model to identify unmet constraints and correct its output, yielding ~15% improvement in constraint translation accuracy and surpassing NMT baselines.
RLAIF-SPA: Optimizing LLM-based Emotional Speech Synthesis via RLAIF
Qing Yang, Zhenghao Liu†, Junxin Wang, Yangfan Du, Pengcheng Huang, Tong Xiao
- This paper proposes RLAIF-SPA, a framework that enhances the emotional expressiveness and prosodic naturalness of LLM-based speech synthesis by using Reinforcement Learning from AI Feedback to optimize semantic accuracy and fine-grained prosodic alignment.
MemoryCard: Topic-Aware Multi-Modal Clue Compression for Long-Video Question Answering
Qing Yang, Pengcheng Huang, Xinze Li, Zhenghao Liu†, Yukun Yan, Yu Gu, Ge Yu, Gang Li, Maosong Sun
- This work proposes MemoryCard, which self-reads a long video and its aligned utterances to split it into topic- or event-level units and renders each unit into a Memory Card (an event-level gist plus representative frames) for retrieval and answering, improving long-video QA accuracy by up to 21.8% (relative) under comparable visual-token budgets.
MemoSight: Unifying Context Compression and Multi Token Prediction for Reasoning Acceleration
Xinyu Liu, Xin Liu, Bo Jin, Runsong Zhao, Pengcheng Huang, Junhao Ruan, Bei Li, Chunyang Xiao, Chenglong Wang, Tong Xiao, Jingbo Zhu
- This work proposes MemoSight, which unifies context compression (memory tokens) and multi-token prediction (foresight tokens) in one minimalist special-token design with token-specific positional layouts; on four reasoning benchmarks it cuts the KV cache by up to 66% and speeds up inference by 56% with under a 3% drop in average accuracy.
HybridThinker: Efficient Chain-of-Thought Reasoning via Compressed Memory and Transient Thought Steps
Xin Liu*, Runsong Zhao*, Xinyu Liu, Junhao Ruan, Pengcheng Huang, Shichao Dong, Chunyang Xiao, Chenglong Wang, Changliang Li, Jingbo Zhu, Tong Xiao†
- This work proposes HybridThinker, which compresses chain-of-thought steps into memory tokens while temporarily keeping the raw steps for fine-grained detail, and trains with a hybrid attention scheme that masks some steps so the model must compress and retrieve through memory tokens; it matches the uncompressed baseline on 4 reasoning benchmarks and beats prior CoT compression by 5.8 points on average.
🏆 Awards
- 2026.10 🏅 CAST Young Talent Cultivation Program for Doctoral Students (中国科协青年科技人才培育工程博士生专项计划), China Association for Science and Technology (CAST).
- 2025.06 🏆 Highlight Poster Award, YSSNLP 2025.
- 2024.05 🥇First Prize of Excellent Student Scholarship.
- 2023.05 🥇First Prize of Excellent Student Scholarship.
- 2022.05 🥇First Prize of Excellent Student Scholarship.
📖 Educations
- 2025.09 - now, Ph.D. School of Computer Science and Engineering, Northeastern University
- 2022.09 - 2025.07, M.S. School of Computer Science and Engineering, Northeastern University
- 2018.09 - 2022.07, B.S. School of Computer Science and Engineering, Northeastern University