Boosting Large Language Model for Speech Synthesis: An Empirical Study
Hongkun Hao, Long Zhou, Shujie Liu, Jinyu Li, Shujie Hu, Rui Wang, Furu Wei
ICASSP | April 2025
Furu Wei
Distinguished Scientist
Publications
Publications by Year
-
Examining False Positives under Inference Scaling for Mathematical Reasoning
Yu Wang, Nan Yang, Liang Wang, Furu Wei
MSR-TR-2025-2 | February 2025
Published by Microsoft
Chain-of-Retrieval Augmented Generation
Liang Wang, Haonan Chen, Nan Yang, Xiaolong Huang, Zhicheng Dou, Furu Wei
January 2025
PEACE: Empowering Geologic Map Holistic Understanding with MLLMs
Yangyu Huang, Tianyi Gao, Haoran Xu, Qihao Zhao, Yang Song, Zhipeng Gui, Tengchao Lv, Hao Cheng, Lei Cui, Scarlett Li, Furu Wei
January 2025
-
Bootstrap Your Own Context Length
Liang Wang, Nan Yang, Xingxing Zhang, Xiaolong Huang, Furu Wei
ArXiv | December 2024, Vol abs/2412.18860
REDSTONE: Curating General, Code, Math, and QA Data for Large Language Models
Yaoyao Chang, Lei Cui, Li Dong, Shaohan Huang, Yangyu Huang, Yupan Huang, Scarlett Li, Tengchao Lv, Shuming Ma, Qinzheng Sun, Wenhui Wang, Furu Wei, Ying Xin, Mao Yang, Qiufeng Yin, Xingxing Zhang
December 2024
MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark
Qihao Zhao, Yangyu Huang, Tengchao Lv, Lei Cui, Furu Wei, Qinzheng Sun, Ying Xin, Shaoguang Mao, Xin Zhang, Qiufeng Yin, Scarlett Li
December 2024
BitNet a4.8: 4-bit Activations for 1-bit LLMs
Hongyu Wang, Shuming Ma, Furu Wei
November 2024
Work in progress
Little Giants: Synthesizing High-Quality Embedding Data at Scale
Haonan Chen, Liang Wang, Nan Yang, Yutao Zhu, Ziliang Zhao, Furu Wei, Zhicheng Dou
NAACL 2025 | October 2024
1-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUs
Jinheng Wang, Hansong Zhou, Ting Song, Shaoguang Mao, Shuming Ma, Hongyu Wang, Yan Xia, Furu Wei
October 2024
Differential Transformer
Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, Furu Wei
MSR-TR-2024-42 | October 2024
Published by Microsoft
ARLON: Boosting Diffusion Transformers with Autoregressive Models for Long Video Generation
Zongyi Li, Shujie Hu, Shujie Liu, Long Zhou, Jeongsoo Choi, Lingwei Meng, Xun Guo, Jinyu Li, Hefei Ling, Furu Wei
October 2024
October 2024
Boosting Text-to-Video Generative Model with MLLMs Feedback
Xun Wu, Shaohan Huang, Furu Wei
October 2024
Multimodal Large Language Models Make Text-to-Image Generative Models Align Better
Xun Wu, Shaohan Huang, Furu Wei
October 2024
Q-Sparse: All Large Language Models can be Fully Sparsely-Activated
Hongyu Wang, Shuming Ma, Ruiping Wang, Furu Wei
July 2024
Autoregressive Speech Synthesis without Vector Quantization
Lingwei Meng, Long Zhou, Shujie Liu, Sanyuan Chen, Bing Han, Shujie Hu, Yanqing Liu, Jinyu Li, Sheng Zhao, Xixin Wu, Helen Meng, Furu Wei
July 2024
VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment
Bing Han, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Yanming Qian, Yanqing Liu, Sheng Zhao, Jinyu Li, Furu Wei
June 2024
VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers
Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, Furu Wei
June 2024
Direct Preference Knowledge Distillation for Large Language Models
Yixing Li, Yuxian Gu, Li Dong, Dequan Wang, Yu Cheng, Furu Wei
June 2024
Instruction Pre-Training: Language Models are Supervised Multitask Learners
Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, Furu Wei
June 2024
VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers
Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, Furu Wei
June 2024
A whole-slide foundation model for digital pathology from real-world data
Hanwen Xu, Naoto Usuyama, Jaspreet Bagga, Sheng Zhang, Rajesh Rao, Tristan Naumann, Cliff Wong, Zelalem Gero, Javier González, Yu Gu, Yanbo Xu, Mu-Hsin Wei, Wenhui Wang, Shuming Ma, Furu Wei, Jianwei Yang, Chunyuan Li, Jianfeng Gao, Jaylen Rosemon, Tucker Bower, Soohee Lee, R. Weerasinghe, Bill Wright, Ari Robicsek, Brian Piening, Carlo Bifulco, Sheng Wang, Hoifung Poon
Nature | May 2024
xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token
Xin Cheng, Xun Wang, Xingxing Zhang, Tao Ge, Si-Qing Chen, Furu Wei, Huishuai Zhang, Dongyan Zhao
NeurIPS 2024 | May 2024
MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, Fuzhen Zhuang
May 2024
You Only Cache Once: Decoder-Decoder Architectures for Language Models
Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, Furu Wei
NeurIPS 2024 | May 2024
Kosmos-G: Generating Images in Context with Multimodal Large Language Models
Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, Furu Wei
ICLR 2024 | May 2024
Multi-Head Mixture-of-Experts
Xun Wu, Shaohan Huang, Wenhui Wang, Furu Wei
NeurIPS 2024 | April 2024
LongEmbed: Extending Embedding Models for Long Context Retrieval
Dawei Zhu, Liang Wang, Nan Yang, Yifan Song, Wenhao Wu, Furu Wei, Sujian Li
EMNLP 2024 | April 2024
Mind’s Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models
Wenshan Wu, Shaoguang Mao, Yadong Zhang, Yan Xia, Li Dong, Lei Cui, Furu Wei
NeurIPS 2024 | April 2024
Mixture of LoRA Experts
ICLR 2024 | April 2024
LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Models
Yadong Zhang, Shaoguang Mao, Tao Ge, Xun Wang, Adrian de Wynter, Yan Xia, Wenshan Wu, Ting Song, Man Lan, Furu Wei
March 2024
WavLLM: Towards Robust and Adaptive Speech Large Language Model
Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, Furu Wei
March 2024
SpeechLM: Enhanced Speech Pre-Training With Unpaired Textual Data
Ziqiang Zhang, Sanyuan Chen, Long Zhou, Yu Wu, Shuo Ren, Shujie Liu, Zhuoyuan Yao, Xun Gong, Lirong Dai, Jinyu Li, Furu Wei
IEEE/ACM Transactions on Audio, Speech, and Language Processing | March 2024
MathScale: Scaling Instruction Tuning for Mathematical Reasoning
Zhengyang Tang, Xingxing Zhang, Benyou Wang, Furu Wei
ICML 2024 | March 2024
ResLoRA: Identity Residual Mapping in Low-Rank Adaption
Shuhua Shi, Shaohan Huang, Minghui Song, Zhoujun Li, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
February 2024
The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Lifeng Dong, Ruiping Wang, Jilong Xue, Furu Wei
February 2024
Work in progress
Towards Optimal Learning of Language Models
Yuxian Gu, Li Dong, Yaru Hao, Qingxiu Dong, Minlie Huang, Furu Wei
February 2024
Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models
Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, Ji-Rong Wen
ACL 2024 | February 2024
Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models
Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, Yuxian Gu, Xin Cheng, Xun Wang, Si-Qing Chen, Li Dong, Wei Lu, Zhifang Sui, Benyou Wang, Wai Lam, Furu Wei
February 2024
Work in progress
Generative Representational Instruction Tuning
Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, Douwe Kiela
ICLR 2025 | February 2024
Multilingual E5 Text Embeddings: A Technical Report
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei
MSR-TR-2024-45 | February 2024
Published by Microsoft
Text Diffusion with Reinforced Conditioning
Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
AAAI 2024 | February 2024
Improving Text Embeddings with Large Language Models
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei
ACL 2024 | January 2024
-
When an Image is Worth 1,024 x 1,024 Words: A Case Study in Computational Pathology
Wenhui Wang, Shuming Ma, Hanwen Xu, Naoto Usuyama, Jiayu Ding, Hoifung Poon, Furu Wei
December 2023
Large Search Model: Redefining Search Stack in the Era of LLMs
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei
SIGIR Forum | December 2023, Vol 57(2)
BitNet: Scaling 1-bit Transformers for Large Language Models
Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, Furu Wei
October 2023
Fine-Tuning LLaMA for Multi-Stage Text Retrieval
Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, Jimmy Lin
SIGIR 2024 | October 2023
Augmenting Language Models with Long-Term Memory
Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, Furu Wei
NeurIPS 2023 | October 2023
PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training
Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, Sujian Li
ICLR 2024 | October 2023
Adapting Large Language Models via Reading Comprehension
Daixuan Cheng, Shaohan Huang, Furu Wei
ICLR 2024 | September 2023
Kosmos-2.5: A Multimodal Literate Model
Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, Shaoxiang Wu, Guoxin Wang, Cha Zhang, Furu Wei
September 2023
Retentive Network: A Successor to Transformer for Large Language Models
Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, Furu Wei
August 2023
Generative Retrieval for Conversational Question Answering
Yongqi Li, Nan Yang, Liang Wang, Furu Wei, Wenjie Li
Information Processing and Management | August 2023, Vol 60(5)
LongNet: Scaling Transformers to 1,000,000,000 Tokens
Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Furu Wei
July 2023
In-context Autoencoder for Context Compression in a Large Language Model
Tao Ge, Jing Hu, Haixun Wang, Si-Qing Chen, Furu Wei
ICLR 2024 | July 2023
Learning to Retrieve In-Context Examples for Large Language Models
Liang Wang, Nan Yang, Furu Wei
EACL 2024 | July 2023
Knowledge Distillation of Large Language Models
Yuxian Gu, Li Dong, Furu Wei, Minlie Huang
ICLR 2024 | June 2023
Learning to Rank in Generative Retrieval
Yongqi Li, Nan Yang, Liang Wang, Furu Wei, Wenjie Li
AAAI 2024 | June 2023
Multiview Identifiers Enhanced Generative Retrieval
Yongqi Li, Nan Yang, Liang Wang, Furu Wei, Wenjie Li
ACL 2023 | May 2023
TextDiffuser: Diffusion Models as Text Painters
Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, Furu Wei
NeurIPS 2023 | May 2023
VatLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning
Qiushi Zhu, Long Zhou, Ziqiang Zhang, Shujie Liu, Binxing Jiao, Jie Zhang, Lirong Dai, Daxin Jiang, Jinyu Li, Furu Wei
IEEE Transactions on Multimedia | May 2023
Inference with Reference: Lossless Acceleration of Large Language Models
Nan Yang, Tao Ge, Liang Wang, Binxing Jiao, Daxin Jiang (姜大昕), Linjun Yang, Rangan Majumder, Furu Wei
MSR-TR-2023-17 | April 2023
Published by Microsoft
Language Is Not All You Need: Aligning Perception with Language Models
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Wei
NeurIPS 2023 | March 2023
Query2doc: Query Expansion with Large Language Models
Liang Wang, Nan Yang, Furu Wei
EMNLP 2023 | March 2023
TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models
Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, Furu Wei
AAAI 2023 | February 2023
Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, Furu Wei
January 2023
-
Text Embeddings by Weakly-Supervised Contrastive Pre-training
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang (姜大昕), Rangan Majumder, Furu Wei
December 2022
Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning
Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, Xia Song
ACL 2023 | October 2022
Work in progress
DiT: Self-supervised Pre-training for Document Image Transformer
Junlong Li, Yiheng Xu, Tengchao Lv, Lei Cui, Cha Zhang, Furu Wei
ACM Multimedia 2022 | October 2022
LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking
Yupang Huang, Tengchao Lv, Lei Cui, Yutong Lu, Furu Wei
ACM Multimedia 2022 | October 2022
SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training
Ziqiang Zhang, Long Zhou, Junyi Ao, Shujie Liu, Lirong Dai, Jinyu Li, Furu Wei
EMNLP 2022 | October 2022
Separating Long-Form Speech with Group-Wise Permutation Invariant Training
Wangyou Zhang, Zhuo Chen, Naoyuki Kanda, Shujie Liu, Jinyu Li, Sefik Emre Eskimez, Takuya Yoshioka, Xiong Xiao, Zhong Meng, Yanmin Qian, Furu Wei
Interspeech 2022 | September 2022
Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks
Wenhui Wang, Hangbo Bao, Li Dong, Kriti Aggarwal, Saksham Singhal, Subhojit Som, Furu Wei, Johan Bjorck, Zhiliang Peng, Qiang Liu, Owais Khan Mohammed
August 2022
Learning Diverse Document Representations with Deep Query Interactions for Dense Retrieval
Zehan Li, Nan Yang, Liang Wang, Furu Wei
August 2022
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, Furu Wei
IEEE Journal of Selected Topics in Signal Processing | July 2022, Vol 16(6): pp. 1505-1518
SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang (姜大昕), Rangan Majumder, Furu Wei
ACL 2023 | July 2022
Lossless Acceleration for Seq2seq Generation with Aggressive Decoding
Tao Ge, Heming Xia, Xin Sun, Si-Qing Chen, Furu Wei
May 2022
UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Training
Sanyuan Chen, Yu Wu, Chengyi Wang, Zhengyang Chen, Zhuo Chen, Shujie Liu, Jian Wu, Yao Qian, Furu Wei, Jinyu Li, Xiangzhan Yu
ICASSP 2022 | May 2022
Attention Temperature Matters in Abstractive Summarization Distillation
Shengqiang Zhang, Xingxing Zhang, Hangbo Bao, Furu Wei
ACL 2022 | May 2022
CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailment
Haoyu Song, Li Dong, Wei-Nan Zhang, Ting Liu, Furu Wei
ACL 2022 | May 2022
Controllable Natural Language Generation with Contrastive Prefixes
Jing Qian, Li Dong, Yelong Shen, Furu Wei, Weizhu Chen
ACL 2022 | May 2022
Knowledge Neurons in Pretrained Transformers
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, Furu Wei
ACL 2022 | May 2022
MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Understanding
Junlong Li, Yiheng Xu, Lei Cui, Furu Wei
ACL 2022 | May 2022
Neural Label Search for Zero-Shot Multi-Lingual Extractive Summarization
Ruipeng Jia, Xingxing Zhang, Yanan Cao, Shi Wang, Zheng Lin, Furu Wei
ACL 2022 | May 2022
SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing
Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, Furu Wei
ACL 2022 | May 2022
StableMoE: Stable Routing Strategy for Mixture of Experts
Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baobao Chang, Furu Wei
ACL 2022 | May 2022
THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption
Tianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong, Binxing Jiao, Daxin Jiang (姜大昕), Haoyi Zhou, Jianxin Li, Furu Wei
May 2022
Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation
Guanhua Chen, Shuming Ma, Yun Chen, Dongdong Zhang, Jia Pan, Wenping Wang, Furu Wei
ACL 2022 | May 2022
XFUND: A Benchmark Dataset for Multilingual Visually Rich Form Understanding
Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Furu Wei
ACL 2022 | May 2022
BEiT: BERT Pre-Training of Image Transformers
Hangbo Bao, Li Dong, Songhao Piao, Furu Wei
ICLR 2022 | April 2022
Sequence Level Contrastive Learning for Text Summarization
Shusheng Xu, Xingxing Zhang, Yi Wu, Furu Wei
AAAI 2022 | January 2022
-
Document AI: Benchmarks, Models and Applications
Lei Cui, Yiheng Xu, Tengchao Lv, Furu Wei
November 2021
Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task.
Jian Yang, Shuming Ma, Haoyang Huang, Dongdong Zhang, Li Dong, Shaohan Huang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
WMT | November 2021
LayoutReader: Pre-training of Text and Layout for Reading Order Detection
Zilong Wang, Yiheng Xu, Lei Cui, Jingbo Shang, Furu Wei
EMNLP 2021 | November 2021
MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs
Zewen Chi, Li Dong, Shuming Ma, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei
EMNLP 2021 | October 2021
Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders
Guanhua Chen, Shuming Ma, Yun Chen, Li Dong, Dongdong Zhang, Jia Pan, Wenping Wang, Furu Wei
EMNLP 2021 | October 2021
Jointly Learning to Repair Code and Generate Commit Message
Jiaqi Bai, Long Zhou, Ambrosio Blanco, Shujie Liu, Furu Wei, Ming Zhou, Zhoujun Li
2021 Empirical Methods in Natural Language Processing | September 2021
Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting
Wangchunshu Zhou, Tao Ge, Ke Xu, Furu Wei
2021 Empirical Methods in Natural Language Processing | September 2021
Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression
Canwen Xu, Wangchunshu Zhou, Tao Ge, Ke Xu, Julian McAuley, Furu Wei
2021 Empirical Methods in Natural Language Processing | September 2021
Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training
Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
EMNLP 2021 | September 2021
Document AI: Benchmarks, Models and Applications (In Chinese)
Lei Cui, Yiheng Xu, Tengchao Lv, Furu Wei
CCL 2021 | August 2021
UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data
Chengyi Wang, Yu Wu, Yao Qian, Kenichi Kumatani, Shujie Liu, Furu Wei, Michael Zeng, Xuedong Huang
2021 International Conference on Machine Learning | July 2021
Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
Yunzhi Yao, Shaohan Huang, Wenhui Wang, Li Dong, Furu Wei
ACL-IJCNLP 2021 | July 2021
Grammar-Based Patches Generation for Automated Program Repair
Yu Tang, Long Zhou, Ambrosio Blanco, Shujie Liu, Furu Wei, Ming Zhou, Muyun Yang
ACL-IJCNLP 2021 | July 2021
Multilingual Agreement for Multilingual Neural Machine Translation
Jian Yang, Yuwei Yin, Shuming Ma, Haoyang Huang, Dongdong Zhang, Zhoujun Li, Furu Wei
July 2021
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, Furu Wei
ACL 2022 | June 2021
Learning to Sample Replacements for ELECTRA Pre-Training
Yaru Hao, Li Dong, Hangbo Bao, Ke Xu, Furu Wei
ACL-IJCNLP 2021 | June 2021
DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
arXiv: Computation and Language | June 2021
DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
June 2021
Consistency Regularization for Cross-Lingual Fine-Tuning
Bo Zheng, Li Dong, Shaohan Huang, Wenhui Wang, Zewen Chi, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
ACL-IJCNLP 2021 | June 2021
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
Zewen Chi, Li Dong, Bo Zheng, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei
ACL-IJCNLP 2021 | June 2021
Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding
Xin Sun, Tao Ge, Furu Wei, Houfeng Wang
ACL-IJCNLP 2021 | June 2021
VT-SSum: A Benchmark Dataset for Video Transcript Segmentation and Summarization
Tengchao Lv, Lei Cui, Momcilo Vasilijevic, Furu Wei
June 2021
Memory-Efficient Differentiable Transformer Architecture Search
Yuekai Zhao, Li Dong, Yelong Shen, Zhihua Zhang, Furu Wei, Wei Chen
ACL-IJCNLP 2021 | May 2021
Blow the Dog Whistle: A Chinese Dataset for Cant Understanding with Common Sense and World Knowledge
Canwen Xu, Wangchunshu Zhou, Tao Ge, Ke Xu, Julian J. McAuley, Furu Wei
April 2021
LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding
Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Furu Wei
April 2021
LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding
Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, Lidong Zhou
ACL-IJCNLP 2021 | January 2021
-
MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, Furu Wei
ACL-IJCNLP 2021 | December 2020
XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders.
Shuming Ma, Jian Yang, Haoyang Huang, Zewen Chi, Li Dong, Dongdong Zhang, Hany Hassan Awadalla, Alexandre Muzio, Akiko Eriguchi, Saksham Singhal, Xia Song, Arul Menezes, Furu Wei
December 2020
DocBank: A Benchmark Dataset for Document Layout Analysis
Minghao Li, Yiheng Xu, Lei Cui, Shaohan Huang, Furu Wei, Zhoujun Li, Ming Zhou
COLING 2020 | December 2020
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, Jianfeng Gao
ECCV | August 2020
LayoutLM: Pre-training of Text and Layout for Document Image Understanding
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou
KDD 2020 | August 2020
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, Ming Zhou
July 2020
Multimodal Matching Transformer for Live Commenting
Chaoqun Duan, Lei Cui, Shuming Ma, Furu Wei, Conghui Zhu, Tiejun Zhao
ECAI 2020 | July 2020
TableBank: Table Benchmark for Image-based Table Detection and Recognition
Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou, Zhoujun Li
LREC 2020 | May 2020
Self-Attention Attribution: Interpreting Information Interactions Inside Transformer
Yaru Hao, Li Dong, Furu Wei, Ke Xu
AAAI 2021 | April 2020
AAAI 2021 Best Paper Runner Up
Self-Adversarial Learning with Comparative Discrimination for Text Generation
Wangchunshu Zhou, Tao Ge, Ke Xu, Furu Wei, Ming Zhou
Eighth International Conference on Learning Representations (ICLR) | April 2020
VL-BERT: Pre-training of Generic Visual-Linguistic Representations
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, Jifeng Dai
Eighth International Conference on Learning Representations (ICLR) | April 2020
UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Songhao Piao, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon
February 2020
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, Ming Zhou
NeurIPS 2020 | February 2020
-
Unified Language Model Pre-training for Natural Language Understanding and Generation
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon
33rd Conference on Neural Information Processing Systems (NeurIPS 2019) | December 2019
Neural Melody Composition from Lyrics
Hangbo Bao, Shaohan Huang, Furu Wei, Lei Cui, Yu Wu, Chuanqi Tan, Songhao Piao, Ming Zhou
NLPCC 2019 | September 2019
LiveBot: Generating Live Video Comments Based on Visual and Textual Contexts
Shuming Ma, Lei Cui, Damai Dai, Furu Wei, Xu Sun
AAAI 2019 | July 2019
Retrieval-Enhanced Adversarial Training for Neural Response Generation
Qingfu Zhu, Lei Cui, Wei-Nan Zhang, Furu Wei, Ting Liu
ACL 2019 | July 2019
-
Fine-grained Coordinated Cross-lingual Text Stream Alignment for Endless Language Knowledge Acquisition
Tao Ge, Qing Dou, Heng Ji, Lei Cui, Baobao Chang, Zhifang Sui, Furu Wei, Ming Zhou
EMNLP 2018 | November 2018
SeRI: A Dataset for Sub-event Relation Inference from an Encyclopedia
Tao Ge, Lei Cui, Baobao Chang, Zhifang Sui, Furu Wei, Ming Zhou
NLPCC 2018 | August 2018
Neural Open Information Extraction
ACL 2018 | July 2018
Attention-Fused Deep Matching Network for Natural Language Inference
Chaoqun Duan, Lei Cui, Xinchi Chen, Furu Wei, Conghui Zhu, Tiejun Zhao
IJCAI 2018 | July 2018
EventWiki: A Knowledge Base of Major Events
Tao Ge, Lei Cui, Baobao Chang, Zhifang Sui, Furu Wei, Ming Zhou
LREC 2018 | May 2018
-
Leveraging Contextual Sentence Relations for Extractive Summarization Using a Neural Attention Model
Pengjie Ren, Zhumin Chen, Zhaochun Ren, Furu Wei, Jun Ma, Maarten de Rijke
SIGIR '17, Shinjuku, Tokyo, Japan, August 07-11, 2017 | August 2017
SuperAgent: A Customer Service Chatbot for E-commerce Websites
Lei Cui, Furu Wei, Shaohan Huang, Chuanqi Tan, Chaoqun Duan, Ming Zhou
ACL 2017 | July 2017
-
QuickView: advanced search of tweets
Xiaohua Liu, Long Jiang, Furu Wei, Ming Zhou, QuickView Team Microsoft
SIGIR (Demo paper) | January 2011
ISBN: 978-1-4503-0757-4
-
OpinionSeer: Interactive Visualization of Hotel Customer Feedback
Yingcai Wu, Furu Wei, Shixia Liu, Norman Au, Weiwei Cui, Hong Zhou, Huamin Qu
IEEE Transactions on Visualization and Computer Graphics | November 2010, pp. 1109-1118
Context-Preserving, Dynamic Word Cloud Visualization
Weiwei Cui, Yingcai Wu, Shixia Liu, Furu Wei, Michelle Zhou, Huamin Qu
IEEE Computer Graphics and Applications | September 2010, Vol 30(6): pp. 42
Context Preserving Dynamic Word Cloud Visualization
Weiwei Cui, Yingcai Wu, Shixia Liu, Furu Wei, Michelle Zhou, Huamin Qu
2010 IEEE Pacific Visualization Symposium (PacificVis) | March 2010
Publications by Research Area
-
Boosting Large Language Model for Speech Synthesis: An Empirical Study
Hongkun Hao, Long Zhou, Shujie Liu, Jinyu Li, Shujie Hu, Rui Wang, Furu Wei
ICASSP | April 2025
Examining False Positives under Inference Scaling for Mathematical Reasoning
Yu Wang, Nan Yang, Liang Wang, Furu Wei
MSR-TR-2025-2 | February 2025
Published by Microsoft
Chain-of-Retrieval Augmented Generation
Liang Wang, Haonan Chen, Nan Yang, Xiaolong Huang, Zhicheng Dou, Furu Wei
January 2025
Bootstrap Your Own Context Length
Liang Wang, Nan Yang, Xingxing Zhang, Xiaolong Huang, Furu Wei
ArXiv | December 2024, Vol abs/2412.18860
Little Giants: Synthesizing High-Quality Embedding Data at Scale
Haonan Chen, Liang Wang, Nan Yang, Yutao Zhu, Ziliang Zhao, Furu Wei, Zhicheng Dou
NAACL 2025 | October 2024
VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers
Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, Furu Wei
June 2024
LongEmbed: Extending Embedding Models for Long Context Retrieval
Dawei Zhu, Liang Wang, Nan Yang, Yifan Song, Wenhao Wu, Furu Wei, Sujian Li
EMNLP 2024 | April 2024
Mixture of LoRA Experts
ICLR 2024 | April 2024
SpeechLM: Enhanced Speech Pre-Training With Unpaired Textual Data
Ziqiang Zhang, Sanyuan Chen, Long Zhou, Yu Wu, Shuo Ren, Shujie Liu, Zhuoyuan Yao, Xun Gong, Lirong Dai, Jinyu Li, Furu Wei
IEEE/ACM Transactions on Audio, Speech, and Language Processing | March 2024
Generative Representational Instruction Tuning
Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, Douwe Kiela
ICLR 2025 | February 2024
Improving Text Embeddings with Large Language Models
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei
ACL 2024 | January 2024
Large Search Model: Redefining Search Stack in the Era of LLMs
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei
SIGIR Forum | December 2023, Vol 57(2)
Fine-Tuning LLaMA for Multi-Stage Text Retrieval
Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, Jimmy Lin
SIGIR 2024 | October 2023
PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training
Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, Sujian Li
ICLR 2024 | October 2023
Kosmos-2.5: A Multimodal Literate Model
Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, Shaoxiang Wu, Guoxin Wang, Cha Zhang, Furu Wei
September 2023
Generative Retrieval for Conversational Question Answering
Yongqi Li, Nan Yang, Liang Wang, Furu Wei, Wenjie Li
Information Processing and Management | August 2023, Vol 60(5)
Learning to Retrieve In-Context Examples for Large Language Models
Liang Wang, Nan Yang, Furu Wei
EACL 2024 | July 2023
Learning to Rank in Generative Retrieval
Yongqi Li, Nan Yang, Liang Wang, Furu Wei, Wenjie Li
AAAI 2024 | June 2023
VatLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning
Qiushi Zhu, Long Zhou, Ziqiang Zhang, Shujie Liu, Binxing Jiao, Jie Zhang, Lirong Dai, Daxin Jiang, Jinyu Li, Furu Wei
IEEE Transactions on Multimedia | May 2023
TextDiffuser: Diffusion Models as Text Painters
Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, Furu Wei
NeurIPS 2023 | May 2023
Multiview Identifiers Enhanced Generative Retrieval
Yongqi Li, Nan Yang, Liang Wang, Furu Wei, Wenjie Li
ACL 2023 | May 2023
Inference with Reference: Lossless Acceleration of Large Language Models
Nan Yang, Tao Ge, Liang Wang, Binxing Jiao, Daxin Jiang (姜大昕), Linjun Yang, Rangan Majumder, Furu Wei
MSR-TR-2023-17 | April 2023
Published by Microsoft
Query2doc: Query Expansion with Large Language Models
Liang Wang, Nan Yang, Furu Wei
EMNLP 2023 | March 2023
TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models
Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, Furu Wei
AAAI 2023 | February 2023
Text Embeddings by Weakly-Supervised Contrastive Pre-training
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang (姜大昕), Rangan Majumder, Furu Wei
December 2022
Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning
Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, Xia Song
ACL 2023 | October 2022
Work in progress
LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking
Yupang Huang, Tengchao Lv, Lei Cui, Yutong Lu, Furu Wei
ACM Multimedia 2022 | October 2022
SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training
Ziqiang Zhang, Long Zhou, Junyi Ao, Shujie Liu, Lirong Dai, Jinyu Li, Furu Wei
EMNLP 2022 | October 2022
Separating Long-Form Speech with Group-Wise Permutation Invariant Training
Wangyou Zhang, Zhuo Chen, Naoyuki Kanda, Shujie Liu, Jinyu Li, Sefik Emre Eskimez, Takuya Yoshioka, Xiong Xiao, Zhong Meng, Yanmin Qian, Furu Wei
Interspeech 2022 | September 2022
Learning Diverse Document Representations with Deep Query Interactions for Dense Retrieval
Zehan Li, Nan Yang, Liang Wang, Furu Wei
August 2022
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, Furu Wei
IEEE Journal of Selected Topics in Signal Processing | July 2022, Vol 16(6): pp. 1505-1518
SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang (姜大昕), Rangan Majumder, Furu Wei
ACL 2023 | July 2022
XFUND: A Benchmark Dataset for Multilingual Visually Rich Form Understanding
Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Furu Wei
ACL 2022 | May 2022
Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation
Guanhua Chen, Shuming Ma, Yun Chen, Dongdong Zhang, Jia Pan, Wenping Wang, Furu Wei
ACL 2022 | May 2022
THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption
Tianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong, Binxing Jiao, Daxin Jiang (姜大昕), Haoyi Zhou, Jianxin Li, Furu Wei
May 2022
StableMoE: Stable Routing Strategy for Mixture of Experts
Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baobao Chang, Furu Wei
ACL 2022 | May 2022
SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing
Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, Furu Wei
ACL 2022 | May 2022
Neural Label Search for Zero-Shot Multi-Lingual Extractive Summarization
Ruipeng Jia, Xingxing Zhang, Yanan Cao, Shi Wang, Zheng Lin, Furu Wei
ACL 2022 | May 2022
Knowledge Neurons in Pretrained Transformers
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, Furu Wei
ACL 2022 | May 2022
Controllable Natural Language Generation with Contrastive Prefixes
Jing Qian, Li Dong, Yelong Shen, Furu Wei, Weizhu Chen
ACL 2022 | May 2022
CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailment
Haoyu Song, Li Dong, Wei-Nan Zhang, Ting Liu, Furu Wei
ACL 2022 | May 2022
Attention Temperature Matters in Abstractive Summarization Distillation
Shengqiang Zhang, Xingxing Zhang, Hangbo Bao, Furu Wei
ACL 2022 | May 2022
MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Understanding
Junlong Li, Yiheng Xu, Lei Cui, Furu Wei
ACL 2022 | May 2022
Document AI: Benchmarks, Models and Applications
Lei Cui, Yiheng Xu, Tengchao Lv, Furu Wei
November 2021
Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task.
Jian Yang, Shuming Ma, Haoyang Huang, Dongdong Zhang, Li Dong, Shaohan Huang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
WMT | November 2021
LayoutReader: Pre-training of Text and Layout for Reading Order Detection
Zilong Wang, Yiheng Xu, Lei Cui, Jingbo Shang, Furu Wei
EMNLP 2021 | November 2021
MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs
Zewen Chi, Li Dong, Shuming Ma, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei
EMNLP 2021 | October 2021
Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders
Guanhua Chen, Shuming Ma, Yun Chen, Li Dong, Dongdong Zhang, Jia Pan, Wenping Wang, Furu Wei
EMNLP 2021 | October 2021
Jointly Learning to Repair Code and Generate Commit Message
Jiaqi Bai, Long Zhou, Ambrosio Blanco, Shujie Liu, Furu Wei, Ming Zhou, Zhoujun Li
2021 Empirical Methods in Natural Language Processing | September 2021
Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting
Wangchunshu Zhou, Tao Ge, Ke Xu, Furu Wei
2021 Empirical Methods in Natural Language Processing | September 2021
Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training
Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
EMNLP 2021 | September 2021
Document AI: Benchmarks, Models and Applications (In Chinese)
Lei Cui, Yiheng Xu, Tengchao Lv, Furu Wei
CCL 2021 | August 2021
UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data
Chengyi Wang, Yu Wu, Yao Qian, Kenichi Kumatani, Shujie Liu, Furu Wei, Michael Zeng, Xuedong Huang
2021 International Conference on Machine Learning | July 2021
Grammar-Based Patches Generation for Automated Program Repair
Yu Tang, Long Zhou, Ambrosio Blanco, Shujie Liu, Furu Wei, Ming Zhou, Muyun Yang
ACL-IJCNLP 2021 | July 2021
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, Furu Wei
ACL 2022 | June 2021
Learning to Sample Replacements for ELECTRA Pre-Training
Yaru Hao, Li Dong, Hangbo Bao, Ke Xu, Furu Wei
ACL-IJCNLP 2021 | June 2021
DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
arXiv: Computation and Language | June 2021
DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
June 2021
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
Zewen Chi, Li Dong, Bo Zheng, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei
ACL-IJCNLP 2021 | June 2021
VT-SSum: A Benchmark Dataset for Video Transcript Segmentation and Summarization
Tengchao Lv, Lei Cui, Momcilo Vasilijevic, Furu Wei
June 2021
LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding
Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Furu Wei
April 2021
LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding
Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, Lidong Zhou
ACL-IJCNLP 2021 | January 2021
XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders.
Shuming Ma, Jian Yang, Haoyang Huang, Zewen Chi, Li Dong, Dongdong Zhang, Hany Hassan Awadalla, Alexandre Muzio, Akiko Eriguchi, Saksham Singhal, Xia Song, Arul Menezes, Furu Wei
December 2020
DocBank: A Benchmark Dataset for Document Layout Analysis
Minghao Li, Yiheng Xu, Lei Cui, Shaohan Huang, Furu Wei, Zhoujun Li, Ming Zhou
COLING 2020 | December 2020
LayoutLM: Pre-training of Text and Layout for Document Image Understanding
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou
KDD 2020 | August 2020
Multimodal Matching Transformer for Live Commenting
Chaoqun Duan, Lei Cui, Shuming Ma, Furu Wei, Conghui Zhu, Tiejun Zhao
ECAI 2020 | July 2020
TableBank: Table Benchmark for Image-based Table Detection and Recognition
Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou, Zhoujun Li
LREC 2020 | May 2020
Self-Attention Attribution: Interpreting Information Interactions Inside Transformer
Yaru Hao, Li Dong, Furu Wei, Ke Xu
AAAI 2021 | April 2020
AAAI 2021 Best Paper Runner Up
VL-BERT: Pre-training of Generic Visual-Linguistic Representations
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, Jifeng Dai
Eighth International Conference on Learning Representations (ICLR) | April 2020
Self-Adversarial Learning with Comparative Discrimination for Text Generation
Wangchunshu Zhou, Tao Ge, Ke Xu, Furu Wei, Ming Zhou
Eighth International Conference on Learning Representations (ICLR) | April 2020
Unified Language Model Pre-training for Natural Language Understanding and Generation
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon
33rd Conference on Neural Information Processing Systems (NeurIPS 2019) | December 2019
Neural Melody Composition from Lyrics
Hangbo Bao, Shaohan Huang, Furu Wei, Lei Cui, Yu Wu, Chuanqi Tan, Songhao Piao, Ming Zhou
NLPCC 2019 | September 2019
LiveBot: Generating Live Video Comments Based on Visual and Textual Contexts
Shuming Ma, Lei Cui, Damai Dai, Furu Wei, Xu Sun
AAAI 2019 | July 2019
Retrieval-Enhanced Adversarial Training for Neural Response Generation
Qingfu Zhu, Lei Cui, Wei-Nan Zhang, Furu Wei, Ting Liu
ACL 2019 | July 2019
Fine-grained Coordinated Cross-lingual Text Stream Alignment for Endless Language Knowledge Acquisition
Tao Ge, Qing Dou, Heng Ji, Lei Cui, Baobao Chang, Zhifang Sui, Furu Wei, Ming Zhou
EMNLP 2018 | November 2018
SeRI: A Dataset for Sub-event Relation Inference from an Encyclopedia
Tao Ge, Lei Cui, Baobao Chang, Zhifang Sui, Furu Wei, Ming Zhou
NLPCC 2018 | August 2018
Neural Open Information Extraction
ACL 2018 | July 2018
Attention-Fused Deep Matching Network for Natural Language Inference
Chaoqun Duan, Lei Cui, Xinchi Chen, Furu Wei, Conghui Zhu, Tiejun Zhao
IJCAI 2018 | July 2018
EventWiki: A Knowledge Base of Major Events
Tao Ge, Lei Cui, Baobao Chang, Zhifang Sui, Furu Wei, Ming Zhou
LREC 2018 | May 2018
SuperAgent: A Customer Service Chatbot for E-commerce Websites
Lei Cui, Furu Wei, Shaohan Huang, Chuanqi Tan, Chaoqun Duan, Ming Zhou
ACL 2017 | July 2017
-
Chain-of-Retrieval Augmented Generation
Liang Wang, Haonan Chen, Nan Yang, Xiaolong Huang, Zhicheng Dou, Furu Wei
January 2025
Little Giants: Synthesizing High-Quality Embedding Data at Scale
Haonan Chen, Liang Wang, Nan Yang, Yutao Zhu, Ziliang Zhao, Furu Wei, Zhicheng Dou
NAACL 2025 | October 2024
LongEmbed: Extending Embedding Models for Long Context Retrieval
Dawei Zhu, Liang Wang, Nan Yang, Yifan Song, Wenhao Wu, Furu Wei, Sujian Li
EMNLP 2024 | April 2024
Generative Representational Instruction Tuning
Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, Douwe Kiela
ICLR 2025 | February 2024
Improving Text Embeddings with Large Language Models
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei
ACL 2024 | January 2024
Large Search Model: Redefining Search Stack in the Era of LLMs
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei
SIGIR Forum | December 2023, Vol 57(2)
Fine-Tuning LLaMA for Multi-Stage Text Retrieval
Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, Jimmy Lin
SIGIR 2024 | October 2023
Text Embeddings by Weakly-Supervised Contrastive Pre-training
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang (姜大昕), Rangan Majumder, Furu Wei
December 2022
Leveraging Contextual Sentence Relations for Extractive Summarization Using a Neural Attention Model
Pengjie Ren, Zhumin Chen, Zhaochun Ren, Furu Wei, Jun Ma, Maarten de Rijke
SIGIR '17, Shinjuku, Tokyo, Japan, August 07-11, 2017 | August 2017
SuperAgent: A Customer Service Chatbot for E-commerce Websites
Lei Cui, Furu Wei, Shaohan Huang, Chuanqi Tan, Chaoqun Duan, Ming Zhou
ACL 2017 | July 2017
QuickView: advanced search of tweets
Xiaohua Liu, Long Jiang, Furu Wei, Ming Zhou, QuickView Team Microsoft
SIGIR (Demo paper) | January 2011
ISBN: 978-1-4503-0757-4
-
MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark
Qihao Zhao, Yangyu Huang, Tengchao Lv, Lei Cui, Furu Wei, Qinzheng Sun, Ying Xin, Shaoguang Mao, Xin Zhang, Qiufeng Yin, Scarlett Li
December 2024
BitNet a4.8: 4-bit Activations for 1-bit LLMs
Hongyu Wang, Shuming Ma, Furu Wei
November 2024
Work in progress
1-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUs
Jinheng Wang, Hansong Zhou, Ting Song, Shaoguang Mao, Shuming Ma, Hongyu Wang, Yan Xia, Furu Wei
October 2024
Differential Transformer
Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, Furu Wei
MSR-TR-2024-42 | October 2024
Published by Microsoft
ARLON: Boosting Diffusion Transformers with Autoregressive Models for Long Video Generation
Zongyi Li, Shujie Hu, Shujie Liu, Long Zhou, Jeongsoo Choi, Lingwei Meng, Xun Guo, Jinyu Li, Hefei Ling, Furu Wei
October 2024
October 2024
Boosting Text-to-Video Generative Model with MLLMs Feedback
Xun Wu, Shaohan Huang, Furu Wei
October 2024
Q-Sparse: All Large Language Models can be Fully Sparsely-Activated
Hongyu Wang, Shuming Ma, Ruiping Wang, Furu Wei
July 2024
Autoregressive Speech Synthesis without Vector Quantization
Lingwei Meng, Long Zhou, Shujie Liu, Sanyuan Chen, Bing Han, Shujie Hu, Yanqing Liu, Jinyu Li, Sheng Zhao, Xixin Wu, Helen Meng, Furu Wei
July 2024
VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment
Bing Han, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Yanming Qian, Yanqing Liu, Sheng Zhao, Jinyu Li, Furu Wei
June 2024
Direct Preference Knowledge Distillation for Large Language Models
Yixing Li, Yuxian Gu, Li Dong, Dequan Wang, Yu Cheng, Furu Wei
June 2024
Instruction Pre-Training: Language Models are Supervised Multitask Learners
Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, Furu Wei
June 2024
VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers
Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, Furu Wei
June 2024
A whole-slide foundation model for digital pathology from real-world data
Hanwen Xu, Naoto Usuyama, Jaspreet Bagga, Sheng Zhang, Rajesh Rao, Tristan Naumann, Cliff Wong, Zelalem Gero, Javier González, Yu Gu, Yanbo Xu, Mu-Hsin Wei, Wenhui Wang, Shuming Ma, Furu Wei, Jianwei Yang, Chunyuan Li, Jianfeng Gao, Jaylen Rosemon, Tucker Bower, Soohee Lee, R. Weerasinghe, Bill Wright, Ari Robicsek, Brian Piening, Carlo Bifulco, Sheng Wang, Hoifung Poon
Nature | May 2024
xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token
Xin Cheng, Xun Wang, Xingxing Zhang, Tao Ge, Si-Qing Chen, Furu Wei, Huishuai Zhang, Dongyan Zhao
NeurIPS 2024 | May 2024
MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, Fuzhen Zhuang
May 2024
You Only Cache Once: Decoder-Decoder Architectures for Language Models
Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, Furu Wei
NeurIPS 2024 | May 2024
Kosmos-G: Generating Images in Context with Multimodal Large Language Models
Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, Furu Wei
ICLR 2024 | May 2024
Multi-Head Mixture-of-Experts
Xun Wu, Shaohan Huang, Wenhui Wang, Furu Wei
NeurIPS 2024 | April 2024
Mind’s Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models
Wenshan Wu, Shaoguang Mao, Yadong Zhang, Yan Xia, Li Dong, Lei Cui, Furu Wei
NeurIPS 2024 | April 2024
LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Models
Yadong Zhang, Shaoguang Mao, Tao Ge, Xun Wang, Adrian de Wynter, Yan Xia, Wenshan Wu, Ting Song, Man Lan, Furu Wei
March 2024
WavLLM: Towards Robust and Adaptive Speech Large Language Model
Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, Furu Wei
March 2024
MathScale: Scaling Instruction Tuning for Mathematical Reasoning
Zhengyang Tang, Xingxing Zhang, Benyou Wang, Furu Wei
ICML 2024 | March 2024
ResLoRA: Identity Residual Mapping in Low-Rank Adaption
Shuhua Shi, Shaohan Huang, Minghui Song, Zhoujun Li, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
February 2024
Towards Optimal Learning of Language Models
Yuxian Gu, Li Dong, Yaru Hao, Qingxiu Dong, Minlie Huang, Furu Wei
February 2024
The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Lifeng Dong, Ruiping Wang, Jilong Xue, Furu Wei
February 2024
Work in progress
Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models
Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, Ji-Rong Wen
ACL 2024 | February 2024
Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models
Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, Yuxian Gu, Xin Cheng, Xun Wang, Si-Qing Chen, Li Dong, Wei Lu, Zhifang Sui, Benyou Wang, Wai Lam, Furu Wei
February 2024
Work in progress
Multilingual E5 Text Embeddings: A Technical Report
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei
MSR-TR-2024-45 | February 2024
Published by Microsoft
Text Diffusion with Reinforced Conditioning
Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
AAAI 2024 | February 2024
When an Image is Worth 1,024 x 1,024 Words: A Case Study in Computational Pathology
Wenhui Wang, Shuming Ma, Hanwen Xu, Naoto Usuyama, Jiayu Ding, Hoifung Poon, Furu Wei
December 2023
BitNet: Scaling 1-bit Transformers for Large Language Models
Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, Furu Wei
October 2023
Augmenting Language Models with Long-Term Memory
Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, Furu Wei
NeurIPS 2023 | October 2023
Adapting Large Language Models via Reading Comprehension
Daixuan Cheng, Shaohan Huang, Furu Wei
ICLR 2024 | September 2023
Kosmos-2.5: A Multimodal Literate Model
Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, Shaoxiang Wu, Guoxin Wang, Cha Zhang, Furu Wei
September 2023
Retentive Network: A Successor to Transformer for Large Language Models
Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, Furu Wei
August 2023
LongNet: Scaling Transformers to 1,000,000,000 Tokens
Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Furu Wei
July 2023
In-context Autoencoder for Context Compression in a Large Language Model
Tao Ge, Jing Hu, Haixun Wang, Si-Qing Chen, Furu Wei
ICLR 2024 | July 2023
Knowledge Distillation of Large Language Models
Yuxian Gu, Li Dong, Furu Wei, Minlie Huang
ICLR 2024 | June 2023
TextDiffuser: Diffusion Models as Text Painters
Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, Furu Wei
NeurIPS 2023 | May 2023
Language Is Not All You Need: Aligning Perception with Language Models
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Wei
NeurIPS 2023 | March 2023
TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models
Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, Furu Wei
AAAI 2023 | February 2023
Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning
Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, Xia Song
ACL 2023 | October 2022
Work in progress
LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking
Yupang Huang, Tengchao Lv, Lei Cui, Yutong Lu, Furu Wei
ACM Multimedia 2022 | October 2022
DiT: Self-supervised Pre-training for Document Image Transformer
Junlong Li, Yiheng Xu, Tengchao Lv, Lei Cui, Cha Zhang, Furu Wei
ACM Multimedia 2022 | October 2022
SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training
Ziqiang Zhang, Long Zhou, Junyi Ao, Shujie Liu, Lirong Dai, Jinyu Li, Furu Wei
EMNLP 2022 | October 2022
Separating Long-Form Speech with Group-Wise Permutation Invariant Training
Wangyou Zhang, Zhuo Chen, Naoyuki Kanda, Shujie Liu, Jinyu Li, Sefik Emre Eskimez, Takuya Yoshioka, Xiong Xiao, Zhong Meng, Yanmin Qian, Furu Wei
Interspeech 2022 | September 2022
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, Furu Wei
IEEE Journal of Selected Topics in Signal Processing | July 2022, Vol 16(6): pp. 1505-1518
Lossless Acceleration for Seq2seq Generation with Aggressive Decoding
Tao Ge, Heming Xia, Xin Sun, Si-Qing Chen, Furu Wei
May 2022
Neural Label Search for Zero-Shot Multi-Lingual Extractive Summarization
Ruipeng Jia, Xingxing Zhang, Yanan Cao, Shi Wang, Zheng Lin, Furu Wei
ACL 2022 | May 2022
MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Understanding
Junlong Li, Yiheng Xu, Lei Cui, Furu Wei
ACL 2022 | May 2022
BEiT: BERT Pre-Training of Image Transformers
Hangbo Bao, Li Dong, Songhao Piao, Furu Wei
ICLR 2022 | April 2022
Sequence Level Contrastive Learning for Text Summarization
Shusheng Xu, Xingxing Zhang, Yi Wu, Furu Wei
AAAI 2022 | January 2022
Document AI: Benchmarks, Models and Applications
Lei Cui, Yiheng Xu, Tengchao Lv, Furu Wei
November 2021
Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task.
Jian Yang, Shuming Ma, Haoyang Huang, Dongdong Zhang, Li Dong, Shaohan Huang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
WMT | November 2021
LayoutReader: Pre-training of Text and Layout for Reading Order Detection
Zilong Wang, Yiheng Xu, Lei Cui, Jingbo Shang, Furu Wei
EMNLP 2021 | November 2021
Jointly Learning to Repair Code and Generate Commit Message
Jiaqi Bai, Long Zhou, Ambrosio Blanco, Shujie Liu, Furu Wei, Ming Zhou, Zhoujun Li
2021 Empirical Methods in Natural Language Processing | September 2021
Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting
Wangchunshu Zhou, Tao Ge, Ke Xu, Furu Wei
2021 Empirical Methods in Natural Language Processing | September 2021
Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression
Canwen Xu, Wangchunshu Zhou, Tao Ge, Ke Xu, Julian McAuley, Furu Wei
2021 Empirical Methods in Natural Language Processing | September 2021
Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training
Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
EMNLP 2021 | September 2021
Document AI: Benchmarks, Models and Applications (In Chinese)
Lei Cui, Yiheng Xu, Tengchao Lv, Furu Wei
CCL 2021 | August 2021
UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data
Chengyi Wang, Yu Wu, Yao Qian, Kenichi Kumatani, Shujie Liu, Furu Wei, Michael Zeng, Xuedong Huang
2021 International Conference on Machine Learning | July 2021
Multilingual Agreement for Multilingual Neural Machine Translation
Jian Yang, Yuwei Yin, Shuming Ma, Haoyang Huang, Dongdong Zhang, Zhoujun Li, Furu Wei
July 2021
Grammar-Based Patches Generation for Automated Program Repair
Yu Tang, Long Zhou, Ambrosio Blanco, Shujie Liu, Furu Wei, Ming Zhou, Muyun Yang
ACL-IJCNLP 2021 | July 2021
Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
Yunzhi Yao, Shaohan Huang, Wenhui Wang, Li Dong, Furu Wei
ACL-IJCNLP 2021 | July 2021
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, Furu Wei
ACL 2022 | June 2021
Learning to Sample Replacements for ELECTRA Pre-Training
Yaru Hao, Li Dong, Hangbo Bao, Ke Xu, Furu Wei
ACL-IJCNLP 2021 | June 2021
DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
arXiv: Computation and Language | June 2021
DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
June 2021
Consistency Regularization for Cross-Lingual Fine-Tuning
Bo Zheng, Li Dong, Shaohan Huang, Wenhui Wang, Zewen Chi, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
ACL-IJCNLP 2021 | June 2021
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
Zewen Chi, Li Dong, Bo Zheng, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei
ACL-IJCNLP 2021 | June 2021
Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding
Xin Sun, Tao Ge, Furu Wei, Houfeng Wang
ACL-IJCNLP 2021 | June 2021
VT-SSum: A Benchmark Dataset for Video Transcript Segmentation and Summarization
Tengchao Lv, Lei Cui, Momcilo Vasilijevic, Furu Wei
June 2021
Memory-Efficient Differentiable Transformer Architecture Search
Yuekai Zhao, Li Dong, Yelong Shen, Zhihua Zhang, Furu Wei, Wei Chen
ACL-IJCNLP 2021 | May 2021
Blow the Dog Whistle: A Chinese Dataset for Cant Understanding with Common Sense and World Knowledge
Canwen Xu, Wangchunshu Zhou, Tao Ge, Ke Xu, Julian J. McAuley, Furu Wei
April 2021
LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding
Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Furu Wei
April 2021
XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders.
Shuming Ma, Jian Yang, Haoyang Huang, Zewen Chi, Li Dong, Dongdong Zhang, Hany Hassan Awadalla, Alexandre Muzio, Akiko Eriguchi, Saksham Singhal, Xia Song, Arul Menezes, Furu Wei
December 2020
MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, Furu Wei
ACL-IJCNLP 2021 | December 2020
DocBank: A Benchmark Dataset for Document Layout Analysis
Minghao Li, Yiheng Xu, Lei Cui, Shaohan Huang, Furu Wei, Zhoujun Li, Ming Zhou
COLING 2020 | December 2020
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, Jianfeng Gao
ECCV | August 2020
LayoutLM: Pre-training of Text and Layout for Document Image Understanding
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou
KDD 2020 | August 2020
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, Ming Zhou
July 2020
Multimodal Matching Transformer for Live Commenting
Chaoqun Duan, Lei Cui, Shuming Ma, Furu Wei, Conghui Zhu, Tiejun Zhao
ECAI 2020 | July 2020
TableBank: Table Benchmark for Image-based Table Detection and Recognition
Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou, Zhoujun Li
LREC 2020 | May 2020
Self-Attention Attribution: Interpreting Information Interactions Inside Transformer
Yaru Hao, Li Dong, Furu Wei, Ke Xu
AAAI 2021 | April 2020
AAAI 2021 Best Paper Runner Up
UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Songhao Piao, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon
February 2020
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, Ming Zhou
NeurIPS 2020 | February 2020
Unified Language Model Pre-training for Natural Language Understanding and Generation
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon
33rd Conference on Neural Information Processing Systems (NeurIPS 2019) | December 2019
Neural Melody Composition from Lyrics
Hangbo Bao, Shaohan Huang, Furu Wei, Lei Cui, Yu Wu, Chuanqi Tan, Songhao Piao, Ming Zhou
NLPCC 2019 | September 2019
LiveBot: Generating Live Video Comments Based on Visual and Textual Contexts
Shuming Ma, Lei Cui, Damai Dai, Furu Wei, Xu Sun
AAAI 2019 | July 2019
Retrieval-Enhanced Adversarial Training for Neural Response Generation
Qingfu Zhu, Lei Cui, Wei-Nan Zhang, Furu Wei, Ting Liu
ACL 2019 | July 2019
SeRI: A Dataset for Sub-event Relation Inference from an Encyclopedia
Tao Ge, Lei Cui, Baobao Chang, Zhifang Sui, Furu Wei, Ming Zhou
NLPCC 2018 | August 2018
SuperAgent: A Customer Service Chatbot for E-commerce Websites
Lei Cui, Furu Wei, Shaohan Huang, Chuanqi Tan, Chaoqun Duan, Ming Zhou
ACL 2017 | July 2017
-
REDSTONE: Curating General, Code, Math, and QA Data for Large Language Models
Yaoyao Chang, Lei Cui, Li Dong, Shaohan Huang, Yangyu Huang, Yupan Huang, Scarlett Li, Tengchao Lv, Shuming Ma, Qinzheng Sun, Wenhui Wang, Furu Wei, Ying Xin, Mao Yang, Qiufeng Yin, Xingxing Zhang
December 2024
OpinionSeer: Interactive Visualization of Hotel Customer Feedback
Yingcai Wu, Furu Wei, Shixia Liu, Norman Au, Weiwei Cui, Hong Zhou, Huamin Qu
IEEE Transactions on Visualization and Computer Graphics | November 2010, pp. 1109-1118
Context-Preserving, Dynamic Word Cloud Visualization
Weiwei Cui, Yingcai Wu, Shixia Liu, Furu Wei, Michelle Zhou, Huamin Qu
IEEE Computer Graphics and Applications | September 2010, Vol 30(6): pp. 42
Context Preserving Dynamic Word Cloud Visualization
Weiwei Cui, Yingcai Wu, Shixia Liu, Furu Wei, Michelle Zhou, Huamin Qu
2010 IEEE Pacific Visualization Symposium (PacificVis) | March 2010
-
PEACE: Empowering Geologic Map Holistic Understanding with MLLMs
Yangyu Huang, Tianyi Gao, Haoran Xu, Qihao Zhao, Yang Song, Zhipeng Gui, Tengchao Lv, Hao Cheng, Lei Cui, Scarlett Li, Furu Wei
January 2025
Multimodal Large Language Models Make Text-to-Image Generative Models Align Better
Xun Wu, Shaohan Huang, Furu Wei
October 2024
Kosmos-2.5: A Multimodal Literate Model
Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, Shaoxiang Wu, Guoxin Wang, Cha Zhang, Furu Wei
September 2023
TextDiffuser: Diffusion Models as Text Painters
Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, Furu Wei
NeurIPS 2023 | May 2023
Language Is Not All You Need: Aligning Perception with Language Models
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Wei
NeurIPS 2023 | March 2023
TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models
Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, Furu Wei
AAAI 2023 | February 2023
LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking
Yupang Huang, Tengchao Lv, Lei Cui, Yutong Lu, Furu Wei
ACM Multimedia 2022 | October 2022
DiT: Self-supervised Pre-training for Document Image Transformer
Junlong Li, Yiheng Xu, Tengchao Lv, Lei Cui, Cha Zhang, Furu Wei
ACM Multimedia 2022 | October 2022
Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks
Wenhui Wang, Hangbo Bao, Li Dong, Kriti Aggarwal, Saksham Singhal, Subhojit Som, Furu Wei, Johan Bjorck, Zhiliang Peng, Qiang Liu, Owais Khan Mohammed
August 2022
Document AI: Benchmarks, Models and Applications
Lei Cui, Yiheng Xu, Tengchao Lv, Furu Wei
November 2021
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, Jianfeng Gao
ECCV | August 2020
TableBank: Table Benchmark for Image-based Table Detection and Recognition
Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou, Zhoujun Li
LREC 2020 | May 2020
VL-BERT: Pre-training of Generic Visual-Linguistic Representations
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, Jifeng Dai
Eighth International Conference on Learning Representations (ICLR) | April 2020
-
PEACE: Empowering Geologic Map Holistic Understanding with MLLMs
Yangyu Huang, Tianyi Gao, Haoran Xu, Qihao Zhao, Yang Song, Zhipeng Gui, Tengchao Lv, Hao Cheng, Lei Cui, Scarlett Li, Furu Wei
January 2025
-
A whole-slide foundation model for digital pathology from real-world data
Hanwen Xu, Naoto Usuyama, Jaspreet Bagga, Sheng Zhang, Rajesh Rao, Tristan Naumann, Cliff Wong, Zelalem Gero, Javier González, Yu Gu, Yanbo Xu, Mu-Hsin Wei, Wenhui Wang, Shuming Ma, Furu Wei, Jianwei Yang, Chunyuan Li, Jianfeng Gao, Jaylen Rosemon, Tucker Bower, Soohee Lee, R. Weerasinghe, Bill Wright, Ari Robicsek, Brian Piening, Carlo Bifulco, Sheng Wang, Hoifung Poon
Nature | May 2024
When an Image is Worth 1,024 x 1,024 Words: A Case Study in Computational Pathology
Wenhui Wang, Shuming Ma, Hanwen Xu, Naoto Usuyama, Jiayu Ding, Hoifung Poon, Furu Wei
December 2023
-
Autoregressive Speech Synthesis without Vector Quantization
Lingwei Meng, Long Zhou, Shujie Liu, Sanyuan Chen, Bing Han, Shujie Hu, Yanqing Liu, Jinyu Li, Sheng Zhao, Xixin Wu, Helen Meng, Furu Wei
July 2024
VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment
Bing Han, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Yanming Qian, Yanqing Liu, Sheng Zhao, Jinyu Li, Furu Wei
June 2024
VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers
Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, Furu Wei
June 2024
WavLLM: Towards Robust and Adaptive Speech Large Language Model
Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, Furu Wei
March 2024
Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, Furu Wei
January 2023
SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training
Ziqiang Zhang, Long Zhou, Junyi Ao, Shujie Liu, Lirong Dai, Jinyu Li, Furu Wei
EMNLP 2022 | October 2022
Separating Long-Form Speech with Group-Wise Permutation Invariant Training
Wangyou Zhang, Zhuo Chen, Naoyuki Kanda, Shujie Liu, Jinyu Li, Sefik Emre Eskimez, Takuya Yoshioka, Xiong Xiao, Zhong Meng, Yanmin Qian, Furu Wei
Interspeech 2022 | September 2022
UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Training
Sanyuan Chen, Yu Wu, Chengyi Wang, Zhengyang Chen, Zhuo Chen, Shujie Liu, Jian Wu, Yao Qian, Furu Wei, Jinyu Li, Xiangzhan Yu
ICASSP 2022 | May 2022
-
MathScale: Scaling Instruction Tuning for Mathematical Reasoning
Zhengyang Tang, Xingxing Zhang, Benyou Wang, Furu Wei
ICML 2024 | March 2024
-
Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding
Xin Sun, Tao Ge, Furu Wei, Houfeng Wang
ACL-IJCNLP 2021 | June 2021
-
Self-Attention Attribution: Interpreting Information Interactions Inside Transformer
Yaru Hao, Li Dong, Furu Wei, Ke Xu
AAAI 2021 | April 2020
AAAI 2021 Best Paper Runner Up
Publications by Type
-
Boosting Large Language Model for Speech Synthesis: An Empirical Study
Hongkun Hao, Long Zhou, Shujie Liu, Jinyu Li, Shujie Hu, Rui Wang, Furu Wei
ICASSP | April 2025
Little Giants: Synthesizing High-Quality Embedding Data at Scale
Haonan Chen, Liang Wang, Nan Yang, Yutao Zhu, Ziliang Zhao, Furu Wei, Zhicheng Dou
NAACL 2025 | October 2024
xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token
Xin Cheng, Xun Wang, Xingxing Zhang, Tao Ge, Si-Qing Chen, Furu Wei, Huishuai Zhang, Dongyan Zhao
NeurIPS 2024 | May 2024
You Only Cache Once: Decoder-Decoder Architectures for Language Models
Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, Furu Wei
NeurIPS 2024 | May 2024
Kosmos-G: Generating Images in Context with Multimodal Large Language Models
Xichen Pan, Li Dong, Shaohan Huang, Zhiliang Peng, Wenhu Chen, Furu Wei
ICLR 2024 | May 2024
Multi-Head Mixture-of-Experts
Xun Wu, Shaohan Huang, Wenhui Wang, Furu Wei
NeurIPS 2024 | April 2024
LongEmbed: Extending Embedding Models for Long Context Retrieval
Dawei Zhu, Liang Wang, Nan Yang, Yifan Song, Wenhao Wu, Furu Wei, Sujian Li
EMNLP 2024 | April 2024
Mind’s Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models
Wenshan Wu, Shaoguang Mao, Yadong Zhang, Yan Xia, Li Dong, Lei Cui, Furu Wei
NeurIPS 2024 | April 2024
Mixture of LoRA Experts
ICLR 2024 | April 2024
MathScale: Scaling Instruction Tuning for Mathematical Reasoning
Zhengyang Tang, Xingxing Zhang, Benyou Wang, Furu Wei
ICML 2024 | March 2024
Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models
Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, Ji-Rong Wen
ACL 2024 | February 2024
Generative Representational Instruction Tuning
Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, Douwe Kiela
ICLR 2025 | February 2024
Text Diffusion with Reinforced Conditioning
Yuxuan Liu, Tianchi Yang, Shaohan Huang, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
AAAI 2024 | February 2024
Improving Text Embeddings with Large Language Models
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei
ACL 2024 | January 2024
Fine-Tuning LLaMA for Multi-Stage Text Retrieval
Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, Jimmy Lin
SIGIR 2024 | October 2023
Augmenting Language Models with Long-Term Memory
Weizhi Wang, Li Dong, Hao Cheng, Xiaodong Liu, Xifeng Yan, Jianfeng Gao, Furu Wei
NeurIPS 2023 | October 2023
PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training
Dawei Zhu, Nan Yang, Liang Wang, Yifan Song, Wenhao Wu, Furu Wei, Sujian Li
ICLR 2024 | October 2023
Adapting Large Language Models via Reading Comprehension
Daixuan Cheng, Shaohan Huang, Furu Wei
ICLR 2024 | September 2023
In-context Autoencoder for Context Compression in a Large Language Model
Tao Ge, Jing Hu, Haixun Wang, Si-Qing Chen, Furu Wei
ICLR 2024 | July 2023
Learning to Retrieve In-Context Examples for Large Language Models
Liang Wang, Nan Yang, Furu Wei
EACL 2024 | July 2023
Knowledge Distillation of Large Language Models
Yuxian Gu, Li Dong, Furu Wei, Minlie Huang
ICLR 2024 | June 2023
Learning to Rank in Generative Retrieval
Yongqi Li, Nan Yang, Liang Wang, Furu Wei, Wenjie Li
AAAI 2024 | June 2023
TextDiffuser: Diffusion Models as Text Painters
Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, Furu Wei
NeurIPS 2023 | May 2023
Multiview Identifiers Enhanced Generative Retrieval
Yongqi Li, Nan Yang, Liang Wang, Furu Wei, Wenjie Li
ACL 2023 | May 2023
Language Is Not All You Need: Aligning Perception with Language Models
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, Furu Wei
NeurIPS 2023 | March 2023
Query2doc: Query Expansion with Large Language Models
Liang Wang, Nan Yang, Furu Wei
EMNLP 2023 | March 2023
TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models
Minghao Li, Tengchao Lv, Jingye Chen, Lei Cui, Yijuan Lu, Dinei Florencio, Cha Zhang, Zhoujun Li, Furu Wei
AAAI 2023 | February 2023
Text Embeddings by Weakly-Supervised Contrastive Pre-training
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang (姜大昕), Rangan Majumder, Furu Wei
December 2022
Beyond English-Centric Bitexts for Better Multilingual Language Representation Learning
Barun Patra, Saksham Singhal, Shaohan Huang, Zewen Chi, Li Dong, Furu Wei, Vishrav Chaudhary, Xia Song
ACL 2023 | October 2022
Work in progress
LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking
Yupang Huang, Tengchao Lv, Lei Cui, Yutong Lu, Furu Wei
ACM Multimedia 2022 | October 2022
DiT: Self-supervised Pre-training for Document Image Transformer
Junlong Li, Yiheng Xu, Tengchao Lv, Lei Cui, Cha Zhang, Furu Wei
ACM Multimedia 2022 | October 2022
SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training
Ziqiang Zhang, Long Zhou, Junyi Ao, Shujie Liu, Lirong Dai, Jinyu Li, Furu Wei
EMNLP 2022 | October 2022
Separating Long-Form Speech with Group-Wise Permutation Invariant Training
Wangyou Zhang, Zhuo Chen, Naoyuki Kanda, Shujie Liu, Jinyu Li, Sefik Emre Eskimez, Takuya Yoshioka, Xiong Xiao, Zhong Meng, Yanmin Qian, Furu Wei
Interspeech 2022 | September 2022
SimLM: Pre-training with Representation Bottleneck for Dense Passage Retrieval
Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang (姜大昕), Rangan Majumder, Furu Wei
ACL 2023 | July 2022
UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Training
Sanyuan Chen, Yu Wu, Chengyi Wang, Zhengyang Chen, Zhuo Chen, Shujie Liu, Jian Wu, Yao Qian, Furu Wei, Jinyu Li, Xiangzhan Yu
ICASSP 2022 | May 2022
XFUND: A Benchmark Dataset for Multilingual Visually Rich Form Understanding
Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Furu Wei
ACL 2022 | May 2022
Towards Making the Most of Multilingual Pretraining for Zero-Shot Neural Machine Translation
Guanhua Chen, Shuming Ma, Yun Chen, Dongdong Zhang, Jia Pan, Wenping Wang, Furu Wei
ACL 2022 | May 2022
StableMoE: Stable Routing Strategy for Mixture of Experts
Damai Dai, Li Dong, Shuming Ma, Bo Zheng, Zhifang Sui, Baobao Chang, Furu Wei
ACL 2022 | May 2022
SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing
Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, Furu Wei
ACL 2022 | May 2022
Neural Label Search for Zero-Shot Multi-Lingual Extractive Summarization
Ruipeng Jia, Xingxing Zhang, Yanan Cao, Shi Wang, Zheng Lin, Furu Wei
ACL 2022 | May 2022
Knowledge Neurons in Pretrained Transformers
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, Furu Wei
ACL 2022 | May 2022
Controllable Natural Language Generation with Contrastive Prefixes
Jing Qian, Li Dong, Yelong Shen, Furu Wei, Weizhu Chen
ACL 2022 | May 2022
CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailment
Haoyu Song, Li Dong, Wei-Nan Zhang, Ting Liu, Furu Wei
ACL 2022 | May 2022
Attention Temperature Matters in Abstractive Summarization Distillation
Shengqiang Zhang, Xingxing Zhang, Hangbo Bao, Furu Wei
ACL 2022 | May 2022
MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Understanding
Junlong Li, Yiheng Xu, Lei Cui, Furu Wei
ACL 2022 | May 2022
BEiT: BERT Pre-Training of Image Transformers
Hangbo Bao, Li Dong, Songhao Piao, Furu Wei
ICLR 2022 | April 2022
Sequence Level Contrastive Learning for Text Summarization
Shusheng Xu, Xingxing Zhang, Yi Wu, Furu Wei
AAAI 2022 | January 2022
Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task.
Jian Yang, Shuming Ma, Haoyang Huang, Dongdong Zhang, Li Dong, Shaohan Huang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
WMT | November 2021
LayoutReader: Pre-training of Text and Layout for Reading Order Detection
Zilong Wang, Yiheng Xu, Lei Cui, Jingbo Shang, Furu Wei
EMNLP 2021 | November 2021
MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs
Zewen Chi, Li Dong, Shuming Ma, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei
EMNLP 2021 | October 2021
Zero-shot Cross-lingual Transfer of Neural Machine Translation with Multilingual Pretrained Encoders
Guanhua Chen, Shuming Ma, Yun Chen, Li Dong, Dongdong Zhang, Jia Pan, Wenping Wang, Furu Wei
EMNLP 2021 | October 2021
Jointly Learning to Repair Code and Generate Commit Message
Jiaqi Bai, Long Zhou, Ambrosio Blanco, Shujie Liu, Furu Wei, Ming Zhou, Zhoujun Li
2021 Empirical Methods in Natural Language Processing | September 2021
Improving Sequence-to-Sequence Pre-training via Sequence Span Rewriting
Wangchunshu Zhou, Tao Ge, Ke Xu, Furu Wei
2021 Empirical Methods in Natural Language Processing | September 2021
Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compression
Canwen Xu, Wangchunshu Zhou, Tao Ge, Ke Xu, Julian McAuley, Furu Wei
2021 Empirical Methods in Natural Language Processing | September 2021
Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training
Bo Zheng, Li Dong, Shaohan Huang, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
EMNLP 2021 | September 2021
Document AI: Benchmarks, Models and Applications (In Chinese)
Lei Cui, Yiheng Xu, Tengchao Lv, Furu Wei
CCL 2021 | August 2021
UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data
Chengyi Wang, Yu Wu, Yao Qian, Kenichi Kumatani, Shujie Liu, Furu Wei, Michael Zeng, Xuedong Huang
2021 International Conference on Machine Learning | July 2021
Grammar-Based Patches Generation for Automated Program Repair
Yu Tang, Long Zhou, Ambrosio Blanco, Shujie Liu, Furu Wei, Ming Zhou, Muyun Yang
ACL-IJCNLP 2021 | July 2021
Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains
Yunzhi Yao, Shaohan Huang, Wenhui Wang, Li Dong, Furu Wei
ACL-IJCNLP 2021 | July 2021
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA
Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Saksham Singhal, Payal Bajaj, Xia Song, Furu Wei
ACL 2022 | June 2021
Learning to Sample Replacements for ELECTRA Pre-Training
Yaru Hao, Li Dong, Hangbo Bao, Ke Xu, Furu Wei
ACL-IJCNLP 2021 | June 2021
Consistency Regularization for Cross-Lingual Fine-Tuning
Bo Zheng, Li Dong, Shaohan Huang, Wenhui Wang, Zewen Chi, Saksham Singhal, Wanxiang Che, Ting Liu, Xia Song, Furu Wei
ACL-IJCNLP 2021 | June 2021
Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment
Zewen Chi, Li Dong, Bo Zheng, Shaohan Huang, Xian-Ling Mao, Heyan Huang, Furu Wei
ACL-IJCNLP 2021 | June 2021
Instantaneous Grammatical Error Correction with Shallow Aggressive Decoding
Xin Sun, Tao Ge, Furu Wei, Houfeng Wang
ACL-IJCNLP 2021 | June 2021
Memory-Efficient Differentiable Transformer Architecture Search
Yuekai Zhao, Li Dong, Yelong Shen, Zhihua Zhang, Furu Wei, Wei Chen
ACL-IJCNLP 2021 | May 2021
LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding
Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, Lidong Zhou
ACL-IJCNLP 2021 | January 2021
MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformers
Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, Furu Wei
ACL-IJCNLP 2021 | December 2020
DocBank: A Benchmark Dataset for Document Layout Analysis
Minghao Li, Yiheng Xu, Lei Cui, Shaohan Huang, Furu Wei, Zhoujun Li, Ming Zhou
COLING 2020 | December 2020
Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, Jianfeng Gao
ECCV | August 2020
LayoutLM: Pre-training of Text and Layout for Document Image Understanding
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou
KDD 2020 | August 2020
Multimodal Matching Transformer for Live Commenting
Chaoqun Duan, Lei Cui, Shuming Ma, Furu Wei, Conghui Zhu, Tiejun Zhao
ECAI 2020 | July 2020
TableBank: Table Benchmark for Image-based Table Detection and Recognition
Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou, Zhoujun Li
LREC 2020 | May 2020
Self-Attention Attribution: Interpreting Information Interactions Inside Transformer
Yaru Hao, Li Dong, Furu Wei, Ke Xu
AAAI 2021 | April 2020
AAAI 2021 Best Paper Runner Up
VL-BERT: Pre-training of Generic Visual-Linguistic Representations
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, Jifeng Dai
Eighth International Conference on Learning Representations (ICLR) | April 2020
Self-Adversarial Learning with Comparative Discrimination for Text Generation
Wangchunshu Zhou, Tao Ge, Ke Xu, Furu Wei, Ming Zhou
Eighth International Conference on Learning Representations (ICLR) | April 2020
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, Ming Zhou
NeurIPS 2020 | February 2020
Unified Language Model Pre-training for Natural Language Understanding and Generation
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon
33rd Conference on Neural Information Processing Systems (NeurIPS 2019) | December 2019
Neural Melody Composition from Lyrics
Hangbo Bao, Shaohan Huang, Furu Wei, Lei Cui, Yu Wu, Chuanqi Tan, Songhao Piao, Ming Zhou
NLPCC 2019 | September 2019
LiveBot: Generating Live Video Comments Based on Visual and Textual Contexts
Shuming Ma, Lei Cui, Damai Dai, Furu Wei, Xu Sun
AAAI 2019 | July 2019
Retrieval-Enhanced Adversarial Training for Neural Response Generation
Qingfu Zhu, Lei Cui, Wei-Nan Zhang, Furu Wei, Ting Liu
ACL 2019 | July 2019
Fine-grained Coordinated Cross-lingual Text Stream Alignment for Endless Language Knowledge Acquisition
Tao Ge, Qing Dou, Heng Ji, Lei Cui, Baobao Chang, Zhifang Sui, Furu Wei, Ming Zhou
EMNLP 2018 | November 2018
SeRI: A Dataset for Sub-event Relation Inference from an Encyclopedia
Tao Ge, Lei Cui, Baobao Chang, Zhifang Sui, Furu Wei, Ming Zhou
NLPCC 2018 | August 2018
Neural Open Information Extraction
ACL 2018 | July 2018
Attention-Fused Deep Matching Network for Natural Language Inference
Chaoqun Duan, Lei Cui, Xinchi Chen, Furu Wei, Conghui Zhu, Tiejun Zhao
IJCAI 2018 | July 2018
EventWiki: A Knowledge Base of Major Events
Tao Ge, Lei Cui, Baobao Chang, Zhifang Sui, Furu Wei, Ming Zhou
LREC 2018 | May 2018
Leveraging Contextual Sentence Relations for Extractive Summarization Using a Neural Attention Model
Pengjie Ren, Zhumin Chen, Zhaochun Ren, Furu Wei, Jun Ma, Maarten de Rijke
SIGIR '17, Shinjuku, Tokyo, Japan, August 07-11, 2017 | August 2017
SuperAgent: A Customer Service Chatbot for E-commerce Websites
Lei Cui, Furu Wei, Shaohan Huang, Chuanqi Tan, Chaoqun Duan, Ming Zhou
ACL 2017 | July 2017
QuickView: advanced search of tweets
Xiaohua Liu, Long Jiang, Furu Wei, Ming Zhou, QuickView Team Microsoft
SIGIR (Demo paper) | January 2011
ISBN: 978-1-4503-0757-4
Context Preserving Dynamic Word Cloud Visualization
Weiwei Cui, Yingcai Wu, Shixia Liu, Furu Wei, Michelle Zhou, Huamin Qu
2010 IEEE Pacific Visualization Symposium (PacificVis) | March 2010
-
Examining False Positives under Inference Scaling for Mathematical Reasoning
Yu Wang, Nan Yang, Liang Wang, Furu Wei
MSR-TR-2025-2 | February 2025
Published by Microsoft
Differential Transformer
Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, Furu Wei
MSR-TR-2024-42 | October 2024
Published by Microsoft
Multilingual E5 Text Embeddings: A Technical Report
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei
MSR-TR-2024-45 | February 2024
Published by Microsoft
Inference with Reference: Lossless Acceleration of Large Language Models
Nan Yang, Tao Ge, Liang Wang, Binxing Jiao, Daxin Jiang (姜大昕), Linjun Yang, Rangan Majumder, Furu Wei
MSR-TR-2023-17 | April 2023
Published by Microsoft
-
Chain-of-Retrieval Augmented Generation
Liang Wang, Haonan Chen, Nan Yang, Xiaolong Huang, Zhicheng Dou, Furu Wei
January 2025
MMLU-CF: A Contamination-free Multi-task Language Understanding Benchmark
Qihao Zhao, Yangyu Huang, Tengchao Lv, Lei Cui, Furu Wei, Qinzheng Sun, Ying Xin, Shaoguang Mao, Xin Zhang, Qiufeng Yin, Scarlett Li
December 2024
BitNet a4.8: 4-bit Activations for 1-bit LLMs
Hongyu Wang, Shuming Ma, Furu Wei
November 2024
Work in progress
1-bit AI Infra: Part 1.1, Fast and Lossless BitNet b1.58 Inference on CPUs
Jinheng Wang, Hansong Zhou, Ting Song, Shaoguang Mao, Shuming Ma, Hongyu Wang, Yan Xia, Furu Wei
October 2024
Q-Sparse: All Large Language Models can be Fully Sparsely-Activated
Hongyu Wang, Shuming Ma, Ruiping Wang, Furu Wei
July 2024
Autoregressive Speech Synthesis without Vector Quantization
Lingwei Meng, Long Zhou, Shujie Liu, Sanyuan Chen, Bing Han, Shujie Hu, Yanqing Liu, Jinyu Li, Sheng Zhao, Xixin Wu, Helen Meng, Furu Wei
July 2024
VALL-E R: Robust and Efficient Zero-Shot Text-to-Speech Synthesis via Monotonic Alignment
Bing Han, Long Zhou, Shujie Liu, Sanyuan Chen, Lingwei Meng, Yanming Qian, Yanqing Liu, Sheng Zhao, Jinyu Li, Furu Wei
June 2024
VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers
Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, Furu Wei
June 2024
Direct Preference Knowledge Distillation for Large Language Models
Yixing Li, Yuxian Gu, Li Dong, Dequan Wang, Yu Cheng, Furu Wei
June 2024
Instruction Pre-Training: Language Models are Supervised Multitask Learners
Daixuan Cheng, Yuxian Gu, Shaohan Huang, Junyu Bi, Minlie Huang, Furu Wei
June 2024
VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers
Sanyuan Chen, Shujie Liu, Long Zhou, Yanqing Liu, Xu Tan, Jinyu Li, Sheng Zhao, Yao Qian, Furu Wei
June 2024
MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, Fuzhen Zhuang
May 2024
LLM as a Mastermind: A Survey of Strategic Reasoning with Large Language Models
Yadong Zhang, Shaoguang Mao, Tao Ge, Xun Wang, Adrian de Wynter, Yan Xia, Wenshan Wu, Ting Song, Man Lan, Furu Wei
March 2024
WavLLM: Towards Robust and Adaptive Speech Large Language Model
Shujie Hu, Long Zhou, Shujie Liu, Sanyuan Chen, Hongkun Hao, Jing Pan, Xunying Liu, Jinyu Li, Sunit Sivasankaran, Linquan Liu, Furu Wei
March 2024
ResLoRA: Identity Residual Mapping in Low-Rank Adaption
Shuhua Shi, Shaohan Huang, Minghui Song, Zhoujun Li, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang
February 2024
Towards Optimal Learning of Language Models
Yuxian Gu, Li Dong, Yaru Hao, Qingxiu Dong, Minlie Huang, Furu Wei
February 2024
The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Lifeng Dong, Ruiping Wang, Jilong Xue, Furu Wei
February 2024
Work in progress
Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models
Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, Yuxian Gu, Xin Cheng, Xun Wang, Si-Qing Chen, Li Dong, Wei Lu, Zhifang Sui, Benyou Wang, Wai Lam, Furu Wei
February 2024
Work in progress
When an Image is Worth 1,024 x 1,024 Words: A Case Study in Computational Pathology
Wenhui Wang, Shuming Ma, Hanwen Xu, Naoto Usuyama, Jiayu Ding, Hoifung Poon, Furu Wei
December 2023
BitNet: Scaling 1-bit Transformers for Large Language Models
Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, Furu Wei
October 2023
Retentive Network: A Successor to Transformer for Large Language Models
Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, Furu Wei
August 2023
LongNet: Scaling Transformers to 1,000,000,000 Tokens
Jiayu Ding, Shuming Ma, Li Dong, Xingxing Zhang, Shaohan Huang, Wenhui Wang, Furu Wei
July 2023
Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers
Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, Lei He, Sheng Zhao, Furu Wei
January 2023
Learning Diverse Document Representations with Deep Query Interactions for Dense Retrieval
Zehan Li, Nan Yang, Liang Wang, Furu Wei
August 2022
Lossless Acceleration for Seq2seq Generation with Aggressive Decoding
Tao Ge, Heming Xia, Xin Sun, Si-Qing Chen, Furu Wei
May 2022
Blow the Dog Whistle: A Chinese Dataset for Cant Understanding with Common Sense and World Knowledge
Canwen Xu, Wangchunshu Zhou, Tao Ge, Ke Xu, Julian J. McAuley, Furu Wei
April 2021
XLM-T: Scaling up Multilingual Machine Translation with Pretrained Cross-lingual Transformer Encoders.
Shuming Ma, Jian Yang, Haoyang Huang, Zewen Chi, Li Dong, Dongdong Zhang, Hany Hassan Awadalla, Alexandre Muzio, Akiko Eriguchi, Saksham Singhal, Xia Song, Arul Menezes, Furu Wei
December 2020
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training
Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, Ming Zhou
July 2020
UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training
Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Songhao Piao, Jianfeng Gao, Ming Zhou, Hsiao-Wuen Hon
February 2020
-
Bootstrap Your Own Context Length
Liang Wang, Nan Yang, Xingxing Zhang, Xiaolong Huang, Furu Wei
ArXiv | December 2024, Vol abs/2412.18860
A whole-slide foundation model for digital pathology from real-world data
Hanwen Xu, Naoto Usuyama, Jaspreet Bagga, Sheng Zhang, Rajesh Rao, Tristan Naumann, Cliff Wong, Zelalem Gero, Javier González, Yu Gu, Yanbo Xu, Mu-Hsin Wei, Wenhui Wang, Shuming Ma, Furu Wei, Jianwei Yang, Chunyuan Li, Jianfeng Gao, Jaylen Rosemon, Tucker Bower, Soohee Lee, R. Weerasinghe, Bill Wright, Ari Robicsek, Brian Piening, Carlo Bifulco, Sheng Wang, Hoifung Poon
Nature | May 2024
SpeechLM: Enhanced Speech Pre-Training With Unpaired Textual Data
Ziqiang Zhang, Sanyuan Chen, Long Zhou, Yu Wu, Shuo Ren, Shujie Liu, Zhuoyuan Yao, Xun Gong, Lirong Dai, Jinyu Li, Furu Wei
IEEE/ACM Transactions on Audio, Speech, and Language Processing | March 2024
Large Search Model: Redefining Search Stack in the Era of LLMs
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, Furu Wei
SIGIR Forum | December 2023, Vol 57(2)
Generative Retrieval for Conversational Question Answering
Yongqi Li, Nan Yang, Liang Wang, Furu Wei, Wenjie Li
Information Processing and Management | August 2023, Vol 60(5)
VatLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning
Qiushi Zhu, Long Zhou, Ziqiang Zhang, Shujie Liu, Binxing Jiao, Jie Zhang, Lirong Dai, Daxin Jiang, Jinyu Li, Furu Wei
IEEE Transactions on Multimedia | May 2023
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, Furu Wei
IEEE Journal of Selected Topics in Signal Processing | July 2022, Vol 16(6): pp. 1505-1518
DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
arXiv: Computation and Language | June 2021
OpinionSeer: Interactive Visualization of Hotel Customer Feedback
Yingcai Wu, Furu Wei, Shixia Liu, Norman Au, Weiwei Cui, Hong Zhou, Huamin Qu
IEEE Transactions on Visualization and Computer Graphics | November 2010, pp. 1109-1118
Context-Preserving, Dynamic Word Cloud Visualization
Weiwei Cui, Yingcai Wu, Shixia Liu, Furu Wei, Michelle Zhou, Huamin Qu
IEEE Computer Graphics and Applications | September 2010, Vol 30(6): pp. 42
-
ARLON: Boosting Diffusion Transformers with Autoregressive Models for Long Video Generation
Zongyi Li, Shujie Hu, Shujie Liu, Long Zhou, Jeongsoo Choi, Lingwei Meng, Xun Guo, Jinyu Li, Hefei Ling, Furu Wei
October 2024
October 2024
-
PEACE: Empowering Geologic Map Holistic Understanding with MLLMs
Yangyu Huang, Tianyi Gao, Haoran Xu, Qihao Zhao, Yang Song, Zhipeng Gui, Tengchao Lv, Hao Cheng, Lei Cui, Scarlett Li, Furu Wei
January 2025
REDSTONE: Curating General, Code, Math, and QA Data for Large Language Models
Yaoyao Chang, Lei Cui, Li Dong, Shaohan Huang, Yangyu Huang, Yupan Huang, Scarlett Li, Tengchao Lv, Shuming Ma, Qinzheng Sun, Wenhui Wang, Furu Wei, Ying Xin, Mao Yang, Qiufeng Yin, Xingxing Zhang
December 2024
Multimodal Large Language Models Make Text-to-Image Generative Models Align Better
Xun Wu, Shaohan Huang, Furu Wei
October 2024
Boosting Text-to-Video Generative Model with MLLMs Feedback
Xun Wu, Shaohan Huang, Furu Wei
October 2024
Kosmos-2.5: A Multimodal Literate Model
Tengchao Lv, Yupan Huang, Jingye Chen, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, Li Dong, Weiyao Luo, Shaoxiang Wu, Guoxin Wang, Cha Zhang, Furu Wei
September 2023
Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks
Wenhui Wang, Hangbo Bao, Li Dong, Kriti Aggarwal, Saksham Singhal, Subhojit Som, Furu Wei, Johan Bjorck, Zhiliang Peng, Qiang Liu, Owais Khan Mohammed
August 2022
THE-X: Privacy-Preserving Transformer Inference with Homomorphic Encryption
Tianyu Chen, Hangbo Bao, Shaohan Huang, Li Dong, Binxing Jiao, Daxin Jiang (姜大昕), Haoyi Zhou, Jianxin Li, Furu Wei
May 2022
Document AI: Benchmarks, Models and Applications
Lei Cui, Yiheng Xu, Tengchao Lv, Furu Wei
November 2021
Multilingual Agreement for Multilingual Neural Machine Translation
Jian Yang, Yuwei Yin, Shuming Ma, Haoyang Huang, Dongdong Zhang, Zhoujun Li, Furu Wei
July 2021
DeltaLM: Encoder-Decoder Pre-training for Language Generation and Translation by Augmenting Pretrained Multilingual Encoders.
Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, Furu Wei
June 2021
VT-SSum: A Benchmark Dataset for Video Transcript Segmentation and Summarization
Tengchao Lv, Lei Cui, Momcilo Vasilijevic, Furu Wei
June 2021
LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding
Yiheng Xu, Tengchao Lv, Lei Cui, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Furu Wei
April 2021