Microsoft at NeurIPS 2024

Reinforcement Learning Under Latent Dynamics: Toward Statistical and Algorithmic Modularity

Philip Amortila, Dylan Foster, Nan Jiang, Akshay Krishnamurthy, Zakaria Mhammedi

Accuracy is Not All You Need

Abhinav Dutta, Sanjeev Krishnan, Nipun Kwatra, Ramachandran Ramjee

Blind Image Restoration via Fast Diffusion Inversion

Hamadi Chihaoui, Abdelhak Lemkhenter, Paolo Favaro

Compositional Generalization Across Distributional Shifts with Sparse Tree Operations

Paul Smolensky, Jianfeng Gao, Roland Fernandez

Gorilla: Teaching LLMs to Use Tools

Xin Wang

Leveraging Visual Tokens for Extended Text Contexts in Multi-Modal Learning

Alex Jinpeng Wang, Linjie Li, Yiqi Lin, Min Li, Lijuan Wang, Mike Zheng Shou

Towards Flexible Visual Relationship Segmentation

Fangrui Zhu, Jianwei Yang, Huaizu Jiang

A Large-Scale Human-Centric Benchmark for Referring Expression Comprehension in the LMM Era

Fangyun Wei

BPQP: A Differentiable Convex Optimization Framework for Efficient End-to-End Learning

Xiao Yang, Xu Yang, Weiqing Liu, Lewen Wang, Jiang Bian

Blind Image Restoration via Fast Diffusion Inversion

Hamadi Chihaoui, Abdelhak Lemkhenter, Paolo Favaro

Cooperation, Competition, and Maliciousness: LLM-Stakeholders Interactive Negotiation

Sahar Abdelnabi, Amr Gomaa, S. Sivaprasad, Lea Schonherr, Mario Fritz

Implicit Curriculum in Procgen Made Explicit

Kaixin Wang, Xinchao Wang

Motion Graph Unleashed: A Novel Approach to Video Prediction

Luming Liang, Ilya Zharkov

Semi-Random Matrix Completion via Flow-Based Adaptive Reweighting

Jerry Li

VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time

Sicheng Xu, Guojun Chen, Yu-Xiao Guo, Jiaolong Yang, Chong Li, Zhenyu Zang, Yizhong Zhang, Xin Tong, Baining Guo

A Study of Plasticity Loss in On-Policy Deep Reinforcement Learning

Arthur Juliani, Jordan Ash

Advancing Spiking Neural Networks for Sequential Modeling with Central Pattern Generators

Changze Lv, Dongqi Han, Yansen Wang, Xiaoqing Zheng, Xuanjing Huang, Dongsheng Li

Approximately Equivariant Neural Processes

Matthew Ashman, Cristiana-Diana Diaconu, Adrian Weller, Wessel Bruinsma, Richard Turner

Can Graph Learning Improve Task Planning?

Xixi Wu, Yifei Shen, Caihua Shan, Kaitao Song, Siwei Wang, Bohang Zhang, Jiarui Feng, Hong Cheng, Wei Chen, Yun Xiong, Dongsheng Li

ElasTST: Towards Robust Varied-Horizon Forecasting with Elastic Time-Series Transformer

Shun Zheng, Xumeng Wen, Jiang Bian

GaussianCube: A Structured and Explicit Radiance Representation for 3D Generative Modeling

Bowen Zhang, Yiji Cheng, Jiaolong Yang, Chunyu Wang, Feng Zhao, Yansong Tang, Dong Chen, Baining Guo

Motion Consistency Model: Accelerating Video Diffusion with Disentangled Motion-Appearance Distillation

Yuanhao Zhai, Kevin Lin, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Chung-Ching Lin, David Doermann, Junsong Yuan, Lijuan Wang

Multimodal Large Language Models Make Text-to-Image Generative Models Align Better

Xun Wu, Shaohan Huang, Furu Wei

Predictor-Corrector Enhanced Transformers with Exponential Moving Average Coefficient Learning

Rui Wang, Junliang Guo, Xu Tan

VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Time

Sicheng Xu, Guojun Chen, Yu-Xiao Guo, Jiaolong Yang, Chong Li, Zhenyu Zang, Yizhong Zhang, Xin Tong, Baining Guo

Autoformalize Mathematical Statements by Symbolic Equivalence and Semantic Consistency

Zenan Li, Yifan Wu, Zhaoyu Li, Xinming Wei, Fan Yang, Xian Zhang, Xiaoxing Ma

Amortized Active Causal Induction with Deep Reinforcement Learning

Yashas Annadani, P. Tigas, Stefan Bauer, Adam Foster, Adam Foster

Imprecise Label Learning: A Unified Framework for Learning with Various Imprecise Label Configurations

Hao Chen, Ankit Shah, Jindong Wang, R. Tao, Yidong Wang, Xing Xie, Masashi Sugiyama, Rita Singh, B. Raj

Online Estimation via Offline Estimation: An Information-Theoretic Framework

Dylan Foster, Yanjun Han, Jian Qian, Alexander Rakhlin

SureMap: Simultaneous mean estimation for single-task and multi-task disaggregated evaluation

Lester Mackey, Miro Dudík, Alex Chouldechova

Not All Tokens Are What You Need for Pretraining

Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Jian Jiao, Nan Duan, Weizhu Chen

Boosting Text-to-Video Generative Model with MLLMs Feedback

Xun Wu, Shaohan Huang, Furu Wei

Bridging Geometric States via Generative Modeling

Shengjie Luo, Yixian Xu, Di He, Shuxin Zheng, Tie-Yan Liu, Liwei Wang

Crafting Interpretable Embeddings by Asking LLMs Questions

Vinamra Benara, Chandan Singh, John X. Morris, Richard Antonello, Ion Stoica, Alexander Huth, Jianfeng Gao

Efficient Adversarial Training in LLMs with Continuous Attacks

Sophie Xhonneux, Alessandro Sordoni, Stephan Günnemann, Gauthier Gidel, Leo Schwinn

Human-Aware Vision-and-Language Navigation: Bridging Simulation to Reality with Dynamic Human Interactions

Minghan Li, Heng Li, Zhi-Qi Cheng, Yifei Dong, Yuxuan Zhou, Jun-Yan He, Qi Dai, Teruko Mitamura, Alexander G. Hauptmann

Identifying Equivalent Training Dynamics

William T. Redman, J. M. Bello-Rivas, M. Fonoberova, Ryan Mohr, I. Kevrekidis, Igor Mezi'c

Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models

Vibhav Vineet, Xin Wang, Neel Joshi

Improving the Training of Rectified Flows

Sangyun Lee, Zinan Lin, Giulia Fanti

MEDIQ: Question-Asking LLMs for Adaptive and Reliable Clinical Reasoning

Shuyue Stella Li, Vidhisha Balachandran, Shangbin Feng, Jonathan Ilgen, Emma Pierson, Pang Wei Koh, Yulia Tsvetkov

Mind's Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models

Wenshan Wu, Shaoguang Mao, Yadong Zhang, Yan Xia, Li Dong, Lei Cui, Furu Wei

Neuro-Symbolic Data Generation for Math Reasoning

Zenan Li, Zhi Zhou, Yuan Yao, Yu-Feng Li, Chun Cao, Fan Yang, Xian Zhang, Xiaoxing Ma

PromptFix: You Prompt and We Fix the Photo

Yongsheng Yu, Ziyun Zeng, Hang Hua, Jianlong Fu, Jiebo Luo

Protecting Your LLMs with Information Bottleneck

Zichuan Liu, Zefan Wang, Linjie Xu, Jinyu Wang, Lei Song, Tianchun Wang, Chunlin Chen, Wei Cheng, Jiang Bian

SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretraining

Andi Han, Jiaxiang Li, Wei Huang, Mingyi Hong, Akiko Takeda, Pratik Jawanpuria, Bamdev Mishra

The Art of Saying No: Contextual Noncompliance in Language Models

Faeze Brahman, Sachin Kumar, Vidhisha Balachandran, Pradeep Dasigi, Valentina Pyatkin, Abhilasha Ravichander, Sarah Wiegreffe, Nouha Dziri, K. Chandu, Jack Hessel, Yulia Tsvetkov, Noah A. Smith, Yejin Choi, Hanna Hajishirzi, Vidhisha Balachandran

VideoGUI: A Benchmark for GUI Automation from Instructional Videos

Kevin Qinghong Lin, Linjie Li, Difei Gao, Qinchen Wu, Mingyi Yan, Zhengyuan Yang, Lijuan Wang, Mike Zheng Shou

How to Solve Contextual Goal-Oriented Problems with Offline Datasets?

Ying Fan, Jingling Li, Adith Swaminathan, Aditya Modi, Ching-An Cheng

Noise-Aware Differentially Private Regression via Meta-Learning

Ossi Räisä, Stratis Markou, Matthew Ashman, Wessel Bruinsma, Marlon Tobaben, Antti Honkela, Richard Turner

ReXTime: A Benchmark Suite for Reasoning-Across-Time in Videos

Jr-Jen Chen, Yu-Chien Liao, Hsi-Che Lin, Yu-Chu Yu, Yen-Chun Chen, Yu-Chiang Frank Wang

TaskBench: Benchmarking Large Language Models for Task Automation

Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dongsheng Li, Y. Zhuang

CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark

David Romero, Chenyang Lyu, Haryo Akbarianto Wibowo, Teresa Lynn, Injy Hamed, Aditya Nanda Kishore, Aishik Mandal, Alina Dragonetti, Artem Abzaliev, A. Tonja, Bontu Fufa Balcha, Chenxi Whitehouse, Christian Salamea, Dan John Velasco, D. Adelani, D. Meur, Emilio Villa-Cueva, Fajri Koto, Fauzan Farooqui, Frederico Belcavello, Ganzorig Batnasan, Gisela Vallejo, Grainne Caulfield, Guido Ivetta, Haiyue Song, Henok Biadglign Ademtew, Hernán Maina, Holy Lovenia, Israel Abebe Azime, Jan Christian Blaise Cruz, Jay Gala, Jiahui Geng, Jesús-Germán Ortiz-Barajas, Jinheon Baek, Jocelyn Dunstan, L. A. Alemany, Kumaranage Ravindu Yasas Nagasinghe, Luciana Benotti, L. F. D’Haro, Marcelo Viridiano, Marcos Estecha-Garitagoitia, Maria Camila Buitrago Cabrera, Mario Rodr'iguez-Cantelar, Mélanie Jouitteau, M. Mihaylov, Mohamed Fazli Mohamed Imam, Muhammad Farid Adilazuarda, Munkhjargal Gochoo, Munkh-Erdene Otgonbold, Naome A. Etori, Olivier Niyomugisha, Paula M'onica Silva, Pranjal A. Chitale, Raj Dabre, Rendi Chevi, Ruochen Zhang, Ryandito Diandaru, Samuel Cahyawijaya, Santiago G'ongora, Soyeong Jeong, Sukannya Purkayastha, Tatsuki Kuribayashi, Thanmay Jayakumar, T. Torrent, Toqeer Ehsan, Vladimir Araujo, Yova Kementchedjhieva, Zara Burzo, Zheng Wei Lim, Zheng-Xin Yong, Oana Ignat, Joan Nwatu, Rada Mihalcea, T. Solorio, Alham Fikri Aji

A Label is Worth a Thousand Images in Dataset Distillation

Tian Qin, Zhiwei Deng, David Alvarez-Melis

Active, anytime-valid risk controlling prediction sets

Nikos Karampatziakis, Paul Mineiro

Aligning LLM Agents by Learning Latent Preference from User Edits

Ge Gao, Alexey Taymanov, Eduardo Salinas, Paul Mineiro, Dipendra Misra

Can large language models explore in-context?

Akshay Krishnamurthy, Keegan Harris, Dylan Foster, Cyril Zhang, Aleksandrs Slivkins

Can LLMs Learn by Teaching for Better Reasoning? A Preliminary Study

Xuefei Ning, Zifu Wang, Shiyao Li, Zinan Lin, Peiran Yao, Tianyu Fu, Matthew B. Blaschko, Guohao Dai, Huazhong Yang, Yu Wang

Diffusion Model with Cross Attention as an Inductive Bias for Disentanglement

Tao Yang, Cuiling Lan, Yan Lu, Nanning Zheng

Instruction-Guided Visual Masking

Jinliang Zheng, Jianxiong Li, Si Cheng, Yinan Zheng, Jiaming Li, Jihao Liu, Yu Liu, Jingjing Liu, Xianyuan Zhan

InversionView: A General-Purpose Method for Reading Information from Neural Activations

Xinting Huang, Madhur Panwar, Navin Goyal, Michael Hahn

Mesa-Extrapolation: A Weave Position Encoding Method for Enhanced Extrapolation in LLMs

Jingjing Liu

Meta-Diffu$B$: A Contextualized Sequence-to-Sequence Text Diffusion Model with Meta-Exploration

Kevin Lin

QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs

Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L. Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, James Hensman, Pashmina Cameron

Understanding Information Storage and Transfer in Multi-modal Large Language Models

Samyadeep Basu, Martin Grayson, Cecily Morrison, Besmira Nushi, S. Feizi, Daniela Massiceti

A Framework for Bilevel Optimization on Riemannian Manifolds

Andi Han, Bamdev Mishra, Pratik Jawanpuria, Akiko Takeda

Easy2Hard-Bench: Standardized Difficulty Labels for Profiling LLM Performance and Generalization

Mucong Ding, Chenghao Deng, Jocelyn Choo, Zichu Wu, Aakriti Agrawal, Avi Schwarzschild, Tianyi Zhou, Tom Goldstein, John Langford, A. Anandkumar, Furong Huang

Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning

Dylan Foster, Adam Block, Dipendra Misra

Policy Improvement using Language Feedback Models

Victor Zhong, Dipendra Misra, Xingdi Yuan, Marc-Alexandre Côté

RedCode: Risky Code Execution and Generation Benchmark for Code Agents

Chengquan Guo, Xun Liu, Chulin Xie, Andy Zhou, Yi Zeng, Zinan Lin, Dawn Song, Bo Li

Topic-Conversation Relevance (TCR) Dataset and Benchmarks

Yaran Fan, Jamie Pool, Senja Filipi, Ross Cutler

Aligning Vision Models with Human Aesthetics in Retrieval: Benchmarks and Algorithms

Miaosen Zhang, Yixuan Wei, Zhen Xing, Yifei Ma, Zuxuan Wu, Ji Li, Zheng Zhang, Qi Dai, Chong Luo, Xin Geng, Baining Guo

CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations

Leying Zhang, Yao Qian, Long Zhou, Shujie Liu (shujliu), Dongmei Wang, Xiaofei Wang, Midia Yousefi, Yanmin Qian, Jinyu Li, Lei He, Sheng Zhao, Michael Zeng

Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding

Zhenyu (Allen) Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, Zhangyang Wang

Improving Context-Aware Preference Modeling for Language Models

Silviu Pitis, Ziang Xiao, Nicolas Le Roux, Alessandro Sordoni

Scaling the Codebook Size of VQ-GAN to 100,000 with a Utilization Rate of 99%

Fangyun Wei, Dong Chen

Slot-VLM: SlowFast Slots for Video-Language Modeling

Jiaqi Xu, Cuiling Lan, Wenxuan Xie, Xuejin Chen, Yan Lu

Understanding and Improving Training-free Loss-based Diffusion Guidance

Yifei Shen, Xinyang Jiang, Yezhen Wang, Yifan Yang, Dongqi Han, Dongsheng Li

Using Noise to Infer Aspects of Simplicity Without Learning

Lesia Semenova

Voila-A: Aligning Vision-Language Models with User's Gaze Attention

Kun Yan, Lei Ji, Zeyu Wang, Yuntao Wang, Nan Duan, Shuai Ma

A Bayesian Approach to Data Point Selection

Xinnuo Xu, Minyoung Kim, Royson Lee, Brais Martinez, Timothy M. Hospedales

CARE: a Benchmark Suite for the Classification and Retrieval of Enzymes

Jason Yang, Ariane Mora, Shengchao Liu, Bruce J. Wittmann, A. Anandkumar, Frances H. Arnold, Yisong Yue

C-GAIL: Stabilizing Generative Adversarial Imitation Learning with Control Theory

Tianjiao Luo, Tim Pearce, Huayu Chen, Jianfei Chen, Jun Zhu

CultureLLM: Incorporating Cultural Differences into Large Language Models

Cheng Li, Mengzhou Chen, Jindong Wang, Sunayana Sitaram, Xing Xie

DISCOVERYWORLD: A Virtual Environment for Developing and Evaluating Automated Scientific Discovery Agents

Peter Alexander Jansen, Marc-Alexandre Côté, Tushar Khot, Erin Bransom, Bhavana Dalvi, Bodhisattwa Prasad Majumder, Oyvind Tafjord, Peter Clark

Diffusion for World Modeling: Visual Details Matter in Atari

Eloi Alonso, Adam Jelley, Vincent Micheli, Anssi Kanervisto, A. Storkey, Tim Pearce, Franccois Fleuret

ERBench: An Entity-Relationship based Automatically Verifiable Hallucination Benchmark for Large Language Models

Jio Oh, Soyeon Kim, Junseok Seo, Jindong Wang, Ruochen Xu, Xing Xie, Steven Euijong Whang

Generalized Linear Bandits with Limited Adaptivity

Ayush Sawarni, Nirjhar Das, Siddharth Barman, Gaurav Sinha

Query-Efficient Correlation Clustering with Noisy Oracle

Yuko Kuroki, Atsushi Miyauchi, Francesco Bonchi, Wei Chen

The Power of Resets in Online Reinforcement Learning

Zakaria Mhammedi, Dylan Foster, Alexander Rakhlin

You Only Cache Once: Decoder-Decoder Architectures for Language Models

Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, Furu Wei

Compositional 3D-aware Video Generation with LLM Director

Hanxin Zhu, Tianyu He, Anni Tang, Junliang Guo, Zhibo Chen, Jiang Bian

CulturePark: Boosting Cross-cultural Understanding in Large Language Models

Cheng Li, Damien Teney, Linyi Yang, Qingsong Wen, Xing Xie, Jindong Wang

Does Reasoning Emerge? Examining the Probabilities of Causation in Large Language Models

Javier González, Aditya Nori

EEG2Video: Towards Decoding Dynamic Visual Perception from EEG Signals

Yansen Wang, Zilong Wang

Maia-2: A Unified Model for Human-AI Alignment in Chess

Zhenwei Tang, Difan Jiao, Reid McIlroy-Young, Jon Kleinberg, Siddhartha Sen, Ashton Anderson

MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention

Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H. Abdi, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, Lili Qiu

Neural P$^3$M: A Long-Range Interaction Modeling Enhancer for Geometric GNNs

Yusong Wang, Chaoran Cheng, Shaoning Li, Yuxuan Ren, Bin Shao, Ge Liu, Pheng-Ann Heng, Nanning Zheng

TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservation

Chenyang Le, Yao Qian, Dongmei Wang, Long Zhou, Shujie Liu, Xiaofei Wang, Midia Yousefi, Yanmin Qian, Jinyu Li, Sheng Zhao, Michael Zeng

xRAG: Extreme Context Compression for Retrieval-augmented Generation with One Token

Xin Cheng, Xun Wang, Xingxing Zhang, Tao Ge, Si-Qing Chen, Furu Wei, Huishuai Zhang, Dongyan Zhao

You Only Cache Once: Decoder-Decoder Architectures for Language Models

Yutao Sun, Li Dong, Yi Zhu, Shaohan Huang, Wenhui Wang, Shuming Ma, Quanlu Zhang, Jianyong Wang, Furu Wei

ALPINE: Unveiling the Planning Capability of Autoregressive Learning in Language Models

Siwei Wang, Yifei Shen, Shi Feng, Haoran Sun, Shang-Hua Teng, Wei Chen

CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models

Peng Xia, Ze Chen, Juanxi Tian, Yangrui Gong, Ruibo Hou, Yue Xu, Zhenbang Wu, Zhiyuan Fan, Yiyang Zhou, Kangyu Zhu, Wenhao Zheng, Zhaoyang Wang, Xiao Wang, Xuchao Zhang, Chetan Bansal, Marc Niethammer, Junzhou Huang, Hongtu Zhu, Yun Li, Jimeng Sun, Zongyuan Ge, Gang Li, James Zou, Huaxiu Yao

Dataset and Lessons Learned from the 2024 SaTML LLM Capture-the-Flag Competition

Edoardo Debenedetti, Javier Rando, Daniel Paleka, Fineas Silaghi, Dragos Albastroiu, Niv Cohen, Yuval Lemberg, Reshmi Ghosh, Ahmed Salem, Rui Wen, Giovanni Cherubin, Santiago Zanella-Béguelin, Robin Schmid, Victor Klemm, Takahiro Miki, Chenhao Li, Stefan Kraft, Mario Fritz, Florian Tramer, Sahar Abdelnabi, Lea Schönherr

Energy-efficient Reinforcement Learning by Discovering Neural Pathways

Alessandro Sordoni

ProbTS: Benchmarking Point and Distributional Forecasting across Diverse Prediction Horizons

Jiawen Zhang, Xumeng Wen, Zhenwei Zhang, Shun Zheng, Jia Li, Jiang Bian

Provably Efficient Interactive-Grounded Learning with Personalized Reward

Mengxiao Zhang, Yuheng Zhang, Haipeng Luo, Paul Mineiro

TorchSpatial: A Location Encoding Framework and Benchmark for Spatial Representation Learning

Nemin Wu, Qian Cao, Zhangyu Wang, Zeping Liu, Yanlin Qi, Jielu Zhang, Joshua Ni, Xiaobai Yao, Hongxu Ma, Lan Mu, Stefano Ermon, Tanuja Ganu, Akshay Nambi, Ni Lao, Gengchen Mai