LITMUS Predictor: An AI Assistant for Building Reliable, High-Performing and Fair Multilingual NLP Systems
- Anirudh Srinivasan ,
- Gauri Kholkar ,
- Rahul Kejriwal ,
- Tanuja Ganu ,
- Sandipan Dandapat ,
- Sunayana Sitaram ,
- Balakrishnan Santhanam ,
- Somak Aditya ,
- Kalika Bali ,
- Monojit Choudhury
Thirty-sixth AAAI Conference on Artificial Intelligence |
Published by AAAI
System Demonstration
Pre-trained multilingual language models are gaining popularity due to their cross-lingual zero-shot transfer ability, but these models do not perform equally well in all languages. Evaluating task-specific performance of a model in a large number of languages is often a challenge due to lack of labeled data, as is targeting improvements in low performing languages through few-shot learning. We present a tool – LITMUS Predictor – that can make reliable performance projections for a fine-tuned task-specific model in a set of languages without test and training data, and help strategize data labeling efforts to optimize performance and fairness objectives.
The demo (opens in new tab) and the code (opens in new tab) of the project are available.
@AAAI 2022
Publication Downloads
LITMUS Predictor
October 21, 2021
LITMUS Predictor provides support for simulating performance in ~100 languages given training observations of the desired task-model. Each training observation specifies the finetuning-datasize + test-performance in different languages. Further, the tool provides support for constructing a data-collection strategy to maximize performance in desired targets subject to different constraints.
What ‘bhasha’ do you want to talk in? With Kalika Bali and Dr. Monojit Choudhury | Podcast
Microsoft Research India Podcast | Episode 03 | June 02, 2020
Many of us who speak multiple languages switch seamlessly between them in conversations and even mix multiple languages in one sentence. For us humans, this is something we do naturally, but it’s a nightmare for computing systems to understand mixed languages.
On this podcast with Kalika Bali and Dr. Monojit Choudhury, we discuss codemixing and the challenges it poses, what makes codemixing so natural to people, some insights into the future of human-computer interaction and more.
See more Microsoft Research India podcast episodes and learn about the research: https://jokerchen.me/en-us/research/lab/microsoft-research-india/