In this blog post, we motivate and lay out my main contributions of my Master Thesis. To read the full thesis, download my thesis here. To read the related work sections as blog posts, see Cross-Lingual Representation, Parameter-Efficient Fine-Tuning in NLP and Knowledge Distillation.

Table of Contents

1. Motivation
2. Research Objective & Contribution

1. Motivation

Natural language processing (NLP) has made significant progress in recent years, achieving impressive performances across diverse tasks. However, these advances are focused on just a tiny fraction of the 7000 languages in the world, e.g., English, where sufficient amounts of text in the respective language are available (high-resource languages). Nevertheless, when the situation arises where text data in a language is scarce, language technologies fail. These languages are called low-resource languages, e.g., Swahili, Basque, or Urdu; see the next Figure for a more apparent distinction between high-, mid-, and low-resource languages:

A conceptual view of the NLP resource hierarchy categorised in availability of task-specific labels and availability of unlabeled language-specific text. Taken from (Ruder et al. (2019)).

This leaves low-resource languages and, therefore, most languages understudied, which further increases the digital language divide¹ on a technological level. Being able to develop technologies for low-resource languages is vital for scientific, social, and economic reasons, e.g., Africa and India are the hosts of around 2000 low-resource languages and are home to more than 2.5 billion inhabitants (Magueresse et al. (2020)). "Opening" the newest NLP technologies for low-resource languages can help bridge the gap, e.g., digital assistants, or help reduce the discrimination against speakers of non-English languages (Tatman (2017); Rabinovich et al. (2018); Zhiltsova et al. (2019)).

To improve language technologies for low-resource languages, the field of Cross-Lingual Representation Learning is focused on creating high-quality representations for these languages by gaining benefit from abundant data in another language via a shared representation space. As static word representations gained in popularity, many multilingual embedding methods have been presented (Mikolov et al. (2013); Hermann et al. (2014); Hu et al. (2020)). The idea behind these methods is to induce embeddings² such that the embeddings for two languages are aligned, i.e., word translations, e.g., cat and Katze, have similar representations. Recently, however, large pre-trained language models, the so-called transformer models, took static word embedding methods over in virtually every aspect, partly because these models induce context-dependent word representations, capturing the rich meaning of a word better (Peters et al. (2018); Howard et al. (2018); Radford et al. (2018); Devlin et al. (2019)). E.g., mBERT and XLM-R, transformer-based multilingual masked language models pre-trained on text in (approximately) 100 languages, can obtain impressive performances for a variety of cross-lingual transfer tasks (Pires et al. (2019); Conneau et al. (2020)). Even though these models were not trained with any cross-lingual objectives, they still produce representations that can generalize well across languages for a wide range of downstream tasks (Wu et al. (2019); Conneau et al. (2020)). To analyze the cross-lingual transfer ability of multilingual models, the model is first fine-tuned on annotated data of a downstream task and then evaluated in the zero or few-shot scenario, i.e., evaluated with the fine-tuned models in the target language (Hu et al. (2020)) with no or few additional labeled target language data.

As impressive as these multilingual transformers might seem, low-resource languages still perform sub-par to high-resource languages (Wu et al. (2019); Conneau et al. (2020)), partly due to the fact of a smaller pre-training corpus (Conneau et al. (2020)) , the curse of multilinguality (Conneau et al. (2020)) and the importance of vocabulary curation and size (Chung et al. (2020); Artetxe et al. (2020)). E.g., the curse of multilinguality argues assuming that the model capacity stays constant that adding more languages leads to better cross-lingual performance on low-resource languages up until a point where the overall performance on monolingual and cross-lingual benchmarks degrades. Intuitively explained, adding more languages to the model has two effects: (1) Positive cross-lingual transfer, especially for low-resource languages, and (2) lower per-language capacity, which then, in turn, can degrade the overall model performance. These two effects of capacity dilution and positive transfer need to be carefully traded against each other. The model either needs to have a large model capacity³ or is specialized (constrained) towards a subset of languages beforehand. For these reasons, it is hard to create a single model that can effectively represent a diverse set of languages. One solution is to create language-specific models (monolingual models) with language-specific vocabulary and model parameters (Virtanen et al. (2019); Antoun et al. (2020)), but in return, monolingual models need enough text to pre-train the model on the language modeling task, which is typically not available for low-resource languages. Additionally, we can not benefit from any cross-lingual transfer from related languages, making it harder to create an adequate representation for low-resource languages (Pires et al. (2019); Lauscher et al. (2020)).

2. Research Objective & Contribution

In this thesis, we explore how one can alleviate the issues of big multilingual transformers for low-resource languages, especially the curse of multilinguality. Specifically, our two main objectives are: (1) Improving the cross-lingual alignment for low-resource languages and (2) improving cross-lingual downstream task performance for low-resource languages. We utilize Knowledge Distillation (KD) by distilling the multilingual model into language-specialized (also called monolingual) language models. We make the following contributions:

• We propose a novel setup to distill multilingual transformers into monolingual components. Based on the setup, we propose two KD strategies: One for improving the alignment between two languages and one to improve cross-lingual downstream task performance. We call the former MonoAlignment and the latter MonoShot.
• MonoAlignment uses a distillation strategy to distill multilingual transformer models into smaller monolingual components which have an improved aligned representation space between a high-resource language and a low-resource language. We demonstrate the effectiveness by distilling XLM-R and experimenting with aligning English with Turkish, Swahili, Urdu, and Basque.
• We compare MonoAlignment to other Knowledge Distillation strategies showing that it outperforms them in the retrieval task for low-resource languages.
• Our work suggests that an increase in the cross-lingual alignment of a multilingual transformer model does not necessarily translate into an increase in cross-lingual downstream task performance.
• Therefore, we propose MonoShot, another Knowledge Distillation strategy to distill multilingual transformer models into smaller monolingual components but which have a strong cross-lingual downstream performance in the zero- and few-shot settings.
• We show that MonoShot performs best among many different Knowledge Distillation strategies, albeit still lacks behind the teacher performance. However, it outperforms models built upon the teacher architecture but is trimmed down to the same size as the distilled components and initialized from parts of the teacher.
• We demonstrate an effective fine-tuning strategy for the zero-shot scenario for aligned monolingual models and compare it against many other strategies.

To conduct our research, we will draw inspiration from the field of Cross-Lingual Representation Learning, Knowledge Distillation, and Parameter-Efficient Fine-tuning. Following different Knowledge Distillation strategies, such as from DistilBert (Sanh et al. (2020)) or TinyBert (Jiao et al. (2020)) , we distill the aligned cross-lingual representation space of the multilingual transformer model XLM-R (Conneau et al. (2017)) into smaller monolingual students. To fine-tune aligned monolingual models in a zero-shot scenario, we study the field of parameter-efficient fine-tuning, i.e., Adapters (Houlsby et al. (2019); Pfeiffer et al. (2021)), BitFit (Zaken et al. (2021)) and Sparse Fine-Tuning (Guo et al. (2020)). Finally, we evaluate the general-purpose cross-lingual representation of our monolingual models in the retrieval, classification, structured prediction, and question-answering task.

References

1. Sebastian Ruder, Anders S{\o}gaard, Ivan Vuli{\'c}. Unsupervised Cross-Lingual Representation Learning. Proceedings of ACL 2019, Tutorial Abstracts. 2019
2. Alexandre Magueresse, Vincent Carles, Evan Heetderks. Low-resource Languages: A Review of Past Work and Future Challenges. arXiv. 2020. link
3. Rachael Tatman. Gender and Dialect Bias in {Y}ou{T}ube{'}s Automatic Captions. Proceedings of the First {ACL} Workshop on Ethics in Natural Language Processing. 2017. link
4. Ella Rabinovich, Yulia Tsvetkov, Shuly Wintner. Native Language Cognate Effects on Second Language Lexical Choice. Transactions of the Association for Computational Linguistics. 2018. link
5. Alina Zhiltsova, Simon Caton, Catherine Mulway. Mitigation of Unintended Biases against Non-Native English Texts in Sentiment Analysis. AICS. 2019
6. Tomas Mikolov, Quoc V. Le, Ilya Sutskever. Exploiting Similarities among Languages for Machine Translation. arXiv. 2013. link
7. Karl Moritz Hermann, Phil Blunsom. Multilingual Models for Compositional Distributed Semantics. Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2014. link
8. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, Melvin Johnson. XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization. arXiv. 2020. link
9. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, Luke Zettlemoyer. Deep contextualized word representations. arXiv. 2018. link
10. Jeremy Howard, Sebastian Ruder. Universal Language Model Fine-tuning for Text Classification. arXiv. 2018. link
11. Alec Radford, Karthik Narasimhan. Improving Language Understanding by Generative Pre-Training. 2018
12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv. 2019. link
13. Telmo Pires, Eva Schlinger, Dan Garrette. How Multilingual is Multilingual {BERT}?. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. 2019. link
14. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, Veselin Stoyanov. Unsupervised Cross-lingual Representation Learning at Scale. arXiv. 2020. link
15. Shijie Wu, Mark Dredze. Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of {BERT}. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019. link
16. Hyung Won Chung, Dan Garrette, Kiat Chuan Tan, Jason Riesa. Improving Multilingual Models with Language-Clustered Vocabularies. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020. link
17. Mikel Artetxe, Sebastian Ruder, Dani Yogatama, Gorka Labaka, Eneko Agirre. A Call for More Rigor in Unsupervised Cross-lingual Learning. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020. link
18. Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, Sampo Pyysalo. Multilingual is not enough: BERT for Finnish. arXiv. 2019. link
19. Wissam Antoun, Fady Baly, Hazem Hajj. {A}ra{BERT}: Transformer-based Model for {A}rabic Language Understanding. Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection. 2020. link
20. Anne Lauscher, Vinit Ravishankar, Ivan Vuli{\'c}, Goran Glava{\v{s}}. From Zero to Hero: {O}n the Limitations of Zero-Shot Language Transfer with Multilingual {T}ransformers. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020. link
21. Victor Sanh, Lysandre Debut, Julien Chaumond, Thomas Wolf. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv. 2020. link
22. Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, Qun Liu. TinyBERT: Distilling BERT for Natural Language Understanding. arXiv. 2020. link
23. Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato, Ludovic Denoyer, Hervé Jégou. Word Translation Without Parallel Data. arXiv. 2017. link
24. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, Sylvain Gelly. Parameter-Efficient Transfer Learning for NLP. arXiv. 2019. link
25. Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, Iryna Gurevych. AdapterFusion: Non-Destructive Task Composition for Transfer Learning. arXiv. 2021. link
26. Elad Ben Zaken, Shauli Ravfogel, Yoav Goldberg. BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models. arXiv. 2021. link
27. Demi Guo, Alexander M. Rush, Yoon Kim. Parameter-Efficient Transfer Learning with Diff Pruning. arXiv. 2020. link