Cross-lingual Visual Pre-training for Multimodal Machine Translation

A novel cross-lingual visual pre- training approach that extends the TLM framework (Lample and Conneau, 2019) with regional features and performs masked language modelling and masked region classification on a three-way parallel corpus.
The architecture of the proposed model: VTLM extends the TLM (Lample and Conneau, 2019) (left side of the dotted line) with regional image features. Masking applies on both linguistic and visual tokens.
paper thumbnail
Paper

Ozan Caglayan, Menekse Kuyu, Mustafa Sercan Amac, Pranava Madhyastha, Erkut Erdem, Aykut Erdem and Lucia Specia. "Cross-lingual Visual Pre-training for Multimodal Machine Translation", Conference of the European Chapter of the Association for Computational Linguistics (EACL 2021), Short Papers, 2021.
Paper | BibTeX


Code & Training

Parallel CC Dataset & Features & Checkpoints


Abstract

Pre-trained language models have been shown to substantially improve performance in many natural language tasks. Although the early focus of such models was single language pre-training, recent advances have resulted in cross-lingual and visual pre-training methods. In this paper, we combine these two approaches to learn visually-grounded cross-lingual representations. Specifically, we extend the translation language modelling (Lample and Conneau, 2019) with masked region classification, and perform pre-training with three-way parallel vision and language corpora. We show that when fine-tuned for multimodal machine translation, these models obtain state-of-the-art performance. We also provide qualitative insights into the usefulness of the learned grounded representations.

Acknowledgements

This work was supported in part by TUBA GEBIP fellowship awarded to Erkut Erdem, and the MMVC project funded by TUBITAK and the British Council via the Newton Fund Institutional Links grant programme (grant ID 219E054 and 352343575). Lucia Specia, Pranava Madhyastha and Ozan Caglayan also received support from MultiMT (H2020 ERC Starting Grant No. 678017) and Lucia Specia from the Air Force Office of Scientific Research (under award number FA8655-20-1-7006).