MarkedBERT: Integrating Traditional IR Cues in Pre-trained Language Models for Passage Retrieval

Lila Boualili (1) , Jose G. Moreno (1) , Mohand Boughanem (1, 2)

1 IRIT-IRIS - Recherche d’Information et Synthèse d’Information
2 UT3 - Université Toulouse III - Paul Sabatier

The Information Retrieval (IR) community has witnessed a flourishing development of deep neural networks, however, only a fewmanaged to beat strong baselines. Among them, models like DRMM and DUET were able to achieve better results thanks to the properhandling of exact match signals. Nowadays, the application of pre-trained language models to IR tasks has achieved impressive resultsexceeding all previous work. In this paper, we assume that established IR cues like exact term-matching, proven to be valuable fordeep neural models, can be used to augment the direct supervision from labeled data for training these pre-trained models. To studythe effectiveness of this assumption, we propose MarkedBERT a modified version of one of the most popular pre-trained modelsvia language modeling tasks, BERT. MarkedBERT integrates exact match signals using a marking technique that locates and highlightsExact Matched query-document terms using marker tokens. Experiments on MS MARCO Passage Ranking task show that our rathersimple approach is actually effective. We find that augmenting the input with marker tokens allows the model to focus on valuabletext sequences for IR.

Mots clés: Deep Learning; Passage Retrieval; Exact Matching

Domaines: Recherche d'information [cs.IR]; Informatique et langage [cs.CL]

Lila Boualili, Jose G. Moreno, Mohand Boughanem. MarkedBERT: Integrating Traditional IR Cues in Pre-trained Language Models for Passage Retrieval. 43rd International ACM SIGIR conference on research and development in Information Retrieval (SIGIR 2020), Jul 2020, Virtual Event, China. pp.1977-1980, ⟨10.1145/3397271.3401194⟩. ⟨hal-03011890⟩