Unifying retrieval and re-ranking: A multimodal approach to detecting fact-checked information

Although recently several fact-checking organizations have emerged to verify disinformation, fake news has continued to proliferate, especially exploiting multimodal data on social media. As a result, the fact-checking verification process cannot keep up with this overwhelming and uncertain content, thus raising the need to adopt improved strategies for disinformation detection. The fact-checking process could be optimised considering the tendency of viral claims to be reshared over time and in different contexts. In other words, verifying whether a (multimodal) claim has been previously fact-checked can ease fact-checkers’ manual effort and would provide reliable evidence for the input claim. In this paper, considering the task's ranking formulation, we propose a novel multimodal information retrieval approach aimed at retrieving and re-ranking a list of verified documents according to their relevance to the input claim. Specifically, we exploit text and image's modalities and leverage modern visual-language models to extract powerful representations that capture their complex relationships. Our experiments on three benchmark datasets prove the superiority of the proposed system: in re-ranking settings, it exceeds competitors by up to 15 NDCG points; in retrieval settings, it is the only one that overcomes the standard BM25 baseline.