Large Language Models as Assessors: On the Impact of Relevance Scales
Poster - The 48th European Conference on Information Retrieval (ECIR 2026). Delft, The Netherlands.
riccardo-zamolo
Gianluca Demartini
•
Mar 30, 2026 13:30 — 14:30
•
1 min read
Large Language Models as Assessors: On the Impact of Relevance Scales
Traditionally, relevance judgments have relied on human annotators, but recent advances in Large Language Models (LLMs) have prompted growing interest in their use as a proxy for …
riccardo-zamolo
Longitudinal Loyalty: Understanding The Barriers To Running Longitudinal Studies On Crowdsourcing Platforms
Crowdsourcing tasks have been widely used to collect a large number of human labels at scale. While some of these tasks are deployed by requesters and performed only once by crowd …
Crowdsourced Fact-checking: Does It Actually Work?
There is an important ongoing effort aimed to tackle misinformation and to perform reliable fact-checking by employing human assessors at scale, with a crowdsourcing-based …
david-la-barbera
Cognitive Biases in Fact-Checking and Their Countermeasures: A Review
The increase of the amount of misinformation spread every day online is a huge threat to the society. Organizations and researchers are working to contrast this misinformation …
How Many Crowd Workers Do I Need? On Statistical Power When Crowdsourcing Relevance Judgments
To scale the size of Information Retrieval collections, crowdsourcing has become a common way to collect relevance judgments at scale. Crowdsourcing experiments usually employ …
kevin-roitero
A Neural Model to Jointly Predict and Explain Truthfulness of Statements
Automated fact-checking (AFC) systems exist to combat disinformation, however their complexity usually makes them opaque to the end user, making it difficult to foster trust in the …
erik-brand
E-BART: Jointly Predicting and Explaining Truthfulness
Automated fact-checking (AFC) systems exist to combat disinformation, however their complexity makes them opaque to the end user, making it difficult to foster trust. In this …
erik-brand
Can The Crowd Judge Truthfulness? A Longitudinal Study on Recent Misinformation About COVID-19
Recently, the misinformation problem has been addressed with a crowdsourcing-based approach: to assess the truthfulness of a statement, instead of relying on a few experts, a crowd …
kevin-roitero
The COVID-19 Infodemic: Can the Crowd Judge Recent Misinformation Objectively?
Misinformation is an ever increasing problem that is difficult to solve for the research community and has a negative impact on the society at large. Very recently, the problem has …
kevin-roitero