Crowdsourcing

How Many Crowd Workers Do I Need? On Statistical Power When Crowdsourcing Relevance Judgments

To scale the size of Information Retrieval collections, crowdsourcing has become a common way to collect relevance judgments at scale. Crowdsourcing experiments usually employ …

kevin-roitero

The Effects of Crowd Worker Biases in Fact-Checking Tasks

Due to the increasing amount of information shared online every day, the need for sound and reliable ways of distinguishing between trustworthy and non-trustworthy information is …

tim-draws
Crowd_Frame: A Simple and Complete Framework to Deploy Complex Crowdsourcing Tasks Off-the-Shelf featured image

Crowd_Frame: A Simple and Complete Framework to Deploy Complex Crowdsourcing Tasks Off-the-Shelf

Conference Talk - The 15th ACM International Conference on Web Search and Data Mining (WSDM 2022). February 23, 2022, Phoenix, U.S.A. (remote). Held remotely as a pre-recorded …

avatar
Michael Soprano

Crowd_Frame: A Simple and Complete Framework to Deploy Complex Crowdsourcing Tasks Off-the-Shelf

Due to their relatively low cost and ability to scale, crowdsourcing based approaches are widely used to collect a large amount of human annotated data. To this aim, multiple …

avatar
Michael Soprano

The Many Dimensions of Truthfulness: Crowdsourcing Misinformation Assessments on a Multidimensional Scale

Recent work has demonstrated the viability of using crowdsourcing as a tool for evaluating the truthfulness of public statements. Under certain conditions such as: (1) having a …

avatar
Michael Soprano

Can The Crowd Judge Truthfulness? A Longitudinal Study on Recent Misinformation About COVID-19

Recently, the misinformation problem has been addressed with a crowdsourcing-based approach: to assess the truthfulness of a statement, instead of relying on a few experts, a crowd …

kevin-roitero

The COVID-19 Infodemic: Can the Crowd Judge Recent Misinformation Objectively?

Misinformation is an ever increasing problem that is difficult to solve for the research community and has a negative impact on the society at large. Very recently, the problem has …

kevin-roitero
The COVID-19 Infodemic: Can the Crowd Judge Recent Misinformation Objectively? featured image

The COVID-19 Infodemic: Can the Crowd Judge Recent Misinformation Objectively?

Conference Talk - The 29th ACM International Conference on Information and Knowledge Management (CIKM 2020). October 19, 2020, Galway, Ireland (remote). Held remotely as a …

kevin-roitero
Can The Crowd Identify Misinformation Objectively? The Effects of Judgment Scale and Assessor’s Background featured image

Can The Crowd Identify Misinformation Objectively? The Effects of Judgment Scale and Assessor’s Background

Conference Talk - The 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2020). July 25, 2020, Xi'an, China (remote). Held remotely …

kevin-roitero

Can The Crowd Identify Misinformation Objectively? The Effects of Judgment Scale and Assessor's Background

Truthfulness judgments are a fundamental step in the process of fighting misinformation, as they are crucial to train and evaluate classifiers that automatically distinguish true …

kevin-roitero