statistical analysis

How Many Crowd Workers Do I Need? On Statistical Power When Crowdsourcing Relevance Judgments

To scale the size of Information Retrieval collections, crowdsourcing has become a common way to collect relevance judgments at scale. Crowdsourcing experiments usually employ 100-10,000 workers, but such a number is often decided in a heuristic way. …