Conducting large-scale and/or high-quality human curation of data

In the creation of datasets or benchmarks it is often the case that, ideally, large quantities of data that underwent high-quality curation by humans are available (e.g., for labeling or annotating data items).

Strategies for incentivizing, funding, organizing and quality-checking such human curation efforts might have high leverage for improving progress directly (via directly increasing the utility of data), and are also likely to improve AI model training and benchmarking.