Truth Inference in Crowdsourcing

Truth Inference in Crowdsourcing: Is the Problem Solved?

An Experimental Evaluation on Truth Inference Methods in Crowdsourcing

Crowdsourcing has emerged as a novel problem-solving paradigm, which facilitates addressing problems that are hard for computers, e.g., entity resolution and sentiment analysis. However due to the openness of crowdsourcing, workers may yield low-quality answers, and a redundancy-based method is widely employed, which first assigns each task to multiple workers and then infers the correct answer (called truth) for the task based on the answers of the assigned workers. A fundamental problem in this method is Truth Inference, which decides how to effectively infer the answer. Recently the database community and data mining community independently study this problem and propose various algorithms. However, these algorithms are not compared extensively under the same framework and it is hard for practitioners to select appropriate algorithms. To alleviate this problem, we provide a detailed survey on 17 existing algorithms and perform a comprehensive evaluation using 5 real datasets. We make all codes and datasets public for future research. Through experiments we find that existing algorithms are not stable across different datasets and there is no algorithm that outperforms others consistently. We believe that the truth inference problem is not fully solved, and identify the limitations of existing algorithms and point out promising research directions.

Program Usage

If you would like to run a method on a crowdsourced dataset (with workers' answers), then a direct way is to just execute run.py in the downloaded project (see Downloads section below) with four parameters: (1) 'method_file_path', (2) 'answer_file_path', (3) 'result_file_path', and (4) 'task type'. Next we introduce the four parameters in detail.

Parameters:

  1. 'method_file_path': the path of the algorithm source file. Note that we provide all 17 algorithms' source codes (as compared in the paper) in the methods folder, and each method's source file is always the file method.py under respective folders mentioned in the table below.
    For example, if you would like to run D&S method as mentioned in the paper, then it is in fact implemented in the file methods/c_EM/method.py. Similarly for other methods, you can just replace c_EM with the corresponding folder (can be found in the table below) in the above file path.
    2. Methods Task Types Folder
    MV Decision-Making, Single_Choice c_MV
    GLAD Decision-Making, Single_Choice c_GLAD
    D&S Decision-Making, Single_Choice c_EM
    Minimax Decision-Making, Single_Choice l_minimax
    BCC Decision-Making, Single_Choice l_BCC
    CBCC Decision-Making, Single_Choice l_CBCC
    LFC Decision-Making, Single_Choice l_LFCbinay, l_LFCmulti
    CATD Decision-Making, Single_Choice, Numeric c_CATD
    PM Decision-Making, Single_Choice, Numeric c_PM-CRH
    Multi Decision-Making c_multidimensional
    KOS Decision-Making l_KOS-1
    VI-BP Decision-Making l-VI-BP-2
    VI-MF Decision-Making l_VI-MF-2
    LFC_N Numeric l_LFCcont
    Mean Numeric l_mean
    Median Numeric l_median
  2. 'answer_file_path': the path of the answer file. The format of answer file (i.e., a csv file) is:
    (1) the first line is always 'question,worker,answer';
    (2) the lines followed will be the instances of 'question,worker,answer', indicating that worker has answered question with the answer.
    Next we show an example answer file: 
    question,worker,answer  
    36618,896,0  
    11619,896,1  
    36620,896,1  
    36621,896,1  
    36622,896,1
  3. 'result_file_path': the path of result file, containing the inferred truth of the algorithm on the above answer file. The result file is a csv file, and each line contains a question and its inferred truth.
  4. 'task type': it can only be Decision-Making, Single_Choice, or Numeric, indicating different task types.

Let us now give an example as follows:
python run.py methods/c_EM/method.py ./demo_answer_file.csv ./demo_result_file.csv decision-making
It will invoke D&S algorithm to read './demo_answer_file.csv' as the input (the tasks are 'decision-making' tasks), and the output will be in the file './demo_result_file.csv'.

Project Structure

The main files of the project are as follows:

Downloads

You can download our project and complete datasets.

Reproducing Results

After downloading our project, you can get the results presented in our paper simply by executing corresponding programs. There are totally 9 python program scripts for 3 different experiments (Sections 6.3.1, 6.3.2, 6.3.3) involved with all different task types. For example, exp1_decisionmaking.py is for the first experiment (Section 6.3.1) w.r.t. decision-making tasks. When the execution finishes, you can find the results separated by datasets under the "output" folder.

Our experiments are run in Ptyhon 2.7 on a Ubuntu server with Intel 2.40 GHz processor and 60 GB memory.

Publications

Yudian Zheng, Guoliang Li, Yuanbing Li, Caihua Shan, Reynold Cheng.
Truth Inference in Crowdsourcing: Is the Problem Solved?
In VLDB 2017, Vol 10, Isuue 5, Pages 541-552, Full Paper, Present in VLDB 2017, Aug 28 - Sep 1, Munich, Germany.

Guoliang Li, Yudian Zheng, Ju Fan, Jiannan Wang, Reynold Cheng.
Crowdsourced Data Management: Overview and Challenges.
In SIGMOD 2017, Tutorial (3 hours), May 14th - 19th, Chicago, IL, USA.

Contacts

The project is done by Yudian Zheng (The University of Hong Kong), Guoliang Li (Tsinghua University), Yuanbing Li (Tsinghua University), Caihua Shan (The University of Hong Kong) and Reynold Cheng (The University of Hong Kong).

If you have any questions, feel free to contact Yudian Zheng (zhydhkcws AT gmail dot com).