Task Routing

Configure feed overlap

Prodigy provides a configuration file that allows you to provide a "feed_overlap"  or an "annotations_per_task"  setting to quickly define how tasks are routed between your pool of annotators.

These settings can be changed over time as your project evolves. High overlap can make sense early on in the project because it allows annotators to reflect on their disagreement. But eventually, you may prefer to have less overlap in favor of getting more annotations.

Custom routing mechanisms

Tasks can also be routed with custom Python code, which allows you to fully customize who will annotate each example. Prodigy even offers some utilities to ensure that annotators are assigned consistently.

Router based on model confidencedef task_router_conf(ctrl: Controller, session_id: str, item: Dict) -> List[str]:
    """Route tasks based on the confidence of a model."""
    all_annotators = ctrl.session_ids
    confidence_score = custom_model(item['text'])

    if confidence_score < 0.3:
        # If the confidence is low, the example might be hard
        # and then everyone needs to check
        return all_annotators
    # Otherwise just one person needs to check. We re-use the task_hash
    # to ensure consistent routing of the task. 
    idx = item['_task_hash'] % len(all_annotators)
    # Return a list with a single annotator reference. 
    return [all_annotators[idx]]

Example
prodigyreviewreviews_finalreviews_2019,reviews2018

Review annotators

Especially early in your machine learning project, it's important to review annotations between different annotators. You want to detect differences in interpretation or understanding of the annotation guidelines as early as possible and review annotator disagreements. Doing this will also provide valuable feedback and allows you to update your annotation instructions.

Prodigy provides a review recipe to make review easy. It will present one or more versions of a given task, annotated in different sessions by different users, and display them with the session information and allow the reviewer to correct or override the decision.

The same recipe can also be configured to automatically accept all the examples where there is no disagreement. To learn more, check the documentation.