Mark entity spans in a text by highlighting them and selecting the respective labels. The model is used to tokenize the text to allow less sensitive highlighting, since the token boundaries are used to set the entity spans. The label set can be defined as a comma-separated list on the command line or as a path to a text file with one label per line. If no labels are specified, Prodigy will check if labels are present in the model. This recipe does not require an entity recognizer, and doesn’t do any active learning.

prodigy
ner.manual
dataset
spacy_model
source
--loader
--label
--patterns
--exclude
--highlight-chars

Example

prodigy
ner.manual
ner_news
en_core_web_sm
./news_headlines.jsonl
--label PERSON,ORG,PRODUCT
prodigy ner.manual ner_news en_core_web_sm ./news_headlines.jsonl --label PERSON,ORG,PRODUCT

Create gold-standard data for NER by correcting the model’s suggestions. The spaCy pipeline will be used to predict entities contained in the text, which the annotator can remove and correct if necessary.

prodigy
ner.correct
dataset
spacy_model
source
--loader
--label
--update
--exclude
--unsegmented
--component

Example

prodigy
ner.correct
gold_ner
en_core_web_sm
./news_headlines.jsonl
--label PERSON,ORG
prodigy ner.correct gold_ner en_core_web_sm ./news_headlines.jsonl --label PERSON,ORG

Collect the best possible training data for a named entity recognition model with the model in the loop. Based on your annotations, Prodigy will decide which questions to ask next. If the suggested entity is fully correct, you can accept it. If it’s entirely or partially wrong, you should reject it. As of v1.11, the recipe will also ask you about examples containing no entities at all, which can improve overall accuracy of your model. So if you see an example with no highlighted suggestions, you can accept it if the text contains no entities, or reject it if it does contain entities of the labels you’re annotating.

prodigy
ner.teach
dataset
spacy_model
source
--loader
--label
--patterns
--exclude
--unsegmented

Example

prodigy
ner.teach
ner_news
en_core_web_sm
./news_headlines.jsonl
--label PERSON,EVENT
prodigy ner.teach ner_news en_core_web_sm ./news_headlines.jsonl --label PERSON,EVENT

Take existing “silver” datasets with binary accept/reject annotations, merge the annotations to find the best possible analysis given the constraints defined in the annotations, and manually edit it to create a perfect and complete “gold” dataset.

prodigy
ner.silver-to-gold
dataset
silver_sets
spacy_model
--label

Load two models and a stream of text, compare their predictions and select which result you prefer. The outputs will be randomized, so you won’t know which model is which. When you stop the server, the results are calculated. This recipe is especially helpful if you’re updating an existing model or if you’re trying out a new strategy on the same problem. Even if two models achieve similar accuracy, one of them can still be subjectively “better”, so this recipe lets you analyze that.

prodigy
ner.eval-ab
dataset
model_a
model_b
source
--loader
--label
--exclude
--unsegmented

Example

prodigy
ner.eval-ab
eval_dataset
en_core_web_sm
./improved_ner_model
./news_headlines.jsonl
prodigy ner.eval-ab eval_dataset en_core_web_sm ./improved_ner_model ./news_headlines.jsonl

Leverage a model to add NER annotations to the database. You can repeat this multiple times with different models so that you may easily compare their predictions using the review recipe and curate examples where models disagree.

For more information on this method of annotating you can consult this guide.

prodigy
ner.model-annotate
dataset
spacy_model
source
model_alias
--labels
--loader
--component

Example

prodigy
ner.model-annotate
ner_headlines
en_core_web_md
./news_headlines.jsonl
spacy_md
--labels PERSON

Getting labels from the 'ner' component 
Using 1 labels: ['PERSON']
100%|███████████████████████████████████████████████████████|
201/201 [00:00<00:00, 395.14it/s] prodigy ner.model-annotate ner_headlines en_core_web_md ./news_headlines.jsonl spacy_md --labels PERSON 
Getting labels from the 'ner' component 
Using 1 labels: ['PERSON']
100%|███████████████████████████████████████████████████████|
201/201 [00:00<00:00, 395.14it/s]

Mark entity spans in a text by highlighting them and selecting the respective labels. The model is used to tokenize the text to allow less sensitive highlighting, since the token boundaries are used to set the entity spans. The label set can be defined as a comma-separated list on the command line or as a path to a text file with one label per line. If no labels are specified, Prodigy will check if labels are present in the model. This recipe does not require an entity recognizer, and doesn’t do any active learning.

prodigy
spans.manual
dataset
spacy_model
source
--loader
--label
--patterns
--suggester
--exclude
--highlight-chars

Example

prodigy
spans.manual
covid_articles
blank:en
./journal_papers.jsonl
--label FACTOR,CONDITION,METHOD,EFFECT
prodigy spans.manual covid_articles blank:en ./journal_papers.jsonl --label FACTOR,CONDITION,METHOD,EFFECT

If you’re using a custom suggester function for the span categorizer, you can provide it via the --suggester argument and Prodigy will validate submitted annotations against it as you annotate. If you’re not using a suggester, data-to-spacy and train will infer the best-matching ngram suggester based on the available span annotations in your data.

Example with suggester validation

prodigy
spans.manual
covid_articles
blank:en
./journal_papers.jsonl
--label CONDITION
--suggester 123_ngram_suggester.v1
-F ./suggester.py
prodigy spans.manual covid_articles blank:en ./journal_papers.jsonl --label CONDITION --suggester 123_ngram_suggester.v1 -F ./suggester.py

suggester.pyfrom spacy import registry
from spacy.pipeline.spancat import build_ngram_suggester

@registry.misc("123_ngram_suggester.v1")
def custom_ngram_suggester():
    return build_ngram_suggester(sizes=[1, 2, 3])  # all ngrams of size 1, 2 and 3

Create gold-standard data for span categorization by correcting the model’s predictions. Requires a spaCy pipeline with a trained span categorizer and will show all spans in the given group. To customize the span group to read from, you can use the --key argument.

prodigy
spans.correct
dataset
spacy_model
source
--loader
--label
--update
--exclude
--component

Example

prodigy
spans.correct
gold_spans
./spancat_model
./journal_papers_new.jsonl
prodigy spans.correct gold_spans ./spancat_model ./journal_papers_new.jsonl

Leverage a model to add span annotations to the database. You can repeat this multiple times with different models so that you may easily compare their predictions using the review recipe and curate examples where models disagree.

For more information on this method of annotating you can consult this guide.

prodigy
spans.model-annotate
dataset
spacy_model
source
model_alias
--labels
--loader
--component

Example

prodigy
spans.model-annotate
spans_headlines
./local_spacy_model
./news_headlines.jsonl
spacy_local
--labels PERSON,COMPANY,CITY

Getting labels from the 'spancat' component 
Using 3 labels: ['PERSON', 'COMPANY', 'CITY']
100%|███████████████████████████████████████████████████████|
201/201 [00:00<00:00, 395.14it/s] prodigy spans.model-annotate spans_headlines ./local_spacy_model ./news_headlines.jsonl spacy_local --labels PERSON,COMPANY,CITY 
Getting labels from the 'spancat' component 
Using 3 labels: ['PERSON', 'COMPANY', 'CITY']
100%|███████████████████████████████████████████████████████|
201/201 [00:00<00:00, 395.14it/s]

Manually annotate categories that apply to a text. If only one label is set, the classification interface is used. If more than one label is specified, the choice interface is used and categories are added as multiple choice options. If the --exclusive flag is set, categories become mutually exclusive, meaning that only one can be selected during annotation.

prodigy
textcat.manual
dataset
source
--loader
--label
--exclusive
--exclude
--accept-empty

Example

prodigy
textcat.manual
news_topics
./news_headlines.jsonl
--label Technology,Politics,Economy,Entertainment
prodigy textcat.manual news_topics ./news_headlines.jsonl --label Technology,Politics,Economy,Entertainment

Create training data for an existing trained text classification model by correcting the model’s suggestions. The --threshold is used to determine whether a label should be pre-selected, e.g. if it’s set to 0.5 (default), all labels with a score of 0.5 and above will be checked automatically. Prodigy will automatically infer whether the categories are mutually exclusive, based on the component configuration.

prodigy
textcat.correct
dataset
spacy_model
source
--loader
--label
--update
--threshold
--component
--exclude
--accept-empty

Example

prodigy
textcat.correct
news_topics
./en_textcat_news
./news_headlines.jsonl
--label Technology,Politics,Economy,Entertainment
prodigy textcat.correct news_topics ./en_textcat_news ./news_headlines.jsonl --label Technology,Politics,Economy,Entertainment

Collect the best possible training data for a text classification model by using a model in the loop. Based on your annotations, Prodigy will decide which questions to ask next. All annotations will be stored in the database. If a patterns file is supplied via the --patterns argument, the matches will be included in the stream and the matched spans are highlighted, so you’re able to tell which words or phrases the selection was based on. Note that the exact pattern matches have no influence when updating the model – they’re only used to help pre-select examples for annotation.

prodigy
textcat.teach
dataset
spacy_model
source
--loader
--label
--patterns
--exclude

Example

prodigy
textcat.teach
news_topics
en_core_web_sm
./news_headlines.jsonl
--label Technology,Politics,Economy,Entertainment
prodigy textcat.teach news_topics en_core_web_sm ./news_headlines.jsonl --label Technology,Politics,Economy,Entertainment

Leverage a model to add texcat annotations to the database. You can repeat this multiple times with different models so that you may easily compare their predictions using the review recipe and curate examples where models disagree.

For more information on this method of annotating you can consult this guide.

prodigy
textcat.model-annotate
dataset
spacy_model
source
model_alias
--labels
--loader
--threshold
--component

Example

prodigy
textcat.model-annotate
textcat_headlines
./local_spacy_model
./news_headlines.jsonl
spacy_local
--labels POLICY,VC,AI

Getting labels from the 'textcat' component.
Using 3 labels: ['POLICY', 'VC', 'AI']
100%|███████████████████████████████████████████████████████|
201/201 [00:00<00:00, 395.14it/s] prodigy textcat.model-annotate textcat_headlines ./local_spacy_model ./news_headlines.jsonl spacy_local --labels POLICY,VC,AI 
Getting labels from the 'textcat' component.
Using 3 labels: ['POLICY', 'VC', 'AI']
100%|███████████████████████████████████████████████████�████|
201/201 [00:00<00:00, 395.14it/s]

Create gold-standard data for part-of-speech tagging by correcting the model’s suggestions. The spaCy pipeline will be used to predict fine-grained part-of-speech tags (Token.tag_), which the annotator can remove and correct if necessary. It’s often more efficient to focus on a few labels at a time, instead of annotating all labels jointly.

prodigy
pos.correct
dataset
spacy_model
source
--loader
--label
--exclude
--unsegmented

Example

prodigy
pos.correct
news_tag
en_core_web_sm
./news_headlines.jsonl
--label NN,NNS,NNP,NNPS
prodigy pos.correct news_tag en_core_web_sm ./news_headlines.jsonl --label NN,NNS,NNP,NNPS

Collect the best possible training data for a part-of-speech tagging model with the model in the loop. Based on your annotations, Prodigy will decide which questions to ask next. It’s often more efficient to focus on a few labels at a time, instead of annotating all labels jointly.

prodigy
pos.teach
dataset
spacy_model
source
--loader
--label
--exclude
--unsegmented

Example

prodigy
pos.teach
tag_news
en_core_web_sm
./news_headlines.jsonl
--label NN,NNS,NNP,NNPS
prodigy pos.teach tag_news en_core_web_sm ./news_headlines.jsonl --label NN,NNS,NNP,NNPS

Create gold-standard data for sentence segmentation by correcting the model’s suggestions. The spaCy pipeline will be used to predict sentence boundaries, which the annotator can correct if necessary. The recipe uses the label S to mark tokens that start a sentence. You can double-click a sentence start token in the UI to add a new sentence boundary, or click on an incorrect prediction to remove it.

prodigy
sent.correct
dataset
spacy_model
source
--loader
--exclude

Example

prodigy
sent.correct
sentence_data
xx_sent_ud_sm
./paragraphs.jsonl
prodigy sent.correct sentence_data xx_sent_ud_sm ./paragraphs.jsonl

Collect the best possible training data for a sentence segmentation model with the model in the loop. Based on your annotations, Prodigy will decide which questions to ask next. The recipe uses S to mark tokens that start sentences and I for all other tokens. You can then hit accept or reject, depending on whether the suggested token is correctly labelled as a sentence start or other token.

prodigy
sent.teach
dataset
spacy_model
source
--loader
--exclude

Example

prodigy
sent.teach
sent_news
xx_sent_ud_sm
./news_headlines.jsonl
prodigy sent.teach sent_news xx_sent_ud_sm ./news_headlines.jsonl

Create gold-standard data for dependency parsing by correcting the model’s suggestions. The spaCy pipeline will be used to predict dependencies for the given labels, which the annotator can remove and correct if necessary. If --update is set, the model in the loop will be updated with the annotations and its updated predictions will be reflected in future batches. The recipe performs no example selection and all texts will be shown as they come in.

prodigy
dep.correct
dataset
spacy_model
source
--loader
--label
--update
--wrap
--unsegmented
--exclude

Example

prodigy
dep.correct
deps_news
en_core_web_sm
./news_headlines.jsonl
--label ROOT,csubj,nsubj,dobj,pboj
--update
prodigy dep.correct deps_news en_core_web_sm ./news_headlines.jsonl --label ROOT,csubj,nsubj,dobj,pboj --update

Collect the best possible training data for a dependency parsing model with the model in the loop. Based on your annotations, Prodigy will decide which questions to ask next. It’s often more efficient to focus on a few most relevant labels at a time, instead of annotating all labels jointly.

prodigy
dep.teach
dataset
spacy_model
source
--loader
--label
--exclude
--unsegmented

Example

prodigy
dep.teach
deps_news
en_core_web_sm
./news_headlines.jsonl
--label csubj,nsubj,dobj,pboj
prodigy dep.teach deps_news en_core_web_sm ./news_headlines.jsonl --label csubj,nsubj,dobj,pboj

Create training data for coreference resolution. Coreference resolution is the challenge of linking ambiguous mentions such as “her” or “that woman” back to an antecedent providing more context about the entity in question. This recipe allows you to focus on nouns, proper nouns and pronouns specifically, by disabling all other tokens. You can customize the labels used to extract those using the recipe arguments. Also see the usage guide on coreference annotation.

prodigy
coref.manual
dataset
spacy_model
source
--loader
--label
--pos-tags
--poss-pron-tags
--ner-labels
--exclude

Example

prodigy
coref.manual
coref_movies
en_core_web_sm
./plot_summaries.jsonl
--label COREF
prodigy coref.manual coref_movies en_core_web_sm ./plot_summaries.jsonl --label COREF

Annotate directional relations and dependencies between tokens and expressions by selecting the head, child and dependency label and optionally assign labelled spans for named entities or other expressions. This workflow is extremely powerful and can be used for basic dependency annotation, as well as joint named entity and entity relation annotation. If --span-label defines additional span labels, a second mode for span highlighting is added.

The recipe lets you take advantage of several efficiency tricks: spans can be pre-defined using an existing NER dataset, entities or noun phrases from a model or fully custom match patterns. You can also disable certain tokens to make them unselectable. This lets you focus on what matters and prevents annotators from introducing mistakes. For more details and examples, check out the usage guide on custom relation annotation and see the task-specific recipes dep.correct and coref.manual that include pre-defined configurations.

prodigy
rel.manual
dataset
spacy_model
source
--loader
--label
--span-label
--patterns
--disable-patterns
--add-ents
--add-nps
--wrap
--exclude
--hide_arrow_heads

Example

prodigy
rel.manual
relation_data
en_core_web_sm
./data.jsonl
--label COREF,OBJECT
--span-label PERSON,PRODUCT,NP
--disable-patterns ./disable_patterns.jsonl
--add-ents
--wrap
prodigy rel.manual relation_data en_core_web_sm ./data.jsonl --label COREF,OBJECT --span-label PERSON,PRODUCT,NP --disable-patterns ./disable_patterns.jsonl --add-ents --wrap

disable_patterns.jsonl{"pattern": [{"is_punct": true}]}
{"pattern": [{"pos": "VERB"}]}
{"pattern": [{"lower": {"in": ["'s", "’s"]}}]}

Annotate images by drawing rectangular bounding boxes and polygon shapes. Each shape will be added to the task’s "spans" with its label and a "points" property containing the [x, y] pixel coordinate tuples. See here for more details on the JSONL format. You can click and drag or click and release to draw boxes. Polygon shapes can also be closed by double-clicking when adding the last point, similar to closing a shape in Photoshop or Illustrator. Clicking on the label will select a shape so you can change the label or delete it.

prodigy
image.manual
dataset
source
--loader
--exclude
--width
--darken
--no-fetch
--remove-base64

Example

prodigy
image.manual
photo_objects
./stock-photos
--label LAPTOP,CUP,PLANT
prodigy image.manual photo_objects ./stock-photos --label LAPTOP,CUP,PLANT

Manually label regions for the given labels in the audio or video file. The recipe expects a directory of audio files as the source argument and will use the audio loader (default) to load the data. To load video files instead, you can set --loader video. Each added region will be added to the "audio_spans" with a start and end timestamp and the selected label.

prodigy
audio.manual
dataset
source
--loader
--label
--autoplay
--keep-base64
--fetch-media
--exclude

Example

prodigy
audio.manual
speaker_data
./recordings
--label SPEAKER_1,SPEAKER_2,NOISE
prodigy audio.manual speaker_data ./recordings --label SPEAKER_1,SPEAKER_2,NOISE

Recipe command

prodigy
audio.manual
speaker_data
./recordings
--loader video
--label SPEAKER_1,SPEAKER_2
prodigy audio.manual speaker_data ./recordings --loader video --label SPEAKER_1,SPEAKER_2

Manually transcribe audio and video files by typing the transcript into a text field. The recipe expects a directory of audio files as the source argument and will use the audio loader (default) to load the data. To load video files instead, you can set --loader video. The transcript will be stored as the key "transcript". To make it easier to toggle play and pause as you transcribe and to prevent clashes with the text input field (like with the default enter), this recipe lets you customize the keyboard shortcuts. To toggle play/pause, you can press command/option/alt/ctrl+enter or provide your own overrides via --playpause-key.

prodigy
audio.transcribe
dataset
source
--loader
--autoplay
--keep-base64
--fetch-media
--playpause-key
--text-rows
--text-rows
--exclude

Example

prodigy
audio.transcribe
speaker_transcripts
./recordings
--text-rows 3
prodigy audio.transcribe speaker_transcripts ./recordings --text-rows 3

Train a model with one or more components (NER, text classification, tagger, parser, sentence recognizer or span categorizer) using one or more Prodigy datasets with annotations. The recipe calls into spaCy directly and can update an existing model or train a new model from scratch. For each component, you can provide optional datasets for evaluation using the eval: prefix, e.g. --ner dataset,eval:eval_dataset. If no evaluation sets are specified, the --eval-split is used to determine the percentage held back for evaluation.

Datasets will be merged and conflicts will be filtered out. If your data contains potentially conflicting annotations, it’s recommended to first use review to resolve them. If you specify an output directory as the first argument, the best model will be saved at the end. You can then load it into spaCy by pointing spacy.load at the directory.

prodigy
train
output_dir
--ner
--textcat
--textcat-multilabel
--tagger
--parser
--senter
--spancat
--coref
--eval-split
--config
--base-model
--lang
--label-stats
--verbose
--silent
--gpu-id
overrides
-F

Example

prodigy
train
--ner fashion_brands_training,eval:fashion_brands_eval
======================== Generating Prodigy config ========================
ℹ Auto-generating config with spaCy
✔ Generated training config
============================ Training pipeline ============================
Components: ner
Merging training and evaluation data for 1 components
  - [ner] Training: 685 | Evaluation: 300 (from datasets)
Training: 685 | Evaluation: 300
Labels: ner (1)

ℹ Pipeline: ['tok2vec', 'ner']
ℹ Initial learn rate: 0.001
============================ Training pipeline ============================

E    #       LOSS TOK2VEC  LOSS NER  ENTS_F  ENTS_P  ENTS_R  SCORE
---  ------  ------------  --------  ------  ------  ------  ------
  0       0          0.00     26.50    0.73    0.39    5.43    0.01
  0     200         33.58    847.68   10.88   44.44    6.20    0.11
  1     400         70.88    267.65   33.50   45.95   26.36    0.33
  2     600         67.56    156.63   45.32   62.16   35.66    0.45
  3     800        138.28    134.12   48.17   74.19   35.66    0.48
  4    1000        177.95    109.77   51.43   66.67   41.86    0.51
  6    1200         94.95     52.13   54.63   67.82   45.74    0.55
  8    1400        126.85     66.19   56.00   65.62   48.84    0.56
 10    1600         38.34     24.16   51.96   70.67   41.09    0.52
 13    1800        105.14     23.23   56.88   69.66   48.06    0.57
 16    2000         32.27     12.44   54.55   71.25   44.19    0.55

The example below shows how you might run the same command but with overrides. Notice how this training run uses less steps and has a different learning rate.

Example with overrides

prodigy
train
--ner fashion_brands_training,eval:fashion_brands_eval
--training.max_steps=1000 --training.optimizer.learn_rate=0.002
======================== Generating Prodigy config ========================
ℹ Auto-generating config with spaCy
✔ Generated training config
============================ Training pipeline ============================
Components: ner
Merging training and evaluation data for 1 components
  - [ner] Training: 685 | Evaluation: 300 (from datasets)
Training: 685 | Evaluation: 300
Labels: ner (1)

ℹ Pipeline: ['tok2vec', 'ner']
ℹ Initial learn rate: 0.002
============================ Training pipeline ============================

E    #       LOSS TOK2VEC  LOSS NER  ENTS_F  ENTS_P  ENTS_R  SCORE
---  ------  ------------  --------  ------  ------  ------  ------
  0       0          0.00     26.50    0.73    0.39    5.43    0.01
  0     200         33.58    847.68   10.88   44.44    6.20    0.11
  1     400         70.88    267.65   33.50   45.95   26.36    0.33
  2     600         67.56    156.63   45.32   62.16   35.66    0.45
  3     800        138.28    134.12   48.17   74.19   35.66    0.48
  4    1000        177.95    109.77   51.43   66.67   41.86    0.51

Train a model with one or more components (NER, text classification, tagger, parser, sentence recognizer or span categorizer) with different portions of the training examples and print the accuracy figures and accuracy improvements with more data. This recipe takes pretty much the same arguments as train. --n-samples sets the number of sample models to train at different stages. For instance, 10 will train models for 10% of the examples, 20%, 30% and so on. This recipe is useful to determine the quality of the collected annotations, and whether more training examples will improve the accuracy. As a rule of thumb, if accuracy improves within the last 25%, training with more examples will likely result in better accuracy.

prodigy
train-curve
--ner
--textcat
--textcat-multilabel
--tagger
--parser
--senter
--spancat
--coref
--eval-split
--config
--base-model
--lang
--gpu-id
--n-samples
--show-plot
overrides
-F

Example

prodigy
train-curve
--ner news_headlines
--show-plot
======================== Generating Prodigy config ========================
ℹ Auto-generating config with spaCy
✔ Generated training config
========================== Train curve diagnostic ==========================
Training 4 times with 25%, 50%, 75%, 100% of the data

%      Score    ner
----   ------   ------
  0%   0.00     0.00
 25%   0.31 ▲   0.31 ▲
 50%   0.44 ▲   0.44 ▲
 75%   0.43 ▼   0.43 ▼
100%   0.56 ▲   0.56 ▲

    ┌──────────────────────────────────┐
0.56┤                                 •│
0.44┤                 •           •••• │
0.43┤              ••• •••••••••••     │
    │           •••                    │
0.31┤        •••                       │
    │      ••                          │
    │    ••                            │
    │  ••                              │
0.00┤••                                │
    └┬───────┬────────┬───────┬───────┬┘
     0%     25%      50%     75%    100%

✔ Accuracy improved in the last sample
As a rule of thumb, if accuracy increases in the last segment, this could indicate that collecting more annotations of the same type might improve the model further.

Combine multiple datasets, merge annotations on the same examples and output training and evaluation data in spaCy’s binary .spacy format, which you can use with spacy train. The command takes an output directory and generates all data required to train a pipeline with spaCy, including the config and pre-generated labels data to speed up the training process. This recipe will merge annotations for the different pipeline components and outputs a combined training corpus. If an example is only present in one dataset type, its annotations for the other components will be missing values. It’s recommended to use the review recipe on the different annotation types first to resolve conflicts properly.

prodigy
data-to-spacy
output_dir
--ner
--textcat
--textcat-multilabel
--tagger
--parser
--senter
--spancat
--coref
--eval-split
--config
--base-model
--lang
--verbose
-F

Example

prodigy
data-to-spacy
./corpus
--ner news_ner_person,news_ner_org,news_ner_product
--textcat news_cats2018,news_cats2019
--eval-split 0.3
ℹ Using language 'en'
============================= Generating data =============================
Components: ner, textcat
Merging training and evaluation data for 2 components
  - [ner] Training: 685 | Evaluation: 300 (from datasets)
  - [textcat] Training: 538 | Evaluation: 230 (30% split)
Training: 1223 | Evaluation: 530
Labels: ner (3) | textcat (5)
✔ Saved 1223 training examples
./corpus/train.spacy
✔ Saved 530 evaluation examples
./corpus/dev.spacy
============================ Generating config ============================
ℹ Auto-generating config with spaCy
✔ Generated training config
======================= Generating cached label data =======================
✔ Saving label data for component 'ner'
./corpus/labels/ner.json
✔ Saving label data for component 'textcat'
./corpus/labels/textcat.json
============================ Finalizing export ============================
✔ Saved training config
./corpus/config.cfg

To use this data for training with spaCy, you can run:
python -m spacy train ./corpus/config.cfg --paths.train ./corpus/train.spacy --paths.dev ./corpus/dev.spacy

Training in spaCy v3

spacy
train
./corpus/config.cfg
--paths.train ./corpus/train.spacy
--paths.dev ./corpus/dev.spacy

Generate a starter config for training from Prodigy datasets which you can use with spacy train. A custom reader will be used that merges annotations on the same examples. It’s recommended to use the review recipe on the different annotation types first to resolve conflicts properly (instead of relying on this recipe to just filter conflicting annotations and decide on one).

Build a terminology list interactively using a model’s word vectors and seed terms, either a comma-separated list or a text file containing one term per line. Based on the seed terms, a target vector is created and only terms similar to that target vector are shown. As you annotate, the recipe iterates over the vector model’s vocab and updates the target vector with the words you accept.

prodigy
terms.teach
dataset
vectors
--seeds
--resume

Example

prodigy
terms.teach
prog_lang_terms
en_core_web_lg
--seeds Python,C++,Ruby

Convert a dataset collected with terms.teach or sense2vec.teach to a JSONL-formatted patterns file. You can optionally provide a spaCy pipeline for tokenization to create token-based patterns and make them case-insensitive. If no model is provided, the patterns will be generated as exact string matches. Pattern files can be used in Prodigy to bootstrap annotation and pre-highlight suggestions, for example in ner.manual. You can also use them with spaCy’s EntityRuler for rule-based named entity recognition.

prodigy
terms.to-patterns
dataset
output_file
--label
--spacy-model
--case-sensitive

Example

prodigy
terms.to-patterns
prog_lang_terms
./prog_lang_patterns.jsonl
--label PROGRAMMING_LANGUAGE
--spacy-model blank:en

✨ Exported 59 patterns
./prog_lang_patterns.jsonl

This recipe marks entity predictions obtained from a large language model configured by spacy-llm and allows you to accept them as correct, or to manually curate them. This recipe may require environment variables to be set, depending on the large language model that you’re using. The details of this are explained in this section

prodigy
ner.llm.correct
dataset
config_path
source
--loader
--segment
--component
--overrides

Example spacy-llm config file for NER[nlp]
lang = "en"
pipeline = ["llm"]

[components]

[components.llm]
factory = "llm"

[components.llm.task]
@llm_tasks = "spacy.NER.v2"
labels = ["DISH", "INGREDIENT", "EQUIPMENT"]

[components.llm.model]
@llm_models = "spacy.GPT-4.v1"
config = {"temperature": 0.3}

[components.llm.cache]
@llm_misc = "spacy.BatchCache.v1"
path = "local-ner-cache"
batch_size = 3
max_batches_in_mem = 10

Example
 
prodigy 
ner.llm.correct
annotated-recipes
config.cfg
examples.jsonl

prodigy
ner.llm.fetch
config_path
source
output
--loader
--segment
--component
--resume
--overrides

Example

prodigy
ner.llm.fetch
config.cfg
examples.jsonl
ner-annotated.jsonl

100%|████████████████████████████| 50/50 [00:12<00:00, 3.88it/s]

prodigy
textcat.llm.correct
dataset
config_path
source
--loader
--component
--overrides

Example spacy-llm config file for textcat
[nlp]
lang = "en"
pipeline = ["llm"]

[components]

[components.llm]
factory = "llm"

[components.llm.task]
@llm_tasks = "spacy.TextCat.v2"
labels = ["RECIPE", "QUESTION", "FEEDBACK"]

[components.llm.model]
@llm_models = "spacy.GPT-4.v1"
config = {"temperature": 0.3}

[components.llm.cache]
@llm_misc = "spacy.BatchCache.v1"
path = "local-ner-cache"
batch_size = 3
max_batches_in_mem = 10

Example
 prodigy
textcat.llm.correct annotated-recipes
config.cfg
examples.jsonl 

prodigy
textcat.llm.fetch
config_path
source
output
--loader
--resume
--component
--overrides

Example

prodigy
textcat.llm.fetch
config.cfg
examples.jsonl
dataset:textcat-annotated

100%|████████████████████████████| 50/50 [00:12<00:00, 3.88it/s]

prodigy
spans.llm.correct
dataset
config_path
source
--loader
--component
--overrides

Basic spacy-llm config for spans[nlp]
lang = "en"
pipeline = ["llm"]

[components]

[components.llm]
factory = "llm"
save_io = true

[components.llm.task]
@llm_tasks = "spacy.SpanCat.v2"
labels = ["DISH", "INGREDIENT", "EQUIPMENT"]

[components.llm.task.label_definitions]
DISH = "Extract the name of a known dish."
INGREDIENT = "Extract the name of a cooking ingredient, including herbs and spices."
EQUIPMENT = "Extract any mention of cooking equipment. e.g. oven, cooking pot, grill"

[components.llm.model]
@llm_models = "spacy.GPT-3-5.v1"
config = {"temperature": 0.3}

[components.llm.task.examples]
@misc = "spacy.FewShotReader.v1"
path = "span_examples.yaml"

[components.llm.cache]
@llm_misc = "spacy.BatchCache.v1"
path = "local-cached"
batch_size = 3
max_batches_in_mem = 10

span_examples.yaml- text: 'Mac and Cheese is a popular American pasta variant.'
  entities:
    INGREDIENT: ['Cheese']
    DISH: ['Mac and Cheese']

Example
 
prodigy 
spans.llm.correct
annotated-recipes
config.cfg
examples.jsonl

prodigy
spans.llm.fetch
config_path
source
output
--loader
--resume
--component
--overrides

Example

prodigy
spans.llm.fetch
config.cfg
examples.jsonl
spancat-annotated.jsonl

100%|████████████████████████████| 50/50 [00:12<00:00, 3.88it/s]

prodigy
terms.llm.fetch
dataset
config_path
output
--n_requests
--auto-accept
--component
--overrides

Example spacy-llm config for terms[nlp]
lang = "en"
pipeline = ["llm"]

[components]

[components.llm]
factory = "llm"

[components.llm.task]
@llm_tasks = "prodigy.Terms.v1"
batch_size = 50

[components.llm.model]
@llm_models = "spacy.GPT-3-5.v1"
config = {"temperature": 0.3}

Example

prodigy
terms.llm.fetch
skateboard-trick-terms
config.cfg
"skateboard tricks"

100%|████████████████████████████| 50/50 [00:12<00:00, 3.88it/s]

prodigy
ab.llm.tournament
dataset
inputs_path
prompt_path
config_path
--display-template-path
--no-random
--resume
--no-meta

Example spacy-llm config for the tournament recipe[nlp]
lang = "en"
pipeline = ["llm"]

[components]

[components.llm]
factory = "llm"

[components.llm.task]
@llm_tasks = "prodigy.TextPrompter.v1"

[components.llm.model]
@llm_models = "spacy.GPT-3-5.v1"
config = {"temperature": 0.3}

Example spacy-llm config for the tournament recipe[nlp]
lang = "en"
pipeline = ["llm"]

[components]

[components.llm]
factory = "llm"

[components.llm.task]
@llm_tasks = "prodigy.TextPrompter.v1"

[components.llm.model]
@llm_models = "spacy.GPT-4.v1"
config = {"temperature": 0.3}

prompt1.jinja2Write a haiku about {{topic}} that rhymes.

prompt2.jinja2Write a super funny haiku about {{topic}} that rhymes.

{"topic": "star wars"}
{"topic": "python"}
{"topic": "stroopwafels"}

Example

    prodigy
    ab.llm.tournament
    haiku-tournament
    inputs.jsonl
    ./prompt_folder
    ./config_folder
    --resume

Output after annotating a few examples


============== Current winner: [prompt1.jinja2 + cfg1.cfg] ==============

comparison                                                  prob   trials
[prompt1.jinja2 + cfg1.cfg] > [prompt1.jinja2 + cfg2.cfg]   0.50        0
[prompt1.jinja2 + cfg1.cfg] > [prompt2.jinja2 + cfg1.cfg]   0.50        0
[prompt1.jinja2 + cfg1.cfg] > [prompt2.jinja2 + cfg1.cfg]   0.71        1

Output after annotating more examples, ratings will converge


============== Current winner: [prompt1.jinja2 + cfg1.cfg] ==============

comparison                                                  prob   trials
[prompt1.jinja2 + cfg1.cfg] > [prompt1.jinja2 + cfg2.cfg]   0.96       15
[prompt1.jinja2 + cfg1.cfg] > [prompt2.jinja2 + cfg1.cfg]   0.94       12
[prompt1.jinja2 + cfg1.cfg] > [prompt2.jinja2 + cfg1.cfg]   0.89       10

prodigy
ner.openai.correct
dataset
source
--labels
--lang
--model
--examples-path
--max-examples
--prompt-path
--batch-size
--segment
--loader
--verbose

prodigy
ner.openai.fetch
dataset
source
output_path
--labels
--lang
--model
--examples-path
--max-examples
--prompt-path
--batch-size
--segment
--resume
--loader
--verbose

Example

prodigy
ner.openai.fetch
examples.jsonl
ner-annotated.jsonl
--labels "dish,ingredient,equipment"

100%|████████████████████████████| 50/50 [00:12<00:00, 3.88it/s]

prodigy
textcat.openai.correct
dataset
source
--labels
--lang
--model
--batch-size
--segment
--prompt-path
--examples-path
--max-examples
--loader
--verbose

Example

prodigy
textcat.openai.correct
food-comments
examples.jsonl
--labels "recipe,feedback,question"

prodigy
textcat.openai.fetch
dataset
source
output_path
--labels
--lang
--model
--prompt-path
--examples-path
--max-examples
--batch-sizes
--segment
--exclusive-classess
--resume
--loader
--verbose

Example

prodigy
textcat.openai.fetch
examples.jsonl
textcat-annotated.jsonl
"recipe,feedback,question"

100%|████████████████████████████| 50/50 [00:12<00:00, 3.88it/s]

prodigy
terms.openai.fetch
query
output_path
--seeds
--n
--model
--prompt-path
--resume
--temperature
--top-p
--best-of
--n-batch
--max-tokens

Example

prodigy
terms.openai.fetch
"skateboard tricks"
./tricks.jsonl
--n 100
"kickflip,ollie"

100%|████████████████████████████| 100/100 [00:12<00:00, 3.88it/s]

prodigy
ab.openai.prompts
dataset
inputs_path
display_template_path
prompt1_template_path
prompt2_template_path
--model
--batch-size
--temperature
--no-random
--repeat
--verbose
--no-meta

Example

prodigy
ab.openai.prompts
haiku
haiku-inputs.jsonl
title.jinja2
prompt1.jinja2
prompt2.jinja2
--repeat 5

Output after annotating


=============== ✨  Evaluation results ===============

✔ You preferred prompt1.jinja2
prompt1.jinja2   11
prompt2.jinja2    5

prodigy
ab.openai.tournament
dataset
inputs_path
display_template_path
prompt_template_folder
--model
--batch-size
--resume
--temperature
--no-random
--verbose
--no-meta

Example

    prodigy
    ab.openai.tournament
    haiku-tournament
    
        input.jsonl
    
    
        title.jinja2
    
    
        prompt_folder
    
    --resume

Output after annotating a few examples


=============== ✨  Current winner: prompt3.jinja2 ===============

comparison                          value  count
P(prompt3.jinja2 > prompt1.jinja2)   0.60      7
P(prompt3.jinja2 > prompt2.jinja2)   0.83      3

Output after annotating more examples, ratings will converge


=============== ✨  Current winner: prompt3.jinja2 ===============

comparison                          value   count
P(prompt3.jinja2 > prompt1.jinja2)   0.95      19
P(prompt3.jinja2 > prompt2.jinja2)   0.97       5

prodigy
review
dataset
in_sets
--label
--view-id
--fetch-media
--show-skipped
--auto-accept
--auto-accept

Example (binary)

prodigy
review
food_reviews_final
food_reviews2019,food_reviews2018

Example (manual)

prodigy
review
ner_final
ner_news,ner_misc
--label ORG,PRODUCT

prodigy
compare
dataset
a_file
b_file
--no-random
--diff

prodigy
compare
eval_translation
./model_a.jsonl
./model_b.jsonl

model_a.jsonl{"id": 1, "input": {"text": "FedEx von weltweiter Cyberattacke getroffen"}, "output": {"text": "FedEx hit by worldwide cyberattack"}}

model_b.jsonl{"id": 1, "input": {"text": "FedEx von weltweiter Cyberattacke getroffen"}, "output": {"text": "FedEx from worldwide Cyberattacke hit"}}

Example metric.iaa.doc output

prodigy
metric.iaa.doc
dataset:textcat
multiclass
-l POSITIVE,NEGATIVE
ℹ Annotation Statistics

Attribute                     Value
--------------------------    -----
Examples                      100
Categories                    3
Coincident Examples*          50
Single Annotation Examples    4
Annotators                    2
Avg. Annotations per Example  2.00
\* (>1 annotation)
ℹ Agreement Statistics 
Statistic                     Value
--------------------------    -----
Percent (Simple) Agreement    0.4167
Krippendorff's Alpha          0.1809
Gwet's AC2                    0.1640

Example metric.iaa.span output

prodigy
metric.iaa.span
dataset:ner
-l PER,ORG
ℹ Annotation Statistics

Attribute                     Value
--------------------------    -----
Examples                      22
Categories                    2
Coincident Examples*          12
Single Annotation Examples    0
Annotators                    2
Avg. Annotations per Example  2.00
\* (>1 annotation)
ℹ Agreement Statistics 
Label                         Value
--------------------------    -----
PER                           0.0
ORG                           0.5

ℹ Confusion matrix

        PER    ORG   NONE
---    ----   ----   ----
PER    0.00   0.00   0.00
ORG    0.00   0.67   0.33
NONE   0.00   0.60   0.40                 

Example metric.iaa.binary output

prodigy
metric.iaa.binary
dataset:ner
ℹ Annotation Statistics

Attribute                     Value
--------------------------    -----
Examples                      37
Categories                    2
Coincident Examples*          17
Single Annotation Examples    3
Annotators                    2
Avg. Annotations per Example  1.85
\* (>1 annotation)
ℹ Agreement Statistics 
Statistic                     Value
--------------------------    -----
Percent (Simple) Agreement    0.5294
Krippendorff's Alpha          0.1833
Gwet's AC2                    0.1681

prodigy
mark
dataset
source
--loader
--label
--view-id
--exclude

prodigy
mark
news_marked
./news_headlines.jsonl
--label INTERESTING
--view-id classification

prodigy
match
dataset
spacy_model
source
--loader
--label
--patterns
--label-task
--label-span
--combine-matches
--exclude

prodigy
match
news_matched
blank:en
./news_headlines.jsonl
--patterns ./news_patterns.jsonl
--label ORG,PRODUCT
--label-span

prodigy
print-stream
spacy_model
source
--loader

prodigy
print-dataset
dataset
--style

Example

prodigy
filter-by-patterns
source
output
spacy_model
--patterns
--label
--loader

Example

prodigy
filter-by-patterns
examples.jsonl
subset.jsonl
en_core_web_sm
--patterns ./patterns.jsonl

100%|████████████████████████████| 250/250 [00:12<00:00, 20.88it/s]

prodigy
db-out
dataset
out_dir
--dry

Example

prodigy
db-out
news_headlines
>
./news_headlines.jsonl
prodigy db-out news_headlines > ./news_headlines.jsonl

prodigy
db-merge
in_sets
out_set
--rehash
--dry