ApplyAnnotator Tutorial

ApplyAnnotator adds labels to documents. These labels can be for a whole document (such as spam) or for portions of documents such as names or places. In other words, ApplyAnnotator accepts the saved result (annotator) of both TrainClassifier and TrainExtractor. ApplyAnnotator may be used on unlabeled documents or labeled documents, which may be helpful for comparing true labels to extracted or classified labels.

This example will use the ClassifierAnnotator which can be saved when running through the TrainClassifier Tutorial. For a quick reference, here is how to create the annotator from the TrainClassifier tutorial using the command line:

% java –Xmx500M edu.cmu.minorthird.ui.TrainClassifier –labels sample3.train –spanType fun –saveAs sample3.ann

This classifier annotator labels documents that are fun.

To run this type of task start with:

java –Xmx500M edu.cmu.minorthird.ui.ApplyAnnotator

Editing Parameters:

Like all ui tasks, all the parameters for ApplyAnnotator may be specified in either the gui or by the command line. To use the gui, simple type the –gui on the command line. It is also possible to mix and match where the parameters are specified; for example: one can specify two parameters on the command line and use the gui to select the rest. For this reason, the step by step process for this experiment will first explain how to select a parameter value in the gui and then how to set the same parameter on the command line.

To view a list of parameters and their function run:

% java –Xmx500M edu.cmu.minorthird.ui.ApplyAnnotator –help

OR

% java –Xmx500M edu.cmu.minorthird.ui.ApplyAnnotator –gui

And click on the “Parameters” button next to Help or and click on the “?” button next to each field in the Property Editor to see what it is used for.

If using the gui, click the edit button next to ApplyAnnotator when a window appears to edit the parameters. A Property Editor window will appear:

1) There are three bunches of parameters that may be altered. A collection of documents (labelsFilename) and an Annotator (loadFrom) are required. All other fields are optional. For more information about any of the fields, click on the ? (Help) next to the field.

1. baseParameters: contains the options for loading the collection of documents.

a) GUI: Enter sample3.test in the labelsFilename textField. Sample3.test contains labeled documents, but it is useful for comparing trueLabels to classified labels.

b) Command Line: use the –labels option followed by the repositoryKey or the directory of files to load. In this case specify –labels sample3.test

2. loadAnnotatorParams: contains one parameter for specifying the annotator to load.

a) GUI: Enter sample3.ann (or the filename you chose for your annotator) in the loadFrom text field.

b) Command Line: -loadFrom sample3.ann (or the filename you chose for your annotator)

3. saveParameters: contains one parameter for specifying a file to save the result to. Saving is optional, but useful for using result in other experiments or for reference. It is useful to save in the format labelsFilename.labels where labelsFilename is the directory entered in the labelsFilename textfield. This way minorthird can automatically load the labels produced by this experiment in another minorthird task.

a) GUI: Type sample3.labels in the saveAs textField

b) Command Line: -saveAs sample3.labels

2) Feel free to try changing any of the other parameters including the ones in advanced options.

a. GUI: Click on the help buttons to get a feeling for what each parameter does and how changing it may affect your results. Once all the parameters are set, click the “OK” button on the PropertyEditor.

b. Command Line: Add other parameters to the command line (use –help option to see other parameter options) If there is an option that can be set in the gui, but there is no specific parameter for setting it in the help parameter definition, the –other option may be used. To see how to use this option, look at the Command Line Other Option Tutorial.htm

Show Labeled Data:

3) If you would like to view the input data for the annotator task, specify the showLabels option. This will pop up the same TextBaseViewer that you would see if you ran ViewLabels on the data.

a. GUI: Press the Show Labels button

b. Command Line: -showLabels

Getting and Interpreting Results:

4) Command Line: to see gui results, specify –showResult.

5) Now press Start Task under execution controls in the gui or enter on the command line. The task will vary in the amount of time it takes depending on the size of the data set and the annotator. When the task is finished, click on the “View Results” button to see how minorthird has labeled the data. This is the same window that appears when specifying the –showResult option on the command line.

Note: the annotator will output the type that was chosen for output in the TrainClassifier experiment; the default is _prediction. To have the annotator output a more informative type (such as predicted_fun) go back to TrainClassifier and either change the output text field in the Property Editor or specify –output on the command line.

In this case, there is both true data and classified data, so it is useful to compare fun to _prediction to see how the classifier performed. This is a simple example, so the classifier predicted every document correctly, which is why each document is highlighted green. In general if there a document has the first type (in this case the true type fun) and not the second type (predicted type _prediction) it will highlight blue, if the document has the second type but not the first, it will appear red, and if the document has both the first and the second type, the document will appear green like above.

If there are no true type labels (the document were unlabeled before running ApplyAnnotator), try highlighting the spanTypes or spanProps to see which documents or words were predicted.