edu.cmu.minorthird.classify.experiments
Class Evaluation

java.lang.Object
  edu.cmu.minorthird.classify.experiments.Evaluation

All Implemented Interfaces:

Visible, Saveable, java.io.Serializable

public class Evaluation
extends java.lang.Object
implements Visible, java.io.Serializable, Saveable

Stores some detailed results of evaluating a classifier on data.

Author:

William Cohen

See Also:

Serialized Form

Nested Class Summary

static class	Evaluation.ConfusionMatrixViewer
static class	Evaluation.ElevenPointPrecisionViewer
static class	Evaluation.Matrix
class	Evaluation.MyTableCellRenderer
static class	Evaluation.PropertyViewer
static class	Evaluation.ROCViewer
class	Evaluation.SummaryViewer

Field Summary

static int	DEFAULT_PARTITION_ID
static java.lang.String	EVAL_EXT
static java.lang.String	EVAL_FORMAT_NAME

Constructor Summary

Evaluation(ExampleSchema schema)
Create an evaluation for databases with this schema

Method Summary

double	averageLogLoss() Average logloss on all examples.
double	averagePrecision() Non-interpolated average precision.
Evaluation.Matrix	confusionMatrix() Return a confusion matrix.
double[]	elevenPointPrecision() Return eleven-point interpolated precision.
double	errorRate() Error rate.
double	errorRateBalanced() Balanced Error rate.
double[]	errorRateByClass() Error rate by Class.
double	errorRateNeg() Error rate on Negative examples.
double	errorRatePos() Error rate on Positive examples.
double	errors() Weighted total errors.
double	errors(int ID) Weighted total errors on examples with partitionID = ID..
double[]	errorsByClass() Weighted total errors for classes 1 to K.
double[]	errorsByClass(int ID) Weighted total errors for classes 1 to K on examples with partitionID = ID.
double	errorsNeg() Weighted total errors on NEGATIVE examples.
double	errorsNeg(int ID) Weighted total errors on NEGATIVE examples with partitionID = ID.
double	errorsPos() Weighted total errors on POSITIVE examples.
double	errorsPos(int ID) Weighted total errors on POSITIVE examples with partitionID = ID.
void	extend(Classifier c, Dataset d, int cvID) Test the classifier on the examples in the dataset and store the results.
void	extend(ClassLabel predicted, Example example, int cvID) Record the result of predicting the give class label on the given example
void	extend(SemiSupervisedClassifier c, SemiSupervisedDataset d, int cvID) Test the classifier on the examples in the dataset and store the results.
void	extend(SequenceClassifier c, SequenceDataset d) Test the SequenceClassifier on the examples in the dataset and store the results.
void	extend4SGM(StackedGraphicalLearner.StackedGraphicalClassifier c, RealRelationalDataset d, int cvID) Test the classifier on the examples in the relational dataset and store the results.
double	f1()
ClassLabel	getActual(int i)
java.lang.String[]	getClasses()
java.lang.String	getExtensionFor(java.lang.String format) Recomended extension for the format with the given name.
java.lang.String[]	getFormatNames() List of formats in which the object can be saved.
ClassLabel	getPrediction(int i)
java.lang.String	getProperty(java.lang.String prop)
ExampleSchema	getSchema() Returns whether the ExampleSchema this Evaluation is based upon
boolean	isBinary() Returns whether this Evaluation refers to a binary classifier
boolean	isCorrect(int i)
double	kappa()
static Evaluation	load(java.io.File file)
static void	main(java.lang.String[] args)
double	maxF1() Max f1 values at any cutoff.
double	maxF1(double minThreshold) Max f1 values for any threshold above the specified cutoff.
double[]	numberOfExamplesByClass() Total weight of examples in all classes 1 to K.
double[]	numberOfExamplesByClass(int ID) Total weight of examples in all classes 1 to K with partitionID = ID.
double	numberOfInstances() Total weight of all instances.
double	numberOfInstances(int ID) Total weight of all instances.
double	numberOfNegativeExamples() Total weight of all NEGATIVE examples.
double	numberOfNegativeExamples(int ID) Total weight of all NEGATIVE examples with partitionID = ID.
double	numberOfPositiveExamples() Total weight of all POSITIVE examples.
double	numberOfPositiveExamples(int ID) Total weight of all POSITIVE examples with partitionID = ID.
double	numErrors()
int	numExamples()
double	precision()
Evaluation.Matrix	precisionRecallScore() Return array of precision,recall,logitScore.
double	recall()
double	recallTopK(int k, double minScore) Recall in the top K, excluding items with score
java.lang.Object	restore(java.io.File file) Restore the object from a file.
void	save(java.io.File file)
void	save(java.io.PrintStream out)
void	saveAs(java.io.File file, java.lang.String formatName) Save this object to the given file, in the given format.
void	setProperty(java.lang.String prop, java.lang.String value)
double	stDevErrors() standard deviation of total errors.
double[]	stDevErrorsByClass() standard deviation of total errors for classes 1 to K.
double	stDevErrorsNeg() standard deviation of total errors on NEGATIVE examples.
double	stDevErrorsPos() standard deviation of total errors on POSITIVE examples.
void	summarize() Print summary statistics
java.lang.String[]	summaryStatisticNames()
double[]	summaryStatistics()
Evaluation.Matrix	thousandPointROC() Return actual ROC curve.
Viewer	toGUI() Create a view of this object
java.lang.String	toString() Detailed view.
Evaluation.Matrix	TPfractionFPfractionScore() Return array of true positive,false positive,logitScore.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

DEFAULT_PARTITION_ID

public static final int DEFAULT_PARTITION_ID

See Also:

Constant Field Values

EVAL_FORMAT_NAME

public static final java.lang.String EVAL_FORMAT_NAME

See Also:

Constant Field Values

EVAL_EXT

public static final java.lang.String EVAL_EXT

See Also:

Constant Field Values

Constructor Detail

Evaluation

public Evaluation(ExampleSchema schema)

Create an evaluation for databases with this schema

Method Detail

extend4SGM

public void extend4SGM(StackedGraphicalLearner.StackedGraphicalClassifier c,
                       RealRelationalDataset d,
                       int cvID)

Test the classifier on the examples in the relational dataset and store the results.

extend

public void extend(Classifier c,
                   Dataset d,
                   int cvID)

Test the classifier on the examples in the dataset and store the results.

extend

public void extend(SequenceClassifier c,
                   SequenceDataset d)

Test the SequenceClassifier on the examples in the dataset and store the results.

extend

public void extend(SemiSupervisedClassifier c,
                   SemiSupervisedDataset d,
                   int cvID)

Test the classifier on the examples in the dataset and store the results.

extend

public void extend(ClassLabel predicted,
                   Example example,
                   int cvID)

Record the result of predicting the give class label on the given example

setProperty

public void setProperty(java.lang.String prop,
                        java.lang.String value)

getProperty

public java.lang.String getProperty(java.lang.String prop)

getPrediction

public ClassLabel getPrediction(int i)

getActual

public ClassLabel getActual(int i)

isCorrect

public boolean isCorrect(int i)

errors

public double errors()

Weighted total errors.

errors

public double errors(int ID)

Weighted total errors on examples with partitionID = ID..

errorsByClass

public double[] errorsByClass()

Weighted total errors for classes 1 to K.

errorsByClass

public double[] errorsByClass(int ID)

Weighted total errors for classes 1 to K on examples with partitionID = ID.

errorsPos

public double errorsPos()

Weighted total errors on POSITIVE examples.

errorsPos

public double errorsPos(int ID)

Weighted total errors on POSITIVE examples with partitionID = ID.

errorsNeg

public double errorsNeg()

Weighted total errors on NEGATIVE examples.

errorsNeg

public double errorsNeg(int ID)

Weighted total errors on NEGATIVE examples with partitionID = ID.

stDevErrors

public double stDevErrors()

standard deviation of total errors.

stDevErrorsByClass

public double[] stDevErrorsByClass()

standard deviation of total errors for classes 1 to K.

stDevErrorsPos

public double stDevErrorsPos()

standard deviation of total errors on POSITIVE examples.

stDevErrorsNeg

public double stDevErrorsNeg()

standard deviation of total errors on NEGATIVE examples.

numberOfInstances

public double numberOfInstances()

Total weight of all instances.

numberOfInstances

public double numberOfInstances(int ID)

Total weight of all instances.

numberOfExamplesByClass

public double[] numberOfExamplesByClass()

Total weight of examples in all classes 1 to K.

numberOfExamplesByClass

public double[] numberOfExamplesByClass(int ID)

Total weight of examples in all classes 1 to K with partitionID = ID.

numberOfPositiveExamples

public double numberOfPositiveExamples()

Total weight of all POSITIVE examples.

numberOfPositiveExamples

public double numberOfPositiveExamples(int ID)

Total weight of all POSITIVE examples with partitionID = ID.

numberOfNegativeExamples

public double numberOfNegativeExamples()

Total weight of all NEGATIVE examples.

numberOfNegativeExamples

public double numberOfNegativeExamples(int ID)

Total weight of all NEGATIVE examples with partitionID = ID.

errorRate

public double errorRate()

Error rate.

errorRateByClass

public double[] errorRateByClass()

Error rate by Class.

errorRatePos

public double errorRatePos()

Error rate on Positive examples.

errorRateNeg

public double errorRateNeg()

Error rate on Negative examples.

errorRateBalanced

public double errorRateBalanced()

Balanced Error rate.

recallTopK

public double recallTopK(int k,
                         double minScore)

Recall in the top K, excluding items with score

averagePrecision

public double averagePrecision()

Non-interpolated average precision.

maxF1

public double maxF1()

Max f1 values at any cutoff.

maxF1

public double maxF1(double minThreshold)

Max f1 values for any threshold above the specified cutoff.

kappa

public double kappa()

numExamples

public int numExamples()

averageLogLoss

public double averageLogLoss()

Average logloss on all examples.

precision

public double precision()

recall

public double recall()

f1

public double f1()

summaryStatistics

public double[] summaryStatistics()

summaryStatisticNames

public java.lang.String[] summaryStatisticNames()

confusionMatrix

public Evaluation.Matrix confusionMatrix()

Return a confusion matrix.

numErrors

public double numErrors()

getClasses

public java.lang.String[] getClasses()

TPfractionFPfractionScore

public Evaluation.Matrix TPfractionFPfractionScore()

Return array of true positive,false positive,logitScore.

thousandPointROC

public Evaluation.Matrix thousandPointROC()

Return actual ROC curve. At most 1000 points are kept.

precisionRecallScore

public Evaluation.Matrix precisionRecallScore()

Return array of precision,recall,logitScore.

elevenPointPrecision

public double[] elevenPointPrecision()

Return eleven-point interpolated precision. Precisely, result is an array p[] of doubles such that p[i] is the maximal precision value for any point with recall>=i/10.

toString

public java.lang.String toString()

Detailed view.

Overrides:

toString in class java.lang.Object

summarize

public void summarize()

Print summary statistics

toGUI

public Viewer toGUI()

Description copied from interface: Visible

Create a view of this object

Specified by:

toGUI in interface Visible

getFormatNames

public java.lang.String[] getFormatNames()

Description copied from interface: Saveable

List of formats in which the object can be saved.

Specified by:

getFormatNames in interface Saveable

getExtensionFor

public java.lang.String getExtensionFor(java.lang.String format)

Description copied from interface: Saveable

Recomended extension for the format with the given name.

Specified by:

getExtensionFor in interface Saveable

saveAs

public void saveAs(java.io.File file,
                   java.lang.String formatName)
            throws java.io.IOException

Description copied from interface: Saveable

Save this object to the given file, in the given format.

Specified by:

saveAs in interface Saveable

Throws:

java.io.IOException

restore

public java.lang.Object restore(java.io.File file)
                         throws java.io.IOException

Description copied from interface: Saveable

Restore the object from a file.

Specified by:

restore in interface Saveable

Throws:

java.io.IOException

save

public void save(java.io.File file)
          throws java.io.IOException

Throws:

java.io.IOException

save

public void save(java.io.PrintStream out)
          throws java.io.IOException

Throws:

java.io.IOException

load

public static Evaluation load(java.io.File file)
                       throws java.io.IOException

Throws:

java.io.IOException

isBinary

public boolean isBinary()

Returns whether this Evaluation refers to a binary classifier

getSchema

public ExampleSchema getSchema()

Returns whether the ExampleSchema this Evaluation is based upon

main

public static void main(java.lang.String[] args)