Precrec

precrec calculates and plots ROC and precision-recall curves for binary classifiers. It is built for the case where the two curves disagree: on an imbalanced dataset a ROC curve can look excellent while the precision-recall curve shows the classifier is not usable. All the main calculations are implemented in C++ through Rcpp.

Quick example

library(precrec)

# Load a test dataset
data(P10N10)

# Calculate ROC and Precision-Recall curves
sscurves <- evalmod(scores = P10N10$scores, labels = P10N10$labels)

# The ggplot2 package is required
library(ggplot2)

# Show ROC and Precision-Recall plots
autoplot(sscurves)

auc(sscurves) gives the areas, and as.data.frame(sscurves) gives the curve points.

Documentation

Everything is on the package website, in short pages:

• Get started – the five-minute tour. Also available offline with vignette("introduction", package = "precrec").
• How-to – one page per task: preparing data, several test sets, cross-validation, more than two classes, large datasets.
• Compare and decide – which model to ship, at which threshold, and how sure you can be.
• Metrics – what each available metric means and when it misleads.
• Plots – every plot the package draws, and how to change it.
• Reference – the help page of every function, each with examples. Also help(package = "precrec").

Why precrec

Accurate curves. Non-linear interpolation, elongation to the y axis where the first point is undefined, and score-wise thresholds instead of fixed bins. Joining raw precision-recall points with straight lines – what most tools do – overestimates the area.

Fast. Curves over a large dataset take seconds. mode = "aucroc" computes the ROC area from the U statistic without building the curve at all.

Many metrics. Fourteen per-cutoff metrics by default and seventeen more on request, plus AUC, partial AUC, the precision-recall break-even point, and the probability-based Brier score, RMSE and log loss. See the metrics overview.

Several models and several test sets. Averaged curves with confidence bands, cross-validation folds, and confidence intervals of the AUC.

More than two classes. One-vs-rest decomposition, with per-class and macro-averaged areas.

Partial curves. Partial AUCs for any x or y range, standardized so ranges of different widths can be compared.

Installation

Install the release version from CRAN:

install.packages("precrec")

Or the development version from GitHub, which needs a working compiler (Rtools on Windows, Xcode on macOS, the usual build tools on Linux):

# install.packages("devtools")
devtools::install_github("evalclass/precrec")

Functions

Function	Description
evalmod	Main function to calculate evaluation metrics
mmdata	Reformat input data for performance evaluation calculation
join_scores	Join scores of multiple models into a list
join_labels	Join observed labels of multiple test datasets into a list
create_sim_samples	Create random samples for simulations
format_nfold	Create n-fold cross validation dataset from data frame
prob_metrics	Calculate the Brier score, the RMSE and the log loss
prob_metrics_ci	Calculate CIs of the Brier score, the RMSE and the log loss
metric_curve	Draw one evaluation metric against another
metric_table	Get every evaluation metric at every cutoff
best_cutoff	Pick the cutoff that optimizes one evaluation metric
prbe	Find the precision-recall break-even point
average_precision	Calculate the step estimator of the PRC area
classification_report	Report precision, recall and F-score per class at a threshold
auc_boot	Bootstrap the AUC of a single test set
auc_delong	Analytic standard error of the ROC AUC (DeLong)
auc_diff	Compare the AUCs of two models on the same resamples

Ten S3 generics work on the objects evalmod and metric_curve return.

S3 generic	Package	Description
print	base	Print the calculation results and the summary of the test data
as.data.frame	base	Convert a precrec object to a data frame
as.data.table	data.table	Convert a precrec object to a data.table
plot	graphics	Plot performance evaluation metrics
autoplot	ggplot2	Plot performance evaluation metrics with ggplot2
fortify	ggplot2	Prepare a data frame for ggplot2
auc	precrec	Make a data frame with AUC scores
part	precrec	Calculate partial curves and partial AUC scores
pauc	precrec	Make a data frame with pAUC scores
auc_ci	precrec	Calculate confidence intervals of AUC scores

Citation

Precrec: fast and accurate precision-recall and ROC curve calculations in R

Takaya Saito; Marc Rehmsmeier

Bioinformatics 2017; 33 (1): 145-147.

doi: 10.1093/bioinformatics/btw570

External links

• Classifier evaluation with imbalanced datasets - our web site that contains several pages with useful tips for performance evaluation on binary classifiers.

• The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets - our paper that summarized potential pitfalls of ROC plots with imbalanced datasets and advantages of using precision-recall plots instead.