sheepdog

In the fable, the boy cries "wolf!" and the villagers have to decide whether the wolf is real. sheepdog does that check for automated predictions from BirdNET pretrained and custom classifiers in passive acoustic monitoring (PAM). It runs three stages:

1. Validation data. Build clip sets for manual review from the BirdNET selection tables: top-score, random-occasion or random-recording designs, one folder per species, ready for the review tab of the BirdNET GUI or for the review app of the package (run_app()). The random draws can be stratified along an environmental gradient. Then summarise the reviewed clips: precision by confidence bin, suggested thresholds, confirmed presence by night, clips for retraining and the cells that still need review.
2. Calibration. Turn confidence scores into probabilities of true detection with hierarchical logistic models (species and site x species effects, compared by cross-validation), fitted with 'lme4' or, in a Bayesian framework, with 'brms'. Score every detection and build nightly detection histories.
3. Community matrices. Turn the calibrated histories into effort-standardised site x month matrices for multivariate ordinations.

sheepdog was developed for the FrogNet project (anurans of the Pantanal and Cerrado, Mato Grosso do Sul, Brazil), but it works with the output of any BirdNET pretrained and custom classifier. export_occupancy_data() writes its validation and calibration outputs in a documented format for occupancy models with false positives, such as those fitted with crywolf. The two packages are independent: sheepdog does not depend on crywolf.

Installation

Install the package from r-universe, which provides binaries for Windows, macOS and Linux:

install.packages("sheepdog",
                 repos = c("https://diogoprov.r-universe.dev",
                           "https://cloud.r-project.org"))

The development version can be installed from GitHub:

# install.packages("remotes")
remotes::install_github("diogoprov/sheepdog")

Cutting the clips for review (the 3-s windows that BirdNET analyses, or whole recordings in the random-recording design) needs av (it bundles ffmpeg), the hierarchical calibration needs lme4, and the figures need ggplot2:

install.packages(c("av", "lme4", "ggplot2"))

Calibration performance

In threshold-based approaches, calibration automatically compares global and site-effect models per species; with one site, it keeps only estimable candidates. Explicit model lists remain available in compare_calibration_models().

The performance article summarises the Curiango benchmark, regression checks, computing times and an explicitly simulated multispecies benchmark. The results do not show a consistent advantage of one selection rule across datasets.

Grouped calibration folds

Use calibrate(..., fold_group = "source_audio") to keep recordings intact, or fold_group = "night" to keep nights intact across species at each site. The input validation tables must contain those metadata; each site needs at least k_folds groups. The default remains clip folds. For a manual workflow, retain the metadata with read_validations(..., keep = c("source_audio", "night")) and inspect the assignment with calibration_folds(). See the calibration article for examples and limitations.

Migrating from 2.x

In 3.0.0, calibration defaults to select = "one_se_logloss". Use select = "best_auc" to preserve the 2.x model selection. See the migration guide for details and comparison of downstream results.

This is the beginning of the README. Read the rest at the original. Read the full README ↗