Asimov
Workflow orchestration for complex scientific analyses
A general-purpose framework for managing automated analysis pipelines on HPC clusters
Battle-tested in production by LIGO, Virgo, and KAGRA collaborations
Documentation · Installation · Quick Start · Contributing
Table of Contents
What is Asimov?
Asimov is a general-purpose workflow orchestration platform designed to automate complex scientific analyses on high-throughput computing (HTC) clusters.
At its core, Asimov provides:
- Pipeline-agnostic architecture - Coordinate multiple analysis tools through a unified interface
- Intelligent job management - Submit, monitor, and recover jobs automatically
- Configuration management - Version-controlled, reproducible analysis workflows
- Results organization - Structured handling of outputs with validation
Designed for flexibility: While gravitational wave analysis is the primary use case, Asimov's architecture supports any scientific workflow involving:
- Long-running computational jobs
- Multiple analysis pipelines or tools
- Systematic parameter studies
- HTC cluster environments (HTCondor, Slurm)
- Complex dependencies between analyses
Why Asimov?
Many scientific analyses involve running computationally intensive pipelines on HPC clusters—processes that can take days to weeks and involve hundreds of coordinated jobs. Common challenges include:
- Manual configuration for each pipeline and analysis
- Constant monitoring of job status across clusters
- Error-prone restart and recovery procedures
- Scattered results across different filesystems
- Difficult reproducibility and collaboration
- Pipeline lock-in - custom scripts tied to specific tools
Asimov solves this by providing:
✅ Pipeline-agnostic interface - One framework for multiple analysis tools
✅ Automated job orchestration - Submit, monitor, recover automatically
✅ Version-controlled configuration - Reproducible, auditable workflows
✅ Intelligent error handling - Detect and recover from failures
✅ Results management - Organized, validated outputs
✅ Scalability - From single analyses to large-scale campaigns
🚀 Key Features
🤖 Intelligent Job Management
Asimov interacts with high-throughput computing systems (HTCondor, Slurm) to submit jobs, monitor their progress, detect failures, and automatically initiate recovery or post-processing tasks.
🔌 Pipeline-Agnostic Architecture
Define analyses once, deploy across multiple pipelines. Asimov's extensible plugin system allows you to add support for new analysis tools without modifying the core framework. Current implementations include gravitational wave parameter estimation pipelines (LALInference, Bilby, BayesWave, RIFT), with support for custom pipelines.
📋 Centralized Configuration
All ongoing, completed, and scheduled analyses are recorded in version-controlled ledgers, making it easy to find jobs, configurations, and results. Every analysis is tracked and reproducible.
📊 Reporting & Monitoring
Generate both machine-readable and human-friendly reports of all monitored jobs, with automatic collation of log files, diagnostic outputs, and status summaries.
🗂️ Results Management
Special tools help manage analysis outputs, ensure data integrity, and organize results for publication and archival.
🔄 Workflow Automation
Set up continuous monitoring that automatically:
- Starts jobs when dependencies are met
- Initiates post-processing when jobs complete
- Handles errors and resubmissions
- Generates reports and summaries
🌊 Case Study: Gravitational Waves
Proven at scale in production: Asimov was developed by and is actively used by the LIGO, Virgo, and KAGRA collaborations for gravitational wave parameter estimation.
Real-World Impact
- GWTC-2, GWTC-3, and GWTC-4.0 catalogs - Managed parameter estimation for 90+ gravitational wave events
- O3 & O4 observing runs - Continuous automated analysis of new detections
- Multiple pipelines - Unified interface for LALInference, Bilby, BayesWave, and RIFT
- Production-ready - Battle-tested on hundreds of analyses over multiple years
This production use demonstrates Asimov's capability to handle:
- Complex Bayesian inference (days to weeks per job)
- Hundreds of jobs per event (different parameter configurations)
- Catalog-scale workflows (90+ events × multiple pipelines)
- Critical scientific output (published in major collaboration papers)
See USERS.md for institutions and research groups using Asimov.
This is the beginning of the README. Read the rest at the original. Read the full README ↗
