OpenCLIP

[Paper] [Citations] [Clip Colab] [Coca Colab]

Versions

• 4.x is the current release line, developed on main. 4.2.1 is the first 4.x release on PyPI; earlier 4.x version numbers were development versions only.
• 3.x remains available on the v3 branch. To stay on it, run pip install 'open_clip_torch<4'. Further 3.3.x releases will only be made if needed.
• Upgrading from 3.x: pretrained image/text inference is intended to remain compatible, but training scripts and downstream integrations need review. See the migration guide.

Welcome to an open source implementation of OpenAI's CLIP (Contrastive Language-Image Pre-training). OpenCLIP also includes language-audio (CLAP) models and related generative models.

Using this codebase, we have trained several models on a variety of data sources and compute budgets, ranging from small-scale experiments to larger runs including models trained on datasets such as LAION-400M, LAION-2B and DataComp-1B. Many of our models and their scaling properties are studied in detail in the paper reproducible scaling laws for contrastive language-image learning. Some of the best models we've trained and their zero-shot ImageNet-1k accuracy are shown below, along with the ViT-L model trained by OpenAI and other state-of-the-art open source alternatives (all can be loaded via OpenCLIP). We provide more details about our full collection of pretrained models here, and zero-shot results for 38 datasets here. Experimental helper scripts for multilabel zero-shot evaluation (scripts/multilabel_zeroshot.py) and linear probes (scripts/linear_probe.py) are also available.

Model	Training data	Resolution	# of samples seen	ImageNet zero-shot acc.
ConvNext-Base	LAION-2B	256px	13B	71.5%
ConvNext-Large	LAION-2B	320px	29B	76.9%
ConvNext-XXLarge	LAION-2B	256px	34B	79.5%
ViT-B-32-256	DataComp-1B	256px	34B	72.8%
ViT-B-16	DataComp-1B	224px	13B	73.5%
ViT-L-14	LAION-2B	224px	32B	75.3%
ViT-H-14	LAION-2B	224px	32B	78.0%
ViT-L-14	DataComp-1B	224px	13B	79.2%
ViT-bigG-14	LAION-2B	224px	34B	80.1%
ViT-L-14-quickgelu (Original CLIP)	WIT	224px	13B	75.5%
ViT-SO400M-14-SigLIP (SigLIP)	WebLI	224px	45B	82.0%
ViT-L-14 (DFN)	DFN-2B	224px	39B	82.2%
ViT-L-16-256 (SigLIP2)	WebLI (multi-lang)	256px	40B	82.5%
ViT-SO400M-14-SigLIP-384 (SigLIP)	WebLI	384px	45B	83.1%
ViT-H-14-quickgelu (DFN)	DFN-5B	224px	39B	83.4%
PE-Core-L-14-336

Model cards with additional model specific details can be found on the Hugging Face Hub under the OpenCLIP library tag: https://huggingface.co/models?library=open_clip.

If you found this repository useful, please consider citing. We welcome anyone to submit an issue or send an email if you have any other requests or suggestions.

Note that portions of src/open_clip/ modelling and tokenizer code are adaptations of OpenAI's official repository.

What's new in 4.x

• CLAP 2 (NaFlex CLAP): new pretrained audio-text models that take variable-length audio
• NaFlex CLIP: variable-resolution/aspect image towers (timm naflexvit) with token-budget batching (--use-naflex, naflex_* configs)
• NaFlex GenLIP / GenLAP: generative image/audio captioning with prefix-LM attention (naflexgenlip_*, naflexgenlap_* training configs, no pretrained weights yet)
• Modern text towers: RoPE, SwiGLU/ReLU², and RMSNorm options (moderntext-* configs), per-batch variable-length text, and Hugging Face ModernBERT towers
• MaMMUT and CoCa v2: MaMMUT (mammut_*, mammut2_*) with the released LAION weights, and corrected CoCa configs (coca2_*)
• Training stack: TrainingTask wrappers, dict-based batches, FSDP2 (--fsdp), torch.compile strategies, and length bucketing
• Safer checkpoint loading: PyTorch 2.6+ is required, all checkpoint loads use weights_only=True, and OpenAI weights now load from the Hugging Face Hub instead of JIT archives

This is the beginning of the README. Read the rest at the original. Read the full README ↗