evalscope

⭐ If you like this project, please click the "Star" button in the upper right corner to support us. Your support is our motivation to move forward!

📝 Introduction

EvalScope is a one-stop LLM evaluation framework built by the ModelScope Community. Just one command to start — it supports model capability evaluation, inference performance stress testing, and result visualization.

pip install evalscope
evalscope eval --model your-model-name --api-url $OPENAI_API_BASE_URL --api-key $OPENAI_API_KEY --eval-type openai_api --datasets gsm8k --limit 5

✨ Key Features

• 📚 Comprehensive Evaluation Benchmarks: Built-in multiple industry-recognized evaluation benchmarks including MMLU, C-Eval, GSM8K, and more.
• 🧩 Multi-modal and Multi-domain Support: Supports evaluation of various model types including Large Language Models (LLM), Vision Language Models (VLM), Embedding, Reranker, AIGC, and more.
• 🧭 System One Evaluation: Evaluate text Choice decision models via --eval-type systemone_api, with task instructions, few-shot examples and structured probabilities/confidence. See the usage guide.
• 🚀 Multi-backend Integration: Seamlessly integrates multiple evaluation backends including OpenCompass, VLMEvalKit, RAGEval to meet different evaluation needs.
• 🤖 Agent Evaluation Mode: Drives benchmarks (e.g. GSM8K, AIME, SWE-bench Agentic) inside a controlled multi-turn AgentLoop with pluggable strategies and tools, plus Docker or E2B remote sandboxes; full per-sample Agent Trace is recorded and visualizable.
• ⚡ Inference Performance Testing: Provides powerful model service stress testing tools, supporting multiple performance metrics such as TTFT, TPOT.
• 📊 Interactive Reports: Provides a Web Dashboard for multi-dimensional model comparison, report overview and detailed inspection.
• ⚔️ Arena Mode: Supports multi-model battles (Pairwise Battle), intuitively ranking and evaluating models.
• 🔧 Highly Extensible: Developers can easily add custom datasets, models and evaluation metrics.

📊 Visualization Preview

EvalScope provides an interactive Web Dashboard for multi-dimensional model comparison and in-depth analysis.

For details, please refer to 📖 Visualizing Evaluation Results.

This is the beginning of the README. Read the rest at the original. Read the full README ↗