Sched_ext Schedulers and Tools

sched_ext is a Linux kernel feature which enables implementing kernel thread schedulers in BPF and dynamically loading them. This repository contains various scheduler implementations and support utilities.

sched_ext enables safe and rapid iterations of scheduler implementations, thus radically widening the scope of scheduling strategies that can be experimented with and deployed; even in massive and complex production environments.

You can find more information, links to blog posts and recordings, in the wiki. The following are a few highlights of this repository.

• The scx_layered case study concretely demonstrates the power and benefits of sched_ext.
• For a high-level but thorough overview of the sched_ext (especially its motivation), please refer to the overview document.
• For a description of the schedulers shipped with this tree, please refer to the schedulers document.
• For writing schedulers in Rust BPF, see the scx_rustsched framework.

sched_ext is supported by the upstream kernel starting from version 6.12. Both Meta and Google are fully committed to sched_ext and Meta is in the process of mass production deployment. See #kernel-feature-status for more details.

In all example shell commands, $SCX refers to the root of this repository.

Getting Started

All that's necessary for running sched_ext schedulers is a kernel with sched_ext support and the scheduler binaries along with the libraries they depend on. Switching to a sched_ext scheduler is as simple as running a sched_ext binary:

root@test ~# cat /sys/kernel/sched_ext/state /sys/kernel/sched_ext/*/ops 2>/dev/null
disabled
root@test ~# scx_simple
local=1 global=0
local=74 global=15
local=78 global=32
local=82 global=42
local=86 global=54
^Zfish: Job 1, 'scx_simple' has stopped
root@test ~# cat /sys/kernel/sched_ext/state /sys/kernel/sched_ext/*/ops 2>/dev/null
enabled
simple
root@test ~# fg
Send job 1 (scx_simple) to foreground
local=635 global=179
local=696 global=192
^CEXIT: BPF scheduler unregistered

scx_simple is a very simple global vtime scheduler which can behave acceptably on CPUs with a simple topology (single socket and single L3 cache domain).

Above, we switch the whole system to use scx_simple by running the binary, suspend it with ctrl-z to confirm that it's loaded, and then switch back to the kernel default scheduler by terminating the process with ctrl-c. For scx_simple, suspending the scheduler process doesn't affect scheduling behavior because all that the userspace component does is print statistics. This doesn't hold for all schedulers.

Note: C schedulers like scx_simple were previously included in this repository but have since been moved to scx-c-examples. The schedulers in this repository now use Rust for userspace components.

In addition to terminating the program, there are two more ways to disable a sched_ext scheduler - sysrq-S and the watchdog timer. Ignoring kernel bugs, the worst damage a sched_ext scheduler can do to a system is starving some threads until the watchdog timer triggers.

As illustrated, once the kernel and binaries are in place, using sched_ext schedulers is straightforward and safe. While developing and building schedulers in this repository isn't complicated either, sched_ext makes use of many new BPF features, some of which require build tools which are newer than what many distros are currently shipping. This should become less of an issue in the future. For the time being, the following custom repositories are provided for select distros.

Install Instructions by Distro

• Ubuntu
• Arch Linux
• Gentoo Linux
• Fedora
• Nix
• openSUSE Tumbleweed

Repository Structure

scx
|-- scheds               : Sched_ext scheduler implementations
|   |-- include          : Shared BPF and user C include files including vmlinux.h
|   \-- rust             : Example schedulers - userspace code written Rust
\-- rust                 : Rust support code
    \-- scx_utils        : Common utility library for Rust schedulers

Build & Install

Rust schedulers : use cargo.

Dependencies:

• clang: >=18 required, >=22 for cid-form schedulers
• libbpf: >=1.2.2 required, >=1.3 recommended
• bpftool: Usually available in linux-tools-common or similar packages
• libelf, libz, libzstd: For linking against libbpf
• pkg-config: For finding system libraries
• Rust toolchain: >=1.91

The kernel has to be built with the following configuration:

• CONFIG_BPF=y
• CONFIG_BPF_SYSCALL=y
• CONFIG_BPF_JIT=y
• CONFIG_DEBUG_INFO_BTF=y
• CONFIG_BPF_JIT_ALWAYS_ON=y
• CONFIG_BPF_JIT_DEFAULT_ON=y
• CONFIG_SCHED_CLASS_EXT=y
• CONFIG_KALLSYMS_ALL=y (needed by schedulers that read kernel variables such as sd_llc_id or runqueues; without it they fail to load with ldimm64 failed to find the address for kernel symbol)

The scx/kernel.config file includes all required and other recommended options for using sched_ext. You can append its contents to your kernel .config file to enable the necessary features.

Building and Installing

Rust Schedulers

$ cd $SCX
$ cargo build --release                 # Build all Rust schedulers
$ cargo build --release -p scx_rusty    # Build specific scheduler

Rust schedulers are also published on crates.io:

This is the beginning of the README. Read the rest at the original. Read the full README ↗