go-judge
中文文档 | Documentation
Fast, Simple, Secure
Quick Start
Install & Run
Download compiled executable go-judge for your platform from Release and run.
Or, by docker
docker run -it --rm --privileged --shm-size=256m -p 5050:5050 --name=go-judge criyle/go-judge
REST API
A REST service to run program in restricted environment (Listening on localhost:5050 by default).
- POST /run execute program in the restricted environment
- GET /file list all cached file id to original name map
- POST /file prepare a file in the go judge (in memory), returns fileId (can be referenced in /run parameter)
- GET /file/:fileId downloads file from go judge (in memory), returns file content
- DELETE /file/:fileId delete file specified by fileId
- /ws WebSocket version for /run
- /stream WebSocket for stream run. Supports streaming interface
- GET /version gets build git version (e.g.
v1.9.0) together with runtime information (go version, os, platform)
- GET /config gets some configuration (e.g.
fileStorePath, runnerConfig) together with some supported features
REST API Interface
API Interface Structure Definition
Example Request & Response
Example Request & Response
Documentation
Prerequisite
- Linux Kernel Version >= 3.10
- Cgroup file system mounted at /sys/fs/cgroup. Usually done by systemd
Architecture
+----------------------------------------------------------------------------------+
| Transport Layer (HTTP / WebSocket / FFI / ...) |
+----------------------------------------------------------------------------------+
| Sandbox Worker (Environment Pool w/ Environment Builder ) |
+-----------------------------------------------------------+----------------------+
| EnvExec | File Store |
+--------------------+----------------+---------------------+---------------+------+
| Linux (go-sandbox) | Windows (winc) | macOS (app sandbox) | Shared Memory | Disk |
+--------------------+----------------+---------------------+---------------+------+
Configurations
Server:
- The default binding address for the go judge is
localhost:5050. Can be specified with -http-addr flag.
- By default gRPC endpoint is disabled, to enable gRPC endpoint, add
-enable-grpc flag.
- The default binding address for the gRPC go judge is
localhost:5051. Can be specified with -grpc-addr flag.
- The default log level is info, use
-silent to disable logs or use -release to enable release logger (auto turn on if in docker).
-auth-token to add token-based authentication to REST / gRPC- By default, the GO debug endpoints (
localhost:5052/debug) are disabled, to enable, specifies -enable-debug, and it also enables debug log
- By default, the prometheus metrics endpoints (
localhost:5052/metrics) are disabled, to enable, specifies -enable-metrics
- Monitoring HTTP endpoint is enabled if metrics / debug is enabled, the default addr is
localhost:5052 and can be specified by -monitor-addr
Sandbox:
- The default concurrency equal to number of CPU, Can be specified with
-parallelism flag.
-mount-conf specifies detailed mount configuration, please refer File System Mount as a reference (Linux only)-file-timeout specifies maximum TTL for file created in file store （e.g. 30m)- The default file store is in memory(
/dev/shm/), local cache can be specified with -dir flag.
-output-limit specifies size limit of POSIX rlimit of output (default 256MiB)-copy-out-limit specifies the default file copy out max (default 64MiB)
You can find more available configuration here.
Run Terminal in the Container
Download go-judge-shell from Release and run. It will connect local go-judge, and open an interactive shell in the container for debugging purpose.
Return Status
- Accepted: Program exited with status code 0 within time & memory limits
- Memory Limit Exceeded: Program uses more memory than memory limits
- Time Limit Exceeded: (
exitStatus usually have value 9 as killed by SIGKILL after timeout)
- Program uses more CPU time than cpuLimit
- Or, program uses more clock time than clockLimit
- Output Limit Exceeded:
- Program output more than pipeCollector limits
- Or, program output more than output-limit
- File Error:
- CopyIn file is not existed
- Or, CopyIn file too large for container file system
- Or, CopyOut file is not existed after program exited
- Non Zero Exit Status: Program exited with non 0 status code within time & memory limits
- Signalled: Program exited with signal (e.g.
SIGSEGV)
- Dangerous Syscall: Program denied by the default seccomp filter
- Internal Error:
- Program is not exist
- Or, container create not successful (e.g. not privileged docker)
- Or, other errors
Container Root Filesystem
For linux platform, the default mounts points are bind mounting host's /lib, /lib64, /usr, /bin, /etc/ld.so.cache, /etc/alternatives, /etc/fpc.cfg, /dev/null, /dev/urandom, /dev/random, /dev/zero, /dev/full and mounts tmpfs at /w, /tmp and creates /proc.
To customize mount points, please look at example mount.yaml file.
If mount.yaml is not specified, the size of tmpfs for /w and /tmp is configured through -tmp-fs-param with default value size=128m,nr_inodes=4k
If a file named /.env exists in the container rootfs, the container will load the file as environment variable line by line.
If a bind mount is specifying a target within the previous mounted one, please ensure the target exists in the previous mount point.
Metrics Monitoring Endpoint
Prometheus Metrics Monitoring Endpoint
Notice
[!WARNING]
Windows and macOS support are experimental and should not be used in production environments
cgroup usage
For cgroup v1, the go-judge need root privilege to create cgroup. Either creates sub-directory /sys/fs/cgroup/cpuacct/gojudge, /sys/fs/cgroup/memory/gojudge, /sys/fs/cgroup/pids/gojudge and make execution user readable or use sudo to run it.
For cgroup v2, systemd dbus will be used to create a transient scope for cgroup integration.
If no permission to create cgroup, the cgroup related limit will not be effective.
cgroup v2 support
The cgroup v2 is supported by go-judge now when running as root since more Linux distribution are enabling cgroup v2 by default (e.g. Ubuntu 21.10+, Fedora 31+). However, for kernel < 5.19, due to missing memory.max_usage_in_bytes in memory controller, the memory usage is now accounted by maxrss returned by wait4 syscall. Thus, the memory usage appears higher than those who uses cgroup v1. For kernel >= 5.19, memory.peak is being used.
When running in containers, the go-judge will migrate all processed into /api hierarchy to enable nesting support.
When running in Linux distributions powered by systemd, the go-judge will contact systemd via dbus to create a transient scope as cgroup root.
When running with kernel >= 5.7, the go-judge will try faster clone3(CLONE_INTO_CGROUP) and vfork method.
Memory Usage
The controller usually consumes 20M memory and each container usually consumes 20M + size of tmpfs 2 * 128M. For each request, it consumes as much as user program limit + extra limit (16k) + total copy out max. Notice that the cached file stores in the shared memory (/dev/shm) of the host, so please ensure enough size allocated.
For example, when concurrency = 4, the container itself can consume as much as 60 + (20+32) * 4M = 268M + 4 * total copy out + total max memory of requests.
Due to limitation of GO runtime, the memory will not return to OS automatically, which could lead to OOM killer. A background worker was introduced to checks heap usage and invokes GC when necessary.
-force-gc-target default 20m, the minimal size to trigger GC-force-gc-interval default 5s, the interval to check memory usage
