Mojibake

Mojibake is a small, fast, self-contained Unicode library written in C11. It gives C, C++ and Rust programs normalization, case mapping, segmentation, bidirectional text, collation, IDNA, emoji, confusable detection, and the full Unicode Character Database, all in one source file and one header, with zero dependencies. It is released under the MIT License.

Live demo and API reference · API.md · C++ wrapper · Rust wrapper · Changelog · Roadmap

Why Mojibake

• Drop-in. Add mojibake.c and mojibake.h to your project and compile. No build system, no generated files, no runtime data to ship.
• Complete. Implements the Unicode algorithms most programs need, targeting Unicode 18.0.0.
• Correct. Passes the official Unicode conformance test suites for every supported algorithm, with millions of assertions run on every change. See the conformance notes
• Encoding-agnostic. Reads and writes UTF-8, UTF-16LE/BE, and UTF-32LE/BE, with explicit policies for malformed input and full support for embedded U+0000.
• Embeddable. Custom allocators, caller-owned output buffers, and build-time switches to drop tables you do not need.
• Portable. Builds as C11 and C++17 on Linux, macOS, Windows, the BSDs, Haiku, Android, and WebAssembly.

Status

Mojibake covers the Unicode algorithms listed below and is close to feature-complete for its scope. It is still pre-1.0: the C API is stable in practice, but a minor release may make breaking changes, which are always listed in the changelog. Streaming APIs, configurable collators, and the remaining UTS #39 identifier checks are planned; see the roadmap.

Quick start

You don't need to install anything. There are two files (mojibake.c, mojibake.h) to add to your C/C++ project. Download it here mojibake-amalgamation-070.zip.

#include <stdio.h>

#include "mojibake.h"

int main(void) {
    mjb_result result;

    // "Cafe" + U+0301 COMBINING ACUTE ACCENT (6 bytes) -> "Café" with U+00E9 (5 bytes)
    mjb_text cafe = mjb_text_utf8("Cafe\xCC\x81", 6);

    if(mjb_normalize(&cafe, MJB_MALFORMED_STOP, MJB_NORMALIZATION_NFC, MJB_ENC_UTF_8, &result,
           NULL) == MJB_STATUS_OK) {
        printf("NFC: %.*s (%zu bytes)\n", (int)result.output_size, result.output,
            result.output_size);
        mjb_result_free(&result);
    }

    // One flag, two codepoints, eight bytes
    mjb_text flag = mjb_text_utf8("🇮🇹", MJB_NUL_TERMINATED);
    size_t codepoints = 0;
    size_t graphemes = 0;

    if(mjb_codepoint_count(&flag, MJB_MALFORMED_STOP, &codepoints, NULL) == MJB_STATUS_OK &&
        mjb_grapheme_count(&flag, MJB_MALFORMED_STOP, &graphemes, NULL) == MJB_STATUS_OK) {
        printf("🇮🇹: %zu codepoints, %zu grapheme\n", codepoints, graphemes);
    }

    // Caseless identifier matching: "Straße" -> "strasse"
    mjb_text street = mjb_text_utf8("Straße", MJB_NUL_TERMINATED);

    if(mjb_nfkc_casefold(&street, MJB_MALFORMED_STOP, MJB_ENC_UTF_8, &result, NULL) ==
        MJB_STATUS_OK) {
        printf("Straße -> %.*s\n", (int)result.output_size, result.output);
        mjb_result_free(&result);
    }

    return 0;
}

cc -std=c11 example.c mojibake.c -o example
./example

NFC: Café (5 bytes)
🇮🇹: 2 codepoints, 1 grapheme
Straße -> strasse

A few conventions apply across the whole API:

• Functions return mjb_status, and MJB_STATUS_OK is zero: always compare against it. Predicates such as mjb_is_utf8 return bool.
• Lengths are byte counts. Pass an exact length when the input may contain U+0000, or MJB_NUL_TERMINATED to scan for a terminator sized for the declared encoding.
• Allocating functions fill an mjb_result that you release with mjb_result_free. Their _into variants write into a buffer you own instead, and can measure the required size first.

API.md explains these rules in depth.

Every function can also be tried in the browser on the demo site, which runs the library compiled to WebAssembly.

Feature highlights

Text transformation

• Normalization: NFC/NFD/NFKC/NFKD (mjb_normalize, mjb_normalize_into), identifier-oriented NFKC case folding (mjb_nfkc_casefold, mjb_nfkc_casefold_into), plus a fast quick-check (mjb_normalization_quick_check) (UAX #15, Unicode 18.0.0)
• Case conversion: uppercase, lowercase, titlecase, and case folding with full special-casing and conditional mappings (mjb_map_case, mjb_map_case_into)
• Internationalized domain names: strict nontransitional ToASCII and ToUnicode processing, including Punycode and validation (mjb_idna_to_ascii, mjb_idna_to_unicode) (UTS #46)
• Filtering: strip controls, spaces, or numeric characters while normalizing (mjb_filter, mjb_filter_into)

Text analysis

여기까지가 README 앞부분입니다. 나머지는 원문에서 읽을 수 있습니다. README 전문 보기 ↗