orthography2ipa

A library for measuring how languages relate to each other, and for turning their spelling into IPA on the way there.

Two languages can be close on one axis and far apart on another. Galician and Portuguese sound nearly the same and are written differently. Galician and Castilian sound different and are written alike. A single "language similarity" number hides that. orthography2ipa keeps the axes apart and measures each one on its own:

Axis	Question	API
Phonological	Do they use the same sounds?	inventory_distance, allophone_overlap, phonological_distance
Reading	Given the same text, do they sound alike?	grapheme_divergence
Spelling	Given the same sound, do they write it alike?	spelling_divergence
Script	Are the writing systems typologically alike?	script_distance, orthographic_distance
Genealogical	Do they descend from the same ancestors?	ancestry_similarity, ancestry_chain
Temporal	Were they spoken at the same time?	timespan (decays ancestry weights)
Geographic	Are they spoken in the same place?	geographic_distance

import orthography2ipa as o2i
from orthography2ipa.distance import grapheme_divergence, spelling_divergence

gl = o2i.get("gl")                  # Galician, RAG norm: writes /ɲ/ as ⟨ñ⟩, as Castilian does
glr = o2i.get("gl-x-reintegrado")   # Galician, reintegrationist norm: as ⟨nh⟩, as Portuguese does

grapheme_divergence(gl, glr).mean_ipa_distance   # 0.0233: reading: same text, same sounds
spelling_divergence(gl, glr).mean_distance       # 0.0659: spelling: same sounds, written differently

The two norms are the same language with the same phonology, and only the spelling axis can see what separates them. That is the library: the axes are independent, so they are measured independently.

The phonology those axes compare comes from a per-language data spec (graphemes, allophones, positional rules, ancestry, sources), and the same specs drive a grapheme-to-IPA engine:

>>> import orthography2ipa
>>> orthography2ipa.transcribe("olá mundo", "pt")
'oˈla ˈmũdu'

Languages and classification-only clade nodes ship with the package as spec files in orthography2ipa/data/, spread across top-level families from Indo-European to Quechuan, plus reconstructed proto-languages, regional dialects and creoles. The count grows over time, so check it live instead of trusting a number in a doc:

len(orthography2ipa.available_codes())                      # languages only
len(orthography2ipa.available_codes(include_clades=True))   # languages + clade nodes

New to the library? docs/index.md routes you by what you are trying to do, and states the known accuracy limits up front.

The engine is data, not weights

Every language is a JSON file describing which graphemes map to which IPA phonemes and how those phonemes surface in context. A shared, language-agnostic engine (tokenizer, beam search, allophone rules, stress, sandhi) turns that data into transcriptions. There are no trained weights to ship, and adding a language means writing cited data, not training a model.

Sounds are not a property of spelling

A spec states its phonemes (the inventory) directly, independently of its graphemes. Most of the world's languages are unwritten or barely written, and a logographic script encodes no sound at all, so reading the inventory out of the spelling cannot work for either. Proto-Indo-European declares its 30 reconstructed phonemes instead of faking an identity orthography to have an inventory. zh-Hani (Chinese in Han script) has no grapheme map at all (there is no rule to write), and a complete phonology regardless.

import orthography2ipa

han = orthography2ipa.get("zh-Hani")
han.graphemes     # {}: honest, not a gap: a Han character encodes no sound
len(han.phonemes) # 60: the phonology is known anyway

When phonemes is absent the inventory is derived from graphemes, so nothing changes for a spec that says nothing: but that derivation is a fallback, not the definition. The invariant is: a spec declares graphemes or a phoneme inventory, and may not be silent about both.

A spec also states its orthography_kind: native, romanization or transliteration: because Han characters, Pinyin and Buckwalter ASCII make three different claims. zh is a romanization: it reads Pinyin (ISO 7098), not Hanzi, and Pinyin's being a plain alphabet is exactly why it is rule-transcribable when the native script is not. ar-Latn-buckwalter is a transliteration: a machine re-encoding of the Arabic script with no standards body, lossless against it, and inheriting the abjad limit it cannot escape: re-encoding does not recover vowels the script never wrote. See docs/orthography_kind.md.

Two maps, kept apart

• A grapheme map tells you which phonemes a spelling can represent. English ⟨th⟩ → ['θ', 'ð'].
• An allophone map tells you how a phoneme surfaces in context. English /t/ → ['t', 'tʰ', 'ʔ', 'ɾ'].

The pre-lexical map (orthography → phoneme) is conditioned by positional_graphemes, vowel-class positions and candidate weights. The post-lexical map (phoneme → surface allophone) is applied by declarative allophone_rules: context-conditioned rewrites keyed on syllable position, stress, word position and neighbouring segments. They compile into a lattice rescorer that runs after phoneme selection and before stress and sandhi. Catalan uses them for word-final obstruent devoicing (fred → [ˈfɾet]) and nasal place assimilation (banc → [ˈbaŋk]). See docs/allophony.md.

The pronunciation lattice

When a spelling is ambiguous the engine does not just pick a guess: it builds a ranked, cost-annotated lattice for the word, and that lattice is what specialised phonemizers stand on. For any registered language you get, per grapheme, every IPA option it can realise and a -log P cost ranking them:

from orthography2ipa import get
from orthography2ipa.phonetok import PhonetokTokenizer

tok = PhonetokTokenizer(get("en-GB"))
for slot in tok.ipa_lattice("cough"):
    print(slot.grapheme, [(c.ipa, c.cost) for c in slot.candidates])
    # c    → [('k', 0.0), ('s', 1.0)]
    # ough → [('ɔː', 0.0), ('oʊ', 1.0), ('ʌf', 2.0), ('ɒf', 3.0), ('aʊ', 4.0), ('uː', 5.0)]

It tells you when it is unsure. A per-word confidence: each slot's top-1 vs top-2 cost margin folded with grapheme coverage: flags which words a downstream engine should spend its expensive lexicon or rules on, and which it can take on trust:

from orthography2ipa import G2P

g = G2P("en-GB")
g.word_confidence("bar")     # 1.0: one reading, take it
g.word_confidence("cough")   # 0.6321: ambiguous, look closer
g.word_confidence("bar你")   # 0.75: one OOV grapheme drops coverage to 3/4

A downstream engine refines the lattice with a rescorer. A rule is a LatticeRescorer: a pure function over one slot and its context that re-costs candidates. A candidate wins by being made strictly cheapest. Here is a complete mini-phonemizer that teaches English ⟨ough⟩ its /ʌf/ reading (rough, tough, enough):

from orthography2ipa import get
from orthography2ipa.phonetok import PhonetokTokenizer, Candidate
from orthography2ipa.rescorer import LatticeRescorer

class RoughOugh(LatticeRescorer):
    def rescore(self, slot, context):
        if slot.grapheme != "ough":
            return slot.candidates                    # no-op elsewhere
        others = [c for c in slot.candidates if c.ipa != "ʌf"]
        return [Candidate("ʌf", 0.0)] + [Candidate(c.ipa, c.cost + 1.0)
                                         for c in others]

tok = PhonetokTokenizer(get("en-GB"))
tok.ipa_best("tough")                        # 'tɔː': generic beam
tok.ipa_best("tough", rescorer=RoughOugh())  # 'tʌf': refined

The beam is the universal fallback: it produces a defensible transcription for every word in every registered language. Refining it with a rescorer is one of two supported ways to build a specialised engine on this library. Keeping a bespoke tokenizer is the other. arbtok, tugaphone, g2p_barranquenho and mwl_phonemizer build on this library. See Refine or fork? for the trade-offs.

Because the library ships no trained weights it is also a clean feature provider for an ML G2P: G2P.features(text) exposes, per grapheme, the phonological-class predicates, word-local neighbours, the ranked (ipa, cost) lattice and the confidence signal as flat, JSON-able dicts: so a CRF trains on linguistically-grounded features and plugs back in as a LatticeRescorer, spending its learned capacity where confidence says the base engine is unsure. See Features for ML / CRF G2P.

Families are nodes, not labels

There is no authored family string. A family is a clade node in the ancestry graph, a spec with "clade": true, carrying classification and nothing else, and spec.family is derived by walking the parent chain up through the clades it passes:

import orthography2ipa

orthography2ipa.get("pt-BR").family_path   # ('Indo-European', 'Italic', 'Romance', 'Ibero-Romance')
orthography2ipa.get("pt-BR").family        # 'Indo-European > Italic > Romance > Ibero-Romance'

Because it is a chain and not a fixed-depth label, a consumer can ask at any depth: --family Romance and --family Ibero-Romance both match pt-BR. Clade nodes are classification-only: never a data source, never inherited from, and excluded from available_codes() by default. An explicit family string stays accepted for groupings that are not genetic clades: creoles, constructed languages, isolates, unclassified languages. See docs/ancestry.md.

Installation

pip install orthography2ipa

For richer language-specific pipelines, install a downstream engine built on this library: arbtok for Arabic, tugaphone for Portuguese.

Quick start

import orthography2ipa

orthography2ipa.transcribe("olá mundo", "pt")     # 'oˈla ˈmũdu'
orthography2ipa.transcribe("hello world", "en")   # 'hɛlɒ wɔːɹld'
orthography2ipa.transcribe("bona nuèit", "oc")    # 'ˈbunɔ ˈnyɛjt'

# Beam search keeps ranked alternatives per word
from orthography2ipa import G2P

engine = G2P("pt-PT")
result = engine.transcribe_detailed("um café", search="beam", beam_width=4)
result.ipa                          # 'ˈũ kɐˈfɛ'
result.words[1].candidates          # ranked IPAPath alternatives

The pipeline is: normalize → tokenize → greedy/beam per word → allophone rules → stress marks (when the spec declares stress rules) → sandhi → dialect transform.

Language specs

import orthography2ipa

en = orthography2ipa.get("en-GB")
en.graphemes["th"]    # ['θ', 'ð']: grapheme → phoneme candidates
en.allophones["t"]    # ['t', 'tʰ', 'ʔ', 'ɾ']: phoneme → surface realisations
en.name               # 'British English (RP)'
en.script             # 'Latin'
en.family             # 'Indo-European > Germanic > Northwest Germanic > West Germanic'

# Regional variants share ancestry but diverge where pronunciation does
orthography2ipa.get("pt-BR").graphemes["t"]   # ['t', 't͡ʃ']: palatalisation before /i/

# Bare tags, ISO 639-3 aliases and near matches all resolve
orthography2ipa.get("eng").name    # 'British English (RP)'
orthography2ipa.resolve("pt")      # 'pt-PT': reference variety
orthography2ipa.resolve("en-NZ")   # 'en-GB': nearest registered

# Discover what's available
orthography2ipa.available_codes()      # language codes (clades excluded)
orthography2ipa.available_families()   # codes grouped by derived family path

Distance metrics

import orthography2ipa as o2i
from orthography2ipa.distance import (phonological_distance, spelling_divergence,
                                      geographic_distance, ancestry_similarity)

pt, es = o2i.get("pt-PT"), o2i.get("es-ES")

d = phonological_distance(pt, es)
d.combined                              # 0.2062: sound-level distance: inventory
                                        #          and allophony, never orthography
d.inventory.feature_mean                # phoneme-inventory distance
d.allophone_sim                         # allophone overlap
d.grapheme.mean_ipa_distance            # 0.1102: reading divergence, reported here
                                        #          but NOT part of `combined`

spelling_divergence(pt, es).mean_distance      # 0.2068: writing divergence
ancestry_similarity(pt, es)                    # 0.5304: shared-ancestor weight
geographic_distance(pt, es, normalize=False)   # 377.8: km between the two points

Every axis, with its caveats, is in docs/distance.md.

Command-line interface

Every subcommand accepts --json for machine-readable output.

# List languages and families
orthography2ipa list
orthography2ipa list --families
orthography2ipa list --family Romance          # any step of the path matches
orthography2ipa list --family Ibero-Romance    # ... including the sub-branch

# Inspect a language
orthography2ipa info pt-BR
orthography2ipa info pt-BR --graphemes
orthography2ipa info pt-BR --json

# Transcribe text to IPA
orthography2ipa transcribe pt "olá mundo"
orthography2ipa transcribe en-GB "through" --search beam --beam-width 8

# Distance between two languages
orthography2ipa distance pt-BR pt-PT
orthography2ipa distance es-ES it-IT --json

What each language carries

Every LanguageSpec provides:

1. Graphemes: orthographic units (characters, digraphs, trigraphs) mapped to canonical IPA phonemes, and an orthography_kind saying whether they are the native script, a romanization or a transliteration.
2. Phonemes: the inventory, stated directly, or derived from the graphemes only as a fallback.
3. Allophones: each phoneme mapped to its positional/contextual surface realisations.
4. Positional graphemes: context-sensitive overrides (word-initial, intervocalic, before a front vowel, …).
5. Allophone rules: post-lexical phoneme → surface rewrites (final devoicing, assimilation, reduction, flapping).
6. Ancestry: weighted multi-ancestor lineage (parent, substrate, superstrate, adstrate, …), through which family is derived.
7. Sandhi rules: cross-word phonological processes.
8. Stress rules, tone inventory and word exceptions, where applicable.
9. Orthography standard: the official published spelling norm, when the language has one.
10. Location and timespan: where and when the variety is spoken.
11. Provenance: QualityTier (stub → skeleton → research → production), ScriptType, bibliographic sources, wikipedia / urls, and cross-reference identifiers (glottolog_code, wikidata_qid, phoible_id, wals_code, iso639_3).

Regional varieties get their own LanguageSpec linked through ancestry, and graphemes_base / allophones_base inheritance lets a dialect declare only what differs from its parent. The full field list is in docs/data_model.md. The authoring reference is orthography2ipa/data/SCHEMA.md.

Benchmarks

The engine is evaluated against the best gold sets available: the Portal da Língua Portuguesa lexicon, WikiPron, Infopédia, CMUdict, the Mirandese gold set, 4CatAc and others. The published scoreboard scores the entire gold set of every registered dataset/language pair, with no cap, so each row's N is the real number of covered gold words, not a sample size. Reproduce any row with python scripts/benchmark.py --scoreboard.

Take every number with a grain of salt. Reliable G2P gold barely exists, so most datasets are semi-automated, dictionary-extracted, community-scraped, or a phonemizer's own output reused as a reference. Read PER as directional, not precise. Every dataset carries a reliability tier in the scoreboard's provenance column, and one of those tiers deserves naming here:

Competitor-derived gold measures agreement, not correctness. The espeak-derived rows (styletts2_phonemes, ipa_babylm, and the phonemizer-phonemized ipa_childes languages) are espeak-ng output, and the epitran-derived rows (the epitran-phonemized ipa_childes languages) are epitran output: both are competitors this library benchmarks against in comparison.md. Diverging from them can mean we are right and they are wrong, which shows up in those rows as a worse score. Such a row can therefore neither qualify a language for a production promotion nor block one. It is breadth, and a directional signal, and nothing more.

llm-generated gold has no error model at all. An LLM has no lexicon and no rules, so its IPA can be confidently wrong with no systematic structure and a disagreement cannot be attributed to anything. It certifies nothing and diagnoses nothing.

Contextual (positional) scoring is still limited, no language sits at the production quality tier, and PER is genuinely mediocre for several languages: see Honest limitations before depending on this for anything accuracy-sensitive. The reliability taxonomy is in docs/benchmarks.md, the per-dataset evidence in docs/benchmark_datasets.md, and the methodology in docs/benchmark_methodology.md.

Candidate ordering defaults to rank order (most common pronunciation first). A spec can attach per-candidate weights (candidate frequencies from cited corpora), so the beam favours the corpus-dominant phoneme and a path's score becomes a real log-probability. See candidate scoring.

Comparison to other G2P systems

scripts/compare_systems.py runs the same gold rows through orthography2ipa, espeak-ng, epitran and gruut with identical normalization and scoring, and commits the result to docs/comparison.md (benchmarks/comparison.json for the machine-readable form). It is an honest table: some languages win against espeak-ng, some lose, and coverage against epitran/gruut is partial by nature of those projects' own language lists. No row is cherry-picked.

Design principles

• Independent axes: phonology, reading, spelling, script, ancestry, time and geography are measured separately, because languages relate along each of them differently.
• Linguistically motivated only: digraphs like English ⟨th⟩, Portuguese ⟨lh⟩ or German ⟨sch⟩ are included because they are standard orthographic units. Arbitrary substrings are not.
• Graphemes ≠ allophones: spelling-to-phoneme and phoneme-to-surface are modelled separately.
• Classification is a graph: families are clade nodes on the ancestry chain, not labels on a spec.
• Pure data, self-contained logic: mappings are declarative JSON, and the engine never loads external G2P implementations.
• Honest numbers: every benchmark row states where its gold came from and how much it can carry.

Building engines on top

G2PPlugin and WordContext are the base types for richer language-specific engines built on this library: arbtok (Arabic: contextual rule cascade + tashkeel diacritization) and tugaphone (Portuguese: lexicon, POS and regional-accent layers). They consume the spec data, tokenizer and stress machinery and own their own pipelines. Whether such an engine expresses its rules as rescorers over the shared lattice or keeps a bespoke tokenizer is a design choice covered in Refine or fork?.

Component plugins that slot into the bundled engine's own logic use dedicated entry-point groups: per-language syllabifiers register under orthography2ipa.syllabify (e.g. silabificador for Portuguese) and are honoured by stress detection automatically.

AI transparency

Most language specs in this repository were written by large language models (Anthropic Claude), directed and reviewed by a human maintainer. This section states how that works and why the project accepts it.

Why. The registry covers more than 900 language codes. No individual speaks them, and no small project can recruit a native-speaker linguist for each one. The real alternative to an AI-written spec was never an expert-written spec. It was no spec at all.

How. A spec is not free-form model output. Every mapping must trace to a published description: a reference grammar, a peer-reviewed paper, or a university dataset. Wikipedia is a reading path, not a source, and is cited next to the reference it points to.

The model reads the sources and encodes them as declarative JSON under the authoring rules in docs/adding_a_language.md, which forbid invented graphemes, enumerated context, and uncited claims. The engine stays language-agnostic, so a spec cannot hide logic. Conformance tests then check the structure, and the benchmarks score the output against external gold data that the model never wrote.

What this does not claim. A cited, AI-written spec is still not native-speaker validation. The per-spec QualityTier records exactly that, and no language currently sits at the production tier. If you speak one of these languages and a spec is wrong, that is a bug: please open an issue, ideally with a citation or a native-speaker judgment. Corrections outrank everything else in this repository.

Contributing

To add a language, create orthography2ipa/data/{code}.json following orthography2ipa/data/SCHEMA.md and the walkthrough in docs/adding_a_language.md. For dialects, use graphemes_base / allophones_base to inherit from the parent.

License

Apache 2.0