sol-tts

A daily podcast pipeline. It reads the opinion columns published at haber.sol.org.tr/kose-yazilari, narrates them in Turkish with ElevenLabs and Google Cloud TTS, renders each one as a video, publishes it to YouTube in a playlist for that day, and archives the audio and transcript to MinIO.

It runs unattended, once every morning.

scrape → normalise → TTS → MP4 → YouTube → MinIO → prune → EMA comparison copy

Every column keeps its author's byline, the publication date, a note that the narration is synthetic, and a link back to the original article.

---

How it works

Stage	Script	What it does
scrape	sol_column_scraper.py	Reads the column index, fetches anything new, stores the transcript verbatim
TTS	tts_converter.py	ElevenLabs while the monthly budget lasts, then Google Chirp 3: HD (two Turkish voices in rotation)
render	mp3_to_video.py	Muxes the audio with a generated title card into an H.264/AAC MP4
publish	youtube_uploader.py	Resumable upload, then adds the episode to that day's playlist
archive	sol_storage.py	Audio, transcript and metadata to MinIO (S3-compatible)
prune	podcast.py prune	Deletes local media after the retention window
compare	ema_compare.py	Re-voices ElevenLabs columns with EMA Lightning, offline; never published
orchestration	run_pipeline.py	Runs the stages in order

Supporting modules: sol_state.py (the ledger), text_normalize.py and tr_readings.py (Turkish text preparation), youtube_meta.py (titles and descriptions), podcast.py (inspection and manual repair).

scraper.py, scrape_sol_fulltext.py and generate_website.py predate the podcast and target the news feed rather than the columns. Only the website generator is still wired in, as a by-product.

---

The ledger

state.json is the single source of truth, keyed by the numeric id at the end of every column URL (.../turkiyenin-rejimi-nedir-413294 → 413294). Each entry carries five stage stamps:

scraped → tts → video → uploaded → stored

Every stage skips what it has already done, so re-running is always safe: nothing is narrated twice (which would cost money), and nothing is published twice (which cannot be undone quietly). Filenames deliberately contain no run timestamp, so a repeated run overwrites rather than accumulates.

The ledger belongs to the server and is not in this repository. A checkout with a stale copy would re-upload the whole archive.

---

Turkish text preparation

Chirp 3 accepts no SSML, so every pronunciation fix has to be a plain-text rewrite. Each rule below was added because the pattern was measured in the corpus, not because it might occur.

Problem	Example	Rewritten as
Apostrophe before a case suffix	Cumhuriyet'in	Cumhuriyetin
Initialisms	TKP, ABD	Te Ka Pe, A Be De
Numbers carrying a suffix	1920'de	bin dokuz yüz yirmide
Ordinals	104. madde	yüz dördüncü madde
Roman numerals	II.	İkinci
Percent, decimals, thousands	%50, 2,5'i, 1.000'e	yüzde 50, iki virgül beşi, bine
Invisible characters, curly quotes, dashes	U+2060, …, —	removed or simplified
Clock times	14.30'da	on dört otuzda
Ranges (ascending, no leading zero)	2019-2023, 10–15 Ekim	2019 ila 2023, 10 ila 15 Ekim
Bare separators	1.250.000, 2,5 milyar	bir milyon iki yüz elli bin, iki buçuk milyar
Currency	500 TL'lik, $5, 10 milyon €	500 liralık, 5 dolar, 10 milyon avro
Fractions (denominator ≤ 10)	3/4'ü, 1/2 asır	dörtte üçü, yarım asır
Reference marks	Bkz. s. 45, M.Ö.	Bakınız sayfa 45, Milattan önce
Links and handles	https://sol.org.tr/..., @x, &	sol nokta org nokta tr, x,

The rows above Clock times were measured in the corpus; the rest were added from known Turkish conventions and are covered by tests/test_readings.py (python -m unittest discover tests).

ElevenLabs layer

eleven_prep.py runs the shared rules and then prepares the text for how ElevenLabs delivers it, which differs from Chirp 3:

• every line ends in punctuation, so headings are not read as unfinished;
• <break> pauses between paragraphs, longer after the title and byline, at most four per request (more destabilises the model; v3 gets none);
• requests of about 2,000 characters, cut on sentences, each sent with previous_text, next_text and previous_request_ids so tone carries over the seams, plus a fixed seed;
• its own initialism table, ELEVEN_SPELLED, empty until an A/B listen (python compare_voices.py --eleven-initialisms --yes) says otherwise.

The monthly budget counts what ElevenLabs bills - its prepared text, break tags included. python eleven_prep.py <txt> shows the requests it would send.

Two details worth knowing:

• Turkish separators are the reverse of English. The dot groups thousands and the comma marks the decimal, so 1.000 is bin, not bir virgül sıfır.
• Unknown initialisms are left alone and reported, never guessed. Spelling out an acronym that is read as a word (NATO, DİSK) makes it worse, and the letters alone do not tell you which kind you have. New ones are added to the table by hand.

Normalisation happens at synthesis time, not at scrape time. Rewriting the stored transcript would destroy the original tokens and no later correction to the tables could ever reach them.

---

EMA Lightning comparison

EMA Lightning is a small Turkish-only TTS model that runs offline on the CPU, at roughly six times real time. It costs nothing per character, so it is a candidate to replace the paid voice - but only once it has been heard on the same material.

This is the beginning of the README. Read the rest at the original. Read the full README ↗