Tanya Data

A discovery layer for Malaysian official data. A user asks a question in plain words (English, Bahasa Malaysia or both) and the platform points to the right official dataset, report, dashboard or live feed, then sends them to the original source.

The platform holds metadata only. It never copies, generates or displays data values. Every result links to the agency's own page.

Pipeline

harvest/harvest.py       datagovmy-meta, BNM, MOH, NAPIC, docs, ...  ->  registry/registry.json
harvest/verify_links.py  checks every page and file URL    ->  registry/registry.json, registry/link_report.md
harvest/build_app.py     registry + app template           ->  dist/index.html

Python 3 standard library plus curl (BNM and ElectionData reject Python's TLS handshake). Run in this order:

python3 harvest/harvest.py          # clones datagovmy-meta and the MOH repos into .cache/ on first run
python3 harvest/verify_links.py     # about 1,400 URLs; results cached 12 hours in .cache/
python3 harvest/build_app.py

What is indexed

Kind	Source	Records
Dataset	data.gov.my / OpenDOSM / KKMNOW catalogue (data-catalogue/)	297
Dataset	Bank Negara Malaysia Open API: one record per statistical table, from BNM's own specification (harvest/bnm.py)	341
Dataset	Ministry of Health open-data repositories covid19-public and data-darah-public on GitHub (harvest/moh.py)	36
Dataset	ElectionData.MY catalogue, an independent project compiling Election Commission results; always Tier 3 (harvest/electiondata.py)	202
Dataset	Archive copies in booluckgmie/sharecode (APIMS air pollutant index, GSO electricity generation, Sabah job portal, NAPIC 2022 property tables, MPOB palm oil prices, Bursa PN17/GN3); allowlisted folders only, always Tier 3 (harvest/sharecode.py)	7
Dataset	Toll sources (harvest/tolls.py): LLM's directories of 33 operating and 3 under-construction highways, read directly from llm.gov.my	2
Dashboard / Dataset	Toll rate lookup (kadartol.llm.gov.my) and two old data.gov.my toll datasets: identified through web search only, not read, flagged unreachable	3
Report series	NAPIC (JPPH) property publications: 14 series across market, stock, status and price/rental indices, read from napic.jpph.gov.my (harvest/napic.py)	14
Dashboard	NAPIC open transaction data, an embedded Tableau Public dashboard	1
Dataset / Report series	MPOB public pages (harvest/mpob.py): daily CPO price (rolling 3 years, Excel per year), the latest monthly industry performance report and year summary, and the annual Overview of the Malaysian Oil Palm Industry (10 PDFs, 2016 to 2025)	4
Report series	DOSM publications (pub-dosm/), editions grouped into series, with technical notes attached	94
Report series	DOSM release archive (harvest/dosm_archive.py): edition history back to 2014 added to 8 labour and wage series (Salaries & Wages 15 editions, Labour Force 194), plus a new series, Job Vacancies Advertised Online	1 new
Dashboard	portal dashboards and explorers (dashboards/, explorers/)	56
Live feed	DOE APIMS public API: hourly Air Pollutant Index by station, last 24 hours (harvest/doe.py)	1
Live feed	data.gov.my realtime APIs read from the developer docs: weather forecast, weather and earthquake warnings, flood warning, GTFS static and realtime for KTMB, Prasarana and BAS.MY (harvest/devdocs.py)	10
API	data.gov.my static query APIs from the same docs: the Data Catalogue API and the OpenDOSM API	2

Of these, 66 are withheld from the page: 6 dashboards whose portal page is gone, and 60 BNM tables that the agency lists but whose endpoint returns no records.

See registry/SCHEMA.md for the record format and registry/tiers.json for the provisional reliability tiers.

Rules the platform keeps

• Metadata and pointers only. No copied data.
• Respect each source's limits. api.data.gov.my allows 4 requests per minute per API, so the pipeline spaces and caches its calls to it, and checks one endpoint per API record.
• A record whose page could not be read says so on its card ("could not be reached when last checked"). GitHub web pages return 403 through this environment's proxy, so a record counts as live when its page or a file link answers.
• Every record carries its agency, a reliability tier with the reason, a licence where the portal states one, and a link-check date.
• A record whose official portal page returns 404/410 is withheld from the app and listed in registry/link_report.md. A network error never hides a record.
• Tiers and fitness scores are estimates made by this tool, not official ratings.

This is the beginning of the README. Read the rest at the original. Read the full README ↗