tz-engine

A small Next.js app for looking up historical OHLC (Open/High/Low/Close) data for global stocks and market indices.

• Index tab: pick a major world index (Nifty 50, Sensex, S&P 500, Dow Jones, Nasdaq 100, FTSE 100, DAX, CAC 40, Nikkei 225, Hang Seng) and pull its own price history directly.
• Stock tab: optionally narrow to a market, then search any stock by name across global exchanges.
• Pick a start and end date, and a daily/weekly/monthly interval.
• View results in a table and download them as CSV.

Deliberately not included: a dropdown of "every stock in every index." Index constituent lists aren't reliably available from Yahoo's free/ unofficial endpoints and go stale as indices rebalance — shipping a wrong or outdated list on a financial data tool is worse than not having one. The curated index list in lib/indices.ts only carries well-known, stable index tickers (e.g. ^NSEI, ^GSPC), used both to fetch the index's own history and as a region hint that narrows stock search to that market.

How it works

The frontend (app/page.tsx) never calls an external API directly. It talks to server-side route handlers, which go through lib/marketData.ts:

• GET /api/search?q=<text>&region=<2-letter code> — resolves a search string to matching stocks/indices globally, optionally weighted toward a market (lib/yahoo.ts → searchSymbols). region is a soft hint, not a hard filter — Yahoo's non-US exchange codes aren't reliably documented enough to filter on exactly.
• GET /api/history?symbol=<sym>&start=YYYY-MM-DD&end=YYYY-MM-DD&interval=1d|1wk|1mo — returns OHLC rows for that symbol (stock or index) and range.

Data sources: Blob storage first, Yahoo Finance as fallback

lib/marketData.ts tries Vercel Blob storage first (lib/blobHistory.ts) for symbols we've bulk-uploaded — faster, and doesn't depend on Yahoo's endpoints staying reachable. It falls back to a live Yahoo Finance fetch (lib/yahoo.ts → fetchHistory) for anything not in Blob storage.

Blob-backed segments: NSE mainboard stocks (real Yahoo .NS suffix, so they can also fall back to a live Yahoo fetch), plus Commodities, Crypto, Forex, Indian indices, International indices, and SME (NSE Emerge) stocks — these six use a pseudo-suffix (.COMM / .CRYPTO / .FX / .IDX / .INTLIDX / .SME) purely to route the lookup to the right Blob segment, since they aren't real Yahoo tickers and have no live-fetch fallback. The curated symbol/name lists for these six live in lib/otherMarkets.ts and back the "Markets" tab on the main page (app/page.tsx).

The Markets tab also has a 7th segment, "F&O Stocks" — a deliberate copy (not a move) of the 210 stocks and 6 index futures that are currently eligible for Futures & Options trading on NSE, so they can be browsed as their own group without duplicating any underlying data. This list is generated by fetching NSE's own live F&O underlying list (scripts/fetch-fno-list.mjs, pulling fo_mktlots.csv from nsearchives.nseindia.com) and intersecting it with whatever's already uploaded — never hand-typed, since NSE revises F&O eligibility periodically. Re-run that script (via its matching GitHub Actions workflow) and regenerate the "fno" segment in lib/otherMarkets.ts if the list needs refreshing.

Blob storage holds the data in consolidated chunk files, not one file per instrument — grouping ~2,578 NSE stocks into ~20-25 files per timeframe (see CHUNK_PLAN in lib/blobHistory.ts and scripts/bulk-upload-consolidated.mjs) instead of one object per stock. This matters: Vercel Blob plans have a monthly cap on operation counts, and one object per instrument (12,776 of them) blew through that cap during initial setup. Reads also construct the blob's URL directly from BLOB_STORE_ID rather than calling list(), since list() is billed at a much lower monthly allowance than plain reads.

To (re)populate Blob storage from scratch: run scripts/cleanup-blob.mjs then scripts/bulk-upload-consolidated.mjs <path-to-data-folder>, either locally (needs BLOB_READ_WRITE_TOKEN in .env.local) or via the "Sync Blob Storage" GitHub Actions workflow (workflow_dispatch, no local setup needed — see .github/workflows/sync-blob-storage.yml).

Keeping the data current: scheduled auto-refresh

Once populated, data stays current on its own via .github/workflows/refresh-blob-data.yml (needs both BLOB_READ_WRITE_TOKEN and BLOB_STORE_ID as GitHub repo secrets — the sync workflow above only needed the token, since it uploads via the @vercel/blob SDK; this one also downloads existing chunks directly by URL, the same way lib/blobHistory.ts does):

• Daily (every day, 05:00 IST — after NSE and US markets have both closed for the day): scripts/refresh-daily.mjs fetches each instrument's latest daily bar(s) from Yahoo Finance and appends only the rows newer than what's already stored. Existing historical rows are never modified or removed, regardless of what happens with any individual ticker's fetch.
• Weekly / monthly / yearly (Saturday / the 1st of the month / Jan 1st, each shortly after that day's daily refresh): scripts/rollup-period.mjs --period <weekly|monthly|yearly> derives one new bar for the just-completed period directly from the (already refreshed) daily data — it never fetches weekly/monthly candles from Yahoo directly, which keeps every timeframe consistent with the same underlying daily series and sidesteps Yahoo's own period-boundary conventions.

NSE and SME tickers are refreshed via their real .NS Yahoo suffix (high confidence — same lookup already proven correct for the app's own reads). The other five segments use a hand-built ticker → Yahoo-symbol map in scripts/otherMarketsYahooMap.mjs; confidence per entry is noted in that file's comments (major commodities/currencies/indices are high-confidence, some Indian sector indices and a couple of others are best-guess and worth spot-checking against the app after the first scheduled run). A wrong or missing mapping only means that one instrument doesn't get new rows — it can't corrupt existing data — and is a one-line fix in that file once noticed.

Each workflow job can also be triggered manually from the Actions tab (workflow_dispatch, with a dropdown to pick daily/weekly/monthly/yearly) to catch up after a missed run or to verify the setup works.

Proxying Yahoo server-side avoids browser CORS issues and keeps the endpoint details out of client code. Yahoo has no official partner API for this data, so its endpoints can change or start rate-limiting without notice — treat failures from them as expected, not as bugs in this app.

Development

npm install
npm run dev   # http://localhost:3000

Note: the sandboxed environment this was originally built in has no outbound network access to Yahoo Finance, so the API routes were verified to build, start, validate input correctly, and fail cleanly (502 with a readable error) rather than crash — but the actual live data path has not been exercised end-to-end in every case. Test against a real network before relying on it in production.

Production considerations

• Yahoo's endpoints are unofficial and can be rate-limited or blocked for a given IP/deployment. For a production website with real traffic, consider a proper licensed market-data vendor instead.
• There is no caching layer yet — every search keystroke and every "Fetch data" click hits Yahoo directly. Add caching (e.g. a short-TTL in-memory or Redis cache) before this sees real traffic.
• The index list in lib/indices.ts is intentionally small and hand-picked for confidence in the ticker symbols. Adding more indices is safe as long as each symbol is verified against Yahoo Finance directly first.