Daily Stock Prediction (S&P 500 + 7 large-caps)

Predicts the next trading day's return/direction for SP500 (^GSPC), AAPL, PLTR, MSFT, GOOGL, AMZN, NVDA, and META using classic ML on engineered technical features. No paid data or API keys required (uses yfinance).

A note on scope while reading this README: the project started with just SP500/AAPL/PLTR, and most of the narrative sections below — the six-point ML checklist, the 80-90% question, the published-research comparison — were written and verified against those three specifically. MSFT, GOOGL, AMZN, NVDA, and META were added later, trained through the same pipeline, and independently leakage-checked (leakage_check.py passes for all eight), but weren't individually re-litigated through every prior finding. Treat the original three as the deeply-audited core and the other five as covered by the same methodology, not as separately re-proven case by case.

Two pieces

The project is split into two clearly separated parts, each with its own job:

• src/engine/ — Part 1, the prediction/backtest/research engine. Everything that fetches data, engineers features, trains and tunes models, verifies there's no leakage, and backtests results. This is all of the work described in the rest of this README.
• src/comparison/ — Part 2, a neutral descriptive-statistics comparison across all eight tickers, built entirely from Part 1's own outputs. It reports how each ticker's model has performed historically — held-out test accuracy, recent backtest accuracy, cost-adjusted returns, long-horizon drift, a live 3-month walk-forward check, raw fundamentals — side by side. It does not train anything new, and it does not recommend a trade. See "Descriptive ticker comparison" below for why that line matters and isn't just a formality.

What it actually predicts

Raw closing price is almost perfectly autocorrelated (tomorrow's price ≈ today's price), so a model that predicts price directly looks deceptively accurate while learning nothing useful. Instead, every model here predicts:

• Regression target: next day's close-to-close % return
• Classification target: next day's direction (up/down)

The predicted price shown in predict.py is just last_close * (1 + predicted_return), reconstructed for readability.

Setup

cd /Users/minsoochoi/Desktop/stock
source venv/bin/activate   # already created; recreate with `python3 -m venv venv` if missing
pip install -r requirements.txt

To train with NYT news-sentiment features (news_features.py, see below), put a free API key from developer.nytimes.com (Archive API enabled on the app) in a .env file at the project root:

NYT_API_KEY=your-key-here

.env is gitignored, so the key never gets committed. Without it, the news feature set is simply unavailable and the pipeline falls back to baseline/macro features only.

Usage

python src/engine/train.py           # fetches data, engineers features (incl. NYT news history on first run), tunes + evaluates models, saves the best ones
python src/engine/predict.py         # loads saved models, prints tomorrow's prediction + live news sentiment for each ticker
python src/engine/news_pulse.py      # just the live news-sentiment readout, on its own
python src/engine/implied_vol.py     # just the live single-stock implied-vol/skew readout (AAPL, PLTR), on its own
python src/engine/backtest_dates.py  # walk-forward accuracy check + transaction-cost reality check over recent days
python src/engine/leakage_check.py   # verifies no feature depends on future data (see the checklist below)
python src/engine/long_horizon_drift.py    # NOT day-to-day forecasting -- see "The 80-90% question" below
python src/engine/predict_long_horizon.py  # ditto -- read that section before running either

python src/comparison/compare_tickers.py   # Part 2: neutral side-by-side stats across tickers, NOT investment advice

train.py re-downloads fresh history every run, so re-run it periodically (e.g. weekly) to keep models current — daily market microstructure drifts. The first train.py run also builds the full NYT news history (~120 monthly archive calls, cached to data/news_cache/ afterward so subsequent runs are fast).

Pipeline

1. src/engine/fetch_data.py — downloads ~10 years of daily OHLCV via yfinance, caches to data/*.csv.
2. src/engine/features.py — builds ~20 "own" features per day from only past data (no look-ahead): lagged returns, SMA ratios (5/10/20/50d), rolling volatility, RSI(14), MACD, Bollinger Band position/width, volume change, day-of-week.
3. src/engine/macro_features.py — builds 24 cross-market/"world" features shared across all tickers: VIX level/z-score (S&P 500's own options-implied vol), VXN (Nasdaq-100 implied vol) for more implied-vol signal, 10Y Treasury yield level/change, the 10Y-vs-13-week yield curve slope (recession-risk signal), dollar index return, oil/gold returns, Dow/Nasdaq/Russell 2000 returns, Nikkei/FTSE/DAX returns (how Asia and Europe already traded before the US session opens), Bitcoin return/volatility (trades weekends, so it reflects global news equities can't price in until Monday), a credit-stress proxy (high-yield bond ETF return minus Treasury bond ETF return), and QQQ return (tech-sector proxy). All same-day (close of day t), which is valid information for predicting day t+1 — no look-ahead.

This is the beginning of the README. Read the rest at the original. Read the full README ↗