stharrold.github.io

Source for Data Science Demos, a blog built with Pelican and deployed to GitHub Pages by GitHub Actions.

Quick start

Requirements: uv (Python 3.12+). All dependencies are pinned in pyproject.toml and uv.lock.

uv sync                                         # install Pelican, plugins, Pagefind, dev tools
uv run pre-commit install                       # once: ruff and file checks on commit
uv run pelican --settings pelicanconf.py        # build the dev site into output/ (drafts included, no analytics)
uv run python -m pagefind --site output         # build the search index (rerun after each build)
uv run pelican --listen                         # serve output/ at http://localhost:8000
uv run pytest                                   # build the publish site in a temp dir and check it

uv run pelican --settings pelicanconf.py --autoreload --listen rebuilds on every change, but each rebuild deletes output/pagefind/, so search only works after rerunning Pagefind. Stop it with Ctrl-C; killing only the parent process leaves its file watcher running.

Branches and deployment

Branch	Purpose
src_master	Production source. Every push builds, tests, and deploys the site (.github/workflows/pages.yml).
src_develop	Integration branch; pushes run the same checks without deploying.
src_feature_*	Feature branches off src_develop (git-flow); merged with --no-ff.
(tag) legacy-ghp-import-master	The former master branch, which held the built site (via ghp-import) until 2026-10-06.

To release: merge src_feature_* into src_develop, then src_develop into src_master, and push. The workflow runs ruff, pytest, pelican --settings publishconf.py --fatal warnings, and Pagefind, then deploys output/. GitHub Pages must be set to Settings > Pages > Source: GitHub Actions. output/ is generated and not tracked.

To verify a deploy, wait for the run (gh run watch <id>; find it with gh run list --workflow pages.yml --json databaseId,headSha), then check the live site with a cache-busting query, e.g. curl -s "https://stharrold.github.io/?v=$(date +%s)". For visual changes, compare before/after screenshots of a local dev build served with python3 -m http.server before merging to src_master.

Automation:

• .github/workflows/links.yml logs internal and external link status every Monday (and on manual dispatch) against a fresh build: a job summary plus a link-report JSON artifact kept 90 days. It is report-only (never fails, files no issues); alerting belongs to whatever reads the log. Dead links in the frozen 2015-2016 posts are left as is. GitHub disables scheduled workflows in public repos after 60 days without repository activity, so the log can stop silently on a quiet blog; re-enable with gh workflow enable links.yml (see #67).
• Dependabot (.github/dependabot.yml) opens weekly grouped PRs for GitHub Actions and Python packages against src_develop; security fixes target src_master, so review them as deploys. Actions are pinned to commit SHAs.
• Vendored theme files and the MathJax CDN version are not covered by Dependabot; see themes/README.md.

pelicanconf.py holds the site settings; publishconf.py extends it for production (absolute URLs, feeds, GA4, no drafts).

Writing posts

Posts are Markdown files in content/ with a metadata header (Title, Date, Modified, Category, Tags, Slug, Summary, optional Related_posts and Alias for old URLs). Pages go in content/pages/. Notebooks, notebook HTML exports, and images go in content/static/<slug>/ and are linked with {static}/static/<slug>/....

• Drafts: Status: draft renders only in dev builds (output/drafts/). Static files are copied regardless of status, so add a draft's static/<slug> folder to STATIC_EXCLUDES in publishconf.py, and remove it when the post is published.
• Math: $...$ inline and $$...$$ display (as its own paragraph); rendered by MathJax 4. Prices such as $20 are left alone.
• Old posts: posts last updated 5+ years before the build year show an "out of date" notice (OUTDATED_AFTER_YEARS).
• Before pushing: run uv run pytest. Deploys stop on any Pelican warning (for example a broken {filename}/{static} link) or failed test. The tests derive what to expect from content/, so new and newly published posts need no test changes; publishing a draft that has a STATIC_EXCLUDES entry fails with a message saying which entry to remove.

Site features

• Theme: themes/datasciencedemos/, Bootstrap 5 with Bootswatch Flatly, forked from pelican-bootstrap3; see themes/README.md.
• Search: Pagefind, built from output/ after Pelican; runs in the browser.
• Comments: giscus (GitHub Discussions, Announcements category; requires the giscus GitHub App on this repository). Comments from Disqus (2016-2017) are archived as static HTML in data/archived-comments/<slug>.html.
• Analytics: Google Analytics 4 (G-K4PX9089RZ, publish builds only, Google signals off), exported daily to BigQuery project stharrold-github-io (dataset analytics_393124094). Universal Analytics history recovered from the GA monthly snapshot emails (2017-11 to 2019-10) is in stharrold-github-io.ua_history.monthly_snapshots.
• Analytics archive: scripts/analytics_archive.py copies every BigQuery table to Parquet in the private Google Drive folder My_Drive/Data/stharrold.github.io/ (ga4/events_YYYYMMDD.parquet, ua_history/), reads each file back, and records rows, BigQuery's last-modified time, a sha256 and the writing commit in manifest.json; each run also writes a README.md there linking back to this repository at that commit. A day is archived once it is 3 days old (GA4 adds late events for 72 hours); a table that changes later is archived again with the old file kept in superseded/; nothing is deleted. It warns if the newest export is over 7 days old, if an archived table has left BigQuery, or if billing is off. A launchd job runs it at 09:00 and 21:00 from a copy outside ~/Documents: install or update with bash scripts/install_analytics_launchd.sh (re-run after changing either script), start a run with --run-now, log in ~/Library/Logs/stharrold-analytics.log. uv run --script scripts/analytics_archive.py --dry-run shows what is due. The read-back checks the write, not Drive's upload. The files include visitor pseudonymous IDs, so they never go in git.
• /analytics skill (.claude/skills/analytics/SKILL.md): checks the archive is healthy, reports readership with uv run --script scripts/analytics_report.py --save (DuckDB over the archive plus the days still only in BigQuery; totals saved to on Drive; pages ranked only once one has 30 engaged visitors), then proposes the next post. Engaged visitors (GA4 or engagement time) screen out most crawlers. The owner's own visits are kept out three ways: a GA4 internal-traffic rule for the home IP with the Internal Traffic filter active (from 2026-10-29; re-enter the IP if it changes), the Google Analytics opt-out add-on in the owner's browsers, and on Drive (browser IDs from the cookie, which the report drops from all past data). How it was built: .

Work tracking

Open work is tracked as GitHub issues grouped in milestones: one per post ("post about predicting income", "post about agentic development", ...) plus Site maintenance for build, theme, and tooling upkeep. Planned posts, in order: a pre-registered causal study on CMS Medicare Shared Savings Program data (#73, pinned), an evaluation harness for a retrieval-augmented QA agent (#74), and an interactive household-income demo built from ACS PUMS data (#68). Published 2015-2016 posts are frozen as written.

License

Code: MIT. Content: CC BY 4.0, except where indicated otherwise. Vendored theme assets keep their own licenses (themes/datasciencedemos/licenses/).