stharrold.github.io
Source for Data Science Demos, a blog built with Pelican
and deployed to GitHub Pages by GitHub Actions.
Quick start
Requirements: uv (Python 3.12+). All dependencies are pinned in pyproject.toml and uv.lock.
uv sync # install Pelican, plugins, Pagefind, dev tools
uv run pre-commit install # once: ruff and file checks on commit
uv run pelican --settings pelicanconf.py # build the dev site into output/ (drafts included, no analytics)
uv run python -m pagefind --site output # build the search index (rerun after each build)
uv run pelican --listen # serve output/ at http://localhost:8000
uv run pytest # build the publish site in a temp dir and check it
uv run pelican --settings pelicanconf.py --autoreload --listen rebuilds on every change, but each rebuild deletes
output/pagefind/, so search only works after rerunning Pagefind. Stop it with Ctrl-C; killing only the parent
process leaves its file watcher running.
Branches and deployment
| Branch | Purpose |
|---|
src_master | Production source. Every push builds, tests, and deploys the site (.github/workflows/pages.yml). |
src_develop | Integration branch; pushes run the same checks without deploying. |
src_feature_* | Feature branches off src_develop (git-flow); merged with --no-ff. |
(tag) legacy-ghp-import-master | The former master branch, which held the built site (via ghp-import) until 2026-10-06. |
To release: merge src_feature_* into src_develop, then src_develop into src_master, and push. The workflow runs
ruff, pytest, pelican --settings publishconf.py --fatal warnings, and Pagefind, then deploys output/.
GitHub Pages must be set to Settings > Pages > Source: GitHub Actions. output/ is generated and not tracked.
To verify a deploy, wait for the run (gh run watch <id>; find it with
gh run list --workflow pages.yml --json databaseId,headSha), then check the live site with a cache-busting query,
e.g. curl -s "https://stharrold.github.io/?v=$(date +%s)". For visual changes, compare before/after screenshots of a
local dev build served with python3 -m http.server before merging to src_master.
Automation:
.github/workflows/links.yml logs internal and external link status every Monday (and on manual dispatch) against
a fresh build: a job summary plus a link-report JSON artifact kept 90 days. It is report-only (never fails, files
no issues); alerting belongs to whatever reads the log. Dead links in the frozen 2015-2016 posts are left as is.
GitHub disables scheduled workflows in public repos after 60 days without repository activity, so the log can
stop silently on a quiet blog; re-enable with gh workflow enable links.yml (see #67).- Dependabot (
.github/dependabot.yml) opens weekly grouped PRs for GitHub Actions and Python packages against
src_develop; security fixes target src_master, so review them as deploys. Actions are pinned to commit SHAs.
- Vendored theme files and the MathJax CDN version are not covered by Dependabot; see
themes/README.md.
pelicanconf.py holds the site settings; publishconf.py extends it for production (absolute URLs, feeds, GA4,
no drafts).
Writing posts
Posts are Markdown files in content/ with a metadata header (Title, Date, Modified, Category, Tags,
Slug, Summary, optional Related_posts and Alias for old URLs). Pages go in content/pages/. Notebooks,
notebook HTML exports, and images go in content/static/<slug>/ and are linked with {static}/static/<slug>/....
- Drafts:
Status: draft renders only in dev builds (output/drafts/). Static files are copied regardless of
status, so add a draft's static/<slug> folder to STATIC_EXCLUDES in publishconf.py, and remove it when the
post is published.
- Math:
$...$ inline and $$...$$ display (as its own paragraph); rendered by MathJax 4.
Prices such as $20 are left alone.
- Old posts: posts last updated 5+ years before the build year show an "out of date" notice
(
OUTDATED_AFTER_YEARS).
- Before pushing: run
uv run pytest. Deploys stop on any Pelican warning (for example a broken
{filename}/{static} link) or failed test. The tests derive what to expect from content/, so new and newly
published posts need no test changes; publishing a draft that has a STATIC_EXCLUDES entry fails with a message
saying which entry to remove.
Site features
- Theme:
themes/datasciencedemos/, Bootstrap 5 with Bootswatch Flatly, forked from pelican-bootstrap3;
see themes/README.md.
- Search: Pagefind, built from
output/ after Pelican; runs in the browser.
- Comments: giscus (GitHub Discussions, Announcements category; requires the giscus
GitHub App on this repository). Comments from Disqus (2016-2017) are archived as static HTML in
data/archived-comments/<slug>.html.
- Analytics: Google Analytics 4 (
G-K4PX9089RZ, publish builds only, Google signals off), exported daily to
BigQuery project stharrold-github-io (dataset analytics_393124094). Universal Analytics history recovered from
the GA monthly snapshot emails (2017-11 to 2019-10) is in stharrold-github-io.ua_history.monthly_snapshots.
- Analytics archive:
scripts/analytics_archive.py copies every BigQuery table to Parquet in the private Google
Drive folder My_Drive/Data/stharrold.github.io/ (ga4/events_YYYYMMDD.parquet, ua_history/), reads each file
back, and records rows, BigQuery's last-modified time, a sha256 and the writing commit in manifest.json; each run
also writes a README.md there linking back to this repository at that commit. A day is archived once it is
3 days old (GA4 adds late events for 72 hours); a table that changes later is archived again with the old file
kept in superseded/; nothing is deleted. It warns if the newest export is over 7 days old, if an archived table
has left BigQuery, or if billing is off. A launchd job runs it at 09:00 and 21:00 from a copy outside ~/Documents:
install or update with bash scripts/install_analytics_launchd.sh (re-run after changing either script), start a
run with --run-now, log in ~/Library/Logs/stharrold-analytics.log. uv run --script scripts/analytics_archive.py --dry-run shows what is due. The read-back checks the write, not Drive's upload. The files include visitor
pseudonymous IDs, so they never go in git.
/analytics skill (.claude/skills/analytics/SKILL.md): checks the archive is healthy, reports readership with
uv run --script scripts/analytics_report.py --save (DuckDB over the archive plus the days still only in BigQuery;
totals saved to on Drive; pages ranked only once one has 30 engaged visitors), then proposes
the next post. Engaged visitors (GA4 or engagement time) screen out most crawlers. The owner's own
visits are kept out three ways: a GA4 internal-traffic rule for the home IP with the Internal Traffic filter active
(from 2026-10-29; re-enter the IP if it changes), the Google Analytics opt-out add-on in the owner's browsers, and
on Drive (browser IDs from the cookie, which the report drops from all past data).
How it was built: .
Work tracking
Open work is tracked as GitHub issues grouped in milestones: one per post ("post about predicting income",
"post about agentic development", ...) plus Site maintenance for build, theme, and tooling upkeep. Planned
posts, in order: a pre-registered causal study on CMS Medicare Shared Savings Program data (#73, pinned), an
evaluation harness for a retrieval-augmented QA agent (#74), and an interactive household-income demo built from ACS
PUMS data (#68). Published 2015-2016 posts are frozen as written.
License
Code: MIT. Content: CC BY 4.0, except where indicated
otherwise. Vendored theme assets keep their own licenses (themes/datasciencedemos/licenses/).
