Futbol Veritabanı
8대 리그 클럽·선수·통계—Transfermarkt 데이터
아직 감지된 업데이트가 없습니다. GitHub 릴리스가 나오면 여기에 보입니다.
개인 계정의 공개 저장소 소유자는 GitHub로 로그인해 운영자를 확인하고 소개를 직접 관리할 수 있습니다. 관리자 승인은 필요하지 않습니다.
로그인한 뒤 확인 파일을 만들어 이 저장소의 기본 브랜치에 30분 안에 커밋해 주세요. 다른 브랜치나 포크의 파일은 인정하지 않습니다. 파일을 올린 뒤 아래에서 확인하면 관리자 승인 없이 관리할 수 있습니다. 자격을 유지하려면 파일을 남겨 두세요.
로그인이 필요하면 로그인 화면으로 이동합니다. 로그인 후 이곳으로 돌아와 원하는 확인 버튼을 다시 눌러 주세요.
프로젝트 폴더에서 /nomorevibe verify를 실행하면 소유권을 확인하고 소개를 직접 관리할 수 있습니다.
1분 만에 확인하기README 전체를 불러오는 중입니다. 아래는 저장된 발췌입니다.
A local data analysis tool for kart racing telemetry exported from a
Unipro laptimer (TSV format). Loads one or more session files, segments
the track into corners from the GPS trace, and produces a ranked
"Top 3 areas to focus on" report -- with a plain-language coaching note
per segment -- plus a kart-setup input form and a correlation engine that
suggests concrete setup changes from telemetry patterns.README 전문 보기 ↗A local data analysis tool for kart racing telemetry exported from a Unipro laptimer (TSV format). Loads one or more session files, segments the track into corners from the GPS trace, and produces a ranked "Top 3 areas to focus on" report -- with a plain-language coaching note per segment -- plus a kart-setup input form and a correlation engine that suggests concrete setup changes from telemetry patterns.
Built for a Rotax Senior EVO junior driver on Danish/Scandinavian tracks, but the analysis core is track/class-agnostic; only the setup-suggestion defaults (peak-power RPM band, carb model) lean Rotax EVO.
Originally built against a written spec only (the sample export it referenced
wasn't actually available in that environment), then validated against a
real 82MB / ~898,000-row / 11-session Unipro export from an actual track day.
That validation pass surfaced and fixed several real-world quirks the
original spec's single sample file didn't cover -- see below. The synthetic
fixture (tests/fixtures/synthetic_session.tsv, generated by
tests/generate_fixture.py) still backs the automated test suite, since a
real export isn't checked into the repo, but the parser and analysis
pipeline have now both been exercised end-to-end against real telemetry.
Real-world quirks found during that validation, beyond the original spec:
Heading/GPS Speed/lat-lon/G-forces
all update together as one atomic GPS fix. A real export instead had two
independent groups: position (Heading, Latitude, Longitude, DOPs,
Altitude) fires together, but GPS Speed and the three G-force channels
fire on their own separate cadence that never once shared a row with the
position fix in that file. telemetry/parser.py now distinguishes
GPS_FIX_COLUMNS (position group) from GPS_MOTION_COLUMNS (speed/G-force
group), and corners.py::lap_gps_trace aligns each independently via
merge_asof rather than assuming they share rows. Getting this wrong
silently produced all-NaN speed and G-force data, which in turn produced
empty braking-zone detection and speed/G-force charts -- worth knowing if
those look empty on a new export.GPS Speed-truly-absent fallback still exists as a backstop
(corners.py::_derive_speed_from_distance): if GPS Speed really isn't
populated anywhere (rather than just being on a different cadence, as
above), speed is derived from GPS Distance's rate of change instead, and
flagged via a gps_speed_is_estimate column / a banner in the app.Inverse Corner Radius and a separate Corner Radius
column not in the original spec). The parser only requires its known
columns to be present; unrecognized extra columns are harmless and
ignored.telemetry/parser.py::split_sessions.Lap Number, not group
by its raw value. Grouping by value silently merges two physically
distinct laps whenever a lap number repeats non-adjacently (a missed
session-restart split, or a stray mistimed reading) -- this was the cause
of a reported "59s lap that doesn't match the dash" bug. Fixed in
telemetry/laps.py::lap_table.segment_track only ever merges a segment
backward into its predecessor, so the very first segment -- which has no
predecessor -- survived regardless of length. Curvature right at the
lap-start distance boundary is a common false-corner artifact (no true
"previous" sample for the heading-gradient calc there), so this reliably
showed up as a bogus few-metre "Corner 1" with a near-zero segment time on
real data, found while building cross-session corner comparison. Fixed to
fold a too-short first segment forward into the second one instead -- see
telemetry/corners.py::segment_track.If you hit a different real-world quirk on your own export, the parser generally fails loudly (missing expected column) rather than silently misinterpreting data -- but channel-level surprises like the ones above (a column present in the header yet structured differently than assumed) won't necessarily raise an error, so a sanity check against a new device/ firmware/config is still worthwhile (see "Validating against a real export").
telemetry/parser.py::COLUMNS for the exact list and order).Start Date/Start Time/Lap Number/Session Time/Lap Time, which are always populated) filled in -- every other
column is blank on that row. There is no fixed sample rate. Do not
read rows as if each one were a complete sample; use
Session.extract_channel() / Session.align_channels() to reconstruct
per-channel time series and align them only where a calculation actually
needs multiple channels together.Session Time and Lap Time are in nanoseconds. Both are divided by
1e9 on load (session_time_s / lap_time_s columns).Lap Time is the authoritative per-lap clock. Lap boundaries come
directly from Lap Number increments -- they are not re-derived from GPS
start/finish crossings.Lap Number and Session Time both
reset to 0 partway through the file. The parser detects this (a drop in
Session Time) and splits the file into separate Session objects
automatically -- see telemetry/parser.py::split_sessions.Inverse Corner Radius directly when a given export has it populated) --
see telemetry/corners.py.telemetry/ Parsing + analysis core (independently testable, no UI code)
parser.py TSV ingestion, session splitting, sparse-channel reconstruction
laps.py Lap segmentation, outlier/anomaly detection, consistency
corners.py GPS-based corner/straight segmentation
metrics.py Speed/RPM/G traces, braking & throttle inference
delta.py Theoretical best lap, delta-time-vs-distance trace
focus_areas.py Top-3 focus area scoring + coaching notes
setup_config.py Kart setup schema (YAML-backed)
setup_engine.py Setup correlation / suggestion engine
comparison.py Reference-lap diff, progression, driver comparison
corner_causal.py Entry/apex/exit extraction + three-zone time deltas
pattern_rules.py Rule-based causal-pattern classification taxonomy
corner_engine.py Orchestrates the two above + complex/noise handling
narrative.py Plain-language phrasing (templated or Anthropic-assisted)
weather.py Track-conditions auto-detect (Open-Meteo, keyless/free)
accounts.py Driver profiles, attribution, claiming, visibility gate, teams
auth.py Registration/login/reset (Supabase or local provider)
mailer.py Email templates + pluggable delivery
storage.py Session library (history + corner/pattern trend logging) -- SQLite locally, Postgres/Supabase when configured
db.py Shared Postgres/Supabase connection helper (see "Row Level Security" below)
analysis.py UI-agnostic façade: analyze_session()/analyze_lap()/compare_laps()
web/ Next.js frontend (Vercel) -- see "The app today" below
worker/ Ingest worker: claims upload_batches, parses, persists (Railway/Render/Fly)
scripts/ingest.py CLI ingestion into the session library (automation-friendly)
tests/ pytest suite + synthetic fixture generator
config/ Example kart setup YAML
The original UI (app.py, a Streamlit app) has been retired now that
web/ + worker/ + Supabase run in production -- see "The app today:
Next.js frontend + background worker" below for the current architecture
and how to run it locally. It's still in git history if you
need to see how a since-removed page worked.
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
This installs telemetry/'s own dependencies, for running scripts/,
the test suite, or unigo_sync from a source checkout. The web app and
worker each have their own separate install step -- see "Running it
locally" under "The app today" below.
On the sections below, through "Session library (history) & automation": they document the full feature set as built in this repo's now-retired Streamlit prototype, plus the underlying
telemetry/analysis engine and data model, which are unchanged and still exactly what the worker (and any page rebuilt inweb/) compute and enforce. Mentions of "the sidebar," "the Settings page," or a specific page name describe that retired UI's layout, notweb/'s -- see "What exists inweb/so far" under "The app today" for what's actually live today. Kept here because the algorithms, data model, and UX decisions they describe are still the spec for anything rebuilt.
Uploading a file on the Settings page requires a driver name and a
track name first -- there are two text fields above the uploader, and the
"Load file(s)" button only appears (and only saves anything) once both are
filled in, plus the five track-conditions fields described below, since
every saved session needs to know who drove it, where, and in what
conditions. Once loaded, every session from
that file is written into the same SQLite-backed library every other page
reads from (SessionLibrary in telemetry/storage.py, the same store the
History page and scripts/ingest.py use) -- so on the next run, whether
that's the next rerun, a page reload, or the app restarting entirely, those
sessions are there already, loaded from disk in load_persisted_sessions_cached
in app.py instead of needing the raw .tsv re-uploaded and re-parsed.
Re-uploading the exact same file under the same driver name is a no-op
(matched by filename + session index + start time + driver, so the same
file honestly re-uploaded by a different driver is still recorded, not
silently dropped as a duplicate).
Because every loaded session -- regardless of which file or which driver's
upload it came from -- ends up in one flat pool, the session/lap pickers
throughout the app (the sidebar's "Session to analyze", and the per-row
pickers on Data Analysis / Corner Comparison) let you pick
any combination across drivers exactly like they let you pick across your
own sessions: there's no separate "driver" concept threaded through the
comparison code, just more sessions to choose from. Session names
(session_label in app.py) reflect this -- driver, session number,
date/time, and best lap, deliberately not the source filename, since two
drivers' exports both showing up as "default_session.tsv" would tell you
nothing about whose lap you're looking at.
This still shares the same disk-persistence caveat as the History page:
data/sessions.db lives on local/container disk, wiped on every
redeploy/reboot on platforms without persistent storage -- a
within-run/within-deploy convenience, not a durable database, unless
SUPABASE_DB_URL/DATABASE_URL is set (the production setup -- see
"Row Level Security" below).
Uploaded the wrong file, or just want to clean one out? The History page's
"Delete a session" picker (SessionLibrary.delete_session) removes a
session's telemetry, lap rows, and cached dataframe permanently -- behind
a confirm/cancel step, since this can't be undone. It deliberately leaves
that session's kart setup history alone (a separate table, keyed by
file/session/start-time rather than this row's id), since that's still a
useful record even once the raw telemetry behind it is gone.
The app is multi-user: everyone signs in, and a driver is not the same thing as an account.
Registering normally creates the matching driver profile automatically -- for the ordinary case this is invisible plumbing.
A single Unipro file can hold several sessions from several drivers. After parsing, the Settings page shows each detected session (track, date, lap count) and asks who drove it. Each one goes to exactly one of:
An invite link is a single-use, 14-day claim token. Claiming is registration -- the same signup path as anyone else, including the under-16 guardian flow -- and then links the existing profile to the new account, so every session already recorded under it becomes theirs with no data movement. Someone who registered independently can also search unclaimed profiles from "Find My Profile"; the original uploader is notified rather than being made an approval gate, with a "report incorrect attribution" escape hatch as the safety valve.
Sessions are shared by default -- the leaderboards and lap-comparison pool are only useful if there's something in them, and a private-by-default world leaves every new driver looking at an empty board. Sharing is a single per-session setting, reversible any time, from "My Sessions & Sharing," with three tiers: private (only you and whoever uploaded it), team (also visible to fellow active members of your team -- see "Teams" below), and shared (also visible to everyone, the default). The Settings upload screen and every consent/invite email say this plainly before anyone commits to it, and the review screen offers a "keep this upload private for now" checkbox for anyone who wants to opt an upload out up front.
A session is only actually visible to others when all of these hold,
checked by one shared predicate (PUBLIC_VISIBILITY_SQL in accounts.py)
that every leaderboard/shared-laps query uses:
Point 3 is a hard gate, not a default, and it's the one place the "shared
by default" choice does not apply. An uploader marking an unclaimed
driver's session as shared has no effect -- that driver has never had the
chance to set their own preference, so their data cannot reach a
leaderboard or be selected as anyone's comparison reference until they
claim the profile and choose for themselves. The asymmetry is deliberate:
defaulting a claimed driver into sharing is reversible because they're
present to reverse it; there's nobody who can do that on an unclaimed
profile's behalf, so it stays gated instead. The same hard gate applies to
team visibility too -- an unclaimed profile's data is never visible to a
team, however a session was marked (team_visibility_sql in
accounts.py carries the identical claimed-profile check).
A team is a second, narrower sharing circle alongside the shared/private toggle above: a driver on a team can mark a session 'Team' visibility so only fellow active members see it, without putting it on any public leaderboard or the shared-laps browser. It exists for a real team (a race team, a club, a group of training partners) that wants to compare telemetry internally without also broadcasting it.
request_to_join_team/resolve_join_request in
accounts.py) -- the same two-sided care (an explicit ask, then a
human accept) the cross-account attribution flow on the Settings page
already takes for a similar reason.visible_sessions_for_user includes them.Auth is behind a provider interface (auth.py):
SUPABASE_URL and SUPABASE_ANON_KEY are set --
the intended production path.users
table), so a single-machine install works with no external service. Not
recommended for a real multi-user deployment.This is a separate switch from where that users table (and everything
else) actually lives -- see "The Postgres/Supabase-backed data layer"
below, controlled independently by SUPABASE_DB_URL/DATABASE_URL. Auth
can move to Supabase Auth (GoTrue) while storage stays on local SQLite, or
vice versa, or both at once.
Email delivery (mailer.py) uses SMTP when SMTP_HOST is set and
otherwise records messages to an email_outbox table without sending. When
no mail transport is configured, email verification is skipped rather than
locking accounts behind a link that can never arrive; password reset needs
a real mailer.
Claim invites are gated off by default. Set
KARTING_ENABLE_INVITE_EMAILS=1 to actually send them. They are
unsolicited contact with someone who has not consented to being on the
platform, often a minor, so that copy wants a legal/privacy review before
it goes anywhere near a real inbox -- until then the invite is generated
and recorded, just not delivered.
For local development, KARTING_DEV_SHOW_EMAIL_LINKS=1 prints
verification/reset/claim links on screen instead of emailing them. Never
set this on a shared deployment: it would show a password reset link for
any address anyone types in.
Every upload also requires track conditions for the session(s) in that file -- a dry/wet/mixed summary plus temperature, humidity, pressure, and altitude, entered once per upload (not per session; if conditions genuinely changed partway through a track day, upload that later batch of sessions separately with different values here). These feed the jetting suggestion on the Kart Setup page and are stored per session for later corner-comparison filtering.
Auto-detected where possible (telemetry/weather.py): as soon as file(s)
are selected, the app looks at the earliest-starting session's own GPS
location and Start Date/Start Time, and queries
Open-Meteo -- a free, keyless weather API with
global historical coverage, chosen specifically so this works anywhere in
the world without an API key or signup. Temperature, humidity, and pressure
come from Open-Meteo's hourly data at the session's own local hour
(timezone=auto resolves this from the GPS coordinates, matching the
logger's tz-naive local timestamp); dry/wet/mixed is inferred from hourly
precipitation at and just before that hour. Altitude comes from the
session's own GPS trace (median Altitude across its fixes), not the
weather API -- it's more accurate for the exact spot than a nearby grid
cell, and it's data the file already contains. A "🔄 Re-fetch weather"
button re-runs the lookup (overwriting manual edits) if the first attempt
failed or looks wrong.
Every field stays editable, and all five are still mandatory even when
auto-detection fails -- no internet access, no GPS fixes in the file, or a
date genuinely outside Open-Meteo's coverage all fall back to blank fields
with a "couldn't auto-detect, enter manually" notice rather than silently
guessing or blocking the upload entirely. Whether a session's values came
from the auto-lookup or manual entry is recorded (conditions_source in
telemetry/storage.py), for traceability.
A library created before this feature shipped gets its sessions table
migrated in place (new columns added via ALTER TABLE, not a fresh table)
the next time it's opened -- existing sessions keep their data with these
new fields simply blank, nothing is lost or needs re-uploading.
.tsv extension. Decimal points, not commas, should be used
for numeric values (this is the Analyser default; don't change locale
settings).Gearing, carburettor, tyre, and chassis settings, plus track/session
context and the engine's peak-power RPM band (none of this exists in the
TSV, so it's all user-supplied) are entered from the Kart Setup page.
Setup is stored per session, not globally -- switch "Session to
analyze" in the sidebar and each one keeps its own setup, since gearing,
jetting, and tyre pressure routinely differ session to session on the same
track day; there's no assumption that one setup applies to everything
you've loaded. A session with nothing saved for it yet starts from
defaults rather than silently inheriting whatever another session had (see
telemetry/storage.py::save_kart_setup / load_latest_kart_setup_for_session).
The History page lists every setup ever saved across every session, and can
copy any past one into the currently active session as a starting point.
Download the active session's setup as YAML to keep a record; see
config/setup_example.yaml for the file format, or load/save it directly
via telemetry.setup_config.KartSetup.
The setup correlation engine (also on that page) cross-references your
stated setup against telemetry patterns -- e.g. peak RPM on the longest
straight vs. your stated peak-power band, RPM hesitation during power-on as
a jetting proxy, lateral-G trend across the session as a tyre-pressure
proxy, left/right corner-speed asymmetry as a chassis-balance proxy.
Medium/high-confidence results here also compete for a slot in the Top 3
Focus Areas headline (see telemetry/focus_areas.py::blended_top_recommendations).
Every suggestion is labeled with a confidence level and phrased as a hypothesis to test, never a certainty -- there's no EGT/lambda, throttle, or brake channel in this export, so these are inferences from lap data, not direct sensor confirmation. Low-confidence jetting suggestions in particular should be verified with a plug chop before changing anything. Gearing direction matters here: for a direct-drive kart, engine RPM = axle RPM × (rear teeth / front teeth), so lowering RPM at a given speed means lowering that ratio (fewer rear teeth or more front teeth) -- the suggestion text spells out the specific tooth-count direction rather than just "gear up/down" to avoid ambiguity.
render_linked_speed_delta), genuine
CSS position: sticky against the page's own scroll can't reach across
that boundary -- the map's "sticky" behavior is hand-rolled instead, via
a small script that reads the component's own position in the page on
every animation frame (window.frameElement.getBoundingClientRect())
and translates the map to match, clamped so it never drifts past the
bottom of the chart column. This desktop layout doesn't work at phone
widths -- see Data Analysis (Mobile) below for the same laps redrawn
for a small screen.session_state, via a shared _render_lap_picker helper --
pick laps here or on desktop Data Analysis and the other view has the
same rows already selected), redrawn around touch instead of a mouse.
The desktop page's side-by-side 62%/38% chart+map split is unusable at
phone width -- squeezed into a few hundred pixels, five stacked
subplots -- so this page shows
(Speed/RPM/Lat G/Lon G/Delta, picked via a segmented control) at a
comfortable reading height, with a compact track map above it rather
than beside it (no need for the desktop version's cross-iframe sticky
hack, either -- one chart at a time is short enough that the whole
component fits on screen without internal scrolling).
Interaction is built around touch, not adapted from the mouse-oriented
desktop version: dragging the chart pans across the lap (the "scroll
across" a phone view needs), pinching zooms into any stretch of it for
detail, and the y-axis re-fits to whatever's actually visible after a
pan/zoom so a zoomed-in corner doesn't look flattened against the whole
lap's range. A tap (not just hover, which touchscreens don't reliably
fire) moves the map marker to that point, so "which part of the track is
this" stays answerable without needing to hover a mouse that isn't
there. The compact map carries corner-number labels for orientation at a
glance; a "🔍 Expand" button opens it larger in a modal for a clearer
look at the track shape and corner layout when the inline
thumbnail isn't enough.Beyond a per-segment time delta, the engine on the Lap Comparison / Recurring Patterns pages tries to explain why time was gained or lost in a corner -- including the case where a driver carries more entry speed, gains time through the corner itself, but loses more than that down the following straight because the entry compromised the exit. Diagnosis is fully deterministic and rule-based (never inferred by an LLM); an optional LLM call is used only to phrase the already-computed facts as a sentence.
Extraction (telemetry/corner_causal.py) reuses the existing
braking-zone (GPS Longitudinal Acceleration < -0.3g) and power-on
(> 0.15g) inference from telemetry/metrics.py rather than re-deriving
entry/exit detection from scratch. For each corner, per lap:
corner_engine.py traces it back
to the earlier corner's exit when the two correlate.Classification (telemetry/pattern_rules.py) is an explicit,
extensible taxonomy -- a list of plain rule functions, each checking the
same per-zone/per-point deltas against configurable SignificanceThresholds
(defaults: ~0.05s per zone, ~1 km/h speed, ~3m distance, refined further by
corner_engine.calibrate_thresholds once a driver has 4+ of their own clean
laps to measure real noise from). Named patterns: compromised exit from a
fast entry, conservative entry/strong exit, early apex with a compromised
exit, late apex rewarded on exit, a braking-point difference with no pace
change either way, and "clean, no significant difference" (deliberately not
manufactured as a finding). Anything real but unclassified is still
surfaced, just at low confidence, rather than silently dropped. Adding a new
pattern is adding one function to pattern_rules.TAXONOMY_RULES.
Phrasing (telemetry/narrative.py) turns one classified corner's facts
into 1-2 sentences, either via a fixed template per pattern (default, no
external dependency) or, if the "Use AI phrasing" checkbox is on and
ANTHROPIC_API_KEY is set in the environment, via a tightly-scoped
Anthropic API call that's given only the already-computed structured facts
and instructed never to invent a cause, number, or recommendation beyond
them -- any failure (no key, no anthropic package, API error) falls back
to the template silently.
Persistence (telemetry/storage.py's corner_metrics /
pattern_instances tables) logs every corner's extracted points and every
classified pattern instance from every comparison run on the Lap Comparison
page, unconditionally -- this is the structured asset the Recurring
Patterns page reads from, and the foundation for the personal-baselining,
outcome-tracking, and eventual cross-driver-benchmarking layers described in
corner_engine.py's module docstring, most of which builds on data that
needs to start accumulating well before there's a UI on top of it.
Unit tests for this engine (tests/test_corner_causal.py,
tests/test_pattern_rules.py, tests/test_corner_engine.py,
tests/test_narrative.py) use hand-built synthetic entry/apex/exit trace
pairs -- one scenario per taxonomy pattern -- rather than real telemetry, so
the diagnosis logic is verified against known-correct answers before it's
trusted on real, noisy GPS data.
Once you have a real Unipro TSV:
source .venv/bin/activate
python -c "
from telemetry.parser import load_sessions
sessions = load_sessions('path/to/your_export.tsv')
for s in sessions:
print(s.session_id, len(s.df), s.df['Lap Number'].nunique())
"
If that runs without error and the lap counts look right, upload the file through the web app and check the lap time table and RPM/speed ranges look plausible for your class before trusting the deeper analysis.
source .venv/bin/activate
python -m pytest
The suite runs against tests/fixtures/synthetic_session.tsv, a fixture
generated by tests/generate_fixture.py from a simulated oval track. It
reproduces the confirmed structural quirks (sparse single-channel rows,
nanosecond time, mid-file session reset, an injected "spin" lap ~3.7x the
median lap time) so the parser and analysis modules are exercised
end-to-end. Regenerate it with:
PYTHONPATH=. python tests/generate_fixture.py
telemetry/storage.py persists parsed sessions and kart setup snapshots
to a local SQLite database (data/sessions.db, lap summaries + a pickled
cache of each session's raw dataframe, plus a timestamped setup history) so
values and trends don't require re-entering/re-uploading every visit. The
app wires this in automatically: whichever session you're actively
analyzing gets saved (not every session in a multi-session file up front --
that pickles a full raw dataframe per session, which was slow enough on an
11-session real file to block the very first render for minutes), and each
session's kart setup is pre-filled from the last one saved for that
specific session (not from whatever session you looked at previously --
see "Filling in the kart setup" above). Both are browsable from the
History view.
Important caveat: by default, this storage lives on local disk, which
is wiped on every redeploy/reboot on most container hosts -- so treat it
as a within-deploy convenience, not durable long-term history, unless
SUPABASE_DB_URL/DATABASE_URL is set (the production setup -- see
"Row Level Security" below), in which case this same storage is backed by
Postgres instead and survives redeploys.
Ingest files from the command line -- e.g. from a script triggered after a race-day upload -- with:
python scripts/ingest.py session1.tsv session2.tsv \
--driver "Your Name" --track "Track Name" --session-type practice \
--db data/sessions.db
Every table this app uses (sessions, laps, kart_setups,
corner_metrics, pattern_instances, users, driver_profiles, teams,
team_memberships, attribution/claim tables, auth_tokens/auth_sessions,
email_outbox) has
a Postgres-backed sibling implementation (Supabase* classes in
telemetry/storage.py, accounts.py, auth.py, mailer.py), selected
automatically over the local SQLite classes whenever SUPABASE_DB_URL (or
DATABASE_URL) is set -- see telemetry/db.py and each module's
*_from_env factory. Nothing about this is opt-in per-feature: setting
that one environment variable moves the whole storage layer, not just
telemetry or just accounts. This is the production path -- the web app and
worker both run against Supabase; the local SQLite classes exist for
offline/single-machine use (see "What's intentionally not covered here"
below).
Why Postgres/Supabase at all, rather than just SQLite everywhere: a
plain SQLite file lives on local disk, which is wiped on every
redeploy/reboot on most container hosts -- durability alone would call
for any managed Postgres. Supabase specifically also means the data is
directly queryable over its REST API (PostgREST) with an official Swift
client, which matters for a native mobile app built against the same data
(see "Row Level Security" below for what that implies), and it's what
web/'s own auth and RLS-scoped queries are built on.
supabase/migrations/0001_init.sql to it -- either
supabase db push (Supabase CLI, linked to the project), or paste the
file into the project's SQL Editor and run it once. This is the single
source of truth for the schema; the Postgres-backed Python classes do
not create tables themselves (unlike their SQLite counterparts) --
see the migration file's own header comment for why.SUPABASE_DB_URL to the project's Postgres connection string
(Project Settings > Database > Connection string -- use the pooled
connection string in production, since Supabase's pgbouncer sits in
front of it).SUPABASE_URL/SUPABASE_ANON_KEY (see "Auth and
email configuration" above) to move auth over to Supabase Auth (GoTrue)
at the same time -- the two are independent switches, so the database
layer can move to Supabase first, with auth staying on the local
provider, if you'd rather stage it.SUPABASE_DB_URL=... python scripts/migrate_sqlite_to_supabase.py --sqlite-db data/sessions.db
once, with the schema from step 2 already applied. It preserves row ids
across the copy (so every foreign key still points at the right row
afterwards) and re-encodes each session's cached raw dataframe from its
local pickle file into the new session_cache table as Parquet (see
below for why not pickle).? placeholder becomes a Postgres %s one, and every
col IS ? null-safe comparison becomes col IS NOT DISTINCT FROM %s
(Postgres's bare IS only accepts a literal NULL/TRUE/FALSE, not a
bound parameter).INTEGER PRIMARY KEY AUTOINCREMENT becomes
BIGINT GENERATED ALWAYS AS IDENTITY, with RETURNING id replacing
SQLite's cursor.lastrowid.SessionLibrary.save_session/load_session wrote/read
session_cache/session_<id>.pkl on local disk, referenced by
sessions.cache_path -- exactly the kind of local-disk state that
doesn't survive a redeploy, and the actual motivation for this whole
migration. The Postgres-backed SupabaseSessionLibrary instead stores it
directly in a new session_cache table, Parquet-encoded rather than
pickled: Parquet is a safe, cross-language columnar format a
non-Python client could also read, whereas unpickling arbitrary bytes is
a code-execution surface once that blob is reachable over a network
rather than only ever written by this one process.The worker's own Postgres connection (SUPABASE_DB_URL) is expected to
use a role that bypasses Row Level Security (Supabase's default
service-role/postgres connection) -- accounts.PUBLIC_VISIBILITY_SQL
is already applied explicitly in every query the Python data-access
classes run, so RLS would just be a redundant second check for that one
process.
It is not redundant for anything else that talks to the database
directly -- which is exactly what web/ does (querying Supabase's
PostgREST API with a user's own JWT, on the authenticated role, never
the service-role one) and what a native mobile client would do too.
supabase/migrations/0001_init.sql
includes a first-pass set of RLS policies for that case (mirroring
PUBLIC_VISIBILITY_SQL for sessions, and narrower rules for laps,
session_cache, corner_metrics, pattern_instances, driver_profiles,
and users), keyed off a current_app_user_id() helper that maps
Supabase Auth's auth.uid() to this schema's internal integer user id via
users.external_auth_id.
Those policies are now tested rather than assumed. They shipped with
0001 but had never gated a single real request -- the Python app connects on
a role that bypasses RLS, and no account had an external_auth_id, so
current_app_user_id() always returned NULL and every policy silently
evaluated to "deny". tests/test_rls_policies.py exercises each one as a
real client would (the authenticated role, a real JWT claim, Supabase's own
default grants reproduced by supabase/testing/simulate_supabase.sql), and
found two classes of problem that 0002_rls_hardening.sql fixes:
auth_tokens
(password-reset and email-verification tokens), auth_sessions (live
login tokens) and email_outbox (every message body, reset links
included). In Supabase that is not "closed by default": anon and
authenticated hold default grants on every table in public, and RLS
only ever restricts an existing grant. Any authenticated user could
have read every other account's reset token the moment a browser client
pointed at PostgREST. Those three are now RLS-enabled with no policy at
all (deny-all) plus an explicit REVOKE, so they are reachable only from
the service-role connection the app and worker use.FOR SELECT, so no client write was possible --
a driver could not change their own session's visibility, nor ask to join
a team. 0002 adds exactly the writes the app performs and no more;
creating sessions stays server-side, and a client cannot grant itself
active team membership.Two operational notes:
0001_init.sql in place
is what broke logins once already (commit 82e3dba). Apply the whole
supabase/migrations/ directory in filename order -- re-running 0001
alone would revert 0002's hardened sessions_select.python scripts/verify_supabase_auth.py signs a
throwaway account in through real GoTrue, confirms the local users row
gets linked, and checks RLS answers correctly for that account's own JWT.SupabaseAuthProvider populates users.external_auth_id, and that column is
the only thing connecting a Supabase identity to this schema. An account
without it authenticates perfectly happily -- the Python app bypasses RLS, so
nothing complains -- while every policy keeps resolving it to NULL and denies
it everything. The symptom is a client that is definitely signed in and
definitely sees no data, with no error raised anywhere.
So the first Supabase sign-in of an account created under the old local
provider backfills that column (_mirror_user), which is what carries
existing accounts across. A second Supabase identity claiming an
already-linked account is refused rather than resolved (AuthMirrorConflict)
-- either outcome would silently hand one driver's telemetry to another.
The local SQLite classes (SessionLibrary, AccountLibrary, AuthStore,
OutboxEmailSender) are untouched and still exist -- this is a second
backend selected by an environment variable, not a replacement. That
matters for this project specifically because working with no network at
all, at the track, is an explicit design goal (see LocalAuthProvider's
docstring) -- a Postgres-only storage layer would break that for a
single-machine/offline install. Deploying with SUPABASE_DB_URL set is the
recommended production setup; running with neither SUPABASE_DB_URL/
DATABASE_URL nor SUPABASE_URL/SUPABASE_ANON_KEY set still works
exactly as it did before, fully offline.
This repo's original prototype was one Streamlit process that did everything: served the UI, parsed uploads, and talked to the database. That put an ~18-second CPU-bound parse inside a web request, which is the one thing a serverless frontend host cannot do -- the reason it was split into three pieces that each do one job:
| Piece | Where | Holds |
|---|---|---|
web/ -- Next.js app | Vercel | Supabase URL + publishable/anon key (both client-safe) |
| Supabase project | unchanged | schema, auth, storage |
worker/ -- ingest worker | any container host (Railway/Render/Fly) | service-role key |
The split is drawn along the service-role key: the frontend never has one,
so every query it makes runs as authenticated and is bounded by the RLS
policies in supabase/migrations/0002_rls_hardening.sql. The worker has
one, bypasses RLS, and is never reachable from a browser.
The retired prototype could hand an uploaded file's bytes straight to
load_sessions() in the same process. A serverless frontend can't do
that handoff implicitly, because a real Unipro export is tens of MB and
~900k rows -- past typical serverless request-body limits, and far past a
sensible function timeout -- so it's explicit instead:
browser Vercel Storage worker
| POST /api/uploads/presign | | |
|---------------------------->| createSignedUploadUrl |
|<-- path + token ------------| | |
| PUT the file (direct) ----------------------->| |
| POST /api/uploads/confirm | | |
|---------------------------->| INSERT upload_batches (pending) |
| GET /api/uploads/confirm?batchId= (poll) | |
| | claim (FOR UPDATE SKIP LOCKED)
| | |<-- download -----|
| | parse + write sessions/laps
|<-- complete | failed -------| | mark complete|
The file never passes through a Vercel function. The server, not the
client, chooses the storage path (<auth uid>/<uuid>.tsv), and the storage
policy independently enforces that the first path segment is the caller's
own uid -- so a client that invents its own path still cannot write into
someone else's folder.
upload_batches is a queue table, not a status field on sessions: one
uploaded file legitimately produces 11 sessions, and it has to be possible
to fail before any session exists. RLS lets a client insert only
status='pending' rows owned by itself and gives it no UPDATE policy at
all, so a client cannot mark an unparsed file complete or stall the queue
(tests/test_rls_policies.py, "the upload queue").
web/ so far/ (Home -- sessions grouped by driver, filters, sortable columns, inline
edit of type/track/conditions/visibility, delete), /sessions/[id] (Lap
Analysis, plus /sessions/[id]/engine for engine analysis), /tracks
(every track in scope with its best/average lap by engine class, and a
per-track page with a personal-best highlight, an all-drivers podium, a
team podium and a track map -- all filterable by class; see
supabase/migrations/0011_track_leaderboards.sql), /teams (create,
join, approve, promote/demote, remove, transfer manager -- see
supabase/migrations/0010_teams_management.sql), /upload, /login,
and /admin. Everything else the retired Streamlit prototype had (Top 3
Focus Areas, Corner/Lap Comparison, Recurring Patterns, Gearing
Simulation, Consistency/Progression, Kart Setup, History, Shared Laps,
Find My Profile -- see the sections above for what each one does) hasn't
been rebuilt here yet; the nav deliberately links only to routes that
exist rather than to a 404.
Lap Analysis reads the tables 0005 added, not the Parquet blob -- which is what makes it possible in a browser at all. Summary cards, a lap table with per-sector splits, per-lap exclusion, and lap selection for the comparison charts that come next.
The sector count is the driver's choice, 3 to 8, defaulting to 4, and it
drives the theoretical best. Sectors are built by grouping whole
corner/straight segments (src/lib/sectors.ts), never by cutting the lap
into equal distances: a split halfway round a corner makes a driver who
takes a different line look slower in one sector and quicker in the next,
for no reason. Where there is a choice, boundaries prefer the start of a
straight, so each corner stays whole inside one sector. npm test in
web/ covers that partitioning.
Home's scope is narrower than what RLS permits, on purpose: your own sessions, plus the whole roster's if you manage or admin a team. RLS would also let you read every publicly shared session in the app, but showing those here would bury your own -- they belong on Leaderboards and Shared Laps. The policies are what make it safe; the scoping is what makes it useful.
session_cache.dataframe_parquet holds each session's whole parsed
dataframe as BYTEA -- 3-5 MB per session, ~46 MB per track day, against a
500 MB database. That caused two problems which turned out to be one
problem: roughly ten track days fills the database, and Lap Analysis
could not be built on the Next.js frontend at all, because every trace,
delta and sector time it draws lived inside that blob and a browser cannot
read Parquet out of BYTEA. laps held only lap number, lap time and the
outlier flag.
0005_analysis_tables.sql writes the analysis down as ordinary rows:
| Table | Holds |
|---|---|
session_analysis | one row per session: best lap, theoretical best, the segment map, per-segment session bests, setup suggestions, lap summary |
lap_segment_times | every lap's time in every segment |
lap_traces | every lap's distance / elapsed-time / speed / RPM / lat / lon / braking / power-on, as parallel float8[] arrays |
Traces are arrays per lap, not a row per sample: a lap is ~310 GPS fixes, so row-per-sample would be ~6k rows per session and tens of millions across a season, always read back whole and in order.
lap_traces.lap_time_s is elapsed time within the lap rather than a
precomputed delta against some fixed reference. That is what lets the delta
trace between any two laps be derived by interpolating both onto a common
distance grid -- so changing the reference lap needs no new data.
Measured on the bundled export: ~335 KB of rows against a 5.4 MB blob,
6% -- and tests/test_analysis_store.py fails if that ratio ever stops
being a clear win. The worker now analyzes at ingest, so new uploads land
with all of this already stored (best-effort: a session whose analysis
raises is still saved, since a backfill can recompute it).
python -m scripts.backfill_analysis # report only
python -m scripts.backfill_analysis --analyze # write the rows
python -m scripts.backfill_analysis --analyze --archive --clear-blobs --vacuum
--clear-blobs refuses to touch a session whose raw data it cannot account
for. Raw data survives two ways: an uploaded session's original TSV is
still in the telemetry bucket (uploads are never deleted), so it can be
re-parsed; and --archive copies any other session's Parquet to Storage,
recording it in session_cache.raw_storage_path. A session with neither is
skipped and reported -- clearing it would make the derived rows the only
copy, and re-running with a corrected corner threshold (which
telemetry/corners.py documents as likely) would then be impossible.
--vacuum is what actually hands the space back: clearing a blob only
marks the row version dead, and the file on disk keeps its size until the
table is rewritten. It runs from the script rather than the Supabase SQL
editor because that editor wraps statements in a transaction, and VACUUM
cannot run inside one (ERROR: 25001). It takes an exclusive lock, so run
it when nothing is uploading.
scripts/ ships in the worker image, so the whole thing runs as a one-off
on the worker service, which already holds the database URL and the
service-role key.
telemetry/rating/ -- an iRating-style competitive number kept strictly
separate from a purely-motivational streak/activity layer:
validity.py): every lap becomes verified,
flagged (implausible time, possible track cut against a consensus GPS
line, or a speed drop with no recorded braking -- held for review, never
silently trusted or dropped), or excluded (reuses the existing
in/out-lap and statistical-outlier detection, plus a driver's own
exclusion). Only verified laps count toward anything below.elo.py): a session is scored against
the other drivers who drove the same track/date/conditions/class that
day (a "cohort"), not an absolute reference time -- every pairing in the
cohort is a virtual head-to-head match, Elo's expected-score formula says
who "should" have won it given current ratings, and mu moves by the
actual-vs-expected gap, averaged across the cohort. A day with no
comparable field falls back to a nightly-refreshed historical reference
bucket (median of the top few verified sessions in that track+class+
conditions bucket) treated as a single virtual opponent. Every
contributing session is recorded in driver_rating_history, tagged
field or reference, for the "why did my rating move" breakdown on
Home.mu - k*sigma, computed at request time in web/src/lib/rating.ts --
nothing writes a "current" sigma to the database, since it would only
ever be advancing a clock.streaks.py): a Monday-start week "counts" with at
least one verified session; a single missed week consumes a banked freeze
and survives, two consecutive misses always breaks it. Architecturally
separate from mu/sigma -- the only coupling is the reduced same-week
sigma-shrink weight above.Every threshold (cohort minimum size, Elo K-factor, sigma shrink rate,
implausible-lap floor, track-cut tolerance, streak freeze rules, decay
grace window) lives in telemetry/rating/config.py (batch-side) or
web/src/lib/rating.ts (display-side decay), not hard-coded -- these are
reasoned starting points meant to be tuned once there's real multi-driver
data to watch, not a guess to lock in.
python -m scripts.compute_ratings # run once
python -m scripts.compute_ratings --recheck-all # also re-run Part 1 on every lap
python -m scripts.compute_ratings --loop # poll forever (default: daily)
Two triggers feed the exact same batch, not two different code paths:
worker/processor.py calls it directly, best-effort, right after a
batch's sessions are saved (see _recompute_ratings_best_effort --
same "never fail an otherwise-successful upload" shape as
_link_to_batch), so a driver sees their own upload reflected without
waiting on a schedule, and so an upload that gives a previously-solo day a
real field to compare against doesn't sit unrated until the next scheduled
pass. The backstop that keeps the historical reference buckets and
reference lines from going stale on a day nobody happens to upload
anything is .github/workflows/daily-rating-recompute.yml -- a scheduled
GitHub Actions workflow (03:00 UTC, plus a manual workflow_dispatch
trigger) that just runs python -m scripts.compute_ratings once, reusing
the SUPABASE_DB_URL repository secret already set up for
build-windows-installer.yml. No standalone host/process to deploy for
this: the --loop mode above still works identically if you'd rather run
it as a long-lived process somewhere (a platform-native cron alongside the
worker, for instance), but isn't required.
At today's data volumes (a club's worth of sessions, not a national
championship's) a full pass is a ~10 second job -- cheap enough to pay on
every upload rather than needing an incremental version; see
_recompute_ratings_best_effort's docstring for where that would go if
the data volume ever demands it.
driver_ratings (leaderboard: /rating) and the two
track_reference_lines/track_pace_reference tables are readable by any
authenticated driver; driver_rating_history/driver_weekly_activity/
driver_streaks are a driver's own only (plus admin) -- none of the six
tables has a client write policy at all, since only the service-role
connection either trigger runs on ever writes them.
Supabase Auth identifies people by UUID; this schema keys everything off an
integer users.id and bridges the two with users.external_auth_id. Every
RLS policy resolves the caller through that bridge
(current_app_user_id()), so an account without it authenticates perfectly
and is invisible to every policy -- signed in, sees nothing, no error
anywhere.
That mirroring used to live only in Python (telemetry/auth.py's
_mirror_user), which meant it only happened for a client that actually
called it. 0004_mirror_auth_users.sql moves it to a trigger on
auth.users, so it happens for every client -- the Next.js app,
unigo_sync, and the iOS app later -- and creates the driver profile
alongside it. Registration details (display name, date of birth,
guardian email) travel as GoTrue user metadata, which is also how the
under-16 guardian-consent rule keeps applying to signups that never touch
Python. _mirror_user itself is still live code, just no longer the only
path: it's what unigo_sync's own login still calls on a Supabase-backed
account (SupabaseAuthProvider.login/register), since that tool talks
to telemetry.auth directly rather than through GoTrue's JS client the
way web/ does.
Accounts created before that trigger existed still have a NULL
external_auth_id. They repair themselves on their next sign-in that
goes through _mirror_user, or can be linked in one go with
supabase/manual/0004_backfill_external_auth_id.sql. That script is
deliberately not a migration: it is the one part of this work that
writes to rows already in the live app's tables, so it is opt-in and shows
you what it would do before it does it.
# Frontend
cd web && cp .env.local.example .env.local # fill in URL + publishable key
npm install && npm run dev # http://localhost:3000
# Worker (needs the service-role key; never put it in web/.env.local)
export SUPABASE_DB_URL=postgresql://... # pooled connection string
export SUPABASE_URL=https://<project>.supabase.co
export SUPABASE_SERVICE_ROLE_KEY=sb_secret_...
python -m worker.main # or WORKER_ONCE=1 to drain and exit
The worker deploys from worker/Dockerfile, built with the repo root as
context (it needs telemetry/ as well as worker/). Its dependency list
is deliberately separate from the repo-root requirements.txt (see
worker/requirements.txt's own header comment) -- so if a UI library
import ever creeps into telemetry/, the worker image stops building
rather than quietly shipping one anyway
(tests/test_analysis_extraction.py::test_analysis_module_imports_no_ui_libraries
asserts the same thing at the source level).
supabase/migrations/. 0003 adds the queue table
and the private telemetry bucket; 0004 adds the signup trigger. Both
are additive. The one thing that touches existing rows,
supabase/manual/0004_backfill_external_auth_id.sql, is deliberately
not in migrations/ -- run it by hand once you've looked at what it
would link.web, and set
NEXT_PUBLIC_SUPABASE_URL and NEXT_PUBLIC_SUPABASE_ANON_KEY. Nothing
else. If you ever find yourself adding a service-role key here, stop:
that key bypasses RLS and NEXT_PUBLIC_-prefixed variables are inlined
into the browser bundle.worker/Dockerfile with the repo root as build
context, and set SUPABASE_DB_URL (pooled), SUPABASE_URL and
SUPABASE_SERVICE_ROLE_KEY. One instance is enough; claiming is atomic
(FOR UPDATE SKIP LOCKED) so a second one needs no coordination.tests/test_worker.py runs the whole loop against a local Postgres using
the bundled 82 MB export, and asserts the 11 real sessions land with their
laps and cached dataframes -- including the failure paths (missing object,
wrong file, one bad batch not stopping the queue) and re-upload
deduplication.
web/ yet -- see "What exists in
web/ so far" above for the current page list, and the earlier
sections of this README for what each retired page did.web/ yet.web/ again.telemetry/corners.py) are reasonable
defaults tuned against the synthetic fixture; they may need adjustment
for a real track's actual corner geometry and GPS noise characteristics.sample_data/default_session.tsv (a real telemetry file, committed
intentionally so the test suite and verification scripts have a real
export to run against -- see sample_data/README.md) means this repo
should not be treated as private even if it isn't in your history yet;
that GPS track, lap times, and RPM data are visible to anyone who can
see the repo while it stays public.scripts/compute_ratings.py now runs on every upload (worker/ processor.py's _recompute_ratings_best_effort), so new sessions get
rated without any manual step, and the daily backstop is
.github/workflows/daily-rating-recompute.yml (reuses the SUPABASE_DB_URL
secret already set up for build-windows-installer.yml, no new hosting
needed) -- so as long as that secret is set on the repo, both triggers
are live with nothing further to deploy. Migrations 0015 and 0016
still need applying to the real Supabase project, same as every
migration noted above them in this list.telemetry/comparison.py::corner_comparison_across_sessions).telemetry/simulation.py for the method, and the "How
this estimate works" expander in the view itself for the same caveats
in-context).