Sharpenly
Bilingual training where every AI-generated question passes a quality gate first.
An EN/ES learning app, a manager dashboard, and the LLM data-quality pipeline behind ~20,000 questions.
The source of the production app is private. This repo is a curated, read-only showcase:
screenshots, a data-quality case study, real sample questions, and trimmed code excerpts.
For AI data and LLM-evaluation reviewers: start here
I built and measured a pipeline that has an LLM write 11,870 bilingual (EN/ES) training questions, then rejects or escalates the weak ones against a written rubric.
- Rubric and gate: 15 written rules, enforced in code wherever a rule can check it, otherwise by human review. RUBRIC.md, gate_excerpt.py
- Measured iteration: first pilot accepted 14% of items; after fixing the prompt and gate, a targeted repilot accepted 84%. Case study
- Failure analysis: a batch reported 617 of 617 requests "succeeded" and produced zero usable questions. Postmortem
- Bilingual QA: English and Spanish answers must line up by position. validators_excerpt.py, 20 real samples
- Run it yourself:
python data-quality/run_checks.py (standard library only) validates the 20 samples and shows why the gate uses a proportional length rule: the strict rule flags 17 of 20 good items, the proportional rule flags 0.
- Honest limits: the gate's agreement with human judges is not yet measured. The case study says so and lays out the study I would run next.
Try it
Install in 30 seconds
| Device | Steps |
|---|
| iPhone (Safari) | Open the app, tap Share, tap Add to Home Screen. |
| Android (Chrome) | Open the app, open the menu, tap Install app. |
| Desktop (Chrome or Edge) | Click the install icon in the address bar. |
The app works offline after the first load.
At a glance
| What | Number |
|---|
| Bilingual EN/ES questions | 20,308 in 71 bank files |
| Study lessons | 168 across 21 categories |
| Difficulty levels per track | 8 |
| App releases shipped | 174 |
| Python test functions | 458 |
| Cloud Functions | 12 |
| Commits | 730+ |
| Questions the AI pipeline generated in its full batch | 11,870 |
| Accepted by the automated gate / sent to human review / rejected | 10,356 / 451 / 1,063 |
| First pilot acceptance, then after the fix | 14% (7/50), then 84% (21/25) |
For managers
The manager dashboard shows who is trained, who is not, and where the skill gaps are. Try it without logging in: https://sharpenlyapp.com/manager-preview.
| Feature | Status |
|---|
| Alerts and KPI tiles | Live |
| Skills-gap heatmap (employees by topic; 9 topics in the demo) | Live |
| Training compliance | Live |
| Employee drill-down | Live |
| Team join code, rotate, deactivate | Live |
| CSV export | Live |
| Candidate invite links | Live |
| Candidate report card with fit rating | Live |
| Integrity trust signals (aggregate tab-hidden and paste counts, shown as "review recommended") | Live |
| Proctored stage 2 exam | Live |
| Convert a candidate to an employee | Live |
| Leaderboard | Live |
| English and Spanish, light and dark | Live (the public preview is English only) |
| Auto-remediation assignments | Demo only |
| Stripe billing | Coded, not live |
Screens show the public demo shop with sample data; the names are fictional.
For employees
Short quizzes, scenario questions, and lessons built from four typed section primitives: flashcard, hotspot, sequence, scenario. Progress, streaks, and a ranking keep people coming back.
For candidates
A manager sends an invite link. The candidate takes a timed assessment. The manager gets a report card.
The AI content pipeline
Questions are generated from a curated corpus of automotive reference material. Nothing reaches a learner until it passes a rubric gate and a human check.
flowchart LR
A[Curated corpus of automotive reference material] --> B[Vector index<br/>MiniLM embeddings]
B --> C[Role-filtered retrieval]
C --> D[LLM generation<br/>Claude Sonnet 4.5]
D --> E{Rubric gate}
E -->|accept| G[Bilingual QA]
E -->|review| F[Human review]
E -->|reject| X[Discarded, reason logged]
F --> G
G --> H[8-level staging]
H --> I[App]
- Retrieval uses Chroma with
sentence-transformers/all-MiniLM-L6-v2, filtered by role so a service-advisor question is grounded in service-advisor material.
- Each item must cite the passage it came from. Citations are stripped from this public repo.
- The gate checks structure, trivia, recall-style stems, theory that does not belong on a shop floor, answer-length giveaways, and near-duplicates by embedding similarity.
- A run only counts as a success if the output parsed and questions survived the gate. See the postmortem below.
Data-quality work (what this proves)
Headline result: first pilot accepted 14% of items. After tightening the prompt and the gate, a smaller, targeted repilot accepted 84% with 0 rejected, and cost per kept item fell about 4.4x. A later batch where the API reported 617 of 617 requests "succeeded" produced 0 usable questions. That led to one rule: product success, not API success.
Engineering
| Layer | Stack |
|---|
| App | Installable single-file web app (PWA) with offline support, Firebase Realtime Database and Auth, Vercel |
| Site and dashboard | Next.js 16, TypeScript, Tailwind CSS 4, next-intl (EN/ES), Playwright and axe-core |
| Backend | 12 Firebase Cloud Functions (Node 22) |
| Content pipeline | Python, Chroma, sentence-transformers, Claude Sonnet 4.5 |
- Testing: 458 Python test functions, Playwright end-to-end tests, visual-regression baselines.
- CI: GitHub Actions for tests, a pre-release launch gate, and a scheduled security scan.
- Security: audits for XSS sinks, silent write failures, and public-data hygiene run before every release. A smoke test runs against the live deploy.
More detail in docs/ARCHITECTURE.md.
About me
Open to AI data annotation, LLM evaluation, and AI training-data roles.
Copyright (c) 2026 Santiago. See LICENSE.md.
