Portal scrape → clean → load
Playwright signs into a legacy portal, extracts what the team used to copy by hand, normalises it, and loads it to the operational DB.
What wasn't working
Legacy portal had no API and rate-limited screen-scraping aggressively.
Ops team copied 200+ records per day by hand, introducing typos into billing.
What we needed to be true
Automate the full sign-in → extract → normalize → load loop.
Cut manual work to zero for the standard case, keep humans in the loop for exceptions.
How it was built
Resilient sign-in
Playwright handles the multi-step login with a retry loop and a persistent session cache to avoid re-auth on every run.
Extraction + validation
Structured extractors per record type, Pydantic validation, and a diff against the last snapshot to skip unchanged rows.
Load + reconciliation
Records are upserted into Postgres inside a transaction; anomalies land in a review queue with the raw HTML for context.
How the pieces connect
Implementation timeline
- Phase 01
Reverse engineering
Mapped every screen and form the ops team touched.
- Phase 02
Worker + scheduler
Containerized Playwright worker on a Docker host, cron-triggered every 15 minutes.
- Phase 03
Observability
Structured logs, per-run screenshots on failure, Slack digest of anomalies.
Stack
What ships to the user
Session persistence
Sign-in only happens on cache expiry — every other run reuses cookies.
Anomaly review queue
Records that fail validation land in a queue with raw HTML + screenshot.
Before → after
20+ hours/week copying by hand
0 hours in the standard case, ~1 hour/week reviewing exceptions
Business impact
Let's map your workflow and see what's worth automating first.
Lessons learned
Browser automation is only reliable when you build the observability first. Screenshots on failure are non-negotiable.
Future improvements
Add a lightweight web UI for the ops team to reprocess anomalies without engineering involvement.
Common questions
More work like this
Research → draft → review → publish
4× editorial output with the same headcount.
Inbound lead → qualified opportunity
Cut lead response time from hours to under 2 minutes.
Have a workflow that looks like this?
Bring me one real bottleneck. I'll tell you whether it's worth automating, and what it would take.