Alex Lewis
Senior Software Engineer · Zapier
Summary
Senior software engineer with 10+ years shipping production software across frontend, backend, platform, and automation. Current focus: applied AI — agent infrastructure, LLM evaluation, retrieval, tool use, and developer tooling — building the systems that make LLM-powered products dependable.
Skills
- Applied AI
- AI agents & orchestration · LLM evaluation (LLM-as-judge, ground-truth measurement) · Retrieval-Augmented Generation · Tool calling & MCP · Prompt architecture · AI observability
- Languages
- TypeScript · Python · JavaScript · SQL
- Frameworks & Platforms
- React · Next.js · Node.js · PostgreSQL · AWS · Vercel
- Practices
- CI/CD · API design · Software architecture · Production observability · Serverless
Experience
Senior Software Engineer Zapier Aug 2025 – Present
- Built an internal agentic end-to-end testing platform — agent runtime, CI integration, and a plugin system that decoupled it from any single app so agentic tests could run against many applications; benchmarked the agent at ~90% first-run pass rate across 50 runs.
- Designed evaluation infrastructure for an LLM-as-judge system: deterministic verdict guards plus a ground-truth measurement layer that cross-tabs eval verdicts against real merge outcomes to correct judge bias and verify verdicts track reality.
- Built an agent-orchestration harness with live permission prompts, an egress sandbox, self-supervision, and cost metering; cleaned it up and handed it off as a shared, contributable team repo.
- Built a semantic-release CI/npm pipeline so any repo could consume the AI testing agent via npx (adopted across 4 repos), and made agent session transcripts first-class test-proof artifacts for observability.
- Shipped inline app authentication to all non-enterprise users, removing auth friction from an AI assistant experience; built and evangelized a Claude Code plugin catalog for the org (demoed to 20+ engineers).
- Cross-team infrastructure fixes: a one-line change unblocking Sentry setup for every Next.js service in the internal deploy pipeline, and diagnosing a cross-service auth failure (Vercel Secure Compute + PrivateLink).
Senior Software Engineer Your Money Line Oct 2023 – Jun 2025
- Built a production Retrieval-Augmented Generation (RAG) system for personal-finance guidance, with chunking and prompt-engineering focused on accuracy and consistency on sensitive topics.
- Ran a lightweight grading/evaluation signal that informed which prompts the system used in production.
- Prototyped an LLM question-answering assistant (serverless) and built a generative-AI product feature.
- Built an AWS Textract document-AI pipeline for loan-payment processing, and a text-to-speech pipeline.
- Designed a serverless HRIS data-sync integration (AWS Lambda, API Gateway, Step Functions, SQS FIFO with idempotent dedupe, Aurora) supporting bulk import and near-real-time webhook delta sync.
Software Consultant Independent 2020 – 2021
Full role history is on LinkedIn.