Receipt Relay — Photo → email, via AI

01The story

Problem, approach, outcome

01Problem

AI extraction demos usually stop at the JSON. In real use the model is sometimes wrong, and a wrong answer delivered confidently is worse than an honest “not sure.” I wanted to see what it takes to put a vision model in front of a real deliverable, an email a customer would actually receive, without letting its mistakes through.

02Approach

One vision-model call turns the photo into a schema.org Order. The output is constrained to a JSON schema and validated before it reaches the UI, so a bad read surfaces as “couldn’t read this receipt,” never as a half-filled form. The model marks every field it’s unsure of, and an arithmetic check (line items → subtotal → total) catches missed lines instead of quietly “fixing” them. A correction screen puts the photo beside every field, so a person fixes what the model got wrong before the email renders.

The model is configuration, not code: one OpenAI-compatible adapter, prototyped locally on Qwen3-VL 8B in LM Studio, now running on hosted models through OpenRouter, with a measured comparison still to pick the production model.

03Outcome

Working end to end in private preview; public launch follows model testing.

02Overview

About Receipt Relay

Receipt Relay is applied AI that ends in something real. Photograph a paper receipt, invoice or ticket; a vision model reads it into structured order data; you check and correct it; and out comes a production-quality transactional email you can copy, download as an .eml, or send to your own inbox.

I built it to show AI extraction and production email working in a single loop. Most of the effort went into the part demos skip: what happens when the model is wrong. Crumpled, rotated, badly lit phone photos are the normal case, so the tool is built around structured output, per-field confidence, a cross-check on the math, and a human who signs off before anything is sent.

The email end is held to the same standard. There are two templates: Modern, built on the MIT-licensed ACORN email framework, and Paper, a line-by-line recreation of the printed receipt. Both carry dark-mode styles and preview at desktop and mobile widths in a sandboxed frame. Sending goes out from its own authenticated subdomain, so a demo can never touch the main domain’s mail reputation.

03Features

What it does

Phone-photo upload with in-browser downscaling, rotate and crop
Vision-model extraction to a validated schema.org Order, with per-field confidence flags
Card numbers cut to the last four digits on the server, before anything is shown or exported
Arithmetic check that flags missed line items rather than hiding them
Correction screen: every field editable, photo alongside with a zoom loupe, flags clear as you fix
Two templates (Modern on ACORN, and Paper) with light/dark and desktop/mobile preview
Export as HTML or .eml (optionally with the original photo), or send to your inbox via SPF/DKIM/DMARC-authenticated mail
Swappable model: changing provider or model is a config edit

04Gallery

A closer look

The correction screen: the receipt photo beside every extracted field, with the handwritten tip and total highlighted because the model was unsure of them — The check-and-correct step. The model was unsure of the handwritten tip and total, so they are flagged for a person to confirm.

The same receipt email shown in light mode and in dark mode, side by side — The Modern template, built on the ACORN email framework, in light and dark mode.

The Paper template: a line-by-line monospace recreation of a printed grocery receipt, including barcode and tax codes — The Paper template recreates the printed receipt line by line.

05Live samples

The real email, live

These are the actual HTML emails the tool produced from sample receipts, not screenshots. They follow your device’s light or dark mode.

Modern template · restaurant receipt with a handwritten tip

Paper template · grocery receipt (scroll inside the frame)

06Stack

How it’s built

Vision LLMs over an OpenAI-compatible API (Qwen3-VL 8B in LM Studio locally; OpenRouter hosted)
JSON Schema + schema.org Order as the extraction contract
HTML, CSS and vanilla JavaScript — no framework
PHP proxy: key handling, SSRF protection, rate limiting
ACORN email framework (MIT)
Resend on a dedicated, authenticated sending subdomain
Strict CSP and a sandboxed email preview

07Roadmap

Where it’s going

Shipped
End-to-end private preview
Photo to extraction, correction, two templates, export and sending.
Next
Model bake-off
Twenty real receipts, measured for accuracy, speed and cost, to pick the production model.
Next
Rendering checks in real inboxes
Gmail, Apple Mail and Outlook, light and dark.
Next
Public launch
Opening the tool at its own subdomain.
Exploring
JSON payload inspector
A tree view of the corrected order data, with validation state.

Hiring for applied AI work?

I build practical AI features that end in something people actually use. If that’s the kind of work your team needs, I’d like to hear about it.

Get in touch See all projects