Inspiration
In Cap-Haïtien's markets, the machann who sells rice and beans keeps her whole business in a school notebook: sales on one page, what the neighbours took on credit on another. About 2 billion people work like her in the informal economy (ILO). Two things go wrong every week:
- Credit given to neighbours gets forgotten, and money quietly disappears.
- There is no written history, so when a microfinance officer asks "show me your sales", there is nothing clean to show.
The notebook is also private: income, debts, the names of people who owe money. And market signal is often weak. An app that uploads every page to a server is the wrong answer twice. On-device AI is the right one: the device reads the page, and the cloud is used only when it adds something, with only what it needs.
Kaye means notebook in Haitian Kreyòl. Our tagline: Kaye ou, ki konn kalkile, "your notebook, now it can count."
What it does
- Snap a page. Photograph a handwritten notebook page or a receipt. Kaye turns it into sales, costs, credit and payments, with item, quantity, price and amount.
- Check the math. Every line and the page total are verified before anything is saved.
- Quick note. "Vann 4 mamit diri 250 chak. Madan Pòl dwe 1200." becomes lines, in Kreyòl, French, English or Spanish.
- Who owes me. A running credit list, with a polite reminder the vendor can share.
- Month record. Sales, costs, profit, sales by week, best sellers, money owed, and a short factual summary for a lender. Kaye keeps records. It never gives financial advice.
- Works with no signal. The notebook lives in Firestore's offline cache and syncs when signal returns.
- Every AI result says where it ran and why, with
inferenceSource, the time it took and the routing reasons.
How we built it
Hybrid routing with Firebase AI Logic
The same design runs on Android (Kotlin + Jetpack Compose, firebase-ai + firebase-ai-ondevice) and on the web (Chrome, the hybrid getGenerativeModel with onDeviceParams and inCloudParams):
| What | Mode | What leaves the device |
|---|---|---|
| Notebook photo | ONLY_ON_DEVICE, Gemini Nano with image input |
Nothing |
| Photo Nano can't read, or the math fails | Asks first, then ONLY_IN_CLOUD |
That one photo, only after "Send this photo" |
| Quick note in a Nano language | ONLY_ON_DEVICE |
Nothing |
| Kreyòl / French note, or no local model | ONLY_IN_CLOUD |
The note with names and phones replaced by C1, [phone] |
| Month record | PREFER_ON_DEVICE / PREFER_IN_CLOUD by language |
Totals only, never entries |
| Offline, no model | Offline rules in 4 languages | Nothing; Firestore syncs later |
The math is the judge
A small model reading handwriting can be wrong, and with money a wrong "250" is worse than no answer. So Kaye never trusts the read: it verifies it. For each line with quantity $q_i$, unit price $p_i$ and amount $a_i$, and for the total $T$ written on the page:
$$ |q_i\,p_i - a_i| \le \max(1,\ \tau a_i) \qquad\text{and}\qquad \Big|\sum_{i \in \text{sales}} a_i - T\Big| \le \max(1,\ \tau T) $$
with tolerance $\tau = 0.02$, set in Remote Config. A page that adds up never needs the cloud. A page that doesn't shows the vendor exactly which number is off, and only then offers the cloud, with consent.
Other Firebase products, each doing real work
- Cloud Firestore: the notebook itself, offline-first on both platforms, locked to its owner by security rules with field validation.
- Authentication: anonymous "Try it now" (no sign-up wall at a market stall), email/password, and linking an anonymous notebook to an email later.
- Remote Config: cloud model, on-device languages per platform, the tolerance $\tau$, the cloud-photo switch and max lines, all changed without a new build.
- Analytics:
ai_call(feature, source, ms, mode) andentries_saved. Never entry text, names or amounts; ad signals off. - App Check: Play Integrity on Android and reCAPTCHA Enterprise on the web, enforced for AI Logic.
No backend of our own: Firebase is the backend. The web app is a static PWA with a service worker, bundled with esbuild.
Challenges we ran into
- A photo can't be redacted. Text can have names replaced; a photo can't. So photos never go to the cloud silently: a dialog explains why and asks.
- Small-model slips. On-device, Gemini Nano sometimes listed the page's "Total" row as a sale, or called a sale a cost. We fixed both with rules, not trust: a "Total" row becomes the page total, and when only the sum of all lines matches the written total, the lines are counted as sales and the trace says so.
- Short on-device output on Android. JSON for six lines can exceed Nano's validated output length, so on Android the device gets a compact
kind|item|qty|price|amount|customerline format, while the cloud gets a JSON schema. - A hidden CSP trap. On-device image input failed with "Failed to fetch": the SDK converts inline images through
fetch("data:..."), which our strict Content-Security-Policy blocked. Allowingdata:inconnect-srcfixed it. - Kreyòl. Nano isn't validated for Kreyòl, the language of the market. Offline rules understand Kreyòl instantly ("vann 3 diri 250 chak", "Madan Pòl dwe 1200"), and the cloud gets Kreyòl notes only with names replaced, then puts the names back on the device.
- Real hardware. Our test phone, a Galaxy S24 Ultra, doesn't support the Prompt API (AICore error 606), so on Android we verified the consent-then-cloud path. On a CPU-only laptop with no graphics card, Chrome's Gemini Nano read a full handwritten page on-device in about a minute, with every line correct and the total verified.
Accomplishments that we're proud of
- A handwritten sales page read entirely on-device in Chrome: 4 lines with quantities, prices and amounts, the page total of 2,100 verified, nothing sent.
- A page with a wrong total caught by the arithmetic check, with the vendor asked before any cloud call.
- A Kreyòl note with 2 customer names hidden before the cloud and restored after, in about 1–2 s.
- A monthly record written from totals only.
- Six Firebase products carrying the whole app, tested live on the web and on a real Android phone.
What we learned
For money, where the AI runs is only half of trust; the other half is verification. Hybrid inference plus a cheap deterministic check lets the device do almost everything, and turns the cloud into a rare, explained and consented exception. We also learned to design for the hardware people actually have: a slow on-device read with nothing uploaded is still worth more than a fast one that leaks the notebook.
What's next for Kaye
- A pilot with vendors at Marché Cluny in Cap-Haïtien and a microfinance partner.
- Kreyòl on-device as soon as Gemini Nano supports it: one Remote Config change, no new build.
- Shared notebooks for family businesses, and a monthly summary ready to share on WhatsApp.
