Voice, Context & Control: The Three Pillars of Useful AI Email

Gmail’s AI email assistant writes like a committee of lawyers designed it.

Pete Koomen’s recent post Horseless Carriages explains why: developers control the AI prompts instead of users. In his post he argues that software developers should expose the prompts and the user should be able to control it.

He inspired me to build my own. I want a system that’s fast, accounts for historical context, & runs locally (because I don’t want my emails to be sent to other servers), & accepts guidance from a locally running voice model.¹

Here’s how it works,

1. I press the keyboard shortcut, F2.
2. I dictate key points of the email.
3. The program finds relevant emails to/from the person I’m writing.²
4. The AI generates an email text using my tone, checks the grammar, ensures that proper spacing & paragraphs exist, & formats lists for readability.
5. It pastes the result back.

Here are two examples : emailing a colleague, Andy, & a hypothetical founder.

Instead of generics, the system learns from my actual email history. It knows how I write to investors vs colleagues vs founders because it’s seen thousands of examples.

The point isn’t that everyone will build their own email system. It’s that these principles will reshape software design.

• Voice dictation feels like briefing an assistant, not programming a machine.
• The context layer - that database of previous emails - becomes the most valuable component because it enables true personalization.
• Local processing, voice control, & personalized training data could transform any application, not just email, because the software learns from my past uses

We’re still in the horseless carriage era of AI applications.

The breakthrough will come when software adapts to us instead of forcing us to adapt to it.

¹ Centered around a command line email client called Neomutt.
² The software hits LanceDB, a vector database with embedded emails & finds the ones that are the most relevant from the sender to match the tone.
³ The code is here.