Building a Governed AI Family Administrator

I wanted an administrator, not an inbox summarizer

The easiest version of this project would have been to give an AI access to Gmail and ask it to summarize what arrived.

That would have been useful for a few days.

Then the same problems would come back.

A summary is transient. A task is not.

A school registration deadline mentioned on Monday is still relevant on Thursday even if no new email arrives. A bill remains unresolved until it is paid or otherwise handled. A school event should stay on the calendar regardless of whether the original message is still fresh in the inbox.

So I designed the system around persistent state instead of repeated interpretation.

The basic flow became:

Incoming email

   ↓

Interpret what it means

   ↓

Route it to the right family workflow

   ↓

Persist the important state

   ↓

Create controlled actions when appropriate

   ↓

Surface only what actually needs attention

The inbox is the source of new information.It is not the family's long-term memory.

One family profile, multiple responsibilities

Rather than creating separate agents for every small family task, I built one Hermes profile called family-admin.

For now, the main family workflows are:

email administration

school administration

household bills

Each one has its own skill and policy.

The email workflow decides what incoming information is relevant and where it belongs. The school workflow decides what matters, what requires family action, and what should become a calendar event. The household-bills workflow understands the difference between a normal bill, an account already covered by autopay, an exception that needs attention, and an ordinary purchase receipt.

Behind those workflows are a few shared services: persistent family state, controlled calendar access, and scheduled automation.

Those pieces are important, but I think of them as infrastructure supporting the family workflows rather than separate family capabilities.

This gives me one family agent without one giant prompt trying to understand everything.

The AI reasons, but it isn't the authority

This is the pattern I care about most in the design.

Hermes reasoning

   ↓

Skill policy

   ↓

Deterministic helper

   ↓

External system

Hermes can reason about an email. It can conclude that a school message contains an actionable deadline, decide that a bill looks routine, or recognize that a calendar event would be useful.

But the model does not directly own Gmail, the family database, or the calendar. The skills define the rules, and deterministic helpers define what operations are technically possible.

That distinction matters. An AI deciding that an email is important is very different from an AI having unrestricted permission to send email, modify a mailbox, write to any calendar, or execute arbitrary commands.

I wanted the first. I deliberately avoided the second.

A dedicated family-admin inbox

I also didn't want the agent reading our normal family inbox. Instead, I created a dedicated Gmail account specifically for family administration. Selected messages are forwarded into it using normal forwarding rules.

The architecture looks like this:

Main family Gmail

   ↓

Selected forwarding rules

   ↓

Dedicated family-admin Gmail

   ↓

Hermes

That creates a very useful boundary.

Hermes only sees messages I intentionally route into this workflow. It does not get unrestricted access to unrelated family email.

The Google authorization is also deliberately narrow. The agent has the scopes required to read the dedicated mailbox, send controlled family briefings, and work with the permitted calendar workflow.

Even those permissions are further restricted by deterministic helpers.

For example, Hermes cannot simply decide who to email.

Family briefing recipients come from configuration. There is no arbitrary To: argument exposed to the model.

The helper also rejects mailbox-modification operations such as deleting, archiving, labeling, forwarding, or replying to messages.

The API might technically support much more. The agent does not.

Turning school email into useful state

School communication turned out to be the best test of whether this architecture was actually useful.

A single school email might contain several completely different kinds of information.

For example:

• a PA day;

• a school trip;

• a transportation change;

• a lunch ordering deadline;

• a newsletter;

• a voluntary activity;

• an early dismissal;

• a form the family needs to complete.

Those should not all produce the same response.

The school workflow separates three decisions:

Should the family know about this?

Does someone need to do something?

Does something belong on the calendar?

Those decisions are independent.

A transportation change might be worth surfacing but not require an action. A PA day belongs on the calendar but may require nothing from us. A lunch order might be optional, yet still have a real deadline that requires a family decision.

That last case exposed an interesting distinction.

Optional doesn't mean non-actionable

One real example involved a school hot-lunch program. Participation was optional. At first glance, that might sound like something the agent should classify as informational. But there was a firm ordering deadline.

The family still had to make a decision:

Order lunch

or

Decide not to order lunch

Doing nothing until after the deadline effectively makes the decision for us.

So the system now distinguishes between optional participation and actionable administration.

The activity can be optional while the administrative deadline remains real.

In that case, the system created an open action and a calendar reminder before the actual deadline.

That is exactly the kind of thing I wanted the agent to help with.

Not because the AI knows something magical about school lunch.

Because it can interpret the communication, apply a defined family policy, and convert it into durable state before the information disappears into the inbox.

The calendar is for awareness, not only confirmed attendance

I also changed how I thought about calendar events. A traditional calendar often represents things the family has committed to attending. For this workflow, that definition is too narrow.

The school calendar is an awareness calendar.

That means it can include things such as:

• school closures;

• PA days;

• trips;

• performances;

• early dismissals;

• transportation disruptions;

• school appointments;

• important activity dates;

• reminders ahead of administrative deadlines.

The calendar helper is intentionally restricted to a specific school calendar.

Hermes cannot choose another calendar.

It also cannot silently create events through an unrestricted calendar API.

The integration defines what is allowed, including explicit confirmation requirements and duplicate protection.

This gives the model enough capability to be useful without turning it into a general-purpose calendar administrator.

Bills needed their own logic

Bills look simple until you try to automate them. An email that says an amount is due does not necessarily mean I need to do anything. Some accounts are on autopay, others are manual. A normally automatic payment might fail.

A provider might send generic language encouraging customers to enroll in autopay even when the account is already configured differently.

So the bills workflow uses family-owned account configuration as part of the decision.

Conceptually:

Known account configuration

        +

Current bill evidence

        +

Existing family state

        ↓

Current bill status

If a bill belongs to an account that is normally paid automatically and nothing unusual has happened, it does not need to become a repeated urgent task.

If an automatic payment fails, the current evidence overrides the normal configuration.

That bill now requires attention.

This is also why I kept ordinary purchase receipts out of the billing workflow.

A completed retail purchase or activity receipt is not the same thing as an outstanding household obligation.

Keeping those domains separate prevents the system from turning every transaction email into a "bill"

And importantly, there is no payment capability.

The agent can understand a bill, track it, and alert us when something needs attention. It cannot pay one.

Persistent state became the real family memory

All of this information eventually needs somewhere durable to live.

For that I use a small SQLite database managed through the family-state layer.

It tracks things such as:

messages

actions

bills

calendar relationships

briefing presentation state

audit history

Hermes does not have arbitrary SQL access. The helper exposes validated operations instead. That means the model can ask the state layer to create or update a supported family entity, but it cannot freely run whatever SQL it generates.

The database also gives the system something an LLM conversation alone cannot provide reliably:

authoritative state.

If an action is still open, it is open because the family state says so, not because the model remembers seeing an email three days ago.

The morning briefing should not reread the inbox

Once persistent state existed, it changed how I built the morning briefing.

My first instinct would normally have been:

Read recent email

   ↓

Summarize it

   ↓

Send the family a morning report

But that would reintroduce the same problem I was trying to solve.

Instead, the briefing is generated from normalized family state.

Family state

   +

Upcoming school calendar

   ↓

Determine what is eligible today

   ↓

Generate briefing

   ↓

Send to fixed family recipients

The briefing does not need to reinterpret the original inbox just to remember what mattered.

That work happened when the message first arrived.

Unresolved doesn't mean repeat it every day

Persistent state created another problem. Suppose there is an open school action due in two weeks. It should remain unresolved until we deal with it. But that doesn't mean the morning briefing should repeat the exact same reminder every single day for fourteen days.

That led to another separation in the design:

Administrative state

Is this still unresolved?

versus:

Presentation state

Should this appear in today's briefing?

Those are different questions.

So in the system:

presented != resolved

snoozed != completed

not shown today != closed

An action can remain open while being temporarily suppressed from the briefing. As the deadline gets closer, it becomes eligible more frequently. If something materially changes, it can immediately become eligible again. If the family snoozes a reminder, that changes presentation behavior without pretending the underlying task is finished. This small distinction made the briefing feel much more like an assistant and much less like a noisy notification system.

Delivery is part of the transaction

There is another detail I didn't want to leave to chance. The system should not mark an item as "presented" simply because it generated a briefing. The family has to actually receive it.

So the workflow is closer to a transaction:

Determine eligible items

   ↓

Generate the final briefing

   ↓

Create the controlled briefing file

   ↓

Send the email

   ↓

Verify the delivery audit

   ↓

Only then mark those items as presented

If email delivery fails, presentation state does not advance. If the briefing file isn't created correctly, presentation state does not advance. If delivery succeeds but presentation-state recording fails, the system treats that as a partially completed workflow rather than pretending everything succeeded. That might seem like an excessive amount of care for a morning family email.

But reliability is the point.

If the system is supposed to help the family remember things, it cannot quietly forget them because one automation step failed.

Email and attachments are untrusted data

Another important rule is that incoming email never becomes instruction. The agent is allowed to interpret an email. It is not allowed to obey an email. That includes attachments. A school PDF can contain dates, instructions for parents, schedules or forms. The content may be useful as data. It does not get to change the agent's permissions. Incoming content cannot redefine who Hermes can email, which calendar it can modify, what files it can access, or what security policy applies.

The system around the model remains authoritative.

What the agent deliberately cannot do

One of the easiest ways to describe the architecture is actually to describe what I refused to give it.

The family-admin agent cannot:

• send arbitrary email;

• reply to or forward incoming email;

• delete, archive or modify Gmail messages;

• access our normal family inbox;

• write to arbitrary calendars;

• run arbitrary SQL against family state;

• execute email attachments or macros;

• make payments;

• treat an incoming message as authorization to expand its own permissions.

Those limitations are not missing features. They are part of the design.

The agent gets enough authority to help with the workflow I built. Nothing more.

Discord for administration, email for the family

I use Discord as the operational interface to the family-admin profile. That is where I can test the skills, inspect state, ask questions, troubleshoot the agent and interact with it directly. But I didn't want the rest of the family to need Discord just to benefit from it.

The family-facing morning briefing is delivered through email to a fixed recipient list.

That separation works well for now:

Discord

   → administration and interaction

Email

   → family-facing briefing

The same Hermes profile can therefore act like an agent when I interact with it while still behaving like normal family infrastructure for everyone else.

What I learned building it

This project reinforced something I've been seeing across the other agents I've built.

Giving an AI access to something is easy. Deciding what authority it should actually have is much more interesting.

The useful part of this family agent is not that an LLM can read a school email. Any capable model can summarize an email.

The useful part is the system around it:

Interpretation

   → domain policy

   → persistent state

   → constrained capability

   → verified outcome

That is what lets an email become a reliable family action instead of another piece of temporary AI output.

It also changed the way I think about "memory" in agents.

For important workflows, I don't want the agent's conversational memory to be authoritative.

I want the agent to reason over deterministic state.

The database remembers whether the action is still open. The calendar remembers the date. The account configuration remembers whether a bill is normally on autopay. The delivery audit remembers whether a briefing actually went out.

The LLM helps make sense of the information connecting those pieces.

Where I want to take it next

School administration and household bills are only the starting point.

The family-admin profile was intentionally designed so additional family domains can be added without creating a completely separate architecture each time.

Possible future areas include:

• family documents;

• contract and legal-document review;

• food and recipe workflows;

• reminders;

• broader household administration;

• media-related family workflows;

• other shared family services.

The important part is that each new domain can inherit the same pattern. The AI can reason, the skill defines the policy.A deterministic integration defines what is technically allowed. And durable state remains the source of truth.