Built solo, start to finish
A production AI agent for Microsoft 365
An assistant that lives in Microsoft Teams. You ask it for something in plain language and it works out which actions to take across your mail, calendar and files, checks what comes back, and keeps going until the job is done.
I did the product, the engineering and the compliance. Each customer's data is kept apart by the database itself, not by application code. The agent passed Microsoft's Marketplace certification and its Teams Store validation, seven review rounds.
It has to choose the right action out of eighty-seven. Sending every tool on every message is expensive, so a retrieval step narrows the list first. The risk is that the short list drops the tool a request needs, so I measure that on test requests written blind. The redesigned selector offers every needed tool 98.1% of the time, up from 87.0%.
How a test case is scoredexpected facts, not similarity
{
id: 'quality-15',
category: 'quality',
prompt: "Draft an email to Tom Nakamura saying I'll review
his investor deck by Thursday.",
expectedFacts: ['tom nakamura', 'thursday', 'investor deck'],
forbiddenFacts:['email sent', 'sent successfully', 'i have sent'],
formatReqs: { forbidFiller: true },
difficulty: 'simple',
}
// passed = accuracy >= 0.8 AND quality >= 0.8 AND no forbiddenFacts present
// accuracy = (expectedFacts matched) / (expectedFacts total)
Each case lists the facts a right answer must contain and the claims it must never make. This one asks for a draft. An agent that sends the email and says “email sent” fails, however well it's written. The suite replays 142 cases offline and checks which tools were called. I run it on demand; it isn't in CI.
What happens when it breaksthe loop and its exits
Graph returns 429/5xx → back off and retry ×3, honouring Retry-After Model rate-limits → wait the provider's interval, bounded to 4 and to 90s Request times out → keep the system prompt + last 15 messages, retry once Content filter trips → ask the user to rephrase rather than fail Same tool, same args, 8× → block that call, tell the model to try another way 35 tool calls in a turn → hard cap, abort before the batch fires // a small model by default; a long request, or one step emitting >3 tool calls, // escalates to the large one for the rest of the turn (87% of eval messages never did)
The repeat guard doesn't end the turn. It blocks the looping call and tells the model to try something else. Rate-limit waits have a hard limit because an early version retried forever. On a 142-case eval, the narrowed list called the right tools on 134 of 142 cases on average over three runs, with 49% fewer prompt tokens than sending every tool.
Where the tenant boundary isclick to enlarge
Everything a user triggers runs inside the boundary, where the database returns only that customer's rows. About twenty system paths, such as cron, retention and metering, need to see every customer, so they run outside it as a separate database role, each listed with its reason. The diagram shows them outside on purpose.
It's one agent in a loop, not a planner and executor, and it works synchronously while streaming progress. It asks before a bulk delete or before sending something it drafted, and that rule lives in the prompt, not in a separate approval system. It's published on the Marketplace and the Teams Store. I've paused the hosted service for now, and it has no paying customers.
TypeScript · Azure OpenAI · Microsoft Graph · Azure Postgres (row-level security) · Azure Container Apps · Bicep · Docker to GHCR via GitHub Actions
