Architecture

Under the hood.

How it all hangs together — Architecture, lifecycle management and continuous deployment. Maintaining a rigorous release standard across code changes and agentic workflows.

System architecture

Endpoints
29
DB tables
49
Unit tests
495
Change records shipped
38
Autonomous handle rate
63%
Production system (sanitised) Data ingestion (scheduled) Dashboard (read-only) AI agent (customer/supplier) Shared database (one source of truth) E-commerce platform Ad platforms Messaging & email Issue tracker

The dashboard, email and Telegram are the three main client/interfaces for this system. Here all information is surfaced, instructions given and actions confirmed for general users. Additional developer interfaces exist directly through Claude Code, Github and JupyterLab (or any Python IDE).

All data ingestion occurs through regular polling of third party systems whilst the agents rely on both polling and webhooks to listen on their channels.

Agents both read and write to the database and talk directly to a subset of the external integrations — sending messages, filing issues, updating orders. Importantly, any agent decision that creates an irreversible action results in an escalation to the team on the business telegram channel and an exception case raised in the dashboard with buttons to either confirm (execute) action or deny it — why our autonomous handle rate is not 100%. Customer communication remains a fundamental part of this workflow at all times.

How it was built

This was built and is maintained by one person, following a consistent discipline: write the spec, turn it into an implementation plan with concrete steps and tests, implement against that plan, and only then ship. Every non-trivial change goes through the same sequence — nothing ships from a first draft.

Change-record lifecycle with the CI/CD pipeline nested inside the build stage The change-record lifecycle runs from idea to done. The build stage expands into the GitHub Actions CI/CD pipeline: feature branch, CI suite, PR review, deploy, live. Deployment feeds back into UAT and done. Change-record lifecycle every feature, every fix — idea through to production Idea status: proposed Spec written, reviewed Plan + tests TDD, up front Build ↓ CI/CD pipeline CI/CD pipeline · GitHub Actions runs inside the build stage, on every change Feature branch never main CI suite pytest · 495 tests PR review merge → main Auto-deploy pull, migrate, restart Live in production deployed UAT external testers, if needed CR → done results published active / terminal state process step

Every one of the 38 Change Records shipped so far took this shape. A CR is raised before any code is written; the design is brainstormed into a written spec; the spec becomes an implementation plan with concrete tasks and tests; the build proceeds against that plan with a review after each task; and only after a test plan — including UAT — has run does the CR get marked done and its results published. Nothing skips a step because the change felt small: the discipline is what keeps 38 changes to a shared production system from turning into 38 opportunities for regression.

The single most valuable step in that discipline is formal user-acceptance testing, with external testers recruited if new agent behaviour is being introduced, before anything reaches a real customer. Self-testing catches the paths you thought to try; external testers, working from their own mental model of the product rather than the implementer's, catch the paths you didn't. Every incident below was caught this way, before it reached a real customer — not after.

Every push to a feature branch runs the same test suite CI runs before merge — nothing lands on main without it passing. Merging to main is itself the deploy trigger: GitHub Actions pulls the new code onto the server, runs any pending database migrations, and restarts the service. There is no separate release step and no staging environment to promote through — which is exactly why the spec-then-plan-then-UAT discipline above carries the weight a staging environment would otherwise carry.

Not every bug has a clearly definable test

Evaluation — how agent behavior is scored and tracked release over release — has its own section: see how this system is evaluated.

What went wrong and how it was fixed

The cross-customer data leak

Pre-launch user-acceptance testing, run with external testers before go-live, surfaced that the customer-service agent's order-lookup tools weren't constrained to the customer who was actually authenticated in the conversation — a tool call could, in principle, return another customer's order data. The fix was architectural, not a patch: a single authentication gate now resolves the caller's identity once, at the start of a conversation, and every downstream tool call is constrained to that identity for the rest of the exchange. Nothing further downstream needs to re-check who's asking, because it structurally can't be asked on behalf of the wrong customer. This is the single most reusable insight from the project: the tool boundary is the security boundary. It generalizes to any agent with real permissions — the question isn't whether each tool checks correctly, it's whether a tool can be called with the wrong identity at all.

The silent data-loss incident

A missed schema migration on the data-ingestion side caused order data to stop arriving through the webhook the platform relied on — silently, for roughly five days, with no error surfaced anywhere in the pipeline. The fix was to stop depending on an inbound push at all: webhook ingestion was replaced with a hardened poller that pulls and reconciles data on its own schedule. The lesson is about failure mode, not the specific bug — a webhook that stops firing looks identical to a quiet day with no orders, so nothing alerts. A poller you control can be checked: did it run, did it get data, does the count look right. For anything where silent loss is unacceptable, a pull you initiate and can audit beats a push you can only hope arrives.

The silent undercounting bug

A count that fed into an operational figure had been quietly undercounting for a period of time, leading to a gradual drift in the inventory count — the number looked plausible, so nothing about it prompted a second look. It was found through systematic reconciliation: cross-checking the figure against an independent total until the discrepancy surfaced, then tracing it back to the naming-convention drift. The lesson is that a number can be wrong without looking wrong — a small, steady undercount doesn't trip any sanity check a human would think to write. Reconciliation against an independent source, not just plausibility, is what actually catches this class of bug.