An ERP implementation is mostly documents: requirements, fit-gap registers, solution designs, configuration plans, test scripts, migration maps, handover packs. This article draws on an AI agent framework OSF Digital is building for a customer across two ERP platforms. It walks through the architecture, the design choices we weighed, and what we’ve learned so far, so you can judge whether the same approach fits your own delivery work.The example is ERP, but the pattern applies to any team that delivers the same kind of project again and again: implementations, onboarding, migrations, or audits, where the work is document-heavy, moves through approval stages, and depends on a few experts.
The problem: delivery that scales with headcount
Today, ERP implementations typically run anywhere from three to nine months, depending on platform complexity, business requirements, and organizational scope. Most of that time goes into documentation that teams build by hand, engagement after engagement: requirements, solution designs, configuration plans, test scripts, migration mappings, and handover packs.That model has three limits. Every project is shaped by the individuals on it, so quality varies. Lessons learned from one engagement rarely carry over to the next. And the only way to deliver more is to hire more consultants.The customer’s goal was explicit: turn implementation from a consultant-led project into a scalable, AI-guided onboarding product. They wanted shorter time-to-value, consistent delivery across micro-verticals, and less manual effort for configuration, migration, and testing.They added one condition that shaped everything else. AI should accelerate repeatable work while remaining auditable and governed where complexity and customer trust are highest.
At a glance: how the factory works
inside-the-agent-factory-architecting-a-governed-ai-delivery-workforce-for-repeatable-delivery-image.jpgWork enters as an event; the gate spine holds the engagement's state; and the Orchestrator selects the next agent. The Critic scores every output. Only policy (for routine work) or a person (for everything that matters) moves it forward. Below the loop, one Markdown record per engagement is the source of truth; everything else is rebuilt from it.
The agents: six tool groups and a governance layer
The roster is organized around the delivery lifecycle, not around what AI can do. Six tool groups follow the phases a project already moves through, from presales to support, so each agent has a clear place in the work and a clear owner on the delivery team. The roster itself changed during discovery: the proof of concept started with 26 agents, and working through real delivery flows reshaped it into 27 capabilities. Treat your first agent list as a hypothesis.One decision mattered early: which agents could be shared and which needed platform-specific versions. Some work, such as requirements analysis, is the same across platforms. Some, such as configuration, depend on how each platform works. Settling that split up front avoids building one agent that serves no platform well.
Test sets derived from the design, automated regression runs, and diagnosed issues ready for support
Around the six groups sits an always-on governance layer of six agents. It touches every tool and every run.
AI Orchestrator reads engagement state and decides which agent runs next where event rules alone cannot.
Critic / Reviewer checks every output and calculates its confidence score independently.
Security & Governance classifies artifacts into four data classes and triages audit anomalies. It does not make access decisions.
Workspace & engagement config stands up each engagement and proposes its roles and data classes for human review.
KB-curator turns completed work into de-identified, reusable lessons.
Documentation ingestion converts product and API documentation into a searchable knowledge base the configuration agents use.
The Solution Design Document is the hinge of the roster. The PRD agents converge on it, and build, configuration, and testing all start from it. That is why the first build batch, now under way, focuses on the PRD tool and the governance agents it depends on.
Orchestration: governance the model cannot talk its way around
The central design rule is simple. Governance lives in deterministic infrastructure, not in prompts. An instruction such as "wait for approval" inside a prompt is a request. A wait state inside a workflow engine is a fact.Four mechanisms carry this rule:
A durable gate spine. Each engagement runs as one long-lived workflow execution. Five human sign-off gates — Presales, Define, Project Ready, Implementation/Delivery, and Support handoff — are durable wait states in that workflow. An engagement can sit at a gate for days or weeks without a process in place. Only an explicit, attributed human action moves it forward. If a stage fails, the workflow restarts from that stage, not from the beginning.
An Orchestrator that routes but does not produce. Events (a new RFP, workshop notes, a support ticket) arrive on an event bus, carried by reference with a data-classification tag rather than as raw payloads. Simple routing is handled by event rules. The AI Orchestrator steps in where rules cannot decide: it reads the engagement state, tracks dependencies, selects the next agent, and requests approvals.
An adversarial Critic. Every producer agent hands its output to the Critic. The Critic derives a confidence score from four checks: an evaluation rubric, schema and traceability-link validation, grounding against the knowledge base, and self-consistency. The producing model never grades its own work. Below threshold, or above a risk class, the output escalates to a human gate.
An autonomy ladder enforced as policy. Every action is assigned a level, and a version-controlled policy engine decides what may run without a person. Permissions combine user role, action class, data class, and delivery phase.
Level
Meaning
Examples
L0 · Human-only
AI may draft and recommend, never approve
Gate approvals, commercial, security, and go-live decisions
L1 · Draft and escalate
AI prepares, a person always reviews
SOW pricing, solution and technical design
L2 · Auto-pass on confidence
Proceeds if the Critic's score clears the threshold and policy allows
Routine analysis
L3 · Autonomous, still audited
Runs on its own, every action logged
Record-keeping and synchronization
The result is a property, not a promise: the AI cannot approve scope, architecture, security, go-live, or commercial commitments, because no path in the system lets it.
Set up. The Workspace agent proposes the engagement's workspace, data classes, and team roles. A person approves them. This creates the authoritative record for the engagement.
Presales. RFPs and workshop notes arrive as events. Discovery & Requirements turns them into individually identified requirements. Fit & Gaps separates what the product does out of the box from what needs configuration, extension, or a partner solution. Solution Recommendation proposes the solution shape, and Scoping assembles the SOW line by line from those artifacts. The SOW is never auto-issued. Gate 1: Presales.
Define. The Solution Design agent consolidates requirements, gaps, recommendations, and scope into the Solution Design Document. Validation & Traceability checks ensure that every requirement reaches a design, a test, and a support artifact. Gate 2: Define.
Project Ready. Build sequencing converts the approved design into an ordered, dependency-aware plan, with the approvals marked where they fall. Gate 3: Project Ready.
Implementation. Configuration agents propose, a person approves, then the agents apply. On platforms that support them, golden images provide a validated baseline. The Data Migration Assistant maps, loads, and reconciles customer data. UAT generation derives tests from the design, and UAT execution runs repeat cycles while customers continue to run their own acceptance tests. Gate 4: Implementation/Delivery, including go-live.
Support handoff. Onboarding agents deliver training and in-product help that match the configured tenant. The Troubleshooting agent packages open issues with diagnoses, ready for support. Gate 5: Support handoff.
Close. The KB-curator distills de-identified lessons into a situation-to-guidance form. A person approves, edits, or rejects each one before it becomes reusable knowledge.
Step 7 is what makes the framework compound. The next engagement starts with the lessons of every previous one, not a blank page.
Design decisions and the alternatives we rejected
Every major choice was made against at least one credible alternative. These are the ones architects ask about most.
Decision
What we chose
What we weighed
Why
Gate enforcement
A durable workflow engine, with gates as wait states
Orchestration logic inside agent code
Deterministic, resumable, and impossible for a model to bypass
Source of truth
Version-controlled Markdown, one record per engagement
A database as the authoritative store
Portable, human-readable and simple to recover; every database becomes a rebuildable index
Output validation
An external, adversarial Critic
Models self-reporting confidence
Confidence a producing model cannot fake; schema, link and grounding checks are objective
Model access
A provider-neutral gateway with a logical model ID per agent
Provider SDKs called directly by each agent
Model choice per agent without code changes; central keys, rate limits, fallbacks and cost tracking
Authorization
One policy engine for access and autonomy
Authorization written into application code
One versioned, testable policy surface
API knowledge
Markdown indexed on platform, module, object, and operation, with semantic search optional
Embedding-only retrieval over developer docs
For an API call, an approximate match is a wrong action; exact lookups are auditable
Agent runtime
A managed agent runtime (Amazon Bedrock AgentCore), with a pre-costed container fallback
Committing to one runtime with no exit
Keeps the one real unknown reversible; agent code is identical on either host
Two of these deserve a closer look.
Markdown as the system of record. Agents write Markdown to versioned storage. A deterministic projector — application code, not a model — rebuilds everything else from it: the state index, the requirement-to-design-to-test traceability graph, the audit hash chain and the knowledge bases. If any store drifts, the Markdown wins, and a reconciliation job raises an alert. Disaster recovery becomes "redeploy and rebuild the projections".
Confidence is derived, never declared. Ask a model how confident it is, and it will tell you, fluently and unreliably. Here the Critic computes the score from evidence and publishes how it got there: every grade opens to its full rubric derivation. Disagreements between the Critic and human reviewers feed an offline calibration process. They never change the knowledge agents retrieve at runtime.
Technical challenges, and what they taught us
Landing the design on the client's own platformThe first blueprint was cloud-native, built from managed AWS services in the customer's own account. During discovery, they asked us to map it onto its existing multi-tenant application platform instead. Most layers moved cleanly: events, storage, the UI, and the projection store each had a platform equivalent. One did not. The platform had no durable workflow primitive, and the gate spine depends on one. The answer is an open-source durable workflow engine running inside a containerized platform service; Flowable and Temporal are the two candidates under evaluation.The lesson: write the architecture as a contract of properties, not a list of services. "Gates are durable wait states the model cannot transition" survived the move. "We use service X" would not have.Data residencySome customer data must stay in the UK or the EU, and model inference must not cross that boundary. Engagements deploy in-region, and the model gateway maps each region to an approved model available there. Agents reach tenant data only through REST services the delivery team owns. That provides a single seam where authorization, classification, and audit are enforced, rather than trusting the agent runtime to behave.Audit that holds upEvery approval and action is written to an append-only hash chain, each entry linked to the previous one, with a copy in write-once storage. Tampering becomes evident by construction. On a shared platform, this means using the platform's own audit service and modeling record versioning through events, with raw storage features not exposed.Shared learning versus strict isolationIsolation is right for customer data. It is wrong for a knowledge base whose whole value is that one engagement improves the next. The resolution: lessons are de-identified by the KB-curator, scoped as engagement, customer, or common, and approved by a person before any agent can retrieve them. The de-identification process itself is open to audit.Independent ownership from day oneProduct teams must be able to enhance, test, and deploy their agents on their own cadence. That requirement shaped the factoring into three layers: common platform controls, product-agnostic coordination, and product-specific bundles. Retrofitting that later would have meant a rewrite.An agent factory is mostly a factoryThe agents depend on shared machinery: the model gateway, an evaluation harness, a documentation-ingestion pipeline, and governance controls. That machinery is on the critical path and needs its own plan.What the framework does not do is worth stating as clearly:
It does not replace ERP platforms or migration tools. It configures and governs; they remain the systems of record.
It does not fix poor source data. It makes data problems visible earlier.
It does not remove people from acceptance. Manual UAT stays, because customer confidence is part of adoption.
It does not touch restricted data before the data-handling architecture is formally approved. The proof of concept runs only on synthetic data.
Projected outcomes and how they will be measured
The build is under way, so these are targets, not results. What is unusual is how they are measured: each agent is accepted on the outcomes it achieves, not on the activities it performs or the documents it produces. With such architecture, an organization can expect the following improvements:
Shorter time to value by automating repeatable documentation, configuration, migration, and testing work across ERP implementations.
Consistent delivery across both platforms and across micro-verticals, with the same governed artifacts from presales to support.
End-to-end traceability from every requirement to its design, its test, and its support artifact.
Capacity without matching headcount. The architecture targets about 20 engagements a month at baseline, scaling about threefold on the same managed footprint.
A reusable foundation, with a path to roll out across 50+ subsidiaries after the initial deployment proves itself.
How each agent is judged:
Measure
What it tells you
Output quality and confidence score
Whether the Critic's independent assessment clears the agreed threshold
Traceability coverage
Whether every requirement reaches a design, a test, and a support artifact
Defect and rework rate
How often a person has to send work back
Human-review pass rate
How often reviewers accept the output as drafted
Confidence calibration
Whether the Critic's scores match human judgment over time
Cycle time and gate aging
Where engagements wait, and for how long
Model cost and latency per agent
What each agent costs to run, so model choices stay economic
All of these are derived from the audited records, so there is a single source of truth for delivery KPIs.Where it stands: discovery ran for seven weeks across 11 workshops. The build phase is currently underway. The first batch is nine agents: six PRD agents from Discovery & Requirements through to the Solution Design and Validation & Traceability, plus the Critic, the KB-curator, and documentation ingestion.
Where to start with your own agent factory
Nothing in this architecture is specific to ERP. Any organization that delivers the same kind of project again and again — implementations, onboardings, migrations, audits — has the same shape of problem: document-heavy, phase-gated, dependent on a few experts, and hard to scale.The starting point is not the agents. It is the delivery flow. Map how a project actually runs, where the documents are created, which decisions must stay with people, and what data each step depends on. The agent roster, the gates, and the autonomy levels follow from that map.If you want to test whether your delivery model is a candidate, book a fit assessment with us. We look at one delivery flow you have in mind and tell you whether an agent factory would pay off, which agents to build first, and what governance it needs before any build budget is committed. If it is a poor fit, we will say so.Book a fit assessment
AuthorGerard “Gerry” SzatvanyiFounder and Chief AI Officer at OSF DigitalGerard “Gerry” Szatvanyi is the Founder and Chief AI Officer at OSF Digital. Since founding the company in 2003, he has helped shape its growth and vision, with his current work focused on advancing enterprise-grade AI adoption and integrating generative AI into real-world delivery, operations, and decision-making. Gerry is the author of The Great Digital Transformation and Workforce 4.0, where he explores the impact of technology, customer experience, and evolving workforce models on business. He also hosts The Great Digital Transformation podcast, produced with Forbes Books, featuring conversations with global leaders on digital transformation, AI, and the future of work. Passionate about the intersection of technology, people, and business, Gerry is committed to turning innovation into measurable impact at scale.
Sign up for the latest news, trends and insightsSUBSCRIBE
Frequently Asked Questions
An agent factory is a framework that coordinates multiple AI agents throughout a delivery process, rather than relying on a single model for isolated tasks. In this architecture, agents are assigned specific responsibilities, including requirements analysis, solution design, configuration, testing, migration, and onboarding. Governance controls, approval gates, and audit records ensure that automation increases delivery capacity without removing human accountability.
Governance is implemented through infrastructure rather than instructions inside prompts. Human approval gates, policy-based controls, confidence scoring, audit trails, and role-based permissions determine what actions agents can take. The design ensures that high-risk decisions remain under human control even when routine activities are automated.
Human approval gates create explicit control points for decisions that affect scope, architecture, security, commercial commitments, or go-live readiness. Because the gates exist as durable workflow states, an AI system cannot bypass them through reasoning or prompt interpretation. This turns governance into an enforceable system property rather than an operational guideline.
The AI Orchestrator coordinates work across the delivery process. It reads engagement state, tracks dependencies, selects the next agent, routes work when event rules are insufficient, and requests approvals when required. The Orchestrator manages flow but does not generate delivery outputs itself.
The Critic independently reviews every output produced by other agents. It calculates confidence scores by combining rubric-based evaluation, schema validation, traceability checks, knowledge-base grounding, and self-consistency analysis. Because the producing agent never evaluates its own work, confidence is derived from evidence rather than self-assessment.
Markdown provides a portable, version-controlled, human-readable source of truth. Rather than treating databases as authoritative, the architecture stores engagement records in Markdown and rebuilds indexes, knowledge bases, traceability graphs, and audit projections from that source. This simplifies recovery, reduces platform dependency, and makes reconciliation easier when systems drift.
The framework separates common controls from platform-specific capabilities. Governance, orchestration, workflow management, and shared services remain consistent, while individual agents can be implemented for specific ERP platforms and business processes. This allows organizations to maintain a common operating model while accommodating platform differences where necessary.
The target outcomes include shorter implementation timelines, more consistent delivery quality, end-to-end traceability, greater delivery capacity without proportional growth in headcount, and a reusable foundation for broader deployment. Performance is measured by confidence scores, traceability coverage, review outcomes, defect rates, cycle times, and operational costs, rather than by document production alone.
Every approval and action is recorded through auditable mechanisms, including append-only records, policy controls, workflow tracking, and data classification rules. The architecture also uses human review, confidence validation, authorization policies, and region-specific deployment controls to support compliance requirements and operational oversight.
Yes. The framework is designed around repeatable, document-heavy delivery processes that rely on human expertise and formal approvals. The same pattern can apply to implementations, onboarding programs, migrations, audits, compliance activities, and other structured delivery models where governance, traceability, and scalability are important.
BROWSE AND READFor More On This Topic
The Workforce of the Future in the Agentic AI Era
Blog
Discover how agentic AI is reshaping the workforce. Learn how to lead hybrid teams of humans + agents.Read more
Ways AI Agents Can Improve Experiences for Your Customers and Teams
White Papers
AI agents can give customers better experiences and boost productivity for companies across industries. Find out exactly what agents can do and how to apply and maximize their capabilities for your business.Download
Reinventing IT Architecture with Multi-Agent AI
Blog
Discover how multi-agent AI is reshaping IT architecture, improving business operations, and enhancing customer service. Learn more about OSF Digital’s AI solutions.Read more
The Workforce of the Future in the Agentic AI Era
Ways AI Agents Can Improve Experiences for Your Customers and Teams
Reinventing IT Architecture with Multi-Agent AI