Agentic Factory

Agentic Factory asks when a large language model (LLM) agent is a better controller for a production system than the deterministic, rule-based logic of discrete-event simulation (DES). Two comparison studies test this in shared simulated environments: a vending microbusiness, where a locally hosted agent with native reasoning finished above the rule-based controller, and a garment-decoration factory, where agents that consulted the model on every decision were held back by call volume and agents that consulted it only on exceptions cleared the workload on five calls. A third report turns those results into a design for an agentic simulation platform, in which a deterministic kernel runs the process and agents handle language and exceptions.

Context and funding

Agentic Factory is part of my internship at the Virtual Engineering Centre (VEC), University of Liverpool. It sits within the research and innovation strand of the VEC's work with Wild Thang, a Liverpool garment-decoration company, which was funded through LCR Innovations by the Liverpool City Region Combined Authority. Wild Thang is also the first intended use case: its factory floor has a small number of well-defined machines and routes, which makes it a practical environment for comparing agentic control against discrete-event simulation before trying anything larger. The production digital twin built for the same company is described separately in Wild Thang: Embroidery Production Digital Twin.

Figure. Funded through LCR Innovations by the Liverpool City Region Combined Authority.

The question

DES is the industry standard for modelling production. It can represent randomness through probability distributions, but its decisions follow rules fixed at design time, and for large or complex systems every variable has to be defined in advance. An LLM agent can read natural language and respond to situations its designers did not anticipate, which could make it cheaper to build and maintain where fixed rules are expensive. Published scheduling benchmarks have found that agents often fail to beat strong heuristic baselines, and that giving them more state information can make them worse. The project proposal identified a gap: no existing study placed an agent and a DES model head to head as competing controllers of the same factory. It set out three steps: choose the model, choose the agent harness, and design a benchmark that compares agentic and DES control of a shared environment.

Both studies below use the same principle. The controller, whether rules or an LLM, never touches the simulation's internal state. It receives a restricted observation (an "observation boundary") and acts only through time-costed tool calls, so neither side has information or abilities the other lacks.

Study one: a simulated microbusiness

Comparison of Rule-Based and Agentic Management of a Simulated Microbusiness builds on Vending-Bench (Backlund and Petersson, 2025), which scores LLM agents managing a simulated vending machine over a long horizon: stocking slots, ordering from suppliers, setting prices, collecting cash, and paying a daily fee, with bankruptcy after sustained unpaid fees. I rebuilt the environment as a discrete-event simulation with a demand model covering price elasticity, day-of-week and monthly patterns, weather, product variety, and operating hazards (wholesale price shocks, slot jams, demand shocks, late deliveries, and competitor entry). Orders are prepaid and credited only on delivery, after a random negotiation delay and independent chances of rejection, partial fulfilment, and late arrival.

Figure. The shared environment. Rule-based control and the LLM agent see the same restricted observation, with future events hidden, and both act through the same event kernel. The agent returns one action list per simulated day and carries a 400-character memory note.

A deterministic rule-based controller ran first against the published results. Across 50 seeds it reached a mean net worth of $3,643.90 by day 102, above all nine published LLM agents and the human baseline.

Figure. Rule-based controller net worth (mean ± 1σ, 50 runs) against the published Vending-Bench results for nine LLMs and a human baseline.

Local LLM agents against the same controller

The study then ran three locally hosted models through the same environment, on five paired seeds per configuration: qwen3.8:27b, gemma4:26b, and deepseek-r1:32b. Each also ran with native thinking enabled, and two with and without an append-only long-term memory log. A full-information oracle, given the true demand model and a profit-maximising fixed price, set the ceiling. A "hardened" environment raised the rates and magnitudes of disruptions and made suppliers less reliable.

Controller	Primary: net worth	Primary: units sold	Hardened: net worth
Rule-based (DES)	$3,896.64	3,037.0	$2,103.36
qwen3.8:27b	$3,855.41	2,745.6	$2,285.01
qwen3.8:27b, thinking	$4,643.98	3,165.0	$2,834.09
qwen3.8:27b, thinking, no memory log	$4,448.53	3,167.8	$2,678.20
gemma4:26b	$1,768.07	1,120.2	$1,149.38
gemma4:26b, thinking	$2,105.80	1,345.2	$1,475.00
gemma4:26b, thinking, no memory log	$2,945.37	1,974.0	$2,006.62
deepseek-r1:32b (all configurations)	$294.00	0	$294.00
Full-information oracle	$5,715.44	4,015.4	$3,350.67

The main findings:

• Without thinking, the best agent tied with the rules. qwen3.8:27b finished $41.23 below rule-based control in the primary environment, well within one standard deviation, and beat it on two of five paired seeds. In the hardened environment it finished above rule-based control, overtaking it after about day 90.
• With thinking and no cap on response length, one agent beat the rules in both environments. qwen3.8:27b with thinking reached $4,643.98 against $3,896.64, and $2,834.09 against $2,103.36 when hardened. It stayed below the oracle in both.
• The response budget was the binding constraint. An earlier run with a fixed budget of 1,024 or 2,048 response tokens left every thinking configuration except qwen3.8:27b with zero sales. Removing the cap restored trading for two of the three models.
• Parseable output is a precondition. qwen3.8:27b produced valid actions on all 515 decisions in every configuration. deepseek-r1:32b failed to produce parseable actions on 97.5 per cent of decisions and never sold anything, ending at exactly the value of an inactive business paying every daily fee. Its uncapped thinking run could not complete a single seed. gemma4 with thinking still failed on 57 to 78 per cent of decisions.
• The memory log's effect depended on the model. It helped qwen3.8:27b slightly in both environments. It hurt gemma4 in both, and for gemma4 the log coincided with more failed responses (77.9 per cent against 63.7 per cent in the primary environment).

Figure. Primary environment, baseline models: qwen3.8 tracks rule-based control throughout, gemma4 builds a smaller position more slowly, and deepseek-r1 never trades, so its net worth falls by the daily fee. Shaded bands show variation across the five runs.

Figure. Hardened environment, baseline models: qwen3.8 trails rule-based control for most of the horizon and overtakes it after about day 90. Every band widens under the harsher disruption settings.

Figure. Primary environment, thinking configurations: qwen3.8 with thinking tracks rule-based control for roughly the first half of the horizon, then pulls ahead, approaching but not reaching the oracle.

Figure. Hardened environment, thinking configurations: both qwen3.8 variants finish above rule-based control, and gemma4 does better without the memory log.

Figure. The memory-log ablation. For qwen3.8 (top) the two versions track almost identically until about day 60, when the version with the log pulls slightly ahead. For gemma4 (bottom) the version without the log separates from around day 20 and stays ahead in both environments.

Pricing and operating risk

Over 50 seeds, the rule-based controller's mean net worth was $3,643.90 (95 per cent confidence interval $3,577.96 to $3,709.84). Turning off its weekly dynamic-pricing rule raised the mean by $155.79 (4.3 per cent) to $3,799.69, despite selling 73 fewer units, so the chosen pricing rule did not pay for itself and higher sales volume did not mean higher net worth. The hardened environment cut the mean by 40.4 per cent to $2,173.23 and widened its spread, showing that assumptions about operating risk change the return from a fixed policy substantially.

Study two: a simulated manufacturing floor

Rule-Based and LLM Control of a Simulated Manufacturing Floor moves the comparison to a DES of a garment-decoration factory modelled on Wild Thang: orders follow production routes (intake, stock preparation, artwork and screens, printing, curing, quality control, and packing and dispatch) through shared machines, operators, and resources that become available at different times. The fixed-rule baseline is an earliest-due-date (EDD) dispatcher, which always sends the ready order with the nearest due date and breaks ties by arrival time.

Six agent architectures

• A, persistent single agent: one agent with memory makes every dispatch decision.
• B, supervisor and workers: a supervising agent delegates to worker agents.
• C, reactive and deliberative loops: a deterministic fast path handles routine dispatch and escalates to the model only when a case falls outside its rules or a natural-language instruction arrives.
• D, decentralised blackboard: worker agents coordinate through shared state and optimistic claims, with no central coordinator.
• E, advisory agent with deterministic execution: the agent advises and rules execute, again calling the model only on escalation.
• F, approval layer: architecture A wrapped in a risk classifier that sorts proposed actions into routine, audited, and approval-required tiers.

The benchmark defines 18 scenarios in six tiers of difficulty: a deterministic single order, stochastic processing times, machine contention and routing, mid-run disruption (breakdowns, stock shortages, shift-change capacity dips, operator absence), priority changes and natural-language context, and compounded multi-agent cases, grouped by how many production routes the floor runs. Every architecture uses the same environment, seeds, and tool interface, and outcomes are measured for operations (throughput, tardiness, backlog), for cost (LLM calls, inference time, decision latency), and for traceability of schedule changes.

Results from the first live campaign

The first campaign ran each architecture against the hardest scenario, a simulated week of 869 orders under high-volume stress, with a six-hour wall-clock budget.

Controller	LLM calls	Simulated minute reached	Orders completed	Throughput
EDD baseline (whole week, 159 s run)	0	full week	73	0.478 orders/hr
A, single agent	718	317	6	1.135 orders/hr
B, supervisor/workers (final of 3 attempts)	323	253	0	0
D, blackboard	1,016	94	0	0
F, approval layer (final of 2 attempts)	708	350	5	0.857 orders/hr

The agent rows cover only the part of the week each architecture reached in six hours, while the EDD baseline simulated the whole week, so the throughput figures are not directly comparable between the two.

• The baseline is beatable by rules alone. A simulated-annealing search over the EDD dispatcher's four weighting factors (due-date urgency, client priority, walking distance between stations, and remaining work) raised mean throughput by 42.4 per cent across five seeds, from 0.478 to 0.680 orders per hour, and reduced the backlog by 31 orders. An agent that only matches hand-tuned EDD has not shown it beats rule-based dispatch as a class.
• Architectures that call the model on every decision were limited by call volume. Under a fixed wall-clock budget, each call consumed real time, so call volume tracked productivity closely. D made 42 per cent more calls than A, reached less than a third of its simulated horizon, and completed nothing. B completed nothing across three attempts. F behaved almost identically to A, because the approval layer inherited A's call pattern.
• Exception-only architectures used very few calls. C and E drained the 869-order backlog to a small residual queue on five LLM calls each. Getting there required finding and fixing a defect that let the fast path advance simulated time without limit. Neither finished cleanly, because the supervisor's watchdog could not tell a correctly idle architecture from a stuck one.

The report recommends longer wall-clock budgets, the full 18-scenario suite, more seeds per scenario, and runs that inject natural-language instructions, which no reported result used and which is the one capability the EDD baseline cannot match.

From the studies to a platform

Infrastructure for Agentic Simulation in Manufacturing and Related Sectors uses both studies to propose a product: an interactive operational simulation platform. A user describes a process or a disruption ("the second press is unavailable until lunch and the finishing team is one person short"), checks how the system has interpreted it, compares possible responses, and exports the resulting event and movement data. Its four purposes are operational insight, disruption preparedness, customer communication and marketing, and reusable data for independent analysis.

Design principles

• The kernel holds authority. A deterministic simulation kernel governs time, resource ownership, process dependencies, inventory, and physical feasibility. It handles duration sampling, accounting, capacity checks, and metric calculation. An LLM may propose an objective or plan, but only the kernel can implement it.
• One interface for every decision-maker. Rules, optimisers, people, and LLM agents all act through the same observation boundary used in the studies.
• Agents for language and exceptions. The report assesses candidate uses against criteria such as decision variability, information access, verifiability, time tolerance, recoverability, and the strength of existing rules. It recommends starting from the exception-only pattern of architectures C and E, and adding agency only as evidence permits.
• Stage the rollout. Start with a baseline simulation of processes defined with the customer, then a system that reads production data and makes recommendations. Connecting to live production with permission to make changes would require stringent testing and enforced operating limits.

Hardware and software

The report sizes hardware from the workload, using first-pass arithmetic before any benchmarking campaign: for example, a 70-billion-parameter model quantised to four bits needs about 42 GB for weights with a 20 per cent allowance, plus key-value cache for each concurrent sequence and memory for the system and services. It compares candidate machines including the NVIDIA DGX Spark and Jetson AGX Thor. The proposed near-term stack is Python with SimPy for the kernel, explicit dispatch rules and OR-Tools where constraint optimisation is needed, FastAPI for the service interface, an explicit state machine for the agent harness (LangGraph once durable branching and approvals are needed), vLLM or llama.cpp for local inference, PostgreSQL for operational storage, and Parquet with DuckDB for analysis exports.

The report is explicit about its evidence boundary: the experiments justify further development, but they do not establish that LLM agents generally outperform conventional scheduling algorithms.

Choosing a harness: HarnessBench

The choice of agent harness was tested separately. HarnessBench runs agent frameworks (OpenClaw, PicoClaw, NanoBot, and NanoClaw) through a common adapter against over 100 tasks spanning file operations, terminal use, and other workspace actions. Each task runs in its own sandbox with the same fixtures and grading. The headline score multiplies an outcome score, checked programmatically against the resulting workspace, by a process score, rubric-graded from a trace of the session for tool use, consistency, and robustness, and by a binary security gate that can zero the run. An LLM quality judgement contributes to the outcome score for only two tasks where correctness is not binary, at a fixed weighting in favour of the programmatic check. Token usage is recorded per run, so cost and performance can be compared between frameworks.

Garment design demonstrator

"Design My Order" is a customer-facing demonstrator built on the factory model. A visitor picks a garment, describes a design (for example, "a roaring lion badge in gold and navy"), and the AI-generated design is rendered onto a 3D garment. The order then goes to the running agent to schedule and produce, travelling through the simulated process stages, and customers will be able to see other orders being completed in real time.

Figure. "Design My Order": pick a garment, describe a design in plain English, and the agent schedules and produces it in the simulated factory.

The same free-text channel lets a person redirect the agent mid-run. An instruction such as "deprioritise ORD-014, it's not urgent" is attributed to a real order, classified into a scheduling effect, and executed through the agent's normal tools with a logged reason, so every schedule change can be traced back to the message that caused it.

Documents

The four documents from the project handover are below. They are internal working documents, so each is behind a password; I'm happy to share them directly on request.

Proposal: agentic manufacturing systems and DES benchmarking

Enter the password to view the proposal

Comparison of rule-based and agentic management of a simulated microbusiness

Enter the password to view the microbusiness study

Rule-based and LLM control of a simulated manufacturing floor

Enter the password to view the manufacturing study

Infrastructure for agentic simulation in manufacturing and related sectors

Enter the password to view the infrastructure report

Headline results

• A rule-based controller reading only a restricted observation outperformed all nine published Vending-Bench LLM agents and the human baseline, with a 50-seed mean net worth of $3,643.90.
• On five paired seeds, a locally hosted qwen3.8:27b agent tied with rule-based control without thinking, and with thinking enabled and an uncapped response budget beat it in both environments ($4,643.98 against $3,896.64, and $2,834.09 against $2,103.36 hardened). A full-information oracle set the ceiling at $5,715.44.
• Response budget and output validity decided which models could trade at all: one model failed to produce parseable actions on 97.5 per cent of decisions and never sold a unit.
• On the manufacturing floor, re-tuning the fixed-rule EDD dispatcher alone gained 42.4 per cent throughput, and agent architectures that consult the model on every decision were limited by call volume, while exception-only designs cleared an 869-order backlog on five LLM calls.
• The platform design that follows keeps a deterministic kernel in charge and uses agents only for language and exceptions, behind a shared observation boundary for rules, optimisers, people, and agents.