AWS Emulator and the engineer’s work when agents write code

In 2004, just a month into my new job at Amazon, I had the opportunity to choose an intern to join my team. I had an advantage over my colleagues: I had already heard of a brilliant young man who was working on his doctorate at the Vrije Universiteit in Amsterdam (where I had also obtained mine :-)). I immediately chose Swaminathan Sivasubramanian, who you all now know as Swami. It was probably one of the best decisions I made in my 22 years as Amazon’s CTO.

After a disastrous Christmas in 2004, when commercial relational databases couldn’t meet Amazon’s needs, I asked Swami to take a deep dive into how relational databases were actually used at Amazon. It turned out that 70% of database operations were not relational at all. They were a key value. This idea led to Dynamo, which Swami helped design. Once his internship ended, I immediately hired him full-time, even before he finished his doctorate.

Bringing in another insightful academic led to many fantastic discussions with Amazon’s hands-on engineers, who had become experts on scale by actually building at scale. Eventual or strong consistency, CAP vs. Paxos, and more. Over time, Swami became one of the key architects of AWS’s data strategy. And as we’ve seen in recent years, he’s also been one of the intellectual drivers of AI innovation at Amazon.

I’m honored that he wanted to write an article about the fundamental distributed systems work behind many AWS services, and how the job of the engineer changes when agents write the code. AWS Emulator, launched today, gives agents the kind of testing environment their work relies on. Swami’s story is another great example of invisible engineering.

Now go build.

-W

Today we’re releasing AWS Emulator, which gives coders a local copy of AWS to test, and it connects to a project from early in my career. In many ways, the launch of this emulator reminds me of the efforts we put into reliable software engineering to improve the quality of the distributed systems we build. To that end, we wanted to share a story:

About 20 years ago, Werner asked me to work with one of Amazon’s early engineers, Al Vermeulen, on a distributed locking service we called “Alf.” Alf was built on Paxos, a protocol based on the work of Leslie Lamport that allows a group of servers to agree on a value, and a replication engine contains consistent copies of the data between servers. Alf, to this day, is the service that powers many of our complex distributed systems like S3, DynamoDB and many others (although it has evolved quite a bit with various modifications on top of that – I’ll save that for another blog!).

Distributed systems are difficult to test

Paxos is the gold standard among distributed systems consensus protocols and was well understood in academia. However, building it at the scale that Amazon and AWS were growing into wasn’t easy. We had to deal not only with replication across servers and data centers, but also with tens of thousands of clients requesting locks at the same time, renewing their leases during network partitions, while respecting the fundamental requirement: always be correct! At that time, the best ideas for testing distributed systems were large-scale system testing and chaos-based testing. Both worked, but they were expensive and could not reproduce many types of conditions on demand. This mattered to Alf, who was supposed to be the store of truth, giving each client a strongly consistent view of the truth, and who also had to continue working despite server crashes and packet loss. Several industry articles have written that Paxos-based locking services were difficult to implement, and the difficulty of testing them was part of the reason.

We solved this problem by simulating Alf’s environment as closely as possible. We ran the servers in a JVM and simulated the network and the rest of the infrastructure they depended on, so the entire system lived in a single program that we controlled. We could drop packages and inject other errors at any time, and we built tools that analyzed the simulation for bugs and scalability issues. Concurrency bugs only appear when events occur in a particular order, and in the simulation we could intentionally configure several of these orders and inspect how Alf handled each one. We also used formal reasoning, via theoretical techniques such as TLA+, and also discovered many problems. By testing this way, we fixed most of Alf’s concurrency bugs before it ran on real servers. Testing in a real environment then revealed different issues, because the real world exposes things that a simulation cannot expose.

Moving beyond hand-built test environments

Both simulation and formal reasoning have improved the software quality of the distributed systems to which we have applied them. The simulation worked because a small team could manually create a copy of a system’s environment, and at the time, AWS only had a small handful of services. But today, AWS offers more than 200 services, and an application that calls AWS needs a test environment for each one it uses. Building all of this by hand would take years of engineering, and agents now write code much faster than any team can verify it in production. We therefore wanted to know if agents could create and test these environments themselves.

The idea came from Matt Goldberg, Senior Principal Engineer on the AWS Developer Tools team. His hypothesis was that, given a written description of how a service should behave and the actual service to be checked, an agent could create a working version of that service itself. For AWS, the written description is the set of public API models, written in Smithy, the AWS format for describing the operations of a service as well as their inputs, outputs and errors. The real service to compare is the AWS cloud itself, the same system that customers use.

Agents work from fidelity tests, which are end-to-end customer scenarios built from public API models. A test can verify, for example, that in Amazon Simple Queue Service, a received message remains hidden from other readers until its visibility timeout has expired. Agents patch each test until it passes against a real AWS, so the test records AWS behavior. Agents then build the emulator until it passes this same test, and a test that still fails indicates a gap in the emulator. Matt started with SQS alone, and when he gave Kiro a prompt describing this approach, Kiro returned a passable first version of an SQS emulator in about 15 minutes. Matt Goldberg’s launch post tells the story of how this prototype became AWS Emulator. All AWS Emulator code was written and tested by agents, and it is built from more than 400 API models covering more than 200 AWS services. It works free and offline on your own machine. Teams of agents test it against real AWS and fix any gaps found. Every change passes through a merge gate, a thorough set of automated checks that each change must pass before entering the codebase, including multiple security guards performing adversarial reviews.

Infrastructure in the coding loop

A coding agent works in a loop: it writes code, executes it, and reads the result before deciding what to modify next. In distributed systems, this loop runs inside the infrastructure, so the services it calls must behave like the real ones. When they do, an agent can check their own work as they write and produce correct code on their first attempt much more often, improving the quality of the software. AWS Emulator puts this infrastructure in the loop on your own machine.

The engineer’s job when agents write code

When agents write the code and check their own work, the engineer’s job is to give them the opportunity to work on their own and create the specifications, tests, and merge gates that decide when that work is correct. This is a lesson I learned the hard way when I used Kiro to rebuild the replication engine that Al and I had built for Alf, which took us about four months the first time. I supervised the officer for four nights before finding that he had no way of verifying his results. Once the specifications and test environment were available, the rebuild took approximately two hours.

The agents building AWS Emulator had both from the start, in the public API models and in the real AWS. An engineer built the scaffolding that makes them work. An internal version was created over a weekend and began to be adopted within a month. Turning this into a robust solution took a lot more work.

The quality of software in the AI ​​era depends on the guardrails that engineers design around agents. In AWS Emulator, these guardrails are fidelity testing verified against real AWS, a merge gate with multiple security agents that each review a change for different ways to attack it alongside a general code reviewer, and deterministic controls that prevent agents from touching code they’re not supposed to touch. We expect agents to build and review most software, with engineers deciding what will be built and when the work will be correct. We’re starting to learn how engineers lead teams of agents to build reliable, high-quality software, and the way AWS Emulator has come to life is a good example of how this works in practice. We plan to continue adding guardrails and refining how agents build them.

What you need to know when launching

The emulator code is released under CC0, a Creative Commons dedication to the public domain.

We will share more about this work on re:Invent. The full commit history and all issues reported by agents are public in the GitHub repository, and its “things that don’t work” folder contains issues describing behaviors found by agents that don’t yet match real AWS.