Why is it hard to build a scalable evals system for legal agents?
A few months ago, we realized that rapid advances in model capabilities necessitated a new kind of agent harness designed for autonomous execution of large legal tasks. Suddenly, the guardrails we had designed to work around intelligence limitations turned into constraints that held the models back. We embarked on a project to completely rip out our old assistant to replace it with our agent. This shift surfaced significant limitations in our evals infrastructure. Addressing them was a prerequisite to transitioning from the assistant to the agent without degrading the user experience.
The first problem was scalability. The assistant was designed for smaller tasks in environments scoped to a few documents, while the agent was expected to run for hours in project matters with thousands of documents. Back then, our evals ran locally which overloaded the computers when we ran them with anything beyond a small number of project documents and a modest number of concurrent runs. The concurrency limitation meant running larger evals was time-consuming or not feasible at all. Today, a typical experiment might have 100 cases, 4 arms, and 5 trials, resulting in 2000 parallel runs.
The second problem was environment-fidelity. This was tied to our “One agent” initiative, the idea of the agent as a surface covering our entire app across the Editor, Tabular Review, the Word Add-in, and more. Compared to filesystem-centric coding harnesses, the environment of the Legora agent spans services such as a headless DOCX editor, self-hosted Turbopuffer instances, PDF conversion services, Postgres and authz databases, blob storage, dynamically spun up code execution sandboxes, and more. Some of these could be run externally, but others needed to run in isolation tied to the specific version of the code we wanted to eval (for example, our Postgres schema changes frequently, and the harness often depends on its exact state).
We therefore separated the eval environment into two parts: a versioned Legora runtime and a reproducible legal matter. We call the latter an Eval World: a snapshot of the documents, database records, permissions, and blob state needed to run one or more cases repeatedly from the same starting point. The core challenge was creating thousands of isolated Legora environments in parallel, restoring the same worlds into them, and unleashing the agent to run ambitious long-running tasks inside.
The evals data model: Worlds, cases, datasets, experiments
The data model is based on worlds, cases, datasets, and experiments. Worlds constitute the project environment the evals run in. This is to simulate the project matters of our users, which typically include uploaded documents, internal playbooks, org DBs, past chats, tabular reviews, editor documents, and more. Inside the worlds, we define cases, which are the tests we run to evaluate the agent. The cases include prompts and scoring criteria, i.e. the rubrics, that we pass to LLM judges that evaluate the result. Datasets are collections of cases selectable from any world.
The experiments are defined in a tensor shape with the cases, number of trials, and experiment arms as the dimensions. An arm is an instantiation of an agent harness, mapped to a commit SHA and a config to select the model, product mode, reasoning mode, and other parameters. Each cell of the experiment tensor is treated as an independent run.
Approaches: From local runs to sandboxes
Running evals through Modal sandboxes and volumes
Since our old approach of running evals on local machines was not scalable, we decided to look into cloud-managed sandboxes. This could potentially solve both the problems of isolated environments and horizontal scalability across concurrent eval runs.
We needed an approach to get our monorepo into the sandboxes from specified commits, load the required secrets in, run the full app inside, restore the state defined in the eval worlds, run the agent end-to-end on the selected cases, and store the results. We also needed to do all this a thousand times in parallel. The system had to be secure, durable, easy to maintain, and fast enough that people want to use it.
We initially explored Harbor for sandboxed eval execution, but found that the more complex components of our application, such as the Legora code sandbox and DOCX services, required a more flexible solution. After experimenting with various sandbox providers, we chose Modal due to the scalability and low latency of their sandboxes.
Modal provides cloud infrastructure in the form of sandboxes that enable programmatically running untrusted code in the cloud. Their sandboxes are designed to handle complex environments, scalable parallelization, and low latency. They recently posted about how they scale to 1 million concurrent sandboxes in seconds, which for our 2000 concurrent runs satisfied our requirements with room for scaling up to larger runs in the future.
To solve the environment-fidelity issue, we used Modal volumes. Volumes are a distributed file system that act like a CDN for Sandboxes to enable fast movement of files. We required scalable storage to handle the large amount of documents in our eval worlds, and to easily get them into the environment of the sandboxes. Volumes solved this, and were also a convenient way of snapshotting the state of the eval worlds to deduplicate shared setup and speed up eval runs. Additionally, we stored cached docker images in volumes, to speed up the docker compose step of running the app (volumes work for this, but we are currently working on moving to filesystem snapshots https://modal.com/docs/guide/sandbox-snapshots).
An early problem we faced was that it was time consuming to build the image that defines our app. To alleviate this, we set up a pipeline that builds this image hourly, and pre-warms the sandboxes. This significantly sped up launch, since the reset to commit stage went from building the full app to just the delta since the previous build.
Finally, we had to securely load the necessary secrets into the sandboxes from our key vault. To do this, we set up an hourly job to sync their values into Modal secrets. We ensure frequent secrets rotation, minimized privileges, and a minimum age for all external dependencies to mitigate supply chain risks. Another security feature we included is to only allowlist the set of domains strictly needed by our agent via Modal’s sandbox domain allowlisting feature.
Achieving durability in long-running eval runs through Temporal Workflows
The biggest challenge with our initial setup was that the eval runs frequently appeared to stall or be dropped, accompanied by intermittent failing network requests. To solve this, we turned to Temporal as it was already extensively used across our platform.
Temporal enabled durable execution of the eval runs, handling redeploys, reconnection, flaky errors, reduced load on the backend through a dedicated worker, and easier distribution of load across the stages in the eval run. We defined the eval experiment as a workflow, with the runs as child workflows. We split activities between the setup, run, and persistence stages of the eval to isolate the retry-able stages of the run and to split activities by load.
A factor to handle was that the memory load of the eval run varies significantly across its stages. The memory required to start the run was relatively light considering that the blobs and documents to load are pre-stored in the volumes. The main run activity is a poller that runs a command inside the sandbox that starts the eval and logs the eval state to a file in the sandbox. With that, on retries or reconnection, the activity simply reads the file to check the state of the eval and decide whether to keep polling, rerun the command, or return an error. The heavy part of the run happens when we pipe the results from the eval from the sandbox back to the worker to store it in a Postgres DB. An agent run can consume many millions of tokens, and we observed the machine fully crash during a run with high concurrency. To solve this, we set up a dedicated queue on the evals worker specifically for the results parsing activity configured with higher memory and lower concurrency.
Moving from Modal gVisor to Modal VMs
With Modal and Temporal, we had a horizontally scalable and durable system for running evals. However, getting the Legora app to run inside the sandboxes originally took a series of workarounds, all stemming from limitations in gVisor, the container runtime of the Modal Sandboxes. The challenges were rooted in our need to run Docker services inside the sandboxes. As the sandboxes themselves are containerized, we required DinD (Docker-in-Docker). This led to complications related to networking and running FUSE in the nested Legora containers where each fix led to a deeper edge case.
gVisor is a library that implements a container runtime named runsc, which works by re-implementing a subset of the Linux kernel in user space. This adds a layer between the syscalls and the OS kernel that narrows what programs inside the gVisor containers can do. The purpose is increased security, which is why it is often chosen for running untrusted LLM-generated code, but it also means that valid features of your program can get rejected with an error like ENOSYS ("Function not implemented").
Our Docker-run services rely on Docker’s embedded DNS server for address resolution, and gVisor does not support Docker’s full networking stack. Workarounds include using the iptables-legacy interface instead of the more modern iptables-nft, as gVisor’s kernel doesn’t fully support nftables. Despite workarounds, gVisor does not have enough coverage to fully support docker bridge networks for nested containers. We therefore had to rely on network host mode, and handle network address resolution ourselves.
This worked for standard docker services where ports were unique and static, meaning we could reliably publish them on the host, but our dynamic code sandbox service was a tougher edge case. The code sandboxes are spun up lazily when the Legora Agent needs them to run the shell tool, and when a new one is created, its port is created through Docker’s ephemeral port mapping. This is not supported when running the containers in network host mode. We therefore needed to implement our own dynamic port allocation logic.
Still, we couldn’t get the shell tool in our eval runs to work. After diving into the logs, we would see “fusermount: mounting over filesystem type 0x01021997 is forbidden”. It turned out another problem caused by gVisor limitations was related to the FUSE mount that the code sandbox relies on to access files stored in blob away from the sandbox pod. We validated that FUSE was supported in the gVisor top-level container, and so tried to forward that FUSE mount into the nested sandboxes, but kept facing similar issues. We finally ran the code sandboxes as Node processes in the Modal sandboxes to get FUSE to work. This solved it, but it was brittle and error-prone.
Luckily, we were given preview access to Modal's new VM Sandboxes around this time. This let the sandboxes run on top of a full virtual machine rather than gVisor. Each Sandbox gets a real Linux kernel, so Docker behaves the way it would on a normal Linux host. We deleted every workaround and got the Legora app fully working inside the sandboxes.
The final system was then Modal VM sandboxes running durably via Temporal Workflows. We began scaling up the usage, migrating old datasets over, and building out a full evals platform around it to enable the whole team to easily run experiments. The results quickly became the basis of critical choices we made around harness design and the underlying models that power it.
Impact
Today, our evals platform is our primary way of evaluating the quality of the agent. It has enabled significant speedups, quality improvements, and discovering what foundation models work well for our use cases and harness. Two highlight unlocks are the Legora BAR and our internal agentic autoresearch workflow.
Legora BAR
Our evals infrastructure enabled us to run the agent on our BAR benchmark. The benchmark composes long-horizon and realistic legal tasks to evaluate the agentic abilities of our harness. After using the evals infrastructure to run the cases in BAR, we conducted a thorough analysis to answer what models performed the best at various cost and latency levels. We used the results to determine which models to use in production.
Autoresearch
Many new use cases were unlocked after we wrapped the evals app in a CLI. Most engineers at Legora currently interface with evals purely through agents that create eval cases, run experiments, and analyze the results autonomously. A key enablement from this is that our agents can now automatically hill climb the harness. This entails giving the agent a concrete metric to optimize or problem to solve, tools to launch eval experiments and analyze their results, and a budget. The agent then iterates for as long as needed to optimize the metric. This has been used for features such as tool search, powerpoint generation, and reproducing research papers to make our harness more cost-efficient.
