Tandem has been working with our model partners to improve LLM performance on engineering work. We have been helping them do long-running reinforcement learning (RL) by providing our tasks and our perspective on what agentic engineering could look like. We wanted to give some insight into the types of tasks we are making for RL and what that process of task creation looks like.
Our approach
Our goal is to help LLMs accomplish complex hardware engineering tasks in the real world.
Currently, the primary way we improve models’ abilities is via reinforcement learning. This consists of making tasks, essentially games, in which the model needs to meet a specific criterion. After thousands of attempts, it gets better at that game, and thereby learns skills that apply to that specific task and generalize to others.
The problem is that current benchmarks and evals for these types of tasks are basic, focused on part creation, CAD as code, simple drawing generation, and other fairly rudimentary operations. Very few address long-horizon engineering tasks that a mechanical engineer would actually have to do, for example, implementing an engineering change order or giving DFM feedback.
We think Tandem is uniquely placed to help solve this problem because we understand what agentic engineering can actually look like. Our platform does it every day in production. We are using what we have learned from our own internal testing and production usage, along with some proprietary data sets, to create these tasks. We are also making some specific bets on where agentic engineering is going.
We really believe in computer use.
We believe that in order for models to succeed at engineering in the real world, they need to be trained with realistic engineering context.
Why computer use
We are particularly bullish on computer use because we think it is a better way to get models to do CAD. Models have been successful at doing some CAD operations via API and CAD as code. But complex operations like surfacing, selecting intricate lines within sketches, and selecting faces are extraordinarily difficult to do via API. Via computer use, they are simply a matter of a few clicks. And while models are fairly slow and not that effective at computer use today, there is a lot of work going into making them better. Based on our internal benchmarking, we believe that as computer use improves, these models will do CAD better through it than through the API.
In fact, from what we have seen, training on CAD tasks significantly improves these models’ ability to do computer use. CAD entails operations like selecting fine lines, which require a high degree of pixel accuracy, and dragging, which is surprisingly hard for some of these models to do. It also requires stronger spatial reasoning. If you look at standard computer-use benchmarks like OSWorld, CAD is routinely the hardest type of task. These operations, and the understanding required to do them, are more complex than the vast majority of other tasks. That pushes the model to its limit, which is what makes CAD such a great learning environment.
The other reason for this bet is that we think computer use is much more generalizable. Many programs do not have an API or a clear way to programmatically access them at all. For the ones that do, that access sometimes needs to be specially granted, which is not easy for the lay user. Computer use requires none of this and works through the same interface a person would use. The skills also seem to transfer. If you teach a model to use SolidWorks or NX, it is then better able to use simulation software or ECAD. That brings us closer to a world where these models can work across all of your tools.
Why context
The second bet we are making is that the models need all the context they would have in a production scenario. Getting an LLM to help with some calculations or do some basic CAD tasks is interesting. But by connecting models with large amounts of context, and having them work across many different programs, we can make them useful for real, complex engineering work.
Models will not be able to do the types of optimizations and tool use required for real-world work without exposure to highly complex, context-rich tasks. The only good way to make tasks of this quality is to derive them from production documentation, assemblies, and experience.
The parts of a task
Here is what a typical CAD computer-use task looks like.
The input. The state the task starts in, usually a CAD assembly and its parts.
The gold. A CAD assembly that is a solution to the task. Many of our tasks are fairly open-ended, and so there are many possible solutions.
The gold trajectory. The clicks an engineer would make to complete the task. Sometimes we train on this to pull the model’s choices closer to it. We do that when we want to encourage a specific strategy, whether that is a more efficient one or one that uses a different tool.
The verifier. How we grade the task. This is the hardest part of creating a task.
The prompt. The instructions we give the model. In some cases the prompt is a document. In others it is a couple of words.
We start the model with the input file open. Then we give it the prompt. The model begins doing computer use. It looks at the screen, takes an action like typing or clicking, looks at the screen again, and repeats that until it believes it has completed the task.
From there we run our verifier, which grades the model’s result from 0 to 1, with 1 being a perfect solution and 0 being what the model scores if it does nothing. We usually count 0.9 or above as a pass.
Making a task
Making the input, gold, and prompt is easy. The hard part is writing a verifier that grades the model’s ability to do that open-ended task. The other hard part is calibrating the task so that it sits at the ideal difficulty for a given model to learn from.
The verifier
Take an open-ended task, say an engineering change order to alter a nozzle to meet a new set of performance criteria. We have to grade an attempt in a way that accommodates the wide variety of potential solutions, while also saying something useful about partial ones. Many different outputs could all earn a perfect 1. But we also need to define what a 0.3 or a 0.5 or a 0.7 looks like, in a way that tells the model how close it got to any of those solutions.
We do this with multi-parameter verifiers specific to each task. A verifier may check any of the following.
Physical, parametric, and geometric properties. For example, the size of the part, the hole count, the fillet radius.
Internal CAD file properties. For example, part mass, specific mates, tools used.
Calculation and simulation. For example, calculating a mass or density. Or something more involved. If you are making a rain cover to go over a camera, simulate whether it will actually keep rain off the lens.
Not all checks are explicitly mentioned in the prompt. We believe some practices should be implicitly enforced, like feature tree optimization or DFM basics such as breaking edges.
Calibration
Once you have made a task, it comes down to calibration. Tasks must be at the optimal level of difficulty, not so easy that the model learns nothing and not so hard that it never succeeds.
The standard metric is pass@10, and model providers are usually looking for something in the 10% to 40% range. Look up this metric if you have not heard it before. A pass@10 of 20% means the model succeeds about 2.2% of the time at the given task. The only way to know where a task sits is to run it 50 to 100 times and see what the pass rate is.
That can be expensive, especially for computer-use tasks, which can use 100x the tokens of a pure text task. For our long-horizon tasks, at around 400 steps, a single run takes 5 to 6 hours. Doing this many runs takes a long time even with a significant amount of parallelization. Many engineering tools like CAD and simulation are also GPU-intensive, so we are spending compute on the environment as well as on tokens.
The process of calibration itself, at a high level, is fairly simple. You run the task, see what its pass rate is, and adjust it to get closer to your target. Adjusting may just mean changing the prompt, or in some cases throwing the task out and starting again.
For example, in one task we have a complex gear mechanism taken entirely apart. The prompt is two words, “Assemble this.” The model has to assemble the whole mechanism with no other input. Models today are actually able to do this pretty well, but to make it easier or harder you can change the number of pieces, or pre-mate or unmate some of them. You can also adjust the prompt and give some hints about the positioning.
Once you start making tasks, you get a clear sense of what these models are good and bad at, and generally what a good task looks like. One thing that surprised us is how rarely they use the drag function. You will almost never see a computer-use model click and drag. The computer-use tasks they were previously given could be done with static clicks alone. Once we noticed this, we built difficult tasks around CAD features that require dragging.
Ultimately, our ability to calibrate a task to the correct difficulty determines whether it will be useful for the model. This ends up being where we spend most of our time.
The environment
The last thing is the environment, where these tasks run. For our CAD tasks, this environment is a version of SolidWorks or NX, hardened to prevent the model from using certain tools. For example, we block the model’s access to the gold assembly, so it cannot copy the solution. We also block the command line and macros, so it cannot gain programmatic access and skip developing computer-use skills.
These models are highly effective at finding novel and unintuitive ways to hack environments. To harden an environment well, the best thing to do is just to run many, many rollouts, identify when the model finds a new exploit, and patch it.
Tandem and agentic engineering
We are excited about this type of work and about how we can use our positioning to help improve agentic engineering. We cannot wait to see models trained on our tasks put the context in our platform to work, assisting engineers with DFM, requirements management, and CAD.
We are interested in working with a wide variety of model providers, including large labs and smaller labs with more unique perspectives. If you are working on this or want to know more about the types of tasks and data discussed, please reach out. We would love to chat.