0 to 1 weeks at Gradiated

Our first revenue, post-training for PostHog AI and why I joined the team.

I'm new to Gradiated. I joined on Monday (three days ago). The team is small (there's six of us), and we're all here in Cambridge this week. It's more or less my job now to understand what the team is working on and write about it.

Write to whom?

Gradiated exists to deliver AI inference that is cheap and energy efficient. Anyone who cares how inference can get cheaper or why tokens per joule is a useful metric should be able to follow along. The interesting developments are in the technical details, and that's hard to follow without a background in chips, inference, and models. We'd like for it not to be hard to follow. I'm not an expert in any of the above, I'm a software engineer, and whatever I need to learn to be able to follow the work, we should likely write about.

What's going on around here?

We've just started selling inference on open-source models through an API directly to customers. We rented 24 top grade GPUs (B300s), self-hosted several open-source models, and set up API access to be self-serve.

With the infrastructure up and running we're focused on building our customer base. Perhaps this is obvious, but we pay for our GPUs per hour whether they're used or not, so while we have GPU capacity to spare we offload it to an inference marketplace. Our ✨first revenue arrived this week, it was from the marketplace.

Selling inference is only part of a much larger plan. It allows us to start learning what the real levers of cost are on serving production inference workflows. We can observe the current drivers of cost, and experiment with optimizations we think would be high yielding. This week I caught up on two such experiments.

Post-training for PostHog AI

PostHog is one of our first customers. They have an agent that lets you chat about your data. It uses Anthropic's Sonnet 5.5 and it costs PostHog $100,000 in tokens to run each month. The spend is significant enough for them to wonder how they can save money without sacrificing performance.

To start, we sent Josh our co-founder as a Forward Deployed Engineer to investigate and benchmark. The idea is to gauge how open-source models perform vs. Sonnet on recent data from production. The expectation is not that open-source models out of the box will match the frontier model. More practically, we want to know how big is the difference in quality on this specific work, and how much can we close that gap?

Open-weight models, by nature of having their weights published, allow you to continue training the model. In pre-training, the model learns to perform well on a wide variety of tasks, and is subsequently post-trained to improve the model's behavior and knowledge in more targeted areas. You can post-train a model to teach it how to call your specific tools, and you can also post-train to teach the model what good looks like on your specific work.

You can try at least. Josh has been running an initial experiment where we post-train Qwen, a smaller open-source model to learn from production traces where Sonnet performed well. The details below.

How did we post-train?

We did Supervised Fine-Tuning (SFT), a common first-try approach. We built a dataset that started with 5000 production traces, anonymized the data (this was hard, if you're curious let me know) and filtered that down to around 1000 conversations where the differences between Sonnet and Qwen would be most observable.

Expectations

There is literature suggesting using SFT in a situation like this probably won't work and might actually degrade model performance, but that's not a guarantee, so it's worth trying anyways.

Results

We did actually observe this. After our SFT experiment, the resulting model performed the same or slightly worse than base Qwen without post-training. With 18 tasks, a difference of one or two is indistinguishable from noise, but there's no sign that SFT helped.

Tasks Qwen passed, out of 18

  • 5Before
  • 41
  • 32
  • 53

Rounds of training (epochs)

Why might it not be working?

The data used in post-training was a paired list where each pair is a user's question and Sonnet's reply. SFT trains Qwen to answer like the frontier model. Ideally Qwen learns the functionality that made Sonnet's answers useful, but this isn't always the case.

For example, Sonnet often replies with confidence, which makes sense given it's a highly capable model. SFT can result in Qwen learning to copy confidence without the functional gains to match it.

Qwen may be able to solve the problem, but if it tries to solve the problem like Sonnet it might end up just sounding like Sonnet.

How sure are we that this is what's happening?

Medium confidence. It makes sense that this is what could be happening. It's a concern with "off-policy" training. "Off" because it's learning from another model and not itself. We could try to probe at why it's performing poorly, but that's a new experiment.

Josh is now working on "on-policy" post-training methods. "On" meaning Qwen can learn from its own good and bad attempts at responding to a question. Results pending.

FPGAs

Aidan is working on a long-term project. It's a knowledge building endeavor, where instead of studying how to optimally use existing hardware, he's focused on what might be possible if every part of the hardware chain was open for exploration.

He gives the team weekly lectures, which feels like a special way to learn. This week he's working with an FPGA (field-programmable gate array) chip. Unlike a CPU or GPU, where the circuit is fixed and you write software for it, an FPGA lets you wire the circuit yourself and re-wire it whenever you like. He's found a small language model that fits in the on-chip memory and the model tells jokes. We're curious how fast it can tell the jokes, how much heat that generates and what's the cost of doing so.

We've yet to see the jokes because finding a charger for the FPGA is reportedly much harder than programming it to serve the model. More on this when we can find one.

The week was fast. The kind of content we'll produce is still being shaped. If there's something you're interested in understanding well, email me at charley@gradiated.com