Internal only

Goal: Figure out if you should build anything.

When to use: You are the only user and are trying to communicate ideas rather than ship usable software.

Models to use: Whatever is fast and good enough (in practice, i find this to be gpt-5.3-codex at medium reasoning effort.)

What you’re allowed to ship: Literally anything. Terrible is fine. Worse is better.

Attention to agent effort: Virtually none. Let it run as long as it wants, ship terrible stuff, expect to throw it away.

Alpha

Goal: Figure out if you’re building something anyone wants.

When to use: When you have < 10 users, and you know most of them directly through 1 degree of separation. You can talk to all of them, and you kind of expect them to churn, because they’re being nice to you more than being a real user.

Models to use: Basically the same fast / good enough.

What you’re allowed to ship: Things that don’t have serious security bugs and unusable performance characteristics.

Attention to agent effort: Slightly more. Don’t let it do anything absolutely terrible, but in practice most modern agents are good enough to not make the sloppiest mistakes.

Private Beta

Goal: Figure out if you’re building something anyone wants enough to use frequently.

When to use: When you have ~10 users but none of them are 1 degree of separation. More importantly: Some of them haven’t churned and are actually getting a quantum of utility.

Models to use: Start thinking about something that’s better at reasoning, and slower.

What you’re allowed to ship: Roughly the same as Alpha, but it should actually be useful for someone. You should still be embarrassed by how bad it is.

Attention to agent effort: I recommend having the agent perform an after action report style summary where it carefully explains all of the changes it made (in a text file) and you should be able to ask questions of your agent to ensure you’re on the same page.

Public Beta

Goal: Figure out if you’re building something people want to use frequently

When to use: When you have enough users that you don’t know all of them / can’t talk to them individually. (Dunbar’s number is about ~150 and is probably a decent guide for consumer products. For B2B, it’s some meaningful amount in your target market.)

Models to use: Slower and more thoughtful for anything that touches all of your users.

What you’re allowed to ship: Something that mostly works, but has a few rough edges. Enough people should be using the product that every minute you spend of effort results in at least 10x saved effort by your users.

Attention to agent effort: More thoroughly code reviewed… not necessarily by a human but there should be some process for maintaining code standards beyond YOLO. (Linters, type checkers, actual tests, playwright tests, etc.)

Production

Goal: Make something people want.

When to use: You have something good enough that it spreads naturally by word of mouth.

Models to use: Ones that are consistent and never break. In practice that means thoroughly vetted and able to be trusted.

What you’re allowed to ship: Something that works in a way that you anticipate will be quality. All of your users should use this part of your software, and every minute you spend should result in 100x saved effort by your users.

Attention to agent effort: Systematic and process driven. You should have an audit trail that proves your software does what you expect, and you shouldn’t have any surprises.

Nobody is shipping production code with agents today.

By my definition, I think the best teams might be shipping public beta quality code. I’m unconvinced that anyone has a robust production level pipeline without thorough human intervention.

It won’t be that way for long, but as of today I think it’s that way.

This contrasts with a command-and-control coordination model, in which a commander is responsible for decision-making, and subordinates are responsible for execution.

The end result is outputs (products) that require less effort to achieve better quality.

This post focuses on the three pillars of product command.

Coding with AI feels like a slot machine. Every once in a while, you hit the jackpot, but most of the time, you’re holding onto hope. Instead of building, you’re babysitting... watching over the shoulder of an AI bound to make mistakes, and it’s your job as the developer to make sure it doesn’t.

That sounds terrible.

The good news: If you can just explain your intent, AI can fill in the gaps. Then you’re creating, designing systems, and building... not babysitting.

Centralize intent

Your goal is to explain what you want and what you don’t want, with as few words as possible. Based on mission command, I’ve adopted these headings:

1. Purpose

2. End State

3. Constraints

4. Tradeoffs

5. Risk tolerance

6. Escalation conditions

7. Verification Steps

8. Activation / Revalidation

9. Appendix

Here’s my copy-pasteable template.

Each of these headings doesn’t tell the AI what to do implementation-wise, it tells it the strategy you’re going after. You don’t have to tell it how to do things, you have to give it the why.

I work with the LLM to come up with the doc... usually by asking it to gather context on the subject first, then build the doc, and 9/10 of the time it gets the majority of it right.

I then review and ensure I agree with every single word in the document.

Distribute execution

Once I’ve built a good enough INTENT.md, I start a new context window, paste in the link, along with this text:

let's implement this. Use your best judgement. Feel free to use subagents if they make your life simpler. Make sure to run through the verification steps thoroughly. I'm not going to be around, so prioritize using your best judgement, making frequent commits but don’t submit them, keep them local. Try to find any runbooks or policies that apply to your work, and make sure you follow them. You can do this! Good luck! -- make sure to build one or more AARs while you're going (if you do a commit, ideally an AAR should accompany it as part of the commit.)

Usually, the AI takes 5-20 minutes to build something, and then at the end, I ask, “Let’s build an AAR.”

Improve through structured feedback

An AAR is military for “After Action Report.” It too has a template:

1. Context

2. Intent

3. What actually happened (facts only)

4. Delta analysis (why it was different)

5. Initiative Assessment (When the AI made its own decisions)

6. Weaknesses in intent (Parts where the intent wasn’t clear enough)

7. What we will sustain

8. What we will improve

I run this after each session. It captures from the existing AI’s current context its analysis of the process it just ran.

Next, I manually review / test the changes. Usually, this means I’m using the product.

If the product is 80% what I expected, I will ship. If there’s a few changes (minor placement issues in UI, for instance) then I will fix them with the LLM and update the AAR.

If the product is < 80% what I expected, I will explain what happened, identify true weaknesses in the intent, then trash the current work, go back to the intent, and have it run the same process.

I’ve only ever had to run this process 3 times to get what I want, normally I just run it once.

Try it, it will work, I promise.

Policy: Long-term memory

One quick side note: I save all the things I want my AI to learn in a folder called “policy” next to the INTENT.md. I use the AAR to keep it updated, but like in a real organization, updating policy can have knock-on effects, so I do it sparingly.

Product command is a coordination model for centralizing intent (the why, success conditions, and constraints), decentralizing execution (letting coworkers decide the less crucial details), and aligning both through explicit verification steps.

This contrasts with a command-and-control coordination model, in which a commander makes decisions, and subordinates execute.

The result is outputs (products) that require less effort to achieve better quality.

This post focuses on building the right environment for AI to find its own context.

LLMs are a communication problem, not a computer science problem.

I haven’t written any code in 6 months. In the last month, I’ve been able to let Codex run unattended for 20-60m and get a feature right the first time.

The key? Quality communication. I’m a startup founder and software engineer, but I started my career as a journalist. That means I actually studied Communication in college, not computer science.

CS and Comm might not be as divergent as you think. Weirdly, these two fields have a shared heritage and patron saint: Claude Shannon.

Shannon describes a scientific definition for communication called A Mathematical Theory of Communication. CS focuses on the math, communication studies focuses on the mechanisms.

Computers are obviously a Computer Science problem. From 1950 to 2015, communication mechanisms were literal. Feedback, signal-to-noise, transmission mechanism, etc., all literally meant wires transmitting electrons. Machine learning isn’t about understanding words, it’s about Bayesian regression. That means math, not communication.

LLMs are different. Training time is still all math and builds the model's instincts. But the intelligence part of artificial intelligence happens at inference time, not training time.

And inference time is communicated in words, not numbers.

Inference engineering

I’d like to introduce a new term: Inference engineer. An inference engineer is someone who manages both sides of Claude Shannon’s model... both the math and the mechanics. That means, yes, writing great prompts, but more importantly, it means clear communication between the person using the AI and the AI implementing it.

A few years back, I left Facebook to build a startup with my best friends. Our startup helped take long videos and turn them into short ones using LLMs.

As we started to scale, the quality of our AI implementation didn’t. Our company’s biggest hurdle ended up being humans reviewing the quality of clips, not developing our product or marketing.

If AI is supposed to save time, it has to actually do that. And if it doesn’t, humans-in-the-loop behind the scenes have to cover for it. Our LLM wasn’t good enough, and so our business couldn’t scale without people. That made our team ask more fundamental questions.

How can we actually know something is reliable? How do we communicate our preferences to the AI? How can we communicate our users’ preferences to the AI?

Sean Grove is one of the smartest people in this area, and he turned me onto Model Spec, Constitutional AI, and other concepts about how to align and verify model behavior.

He gave me the words to communicate what I was feeling: Communication with models defines success with AI. Specs are permanent, code is an artifact of a spec.

Programming will go away, specs will not.

Learning -> building

We took this idea and tried to figure out a practical application. For the last year, our team has been working as sort of consultants... circling ideas like fine-tuning, evals, execution frameworks, and communication theories to fill a growing confidence gap between model performance and consistent outputs.

We’ve worked with some great teams, and were always able to close their trust gaps in surprising ways. (I’ll be posting about those ways! Like and subscribe!)

Finally, we’ve figured out how to explain it. We call it product command.

Product command is a coordination methodology designed to create an environment for an agent to discover the right information at precisely the right moment.

That means agents, rather than humans, tell agents precisely what to do. Humans instead describe their intent and leave the AI to build its own context windows sufficiently to do a task.

People (or other parent agents) are responsible for specifying what the goal is of the next interaction, and the agent is responsible for execution. People shouldn’t be digging in to see which tool calls were made, they should be specifying an intent and then ensuring that intent was met.

The most basic implementation is creating artifacts that say precisely the right amount of information to execute a plan, let agents execute that plan mostly unattended, and then learn from the execution to improve the loop.

We’ll have more tactics and info as we go. In the next post, I’ll explain why this works and why human institutions don’t fundamentally get more effective as they scale, but AI can.

Subscribe now

Writing is thinking, my words are my thoughts.

I’m pretty forward looking, and a lot of my friends have asked why I’m ok with AI writing my code but not writing my words.

To me, modern generative AI represents the p50 of human experience. In a lot of places, I have higher than p50 human experience, and anywhere where I have less than p50, I’m hoping to expand and grow using my own skull.

Anything on this site, or anything on my social media, with my byline, will be written by me. No exceptions.

Anything that doesn’t have my byline but is on something I work with (ie boltfoundry.com etc.) you can probably safely assume AI wrote it.

Why is this your first post?

I want to share my journey from eager 15-year-old would-be journalist at dreamcast.net, to minor tech influencer, to startup founder, big tech employee, and now an Inference Engineer.

This is my story, and I think it’s important for you, dear reader, to know that I’m planning to spend at least 10x more time writing a post than it takes you to read it.

You are important to me, and even though I don’t yet know your name, I’m writing for you.

I care.

I’m building this for you because I think I’ve had a pretty deep breadth of experience ranging from traumatic upbringing to professional success… from the depths of depression and suicidal behavior to the joys of parenthood, self care and emotional management.

I want to share this so you can hopefully speedrun your own trauma, and if you can learn something from my life experiences and perspectives, I want them to be as lossless as possible.

I don’t want to diffuse my experience through the p50 of humanity, I want to present the p100 of Randall Bennett’s life.

I’m mostly going to talk about AI, communication, and mental health.

For me, the three areas of my life I’m most focused on are AI, as defined as post GPT machine learning, communication, as defined as talking to other individual people, computers, and large groups of people at scale, and mental health, as defined as learning from traumatic experiences that someone doesn’t choose, challenging genetic traits and less-than-ideal neurochemistry.

I am not just those things, I just know how to write about them.

I care about a lot of other topics. Feel free to ask! (If you’re a paid subscriber, you can access my chats.) The main things you can ask about: What it’s like to be a parent of 3 autistic boys, and 4 kids in general… moving to a place never having been there before… why having blue hair was fun and why i don’t have it now (i might write about this tho at some point)… how religion is basically the anchor of my entire existence (please ask! I don’t preach, it’s broadly applicable).

I’m not going to write about those topics generally just because I don’t really know how to approach them on a generalized basis yet. They’re things that are kind of unique to me, and require a more custom chat where I understand a person’s needs to tailor the message.

The religion stuff is especially important to me, but I don’t want to have people think I’m prescribing it as something that is universally important to believe the way I do, or that I should convince you of its truth or falseness. Or, that I’m somehow a good illustration of my religious habits / heritage.

I just want to focus on things that I’m able to explain in a free-er manner, that I can generally apply in a way that doesn’t cause me a lot of mental energy to say.

Thanks for coming, tell your friends, like and subscribe!

I’m building this for me as much as you, but I’m really hoping that I’m actually a decent communicator and my perspective will resonate with someone. If that’s you, then welcome! Hopefully you know others like you who I can help.

If that’s not you, then thanks for reading! You should still share it with someone else anyway.

A final note: I will use generate AI to generate ideas, and possibly be an editor, but I’m literally not planning to use it to insert text on this site. Anything read on randallb.com will have come from my fingers, videos, etc.

Sign up now so you don’t miss the first issue.

Subscribe now

In the meantime, tell your friends!