Software used to be a thing you bought. You licensed it, installed it, trained your people on it, and it sat there—a fixed set of features waiting for someone to click. The application was static. The judgment lived entirely in the humans operating around it. That arrangement defined what a software company was, and it quietly defined what every company that ran on software was too.
That arrangement is ending. Not because of a single breakthrough, but because the unit of software is changing from a feature to an agent: a piece of the system that can perceive a situation, decide, act, and account for what it did. When the unit changes, the structure changes. And when the structure of your software changes, the structure of your company follows—whether you design for it or not.
This is the part most leaders miss. They treat AI as a faster tool inside the existing org. It is actually a new kind of org.
From applications to organizations
Think about what an application has always been: a set of frozen decisions. A product manager and an engineer sat down, anticipated what users would need, and encoded those choices into screens and buttons. The software couldn't decide anything. It could only present options a human had pre-decided to offer.
An agent inverts this. Instead of pre-deciding every path, you give it a goal, the data it needs, a set of tools it can use, and guardrails it must respect. Then it decides at runtime. One agent is a clever automation. But that's not where this goes. The interesting systems are made of many agents—each owning a slice of work, handing off to each other, escalating when they're unsure, coordinated by shared goals and shared data.
Look at that shape honestly and you'll notice it resembles something familiar. Roles. Handoffs. Escalation paths. Shared objectives. A division of labor coordinated toward an outcome. That is not an application. That is an organization.
This is why building serious agentic systems—an agentic business operating system, say, where agents handle discovery, outreach, analysis, and operations against shared data and rules—feels less like writing software and more like designing a company. You're not specifying features. You're defining roles, accountabilities, and the conditions under which work moves between people and machines.
The company is becoming part human, part fleet
Here is the structural claim, stated plainly. The software company of the near future is part human team, part fleet of agents, coordinated by shared goals, data, and guardrails. And this won't stay confined to companies that build software. Every company that runs on software inherits the same shape, because the software they run on is becoming agentic underneath them.
So the relevant question for a CEO is no longer "which AI tools should we buy." It's "what is the operating model of a company that is half people and half agents, and how do I design it?"
That question has real, specific implications:
- Roles split into doing and judging. When agents do the doing, humans move up the stack to setting intent, handling exceptions, and owning consequences. The job shifts from execution to direction and judgment.
- Headcount stops being the proxy for capacity. You can no longer read a company's throughput off its org chart. Capacity now includes a fleet you can't see in a seating chart.
- Coordination becomes a design problem, not an HR problem. How agents and humans hand off work, where authority sits, what triggers escalation—these are now things you architect, not things that emerge from culture.
- Data and guardrails become the management system. In a human org you manage through goals, incentives, and review. In an agentic org you also manage through the data agents can see and the rules they must obey. Those rules are your policy.
What humans do when agents do the doing
The anxious version of this conversation is about replacement. The useful version is about relocation. Humans don't disappear from the work; they move to the parts of the work that actually require them.
Three things stay stubbornly human, and they're the three a CEO should be deliberately concentrating people on:
Notice what this does to the meaning of a job. A role stops being "the set of tasks this person performs" and becomes "the set of outcomes this person is responsible for, including the work agents do on their behalf." That is a different definition of a job, and it should change how you hire, evaluate, and promote.
Accountability doesn't move—even when the work does
This is the line a lot of organizations are about to cross carelessly, and it's the one a CEO most needs to own.
When an agent makes a decision—approves a refund, sends an outreach email, flags a transaction, drafts a legal document—who is responsible for it? The honest and only durable answer is: a named human. The work moved to the agent. The accountability did not. If you let accountability dissolve into "the system did it," you have built an organization that cannot be governed and cannot be trusted, and trust is the actual scarce asset here.
So the design discipline is this: every agent, or every cluster of agents, needs a human owner—someone who set its intent, can see what it's doing, and answers for its outcomes. That's not a compliance nicety. It's the load-bearing wall of an agentic organization. The teams that get this right build the visibility and the guardrails first, so a human can always see, understand, and override what the fleet is doing. The teams that get it wrong optimize for autonomy and discover, too late, that they've automated their own loss of control.
This is why AI is an operating-model problem and not a tooling problem, and why it cannot be delegated wholesale to IT. IT can stand up the tools. It cannot decide who is accountable for an agent's decisions, how much autonomy a given function should have, or which judgments must never leave human hands. Those are CEO and board questions. They are questions about the shape of the company.
What the CEO actually has to own
If your company is becoming an agentic organization—and if it runs on software, it is—then a handful of decisions are now squarely yours and cannot be pushed down:
- The autonomy map. Which decisions agents make alone, which they recommend and a human approves, which stay fully human. This is the single most important design choice, and it's a strategy decision wearing an engineering costume.
- The accountability map. Who owns which agents. Where the buck stops when a fleet acts.
- The visibility standard. What you must be able to see and reconstruct about any agent decision, after the fact, under scrutiny.
- The talent shift. Moving your best people from doing to directing, and being explicit that judgment, not output, is now the high-value skill.
None of these are technical questions. They're questions about how your company is structured, governed, and led. They happen to be expressed through software now, which is exactly why they keep getting mistaken for software problems and handed to the wrong people.
The takeaway
Software is no longer a tool your company uses. It is becoming a layer of your company's structure—a set of agents doing real work alongside your people, under your rules, toward your goals. As that happens, the line between "the software" and "the org" blurs, and the company quietly becomes something part human, part machine.
The leaders who treat this as a procurement exercise will end up with a faster version of the org they already have, governed by no one in particular. The leaders who treat it as org design—who decide deliberately what agents do, what humans do, and who answers for all of it—will build companies that are genuinely new, and genuinely theirs.
The technology will keep getting cheaper and more capable. The scarce thing was never the code. It's the clarity to decide what your company should become, and the discipline to keep humans accountable as the machines take on more of the work. That clarity is the job now. This is the work I do with leadership teams.