The AI Cost Stack Nobody Maps

AI products rarely become unprofitable through one bad decision; they get there through dozens of good ones. I have sat in many roadmap reviews for products that ship AI features. A product manager presents a capability, the business case shows the value clearly, and when someone asks what the model usage will cost, the answer is a footnote: small number, manageable, next slide. It gets approved, and a quarter later a different feature arrives with a different footnote and gets approved too.

Every footnote is small, until the quarter when finance asks why margins are shrinking.

Traditional software has trained us to treat marginal cost as a rounding error: each extra customer costs almost nothing to serve. AI features break that habit. Each one carries a recurring cost that grows with something: usage, seats, data, or session length. A roadmap full of AI features is a stack of spend commitments, approved one footnote at a time, owned by nobody as a whole, and the total usually surfaces later as a margin problem.

The post in one image: the price meter and the cost meter are different instruments.

I spent years running monetization for large product portfolios, and much of my work since has been monetization for AI products, where I map this cost side in detail. Two of those products appear through this post as running examples. What follows is the map I use: the number that shows why the map is worth drawing, the four layers of the stack, and the three decisions the map forces.

The same features can cost 10 to 25 times more

For one of those products, I modeled the same monthly workload two ways. Built with discipline about which model does which work, the AI cost lands around eight to twelve dollars per user per month. Built the default way, with a frontier model doing everything and no caching or batching, my estimate for the same feature set is one to three hundred dollars per user per month.

Same features and same user, but a gap of 10 to 25 times, and the whole gap is architecture.

This is why the “AI blew up our costs” stories keep appearing. The cost was never in the feature but in decisions nobody treats as product decisions: which model does which task, what runs overnight versus while the user waits, what gets cached versus recomputed every time. If those decisions happen by default, your margin is set by default.

The four layers of the stack

The labels matter less than the habit of asking, for every AI feature you ship, which layers it touches and what each one grows with.

1. Interactive inference. A user asks, a model answers, and the user waits, which makes this the layer everyone prices for because they can see it. It is also the most inflated, since the default sends everything to the most capable model when most work does not need one. In the cost model I built for that first product, the expensive model only handles the small set of outputs customers judge the product by, routine classification goes to the cheapest model, and anything a plain formula can do never touches a model at all. The biggest reduction came not from negotiating model prices but from asking, for each task, whether it actually deserves the best model available. Most of what looks like frontier-model work is not.

2. Background automation. The work the product does while nobody watches, like research prepared overnight, monitoring, or results computed in advance. This is where AI spend concentrates as products mature, and it is also the cheapest work to serve because nobody is waiting. The work that can wait can be batched, and batch pricing from the major providers runs at roughly half price. In that same product, moving the heavy generation to a bounded overnight batch roughly halved that cost line and put a ceiling on it, since the product now decides how much work happens per day rather than the user’s clicks.

3. Data gravity. This cost grows with the customer’s data rather than their activity, and any product that indexes what customers bring, documents, tickets, call notes, pays it. A customer who does nothing still gets more expensive every quarter, because their data keeps piling up. The other product caps its free tier by data volume rather than by features, because volume is what costs money here; a feature gate would have given the most expensive customers a free ride. When your cost grows with data, your pricing should acknowledge data.

4. Session gravity. Picture a user deep in a long working session, drafting a proposal with the AI over dozens of turns. Most AI products re-send the whole conversation with every turn, so the longer the session runs, the more expensive each turn gets. One real week of heavy use I traced showed about six hundred model calls, averaging well over a hundred thousand tokens of built-up context each. At list prices that week costs about eleven hundred dollars built naively, about a hundred and seventy with the repeated context cached, and fifteen to twenty-five dollars restructured so the AI works from the current document state instead of re-reading the whole history, roughly a 50x difference end to end. Ordinary chat is cheap but the long document grind is the cost center, and most teams have never measured theirs.

The three decisions the map forces

Drawing the stack forces three decisions most product organizations currently make by default, which is to say, do not make at all. They are the cost-side cousins of the portfolio pricing problem I wrote about last time.

First: does your meter match your cost curve? The first product charges a flat monthly rate, usually a margin time bomb once AI costs sit underneath it. It is safe here for one reason: the product, not the user, decides how much expensive AI work happens each day, so no user behavior can produce unbounded spend. The moment a feature lets users trigger any amount of expensive generation, flat pricing stops being a choice and becomes a liability.

The friendlier version is that when cost concentrates where value concentrates, metering gets easier. Expensive working sessions cluster around a customer’s most valuable moments, a live deal, a deadline, and twenty dollars of model spend inside work worth thousands is a rounding error. An allowance sized to the normal rhythm, with a metered valve beyond it, prices the burst without taxing the quiet months.

Second: which layers do you carry, and which do you move off your books? The other product went furthest: customers connect their own AI subscription, or run a local model, and the product charges for the intelligence built on top. Inference is not discounted so much as gone from the company’s cost structure entirely, paid by the customer directly to their model provider. That reshapes the margin structure of the whole business, and it needs to be decided at the architecture stage, long before any pricing conversation.

I am not saying every product should do this, only that “who pays for inference” is a strategic choice with three answers: you carry it, the customer carries it, or you meter it through. Most teams never notice they picked the first one.

Third: who may trade cost against quality, and on what evidence? Cost work has a known failure mode where someone downgrades a model to save money and then quality sags and nobody connects the two for a quarter. The fix is for any routing downgrade to pass the same quality evaluation as the setup it replaces, with a real quality drop blocking the change. Cost work is an experiment with a quality gate rather than a dial in a spreadsheet, and the rule needs a named owner. That owner sits in product, not engineering or finance, because the trade being made is customer value against margin, not dollars against tokens. Most organizations have not named anyone.

The exercise

This takes one working session, not a quarter. List every AI feature, shipped and planned, and for each one mark which layers it touches and what the cost grows with: usage, seats, data, or sessions. Note which model tier it uses, and whether anything cheaper was ever tested against a quality bar. Then put the price meter next to the cost meter and look for rows where they point at different things.

Those rows are your roadmap’s hidden cost center. Some you will keep on purpose, as a subsidy; some you will re-meter; some will send you back to an architecture conversation you thought was not your job.

Do this before the packaging debate and you get to choose your margin structure; skip it and you get one anyway, discovered one margin review at a time.

Which layer of the stack is quietly the biggest in your product? I suspect for most it is not the one being priced.