What Should Be Included in AI COGS

If you’ve added AI features to your SaaS product in the last 18 months, your gross margin probably looks different than it used to. Maybe you haven’t noticed because you haven’t mapped AI costs to COGS. Or maybe you have, and you’re trying to figure out what’s normal. Either way, you need a framework.

My previous post on What Should Be Included in SaaS COGS covers the foundational SaaS COGS departments:

• Customer Support
• Professional Services
• Customer Success
• Variable
• DevOps

Those still apply. You don’t replace them when you add AI. You layer new expense line items on top of your existing COGS stack. Do we need an AI COGS cost center?

When your product runs on inference, vector databases, and model infrastructure, new costs appear that belong in COGS. The evolution of pricing models and software delivery has outpaced our software financial frameworks.

Let’s fix that.

COGS vs OpEx – Where Does AI Spend Live?

The COGS rule has not changed. The costs to deliver your product or revenue belong in COGS. Costs to build your product belong to R&D under OpEx. Other than capped R&D amort in COGS.

Where do things get confusing? AI is both an internal tool and part of our product line. The same vendor, say OpenAI or Anthropic, often shows up in both buckets. Here’s the split:

Here’s an important operational step: separate your production and development AI accounts. If engineers test models in the same account as production, you can’t split the costs. No different than how we allocate our monthly could bill.

Under U.S. GAAP, classification depends on the nature and purpose of the activity. Model training generally falls under ASC 730 (R&D, expensed) or ASC 350-40 (internal-use software, potentially capitalizable). Of course, consult your CPA or audit partner.

Production inference is expensed and belongs in COGS. One update worth flagging: FASB’s ASU 2025-06 modernizes ASC 350-40, effective for fiscal years beginning after Dec 15, 2027. If you’re capitalizing internal-use software development, talk to your auditors about how it affects your AI build. [1]

1) AI Inference Costs

WHAT IT IS: This is your bill from the AI model providers. Every time your product calls a model to answer a customer question, summarize a document, or generate output, you pay for the work. The unit of measurement is tokens. The more tokens your product consumes, the higher the bill.

Check out my post on Salesforce’s new Agentic Work Units AI term.

This is the biggest new line in AI COGS and as variable as it gets. Cloud hosting scales with customers. Inference scales with usage intensity, and heavy users can cost 50 to 100 times more than light users. [2] If you’re not segmenting cost by user cohort, your margins may surprise you when you dig in.

There’s also a forward-looking issue. Per-token prices keep falling, but reasoning models and agentic workflows generate 10 to 100 times more tokens per task than 2023-era completions. Net cost per task is rising even as token prices fall. [3] Be sure to model that accordingly in your forecast.

We saw this with the GPT 5.5 release. New models don’t necessarily mean cheaper models.

TYPICAL EXPENSES: OpenAI, Anthropic, Google Gemini, Azure OpenAI, AWS Bedrock, other LLM APIs.

MARGIN: Hits recurring gross margin directly.

WHAT TO LOOK FOR:

• Your cost per active user and per request
• Whether your heaviest users are still profitable
• Whether you’re using the cheapest model that gets the job done. Frontier models cost a lot more than smaller ones, and they aren’t always necessary. [4]
• Whether you’re charging for AI usage or absorbing it. Companies that package and charge track their cost-to-serve more precisely. [5]

BOTTOM LINE: Inference is your new hosting bill, except it scales with usage and gets more expensive as your features get smarter.

2) Model Hosting & GPU Infrastructure

WHAT IT IS: If you’re running AI models on your own infrastructure instead of calling someone else’s API, you’re paying for GPU compute directly. GPUs are different from the CPUs that run a traditional SaaS app. They cost much more per hour, the supply is constrained, and idle capacity wastes money fast.

This is what powers Llama, Mistral, Qwen, or any fine-tuned model you serve to production customers. Instead of paying per call, you pay for capacity.

This is a substantial commitment. Renting GPU capacity from AWS, GCP, or Azure is expensive, and scaling it is real infrastructure work. Some companies are moving toward dedicated providers like CoreWeave or Lambda Labs at scale. Others are repatriating, which means bringing AI workloads in-house instead of paying per-call API fees. 67% of companies are actively planning some form of repatriation, with another 19% evaluating it. [5] Many AI founders on my podcast mention bringing models in-house.

That’s a structural shift. It moves AI cost from a variable per-call line to a fixed-infrastructure line, with very different margin implications.

TYPICAL EXPENSES: GPU cloud instances (A100, H100, L40S), reserved or on-demand capacity, dedicated providers (CoreWeave, Lambda Labs, Together AI), serving infrastructure (vLLM, TGI, Ray Serve), and DevOps wages.

MARGIN: Hits recurring gross margin.

WHAT TO LOOK FOR:

• Your GPU utilization rate. Cloud waste averages 35%, and AI workloads are particularly prone to idle compute [6]
• Whether you’re locked into reserved capacity or paying on-demand
• Whether you actually need always-on infrastructure, or whether you can scale down between bursts

BOTTOM LINE: GPU infrastructure replaces or supplements your API bill in exchange for fixed-cost commitment and operational complexity. Most early-stage companies aren’t ready for that tradeoff.

3) AI Infrastructure Layer

WHAT IT IS: Modern AI products rarely call a model directly. Between your application and the model sits a stack of supporting infrastructure: vector databases for retrieval, embedding pipelines, agent orchestration, data platforms, and the bandwidth needed to move data between all of it. These costs belong in COGS when they serve production customers.

Here’s a finding most CFOs miss. In the 2025 State of AI Cost Governance survey (N=372), data platform usage (56%) and networking and egress (52%) topped the list of unexpected AI costs. Both were higher than LLM tokens (37%). [5] If you’re focused only on the LLM bill, you could be missing another major line item.

TYPICAL EXPENSES: Vector databases (Pinecone, Weaviate, Qdrant, pgvector), embedding generation, RAG pipeline infrastructure, agent orchestration (LangChain, LlamaIndex), prompt management, caching, data platform usage (Datadog, Snowflake) tied to AI workloads, network egress between cloud regions and model providers.

MARGIN: Hits recurring gross margin.

WHAT TO LOOK FOR:

• How much of your data platform bill (Datadog, Snowflake) is driven by AI workloads
• Egress charges (the fees cloud providers charge when data leaves their network) between cloud regions and out to model providers.
• Whether your vector database is sized correctly for current usage.

BOTTOM LINE: The middleware, data, and egress stack is the invisible cost behind your API bill. Most CFOs are missing it.

4) AI Monitoring & Observability

WHAT IT IS: AI observability tells you whether the AI is producing useful, accurate, and safe outputs. Without it, hallucinations, bad answers, and silent failures often surface only after a customer complains.

LLM observability tools like Langfuse, Arize Phoenix, LangSmith, and Helicone monitor token costs, trace LLM calls, evaluate output quality, and catch guardrail violations. [7] They belong in COGS ,because they’re production infrastructure for AI.

TYPICAL EXPENSES: LLM observability subscriptions (Langfuse, Arize, LangSmith, Helicone, Galileo), output evaluation infrastructure, hallucination and guardrail tooling, prompt version management, AI-specific alerting, cost attribution tooling.

MARGIN: Hits recurring gross margin.

WHAT TO LOOK FOR:

• Whether you find AI quality issues before customers do
• What you spend on AI monitoring tools, and whether it’s classified as AI COGS
• For regulated industries: whether you can demonstrate AI output history for compliance

BOTTOM LINE: AI observability isn’t optional anymore. It belongs in COGS.

5) AI-Related Support Costs

WHAT IT IS: When you add AI features, you add new failure modes. Your support team now handles tickets that didn’t exist before. “The AI gave me wrong information.” “I don’t understand why the AI did that.” That incremental support cost belongs to your AI COGS structure.

This might get confusing. Is this AI-specific support, or forward-deployed engineers? FDEs are in COGS, but they often sit in a Professional Services cost center and don’t impact your recurring gross margin.

But there is demand for your AI product line margins. You’ll need to allocate some support to it.

TYPICAL EXPENSES: Incremental support headcount or hours attributable to AI tickets, AI-specific support training, knowledge base development, output quality investigation time, customer education on AI behavior and limitations.

MARGIN: Hits recurring gross margin.

WHAT TO LOOK FOR:

• Whether your support tickets are tagged for AI
• Share of inbound volume that’s AI-related
• Average handle time for AI tickets versus standard tickets.

BOTTOM LINE: AI tickets aren’t the same as regular tickets. Tag them separately.

AI Gross Margin Benchmarks

The 70% to 80% gross margin benchmark I’ve used for years applies to traditional SaaS. It doesn’t apply to AI-native or heavily AI-featured products.

• AI-native SaaS (inference is core to value delivery, every interaction has compute cost): 50% to 65% gross margin at maturity. Early-stage AI-native companies frequently run 25% to 30%, and Bessemer’s 2025 dataset shows fast-ramping “Supernovas” with negative gross margins. [3]
• AI-enabled SaaS (traditional SaaS with AI features added): 72% to 85% gross margin, depending on AI intensity and whether you charge for AI usage. [8]
• ICONIQ’s 2026 data projects average AI product gross margins at 52%, up from 41% in 2024. Meaningful improvement, still well below traditional SaaS. [4]

In the 2025 State of AI Cost Governance survey, 84% of companies report AI delivery costs cutting product gross margins by more than 6 percentage points, and over a quarter see margin hits of 16 points or more. [5]

Two levers determine where you land:

1. Pricing discipline. Companies that package and charge for AI usage have materially better margins than those absorbing it into the base subscription. Give AI features away free and you’re subsidizing your customers’ compute. We’ve seen Anthropic and GitHub change to platform plus usage-based models.
2. Model routing efficiency. Route most tasks to smaller, cheaper models and escalate only complex cases to frontier models. Burning frontier tokens on simple tasks is an expensive habit.

Recalibrate your gross margin target if you’ve added significant AI features. A 70% gross margin for an AI-enabled product may be excellent, but only if your AI costs are mapped to COGS in the first place.

AI Chart of Accounts Changes

It’s time to review your chart of accounts (COA) and dimensional/department coding. For an accurate SaaS P&L, every expense must be coded to a department. Same for AI spend.

I propose these new general ledger accounts (GLA) that will get tagged to Dev Ops or a new AI COGS cost center.

• Inference Costs
• Model Hosting Infrastructure
• AI Infrastructure Layer
• AI Monitoring & Observability
• AI Support (could be a GLA or a new cost center)

And if you price AI on a subscription, you’ll need to track tokens and token expenses back to each customer.

Your Action Item List

Work these action items in order so it’s not too overwhelming.

If you’ve never mapped AI costs:

1. Separate production and development AI accounts. Without it, every later step is unreliable.
2. Separate internal AI use from AI in your product use.
3. Create the new GLAs and/or cost centers for the five categories above.
4. Tag your support tickets. Start measuring AI ticket volume and handle time.

Once that’s in place:

1. Calculate AI cost per user and per workflow or per Agentic Work Unit. Break down by cohort, average users versus power users.
2. Audit pricing. If you’re absorbing AI, model what consumption-based or hybrid pricing would do to your margin.
3. Tighten model routing. How much frontier-model traffic could be served by a cheaper model? Add caching where it fits. I’m learning this as I build SoftwareMetrics.ai.

At maturity:

1. Recalibrate your gross margin benchmark to AI-native or AI-enabled, whichever applies.
2. Document your cost classification methodology. Make ASC 730, ASC 350-40, and ASU 2025-06 positions explicit. Consult your CPA or audit partner.
3. Evaluate workload repatriation. 67% of companies are already planning it. It’s a structural cost decision, not just a technical one. [5]

Your AI COGS structure tells you whether your AI investment is economically sound. Without it, you’re running blind into one of the most expensive product decisions you’ll make.

Things are moving fast. It’s time to catch up in finance and accounting before we lose all line of sight on AI expense. CFOs, you may be spending a lot of time with your Dev team.

Sources

[1] AI SpendOps. AI Costs, GAAP Classification, and Financial Controls. https://www.aispendops.com/blog/ai-costs-and-financial-classification

[2] Agicent. AI Features Post Launch Costs. https://www.agicent.com/blog/ai-features-post-launch-costs/

[3] SaaStr. Have AI Gross Margins Really Turned the Corner? https://www.saastr.com/have-ai-gross-margins-really-turned-the-corner-the-real-math-behind-openais-70-compute-margin-and-why-b2b-startups-are-still-running-on-a-treadmill/

[4] SaaStr / ICONIQ. 5 Key Takeaways from ICONIQ’s State of AI Report. https://www.saastr.com/the-execution-era-of-ai-5-key-takeaways-from-iconiqs-state-of-ai-report/

[5] Mavvrik + Benchmarkit. State of AI Cost Governance 2025. https://www.mavvrik.ai/wp-content/uploads/State-of-AI-Cost-Governance-2025_FINAL.pdf

[6] CloudZero. Cloud Computing Statistics. https://www.cloudzero.com/blog/cloud-computing-statistics/

[7] CTAIO. Agent Observability: Langfuse vs LangSmith vs Arize Phoenix vs Helicone. https://ctaio.dev/en/labs/agentic-orchestration/observability-tools/

[8] KnowledgeLib. AI-Native SaaS Benchmarks 2026. https://knowledgelib.io/finance/saas-benchmarks/ai-native-saas-benchmarks-2026/2026

Ben Murray

I have worked in finance and accounting for 25+ years.  I’ve been a SaaS CFO for 9+ years and began my career in the FP&A function.  I hold an active Tennessee CPA license and earned my undergraduate degree from the University of Colorado at Boulder and MBA from the University of Iowa.  I offer coaching, fractional CFO services, and SaaS finance courses.