Blog

Last updated: September 4, 2026

This guide defines cost per token clearly and shows how practical teams calculate, measure, and optimize it — with real examples.

What is cost per token?

Cost per token is the fully loaded cost of producing one unit of model output, usually expressed per million tokens. It includes hardware amortization, power, cooling, infrastructure, software, and operational overheads — making it the fundamental unit of inference economics. 

This single metric lets engineers and CTOs connect infrastructure performance to profitability, cost forecasting, and operating margins.

Why does cost per token matter more than FLOPS per dollar?

Many organizations still focus on metrics like cost per GPU-hour or FLOPS per dollar. These measure input capability, not what matters: how many tokens that investment delivers.

NVIDIA analysis shows that its Blackwell architecture delivers 35× lower cost per million tokens and 50× more tokens per megawatt compared to Hopper, even though hardware cost was only 2× higher. That gap drives real-world value by boosting both throughput and margin. ([blogs.nvidia.com](https://blogs.nvidia.com/blog/lowest-token-cost-ai-factories/?utm_source=openai))

In other words, cost per token aligns investments to business outcomes — actual useful output — rather than theoretical compute numbers.

How do you calculate cost per token?

Cost per token can be derived using two perspectives: cloud API pricing or self-hosted infrastructure. Both share the same logic: divide cost by token output.

1. Cloud API pricing

Providers like OpenAI list per‑million‑token rates. Total cost = (input tokens ÷ 1M × input rate) + (output tokens ÷ 1M × output rate). For example, enriching 100,000 product listings at ~800 input and ~400 output tokens each equals ~80M input + 40M output tokens. At a notional $0.20 per million input and $0.80 per million output, the cost is $16 + $32 = $48. ([getotium.ai](https://getotium.ai/learn/tokens-and-pricing?utm_source=openai))

OpenAI’s March 2026 API rates illustrate this: for “gpt‑5.2”, input is $0.875/M, cached input $0.0875/M, output $7.00/M. Break out each token type accordingly. ([developer-openai-com.sitemirror.store](https://developer-openai-com.sitemirror.store/api/docs/pricing/?utm_source=openai))

2. Self‑hosted GPU infrastructure

Here, you compute cost per million tokens based on hardware cost and delivery throughput. The formula is:

cost per 1M tokens = GPU hourly cost ÷ (tokens per second × 3600 ÷ 1e6)

This captures only GPU cost; fully‑loaded cost must also include depreciation, electricity, cooling, utilization, and overhead. For example, Inferecon shows a more complete formula including amortized capex, power, PUE, and actual token throughput. ([inferecon.com](https://www.inferecon.com/writing/what-a-token-actually-costs?utm_source=openai))

Another platform, Serving & Economics, uses:

cost per 1M tokens = hourly serving cost ÷ useful tokens per hour × 1,000,000

This includes GPU or depreciation, CPU, orchestration, monitoring, and distinguishes “useful” token output from synthetic benchmarks. Example: a $4/hour node serving 2M tokens/hour yields ~$2 per million tokens. Drop utilization, and cost doubles. ([rahulkashyap.dev](https://rahulkashyap.dev/learn/llm-serving-economics/03-gpu-cost-per-token.html?utm_source=openai))

What factors drive variation in cost per token?

• Utilization: Idle GPUs still incur cost; low utilization inflates cost per token. ([saturncloud.io](https://saturncloud.io/glossary/cost-per-token/?utm_source=openai))
• Batching strategy: Batching requests massively improves throughput — cost per million tokens on a 70B dense model on H100 drops from ~$15 at batch 1 to ~$0.05 at batch sizes above 1024. ([inference.how](https://inference.how/?utm_source=openai))
• Quantization and model efficiency: Lower-precision formats or MoE models reduce compute per token. ([nvidia.com](https://www.nvidia.com/en-us/solutions/ai/tokenomics-guide/?utm_source=openai))
• Serving engine/software stack: Kernel/tuning updates, serving frameworks can yield multi-fold cost improvements even on the same hardware. ([saturncloud.io](https://saturncloud.io/glossary/cost-per-token/?utm_source=openai))
• Amortization and facility: Cooling, depreciation, power usage effectiveness (PUE), labor — all spread across produced tokens. ([inferecon.com](https://www.inferecon.com/writing/what-a-token-actually-costs?utm_source=openai))

Concrete examples

Example 1: Cloud API use

Suppose generating 1,000 tokens of output with a model priced at $1/M input and $4/M output. If input was 500 tokens, cost ≈ (0.0005×1)+(0.001×4)= $0.000005+$0.004 = $0.004005 total, or $4.005 per million tokens delivered. Simple and instant to calculate before spending.

Example 2: Self‑hosted GPU

Assume an H100 GPU rented at $3/hour. If real throughput (accounting for utilization and batching) yields 12,300 tokens/s, then tokens per hour = 12,300×3600 ≈ 44.3M tokens. Cost per million = $3 ÷ (44.3M ÷ 1M) ≈ $0.068/M tokens. If utilization drops to 50%, that doubles to ~$0.136/M. ([inference.how](https://inference.how/?utm_source=openai))

How to reduce cost per token effectively

1. Improve utilization — keep GPUs busy with concurrent batching and avoid idle time. ([saturncloud.io](https://saturncloud.io/glossary/cost-per-token/?utm_source=openai))
2. Use efficient quantization or model architectures (e.g., MoE) to lower compute per token. ([nvidia.com](https://www.nvidia.com/en-us/solutions/ai/tokenomics-guide/?utm_source=openai))
3. Upgrade serving software — choose optimized kernels, stack improvements, frameworks like vLLM, TensorRT‑LLM. ([blogs.nvidia.com](https://blogs.nvidia.com/blog/lowest-token-cost-ai-factories/?utm_source=openai))
4. Choose high‑throughput infrastructure — e.g., NVIDIA Blackwell vs Hopper delivers dramatically lower cost per token. ([blogs.nvidia.com](https://blogs.nvidia.com/blog/lowest-token-cost-ai-factories/?utm_source=openai))
5. Optimize prompts — trimming unnecessary tokens can cut cost faster than any pricing negotiation. ([getotium.ai](https://getotium.ai/learn/tokens-and-pricing?utm_source=openai))

How does “tokenomics” extend cost per token?

Tokenomics is the broader economics of token usage — linking supply, demand, utility, and monetization in deploying AI at scale. It examines how cost per token interacts with model utility, operational demand forecasting, and revenue pricing. Adjacent metrics include intelligence per token, tokens per task, and cost per task. ([nvidia.com](https://www.nvidia.com/en-us/solutions/ai/tokenomics-guide/?utm_source=openai))

How does this guide connect to broader AI cost optimization?

This topic is part of a broader conversation about AI/LLM cost control and cloud infrastructure economics. For a strategic overview, see our hub on AI cost optimization: AI cost optimization hub.

Understanding cost per token gives teams a sharper measuring stick — it’s the bridge between engineering choices (batching, quantization, GPU selection) and business outcomes (margin, ROI, scalable inference cost).

Frequently Asked Questions

• What’s a token? A token is a subword unit processed by LLMs — roughly 4 characters or 0.75 words for English. Tokenization varies across models, so measure with your target model. ([getotium.ai](https://getotium.ai/learn/tokens-and-pricing?utm_source=openai))
• Why charge separately for input and output tokens? You pay to supply context and to generate text — input and output use different processing and bandwidth; billing reflects that. ([help.openai.com](https://help.openai.com/en/articles/20001415?utm_source=openai))
• Can software make cost per token drop without new hardware? Yes — improvements in serving stacks alone can reduce cost per token several‑fold. ([saturncloud.io](https://saturncloud.io/glossary/cost-per-token/?utm_source=openai))
• How much does batching help? Batch size can collapse cost per token by orders of magnitude — one example dropped from ~$15/M to ~$0.05/M with large batch sizes. ([inference.how](https://inference.how/?utm_source=openai))
• Is cost per token fixed for a chip? No — same GPU can vary greatly depending on utilization, batching, software, overhead — making cost per token a moving metric. ([saturncloud.io](https://saturncloud.io/glossary/cost-per-token/?utm_source=openai))