Understanding AI Tokens and Cost: How API Pricing Works for GPT-5, Claude 4, and Gemini 3

What is a Token and How Do LLMs Process Text?

Large Language Models (LLMs) do not process text by words or characters. Instead, they break input text down into sub-word segments called tokens using algorithms like Byte Pair Encoding (BPE) or SentencePiece. This allows models to build a compact, fixed vocabulary that handles rare words, complex suffixes, and typos efficiently.

In English, a general rule of thumb is that 1 token is roughly equivalent to 4 characters of text, or approximately 0.75 words. However, this ratio varies significantly based on language and text structure. Technical source codes, German with compound words, and non-Latin scripts (like Arabic, Hindi, or Urdu) decompose into much higher numbers of tokens per word, making API requests for these languages more expensive.

How API Token Pricing Works

AI providers charge developers based on two primary dimensions: Input (Prompt) tokens and Output (Completion) tokens. Standard rates are indexed per million tokens ($/1M). Input tokens represent the context you send to the model, while output tokens represent the text generated by the model.

Maximizing Savings with Prompt Caching

Prompt caching is one of the most powerful cost-optimization strategies available to modern AI developers. Offered by models like Google Gemini 3.5 and Anthropic Claude 4.6, prompt caching stores static segments of your prompt (such as massive system prompts, uploaded PDF documents, or RAG contexts) on the provider's server.

Subsequent API calls that reuse this exact same context read it directly from the cache. Providers charge up to 90% less for cache hits compared to standard input token pricing, reducing the cost of static prompts dramatically.

The Quadratic Cost of Chat Session Context Growth

In conversational systems like chatbots, the context window grows dynamically. Because the model has no inherent memory of previous requests, developers must append the entire conversation history (all previous user messages and AI responses) to the prompt of each new turn.

This causes the prompt size to accumulate quadratically. Turn 1 prompts are small, but by Turn 10, the prompt contains all nine prior interactions. This results in high cost escalation. To prevent runaway API bills, developers must implement sliding context windows, summarize older messages, or use prompt caching for the stable parts of the history.

Real-World Case Study: Comparing RAG vs. Support Chatbots

To understand how these concepts affect real budgets, let's compare two common enterprise architectures deployed with Claude 4.6 Sonnet (Input: $3.00/M, Cached Input: $0.30/M, Output: $15.00/M) assuming 10,000 sessions per month.

System A: RAG-Based Document Search (Static Context)

Each request includes a 25,000-token document repository. The user asks a single question and receives a 500-token response.

• Without Caching: 10,000 sessions * (25,000 input tokens * $3/M + 500 output tokens * $15/M) = $750 (Input) + $75 (Output) = $825.00 total monthly cost.
• With Prompt Caching (90% Cache Hit Rate): The cached portion is billed at $0.30/M. Total cost = 10,000 sessions * (25,000 * 90% * $0.30/M + 25,000 * 10% * $3.00/M + 500 * $15.00/M) = $67.50 (Cached Input) + $75.00 (Standard Input) + $75.00 (Output) = $217.50 total monthly cost.
• Net Savings: A monthly savings of $607.50 (73% cost reduction) just by enabling caching on the static documents.

System B: 10-Turn Support Chatbot (Dynamic Context)

The conversation has a 1,000-token system prompt, a 500-token user query per turn, and a 500-token model response per turn.

• Turn 1: Input is 1,500 tokens (System + User). Cost: $0.0120 (Input + Output).
• Turn 5: Input is 5,500 tokens (System + 5 User queries + 4 AI responses). Cost: $0.0240.
• Turn 10: Input is 10,500 tokens (System + 10 User queries + 9 AI responses). Cost: $0.0390.
• Session Total: Over 10 turns, the developer pays for a total of 60,000 input tokens and 5,000 output tokens, costing $0.255 per session. For 10,000 monthly sessions, this totals $2,550.00. Notice how the input cost grows quadratically because past chat history is resent repeatedly.

How to Track Token Usage Programmatically

To monitor API costs in production, developers must calculate token lengths before making API calls. Since model providers use different tokenizer models, you must use the appropriate library for accuracy.

Below is a Python example using the official `tiktoken` library to count prompt tokens for OpenAI models:

```python import tiktoken def count_tokens(text: str, model: str = "gpt-4o") -> int: try: encoding = tiktoken.encoding_for_model(model) except KeyError: encoding = tiktoken.get_encoding("o200k_base") return len(encoding.encode(text)) prompt = "Translate standard instructions to BPE tokens." print(f"Token count: {count_tokens(prompt)}") ```

For Node.js and web environments, developers can use the `@dqbd/tiktoken` package to count tokens in JavaScript:

```javascript const { encoding_for_model } = require("@dqbd/tiktoken"); function countTokensJS(text, model = "gpt-4o") { const encoder = encoding_for_model(model); const tokens = encoder.encode(text); encoder.free(); return tokens.length; } console.log("Token count:", countTokensJS("Hello developer!")); ```

Best Practices for LLM Cost Optimization

Reducing AI token consumption is essential to build profitable SaaS applications. Here is an actionable checklist of cost optimization techniques:

• Implement Sliding Context Windows: Limit the number of past messages resent to the model. Resending only the last 4 to 6 turns prevents quadratic cost growth.
• Use Chat History Summarization: Periodically compress older parts of the chat history using a cheap model, and replace the detailed transcript with a short summary paragraphs.
• Leverage Prompt Compression: Tools like LLMLingua analyze prompt text and remove redundant words and boilerplate instructions without losing semantic meaning, saving up to 25% on input tokens.
• Optimize RAG Chunk Size: Instead of sending entire pages, use advanced chunking strategies (such as semantic chunking or metadata filters) to retrieve and send only the most relevant sentences.
• Router Architecture: Deploy an intent router to direct simple queries to cheap models (like GPT-5.4 Mini or Gemini 3.5 Flash) and only escalate complex coding or math queries to frontier models (Claude 4.6 Opus).

Frequently Asked Questions