How to Reduce Token Usage for AI Marketplace Features

If you're building a marketplace with AI-powered features, token costs are no longer a background concern. Every search result ranked by an LLM, every product description generated automatically, every buyer inquiry routed by a model adds to the bill. This guide explains how to reduce token usage without giving up the quality that makes AI features worth building in the first place.

At Journeyhorizon, we've worked with marketplace founders since 2018, and a common pattern shows up when they add AI: the first version works, but once real traffic arrives, the cost curve gets steep. The good news is that most of that cost is concentrated in a few places, and the fixes are well understood.

How token pricing works for marketplace AI features

When you call an LLM API, you pay for input tokens (everything you send: system prompt, instructions, retrieved context, conversation history) and output tokens (everything the model generates). Output is priced much higher. As of October 2026, Anthropic lists Claude Sonnet 5.5 at $2 per million input tokens and $10 per million output tokens, and OpenAI lists gpt-6.1-sol at the same $2 and $10. Smaller models are far cheaper: Claude Haiku 5.5 and OpenAI's gpt-6-luna are both listed at $0.10 input and $0.50 output per million tokens, and Google lists Gemini 3.8 Flash at $0.75 and $3.75.

Three details catch many founders off guard:

• Output costs about 5x input on these models, so a verbose answer can cost more than a long prompt.
• Reasoning tokens are billed as output. A reasoning model that "thinks" for 2,000 tokens before a 100-token answer bills you for 2,100 output tokens.
• Price gaps between model tiers are large. On list prices, a frontier model can cost around 20x more per token than a small model from the same provider.

For a full side-by-side of providers and the hidden costs behind list prices, see our LLM API pricing comparison for marketplaces, and for a broader primer, read how much an LLM costs in 2026.

For a marketplace, this cost structure creates specific pressure points. If an LLM ranks search results, every query becomes an API call carrying listing data and user signals. Unlike hosting, which is mostly fixed, LLM spend scales with usage, so a traffic spike becomes a cost spike. If you're still mapping which features are worth building, our guide to how marketplaces use AI covers the common use cases.

Step 1: Measure where your tokens go

You can't optimize what you don't measure, and most teams don't look at token consumption until the invoice arrives.

Every major API returns token counts in the response. Log them for each call along with the feature name, model, user or listing ID, and cache usage fields (for example, Anthropic returns cache creation and cache read token counts, and OpenAI returns cached token counts). With that data you can answer the questions that matter:

• Which features consume the most tokens per day?
• What is the average and the 95th-percentile token count per request?
• What is your cache hit rate?

Treat token cost like a performance budget, the same way you track page load time. Set a target per operation (for example, a ceiling of input and output tokens per search ranking call) and alert when a release pushes it over. We cover dashboards, alerts and attribution in detail in our guide to LLM cost monitoring for marketplaces.

Step 2: Architecture choices that control token costs

The biggest savings come from architectural decisions made early.

Send only the context the model needs

Context inflation is the most common source of waste. When ranking search results, it's tempting to send the user's full history, every listing attribute, all reviews and the current marketplace state. That feels thorough, but most of it doesn't change the answer. A focused approach sends the query, a short summary of each candidate listing, and a handful of behavior signals. Practical ways to do this:

• Pre-compute a compact summary for each listing once, store it, and send that instead of the full record.
• Use retrieval to pick the top candidates first, then send only those to the model.
• Strip HTML, boilerplate and duplicate fields. Clean text or markdown uses far fewer tokens than raw page markup.

Don't use the LLM when you don't need it

Many tasks that start as "use AI" are better solved with simpler tools. Sorting by price or distance is a database query. Keyword and semantic retrieval can be handled by a search index or vector search, with the LLM reserved for the final rerank or for interpreting vague queries. Use LLMs where they add real value (understanding intent, nuanced judgments, writing) and avoid them where cheaper, faster tools work.

Route tasks to the right model

Model routing sends each request to the cheapest model that can handle it reliably. Classification, tagging, short extraction and moderation pre-screening usually run well on small models. Complex reasoning, long-form writing or sensitive decisions may justify a frontier model. Research supports this: the RouteLLM paper reported cutting costs by more than 2x in some settings without hurting quality, and the FrugalGPT paper, which escalates from cheap to expensive models only when needed, reported up to 98% cost reduction on its test sets. Your results will vary, so evaluate on your own data.

A simple pattern is a cascade: try the small model first, validate the output (schema checks, confidence score or a rule), and escalate only failures. Remember to count the cost of retries and escalations when you compare options.

Step 3: Prompt engineering to reduce token consumption

How you write prompts directly affects both input and output tokens.

• Be concise and specific. Remove repeated instructions and filler the model doesn't need.
• Use examples deliberately. A good example can replace paragraphs of instructions, but every example adds input tokens on every call. Keep one or two short examples, and put them in the cached part of the prompt.
• Constrain the output. If you only need a category and a score, ask for exactly that in JSON. State a word limit, ask the model not to explain its reasoning when you don't need it, and set a max output tokens limit as a hard cap.
• Control reasoning. On reasoning models, use the lowest reasoning effort that keeps quality acceptable for the task, since hidden reasoning is billed as output.
• Use templates. Reusable templates with placeholders keep prompts consistent, cache-friendly and easy to A/B test for token count and quality.

For large, messy inputs you can't trim at the source, prompt compression tools exist. Microsoft Research's LLMLingua reported compressing prompts up to 20x with little performance loss in its experiments, but compression adds complexity and should come after you've cut unnecessary context.

Step 4: Caching and batching for scalable AI features

Provider prompt caching

Prompt caching lets the provider reuse a repeated prompt prefix, such as your system instructions, ranking rules or category taxonomy, at a steep discount. It's one of the lowest-risk savings available because the output is unchanged.

• Anthropic: according to Anthropic's documentation, cache writes cost 1.25x the base input price with a default 5-minute lifetime (refreshed each time the cache is used), and cache reads on Claude Sonnet 5.5 cost 0.05x base input, which works out to $0.10 per million tokens. Prompts must meet a minimum length (512 tokens on the 5.5 models) to be cached.
• OpenAI: OpenAI's documentation says caching is on by default for supported models. For GPT-5.6 and later, prompts need at least 1,024 tokens, cache reads cost 0.1x the uncached input rate (0.05x for gpt-6.1-sol), and cache writes cost 1.25x.

To get cache hits, keep the stable part of the prompt identical and at the beginning, and put dynamic content (the user's query, the specific listings, timestamps) at the end. Changing a single character in the prefix, reordering tools or inserting the date at the top will break the cache. Note that caching doesn't make first requests slower; the trade-off is a small premium on the cache write, which pays off once the same prefix is reused.

Application-level caching

Your own cache can avoid API calls entirely. Store generated listing descriptions, review summaries and category tags, and regenerate only when the underlying data changes. For search and support, cache answers to frequent identical queries.

Batch APIs

Batch APIs from Anthropic, OpenAI and Google process requests asynchronously, usually within 24 hours, at a 50% discount to standard prices. They're a natural fit for marketplace background jobs: nightly description generation, re-tagging the catalog, moderation backlogs and review summaries. Batch discounts can be combined with prompt caching.

This is different from packing many items into a single prompt. Grouping 20 listings into one request can save on repeated instructions, but it creates longer outputs, makes errors harder to isolate and can hurt quality if the model loses track of items. Test packing size against accuracy rather than assuming bigger is cheaper.

Step 5: Manage conversation and agent context

Chat assistants and AI agents are where token usage grows fastest, because every turn resends the growing history. To keep this in check:

• Keep only the last few turns verbatim and replace older ones with a short running summary.
• Drop or truncate old tool results once the agent has used them.
• Load tool definitions only for the tools the current task needs.
• Set per-task limits on turns and total tokens so a stuck agent can't loop indefinitely.

If you're building agents that act on listings, orders or support tickets, our custom AI agent development team designs these limits into the workflow from day one.

Worked example: review summaries on seller profiles

Consider a marketplace that uses an LLM to summarize each seller's reviews on their profile page. The numbers below are illustrative, using October 2026 list prices.

Naive approach: summarize on every profile view. Assume 10,000 profile views per day, each sending about 4,000 input tokens of reviews and getting a 300-token summary from Claude Sonnet 5.5. That's 40 million input tokens ($80) plus 3 million output tokens ($30), or about $110 per day.

Optimized approach: generate summaries once, store them, and refresh only sellers who received new reviews. If 2,000 sellers change per day, a nightly batch job uses 8 million input tokens and 0.6 million output tokens. On Sonnet 5.5 with the 50% batch discount, that's about $11 per day. If testing shows Claude Haiku 5.5 produces acceptable summaries, the same batch costs roughly $0.55 per day. Profile views now read from your database at no token cost.

The trade-off is freshness: summaries update daily rather than instantly, which is fine for this feature. The same compute-once, refresh-on-change pattern applies to descriptions, tags and other content tasks. To see how savings like these change margins per transaction, read our guide to AI unit economics for marketplaces.

When to optimize and when costs are worth paying

Not every feature deserves optimization effort. Sometimes paying more for a stronger model is the right call if it improves conversion or speeds up your launch.

Optimize when:

• The feature is core and heavily used, such as AI search on every query.
• Your take rate or margins are thin, so AI cost per transaction matters.
• You serve price-sensitive users and can't pass costs on.

Hold off when the feature is experimental or low-traffic. Ship something simple, measure quality and usage, then optimize once the pattern is predictable. Packaged tools can also help: our AI search box and AI product description generator plugins for Sharetribe give you working features to measure before you invest in custom builds.

Token optimization checklist

• Token counts logged per request, feature and model
• Context trimmed to required fields and top candidates
• Stable prompt prefix first, dynamic content last
• Max output tokens and output format set on every call
• Small models used for simple tasks, with escalation rules
• Batch APIs used for all non-urgent jobs
• Generated content stored and refreshed only on change
• Conversation history summarized and agent loops capped
• Quality measured alongside cost after each change

Frequently asked questions

What's the difference between input and output token costs?

Input tokens are what you send to the model (prompt, context and data). Output tokens are what the model generates, including any reasoning tokens. Output is priced higher; on Claude Sonnet 5.5 and gpt-6.1-sol it's $10 per million versus $2 for input, a 5x difference. That's why limiting output length often has an outsized effect on cost.

How much can I realistically reduce token usage?

It depends on your starting point. If you send full records to a frontier model on every request, combining context trimming, caching, batch processing and model routing can cut spend dramatically, as the worked example above shows. If you've already applied those basics, further gains are incremental. Measure a baseline, change one thing at a time, and track cost per successful task.

Does optimizing for token cost reduce quality?

Not necessarily. Prompt caching and batch APIs don't change outputs at all. Smaller models and aggressive compression can reduce quality, so test them on a fixed evaluation set before rolling them out.

Does prompt caching work automatically?

It depends on the provider. OpenAI enables caching by default for supported models once prompts reach the minimum length. With Anthropic, you mark which parts of the request to cache. In both cases, you need a stable prefix to get hits, so check the cache fields in API responses to confirm it's working.

When should I use a more expensive model?

Use a stronger model when a cheaper one produces results that hurt your product, such as wrong categorizations, poor search relevance or unsafe moderation decisions. Default to the cheaper model and escalate to a stronger one only for requests that fail validation or need deeper reasoning.

Build AI features that scale profitably

Reducing token usage isn't a one-time cleanup; it's part of designing AI features that hold up as your marketplace grows. If you're planning new AI capabilities, our AI strategy and architecture work helps you choose models and design pipelines before costs are locked in, and our custom marketplace development team builds them into your platform.

Already running AI features and watching the bill climb? Our AI cost optimization service audits your token usage, prompts and architecture and puts these savings into production.