AI applications can become expensive for a simple reason: every turn in a conversation can carry more history into the model call. Microsoft Mechanics’ short video highlights a practical pattern for reducing that cost pressure: stop sending the entire raw conversation every time, and pass only the context the model actually needs.
For IT and cloud teams building copilots, chat interfaces, support bots, or workflow assistants, this is more than a prompt-engineering detail. Token growth directly affects latency, throughput, and billing predictability. A prototype may feel inexpensive when it handles a few short exchanges, but production usage can expose “context window creep” as conversations get longer and more users arrive.
What the video demonstrates
The example shows a conversation where earlier turns are condensed into a summary instead of being replayed in full with every new request. By turn three, the app sends a compact summary of the previous conversation pairs rather than the complete input and output from those turns. The result is a much smaller prompt footprint while still preserving the key context needed for the next response.
The video also points to a second important pattern: store conversation state outside the model request and retrieve only what is relevant. In the demo, PostgreSQL is used as the backing store, but the architectural idea is broader. Teams can use the database, cache, vector store, or application state layer that best fits their security, performance, and data-governance requirements.
Why this matters operationally
Large context windows are useful, but they should not become a default dumping ground for every prior message, tool result, and system note. Sending too much history can increase token consumption, slow responses, and make costs harder to forecast. It can also complicate troubleshooting because model behavior may be influenced by old or irrelevant context that should have been removed.
A better operating model is to treat context as an actively managed resource. Summaries, retrieval filters, and state stores give engineering teams control over what enters each model call. That control becomes essential when an AI feature moves from pilot to production and must meet real service-level expectations.
Practical takeaways for cloud teams
First, add token measurement early. Track prompt tokens, completion tokens, and total cost by feature, tenant, workflow, and conversation length. Cost surprises are easier to prevent when the telemetry shows where context is growing.
Second, summarize older conversation turns when exact wording is no longer required. A compact state summary can preserve user intent, decisions already made, constraints, and unresolved tasks without resending the entire transcript.
Third, separate durable state from prompt context. Store conversation history, user preferences, workflow state, and retrieved documents in systems designed for storage and querying. At runtime, assemble a focused context package instead of blindly appending everything.
Fourth, define retention and retrieval rules. Sensitive data, outdated tool outputs, and irrelevant prior turns should not automatically re-enter future prompts. This is both a cost-control measure and a governance improvement.
Bottom line
The message is straightforward: context is not free. Summarizing conversation history and retrieving state selectively can reduce token usage while keeping the AI experience coherent. For teams using Microsoft Foundry, Azure AI, or any generative AI stack, controlling context window creep should be part of the production architecture—not an afterthought.
Source: Microsoft Mechanics on YouTube