Productionizing LLMs: The Work of Carving a Waterway

Bringing an LLM into a real production service is much like the work of carving out a waterway.

To deliver the generated output that flows from the vast reservoir of a large language model all the way to the user without interruption, you need more than just calling an API—you need a range of engineering considerations. Because no matter how abundant the water, if the channel meant to carry it is blocked, the user can't drink a single drop.

In this piece, I want to liken LLM productionization to "carving a waterway" and talk through the key points you have to consider in a production environment.

Every waterway begins at its source. The first step in commercializing an LLM is deciding "which water to use."

An LLM's answer doesn't come down all at once; it flows out token by token. To show the user the answer in real time, you need a smooth channel that can carry this flow exactly as it is.

Decompression: If you compress the response with gzip or the like, the stream may not be parsed correctly. It's safer to turn compression off by explicitly setting Content-Encoding: none.

Chunked transfer: Set Transfer-Encoding: chunked so the data can arrive piece by piece. Proxy bypass: You have to configure things so that any proxy or CDN sitting in the middle doesn't buffer or alter the stream. Careful settings, such as managing the Cache-Control header, are required.

Streaming that worked fine in the development environment often suddenly breaks the moment you deploy to a real cloud environment.

These problems usually only surface once real traffic is flowing, so after the initial launch you end up solving them one by one through repeated trial and error.

Once the waterway is open, you now have to control how forcefully to send the water through. An LLM's token generation speed varies with the model and the hardware, and the speed at which it's shown to the user also varies with network conditions.

An LLM service doesn't simply wire the model and the client directly together; it passes through various intermediate layers. If you don't manage these segments well, floods or droughts can occur.

Carving the waterway once isn't the end of it. As the service grows in scale and usage patterns change, problems are bound to crop up all along the channel.

As you work on LLM productionization, you come to realize one interesting fact: AI development desperately needs the help of AI.

Streaming optimization, error debugging, prompt engineering—the vast body of knowledge is too much to shoulder alone. So I ask ChatGPT about code, ask Claude to draft documents, and debug together with Copilot. Without AI, it would have been hard to solve problems and keep improving this quickly.

The engineer who carves a waterway borrows the power of water to build a better waterway. In the same way, an LLM productionization developer makes better AI services with the help of AI. Isn't this very loop the evolution of technology we're experiencing right now?

May the waterways flow smoothly through your LLM project today, too.