How Does an LLM Request and Response Cycle Work? A Full Walkthrough

Facebook X LinkedIn WhatsApp Email Copy Link More In this blog post, we will see how an LLM request and response cycle works, from the second you hit send to the moment the last word lands on your screen. I will not throw a wall of transformer math at you. Instead, we will follow one single prompt through every stage of the journey, so nothing feels abstract.

A single prompt sent to a large language model passes through multiple distinct stages before a response appears on screen. The process begins with the client app building a structured JSON request, which then clears an API gateway before the input text is broken into numeric tokens and assembled into the model’s context window.

The model then runs a forward pass through transformer layers to predict one token at a time, repeating this loop until an end-of-sequence signal is reached. Tokens stream back to the client as they are produced and are converted back into readable text incrementally, which explains the word-by-word appearance of responses.

I got curious about mapping this out properly while building iamspeed.dev, my browser based LLM benchmarking tool. Before I could measure things like tokens per second or time to first token, I had to understand exactly what happens between “you press Enter” and “words start appearing.” Turns out that gap has a lot more steps than most people assume.

The example we will follow Let’s keep one example running through the whole post. Say you type this into a chat app:

And a few seconds later, the model replies:

Assistant: The capital of India is New Delhi.

Simple enough on the surface. But that one exchange touches a client app, an API layer, a tokenizer, a neural network with billions of parameters, a sampling algorithm, and a streaming pipeline, all before those six words show up in your chat window. Let’s go step by step.

The moment you press Enter, your chat app does not just fire off the raw sentence. It builds a structured request, usually JSON, that looks something like this:

{ "model": "some-llm-model", "messages": [ { "role": "system", "content": "You are a helpful assistant." }, { "role": "user", "content": "What's the capital of India?" } ], "temperature": 0.7, "max_tokens": 100, "stream": true } Notice a few things here. The system prompt is bundled in even though you never typed it. Any earlier messages in the conversation get bundled in too, since the model has no memory of its own between calls. And stream: true is already set, which matters a lot later. This payload gets sent over HTTPS to an API endpoint.

Your request does not go straight to a model. It first hits an API gateway that handles the boring but essential stuff:

Think of this layer as the bouncer and the traffic cop combined. Only after it clears the request does anything resembling “AI” actually happen.

Here is where things get interesting. The model does not read English. It reads tokens, which are chunks of text mapped to numbers. A tokenizer breaks your sentence apart, usually using a scheme like byte pair encoding.

Our example sentence, “What’s the capital of India?”, might get tokenized into something like:

["What", "'s", " the", " capital", " of", " India", "?"] Each of these chunks maps to a specific integer ID from the model’s vocabulary, something like [1780, 434, 262, 3139, 286, 4881, 30] . The exact split depends on the tokenizer the model uses, but the idea is always the same: text becomes numbers, because that is the only thing a neural network can actually compute on.

This step matters more than people realize. It is also why providers bill you per token instead of per word or per character. A short word can be one token, a rare word can be split into three or four.

Your new tokens do not go in alone. The system stitches together:

All of this gets packed into the model’s context window, which is just a fancy term for “how many tokens the model can look at in one shot.” Each token also gets a positional encoding attached, so the model knows word order, since without it, “India of capital the” and “capital of India” would look identical to the network.

Now the actual model runs. Your token sequence flows through dozens of transformer layers stacked on top of each other. Inside each layer, two things happen repeatedly:

After all layers, the model produces a probability distribution over its entire vocabulary, essentially a ranked guess of what the very next token should be. For our example, after processing “What’s the capital of India?”, the model’s internal state is now primed to predict something like “The” as a strong first candidate.

Here is the part that surprises people the first time they learn it: the model does not generate the whole sentence at once. It generates one token, feeds that token back into itself as part of the input, and generates the next one. This repeats until it decides to stop.

Parameters like temperature , top_p , and top_k control how the next token gets picked from that probability distribution. Lower temperature means the model plays it safe and picks the highest probability token almost every time. Higher temperature adds randomness, useful for creative writing, less useful for factual answers.

For our example, the autoregressive loop looks roughly like this:

Input so far: "What's the capital of India?" ? predict: "The" Input so far: "...India?" + "The" ? predict: " capital" Input so far: "...The" + " capital" ? predict: " of" ... and so on, until: "The capital of India is New Delhi." + <end-of-sequence token> That end-of-sequence token is the model’s way of saying “I am done,” which is also how it knows when to stop rather than rambling forever.

Remember stream: true from Step 1? This is where it pays off. Instead of waiting for the entire response to finish generating and sending it as one big blob, the server pushes each token to your client the moment it is produced, usually over Server-Sent Events or a chunked HTTP connection.

That is exactly why you see words appear on screen one at a time instead of the whole answer popping in at once. It is not a visual effect, it is literally showing you tokens as the model produces them.

On the client side, each incoming token ID gets converted back into readable text, the reverse of Step 3. The chat app appends each piece to the message bubble as it arrives, re-renders the UI, and by the time the end-of-sequence signal shows up, you are looking at the complete sentence:

Six words. Eight steps. All of it usually finishing in under two seconds.

Given my performance engineering background, I cannot write this post without pointing at the clock. Two numbers matter a lot more than “the response took 3 seconds”:

A model can have a slow TTFT but fast inter-token speed, or the reverse, and the two create very different user experiences even if the total time is identical. This is the exact gap I was trying to measure when building iamspeed.dev, and honestly, understanding this full life cycle is what made the benchmarking numbers actually make sense instead of just being numbers on a dashboard.

Check iamspeed.dev to measure the LLM performance.

So the next time you type a question into a chat app and watch the answer type itself out, you now know the full trip that sentence takes: request formation, gateway checks, tokenization, context assembly, a forward pass through a massive network, token-by-token sampling, streaming, and finally detokenization back into words you can read.

It looks like magic from the outside. From the inside, it is a very well engineered pipeline.

Have you ever watched an LLM response stream in and wondered what was happening under the hood? Let me know in the comments.

Facebook X LinkedIn WhatsApp Email Copy Link More About the Author