LLM Concepts: A Developer Overview

A large language model (LLM) is a next-token predictor with a very large memory of patterns. Chat products wrap that predictor in instructions, tools, and retrieval. If you can separate the model from the product around it, most LLM behavior stops looking mysterious.

This post is a working vocabulary for developers. It covers the pieces you meet when you call an API, debug a bad answer, or design a feature that uses a model.

What an LLM actually does

Given a sequence of text, the model assigns a probability to every token that could come next, then a decoding step picks one. That token is appended, and the process repeats until a stop condition.

The model does not look up a stored answer. It continues a pattern. When the pattern in its training data is strong (common APIs, well-known algorithms, frequent phrasing), the continuation is often correct. When the pattern is weak or the question needs facts the prompt does not contain, the continuation can be fluent and wrong.

That loop is the whole generation process. Everything else in this post is a name for a part of the loop, or a name for a system built around it.

Tokens

Models do not read characters or words. A tokenizer splits text into tokens, small reusable pieces. Most production models use a variant of byte-pair encoding (BPE): frequent character sequences are merged into single tokens.

A rough English average is about four characters per token, or roughly three-quarters of a word. Code, JSON, and non-English text often use more tokens for the same number of characters, because those strings share fewer merges with the tokenizer’s training mix.

You pay for tokens, you fit context in tokens, and you measure latency in tokens. “Summarize this 20-page PDF” is a token-budget problem before it is a quality problem.

"tokenization"  →  ["token", "ization"]
"console.log"   →  ["console", ".", "log"]

Exact splits depend on the model. Treat the provider’s tokenizer as the source of truth when you count cost or check whether a document fits.

Embeddings

Each token ID is mapped to a vector of numbers, an embedding. Similar tokens land near each other in that space, but the useful geometry is learned, not hand-designed. “Bank” does not have one fixed vector for the whole forward pass. Later layers rewrite the vector using the surrounding tokens, so “river bank” and “bank account” diverge.

Embeddings are also a product feature. Many APIs expose a separate embedding model that turns a whole passage into one vector for search. That vector is related to the idea above, and it is a different model from the chat LLM you call for answers.

☕ Embeddings, explained for Java developers

Think of an embedding as a float[] fingerprint of meaning. It is the opposite of hashCode(): a hash deliberately scatters "cat" and "cats" to unrelated values, while an embedding model is trained so texts with similar meaning get nearby vectors.

“Search by meaning” is then a nearest-neighbor lookup over those arrays, usually by cosine similarity. In Spring AI, EmbeddingModel.embed(String) returns the float[], and VectorStore.similaritySearch(...) does the lookup for you.

The transformer

Modern chat LLMs are decoder-only transformers. A transformer is a stack of identical blocks. Each block does two jobs:

1. Self-attention lets every token look at the other tokens in the context and mix in what is relevant.
2. A feed-forward network then transforms each token’s vector on its own, which is where a lot of the model’s stored patterns live.

Residual connections add each block’s output back onto its input, so the stack can be deep without the signal washing out. Layer normalization keeps the numbers in a stable range. You rarely tune these pieces yourself. You do feel them as “the model can use earlier text” (attention) and “a bigger model knows more patterns” (more blocks and wider layers, which means more parameters).

Parameters are the learned weights. A 7-billion-parameter model and a 70-billion-parameter model share this layout. The larger one has more capacity, costs more to run, and is usually stronger at multi-step reasoning. Size is not the only variable. Training data, post-training, and inference settings change behavior as much as parameter count.

Position matters because attention, by itself, does not know order. Models add a position signal (classic positional encodings, or newer schemes such as RoPE) so “dog bites man” and “man bites dog” stay different.

Attention, in plain terms

For each token, the block builds three vectors: a query (what this token is looking for), a key (what this token offers), and a value (the content to pass along). Compatibility between a query and the keys produces weights. Those weights mix the values. The result is a new vector for that token, biased toward the parts of the context that matched.

Multi-head attention runs several of these mixes in parallel, so one head can track syntax while another tracks a name or a number. Causal masking hides future tokens. During training the model still processes the whole sequence at once, but each position may only attend leftward. That is why the same network can be trained on long documents and later generate one token at a time.

Attention cost grows with the square of the sequence length. Doubling the prompt does not double the work inside attention. This is why long context is expensive, and why providers cache previous key/value tensors (the KV cache) so a multi-turn chat does not recompute the whole history on every new token.

Context window

The context window is the maximum number of tokens the model can attend over in one request: system prompt, conversation, retrieved passages, tool results, and the tokens it is about to generate.

Two limits get conflated:

Limit	What it is
Context window	How much text fits in this request.
Knowledge cutoff	How recent the pretraining data is. The model has no live feed of the world.

A 200K context window does not mean the model remembers your last session. Memory across calls is something you store and send back. It also does not mean every token in a long prompt is used equally well. Important instructions placed at the very beginning or very end of a long context are often followed more reliably than the same text buried in the middle.

How a model is trained

Training is a pipeline. The base model and the chat model you call are not the same artifact.

Pretraining

The base model reads a huge corpus and learns to predict the next token. The objective is simple. The result is a model that can continue text in many styles, including text you did not want, because the web contains all of those styles.

Supervised fine-tuning

Supervised fine-tuning (SFT) continues training on curated examples of the behavior you want: questions paired with good answers, tool calls in a fixed format, refusals where a refusal is correct. This is the step that turns a text completer into something that follows a chat template.

Preference tuning

Preference tuning ranks outputs. Humans, or another model, mark one reply as better than another. RLHF (reinforcement learning from human feedback) and direct methods such as DPO both push the model toward the preferred side. The goal is a model that is more helpful and less reckless, not a model that gained new factual knowledge.

Post-training can also narrow the model. Heavy preference tuning sometimes trades away unusual but correct answers. If a small model feels “dumber” after alignment, this tradeoff is a common reason.

Sampling

The model returns a distribution. Sampling turns that distribution into a token. These settings change the pick, not the weights.

Setting	Effect
Temperature	Scales the distribution. Lower values sharpen it toward the top token. Higher values flatten it and increase variety.
Top-k	Considers only the k most likely tokens.
Top-p (nucleus)	Considers the smallest set of tokens whose probabilities sum to p.
Greedy / temperature 0	Always takes the single most likely token. Still not a proof of correctness.

Use low temperature for extraction, classification, and structured output. Use a higher temperature when you want several distinct drafts. None of these settings add knowledge. They only change which plausible continuation you see.

Logprobs, when an API exposes them, are the log probabilities of candidate tokens. They are useful for classification and for spotting a shaky answer. A high-probability wrong token is still wrong.

Prompts are part of the context

A chat request is a sequence of messages with roles, not a single blob of text.

{
  "messages": [
    { "role": "system", "content": "Reply in valid JSON. Do not invent fields." },
    { "role": "user", "content": "Extract the invoice total." },
    { "role": "assistant", "content": "{ \"total\": 42.00 }" }
  ]
}

The system message sets standing instructions. User messages are the request. Assistant messages are prior model turns you send back so the model can continue the thread. Tool messages carry function results.

The model does not inherently obey the system message more than other text. Providers train it to weigh that role highly. Long, conflicting instructions lose. Short rules, plus an output schema, beat a page of tone guidance.

Prompting techniques you will see in the wild:

• Zero-shot: instructions only.
• Few-shot: a handful of input/output examples in the prompt.
• Chain-of-thought: asking the model to write intermediate steps. This can help multi-step tasks and can also produce confident nonsense between the steps. Judge the answer, not the length of the reasoning.

Why fluent answers are sometimes false

Hallucination is the same mechanism as generation. The model samples a likely continuation. If the prompt asks for a citation, a date, or an API that is only partly constrained, invention is a likely continuation.

Common triggers:

• The fact was rare in training, or it changed after the cutoff.
• The prompt implies the fact exists (“what is the error code our service returns for expired tokens?”) when the model has never seen your service.
• The requested format demands a value, so the model fills one in.

You reduce this outside the model more reliably than you reduce it with wording alone: give the source text, constrain the output schema, and reject answers that do not match the source.

Grounding patterns

Three patterns show up in almost every production design.

Retrieval-augmented generation (RAG). Search your own documents, put the best passages into the prompt, and ask the model to answer from those passages. The model is the reader. Your index is the memory.

Tool calling. The model emits a structured request (get_invoice(id)). Your code runs it and returns the result as a tool message. The model then writes the user-facing reply. Use this when the truth lives in a system of record, not in a paragraph.

Structured output. A JSON schema, constrained decoding, or a strict parser turns free text into a value your program can branch on. Validation still belongs to your code. The schema limits the shape. It does not guarantee the fields are true.

A checklist before you trust an answer

• Count tokens for the prompt and the reserved reply. Confirm they fit the context window.
• Put source facts in the prompt, or fetch them with a tool, when the answer must be current or private.
• Lower temperature for tasks with one acceptable shape.
• Validate structured output in code, and retry or fail closed when validation fails.
• Keep system instructions short and non-conflicting.
• Treat chain-of-thought text as scratch work. Do not show it to users as evidence.

The model is a conditional distribution over tokens. Your application decides what goes into the condition, what gets executed for real, and what is allowed to reach the user.