Generative AI: Large Language Models and Diffusion

I prepared this presentation for a friend, with the intention of inspiring them. My goal was to make the foundation of Generative AI understandable — from the Transformers mechanism to file formats — and then move into practice with hands-on exercises. If the material gets technical, just try to understand: there is no unimportant information here, and grasping even a single principle will make your work easier in the future. And if something doesn't click, you can always go back to an earlier section and pick it up from there.

Disclaimer: this post was adapted from the presentation PDF into article form by Claude, an AI assistant.

There is no real prerequisite for Generative AI. Engineering knowledge and principles regarding software are useful, but the assumption that "software requires numerical and mathematical skill" is dangerous and wrong. The computer, as its name suggests, counts and computes information and numbers for us.

Unless you are developing a new model architecture, there is no need for mathematical knowledge — and even then there often isn't, because software, like art, is built on borrowing from one another and building & improving on top of it. Everyone can use Generative AI, tinker with it, and improve it. The only things it requires are patience and time.

Familiarity with the ecosystem also helps. The more of these names ring a bell, the better: OpenAI, Claude, Gemini, LLaMA, Qwen (通義千問), DeepSeek, Mistral, MiniMax, Command R+, Gemma, Grok, EleutherAI, Kumru; on the tooling side llama.cpp, LM Studio, HuggingFace; on the visual side Stability AI, FLUX.1, Midjourney, character.ai, Perplexity, Copilot.

The foundation of Generative AI is the Transformers architecture, introduced in 2017 in the paper "Attention is All You Need". ChatGPT, Gemini, Claude, Qwen, DeepSeek, LLaMA, Mistral, MiniMax, Command R+ and dozens of other models — even if they have small architectural differences between them — are fundamentally based on this architecture, specifically the decoder-only transformers architecture.

In essence, the whole affair is passing the given input through various mathematical operations, looking at the probabilities (temperature, top_p, top_k, etc.) and predicting the next output based on the input values. Most models make this user-friendly with chat templates and reasoning blocks.

This presentation does not cover AI systems that work differently, such as JEPA (LeCun).

Embedding: Tokens are processed as numbers. The word embedding is summed with the positional embedding (sine and cosine values) to produce the final embedding. The difference between "LLaMA 2 is better than LLaMA 1" and "LLaMA 1 is better than LLaMA 2" emerges precisely because the same words receive different positional values.

Parameter: Imagine a water bottle on the table. "Pink-labeled 500g Evian-branded water" is high-parameter, while "500g water" is lower-parameter. As parameter count increases, quality generally increases and the effect of quantization generally decreases. But we cannot generalize — if the architecture is bad, parameter count has no effect. This is called the "Fallacy of Scaling Up", and the concept of "Scaling Up" is criticized especially in the Large Language Model scene.

If we generalize about Large Language Models, there are two important types of Transformers architecture.

Mixture of Experts (MoE): Its most important academic paper is "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer" (https://arxiv.org/abs/1701.06538). The whole model is loaded into memory, but at any given moment only certain experts are active, so resource usage is lower. For example, in early 2025 DeepSeek V3 had 671B parameters and actively used 37B of them. The memory requirement does not change despite the lower resource usage, but since it is well optimized in llama.cpp, it runs much more efficiently than dense models. For this reason MoE models are usually quite large (from 120B GPT-OSS up to 700B Kimi K2). In the open-weight scene it started gaining recognition with Mixtral 8x7B.

Dense: Its most important academic paper is "Attention is All You Need" (https://arxiv.org/abs/1706.03762). The whole model is loaded into memory and all parameters are active at the same time. It is traditional, and its training is easier and more predictable. In the open-weight scene it became known to the general public with the leak of LLaMA, though GPT-2 is the most important early milestone in terms of recognition.

An important note: reasoning models (in the Large Language Model context) have no special architectural difference from normal models.

The operation at the heart of Transformers revolves around the Q, K, V values: Query, Key, Value.

- Query = "What am I looking for?"
- Key = Signifier: this is the "label" or "form" of the data. Just as the sound sequence "A-Ğ-A-Ç" points to a concept, the written word "AĞAÇ" points to the same concept, and the character "木" points to a concept, the Key is the vectorial form that indicates in which context the data can be looked up.

- Value = Signified: this is the actual "meaning" or "content" that the label carries. When the Query matches the Key, the system returns the actual information — the Value.

The classic example: in the sentence "The animal didn't cross the street because it was too tired", the word "it" attaches to "animal" rather than "street" thanks to the attention mechanism.

Pre-Training is costly and requires special hardware (NVIDIA CUDA). Recently, alternative technologies such as ROCm (AMD) and MLX (Apple) have also started to emerge. Training a Transformer's weights — the numbers inside the tensors, representing importance in the attention mechanism; the multiplication of embedding values with the weights is central to how Large Language Models work — is costly and demands a lot of data (it can go up to several hundred GBs, even TBs).

Fine-Tuning, on the other hand, can generally be done in an optimized way on ordinary consumer graphics cards — even for high-parameter models — with various libraries (such as Unsloth). It costs less to train and requires less data, but for good results that data must be in a regular format.

As an example, I showed a cut from a dataset prepared for a Kanbun-Kundoku-focused fine-tune, in the HuggingFace Chat Format: each line holds a system-user-assistant triple, with a <think>...</think> block at the start of the assistant's answer. This is the Reasoning Model format — the model thinks first, then answers according to its thought. Reasoning can sometimes lower performance; sometimes, and in fact in most practical use, it increases it.

A tensor can be thought of as a data container that holds numbers. This container can come in different dimensions: a scalar forms a 0-dimensional, a vector a 1-dimensional, and a matrix a 2-dimensional tensor. To understand the concept of a 3-dimensional tensor we can think of a photograph: while a black-and-white photo is a 2D tensor, a color image forms a 3D tensor, since it consists of a series of 2D tensors representing the red, green and blue (RGB) pixels.

SafeTensor is the file format that holds a model's weights in tensor form, and it is the standard file format for Generative AI. Only tensors are stored — it contains no executable code. Compared to the standard .pkl, .pt and .bin files, SafeTensor models are safe: file formats like Pickle (.pkl) can carry small code fragments under the guise of "how do I reassemble this data", a malicious actor can hide harmful code using this method, and when the file is loaded that code runs automatically without the user's knowledge. This is why SafeTensor files became popular and turned into the standard.

Its architecture is quite simple: the first 8 bytes of the file give us the length of the table-of-contents (header) section. The JSON header holds an identity card for every tensor: dtype (data type — F16, a 16-bit floating point number, is an example), shape (the shape of the tensor), and offsets: [BEGIN, END] (the start and end point of the tensor). The rest of the file holds the actual data and weights.

The GGUF file format makes AI models that use the SafeTensor format accessible. GGUF lets us run AI models on any hardware (NVIDIA, AMD, Apple Silicon) at whatever quantization we want, without an external library (such as bitsandbytes).

GGUF was a project run independently of HuggingFace until 2026, when it joined the HuggingFace platform (more information at https://huggingface.co/blog/ggml-joins-hf). GGUF files are generally used with Large Language Models, but their use has started to appear in Diffusion models as well (ComfyUI-GGUF).

Quantization is the process of optimizing weight files. For variety and convenience (processed file variety, hardware ease and compatibility), GGUF is strongly recommended by the community and has become the standard.

Preferably, a GGUF should not be downloaded directly: if the hardware permits, the SafeTensor file should be converted to GGUF with llama.cpp and then quantized. But someone downloading a GGUF usually doesn't have that hardware in the first place. In SafeTensor files, quantization is done with libraries like BitsAndBytes using 8BIT and 4BIT values; BitsAndBytes does not work in environments like macOS.

In GGUF files, quantization ranges from FP16 down to TQ1_0. A TQ1_0 (Ternary Quantization) file contains only the values -1, 0 and 1 (which is why its Bits Per Weight is generally 1.58). As quantization gets more severe, model performance drops.

A small note: naming and actual weight quantizations usually agree, but the people doing the quantizing (like Unsloth) sometimes use various weights within one type. An example log from a Qwen3.5-397B-A17B-UD-TQ1_0.gguf shows tensors of many types living together, from f32 and q8_0 to iq2_xxs and mxfp4. Mixing these values can get us better performance at severe quantization levels. This is why, when choosing a model, what we should really look at is the BPW (Bits Per Weight) value: below 3-4 serious performance losses begin, while at 4 and above the performance loss can be ignored.

LoRA (QLoRA when it is 4-bit) is the process of partially fine-tuning a specific AI model. It is more practical and cheaper than doing a full fine-tune, and it is the most widely used fine-tuning method.

We can personalize our model by downloading LoRA files separately and loading them whenever we want — meaning we don't need to download a modified copy of the whole model. Every model's LoRA file is different: a LoRA that works on Qwen3.5 will not work on Mistral Large 3.

LoRA plays a big role in personalizing both Large Language Models and Diffusion models. It is even more important in Diffusion models, where RAG (Retrieval Augmented Generation) is practically nonexistent.

This exercise focuses on using AI on local hardware.

First, a llama.cpp build suitable for your hardware must be obtained: for Windows it can be acquired from https://github.com/ggml-org/llama.cpp/releases; on macOS and Linux it is healthier to install it using brew (brew install llama.cpp). The healthiest option is compiling the code from scratch, but it is unnecessary. Since llama.cpp is a command-line application, it requires no installation. Interfaces like LM Studio and Ollama exist as alternatives.

To choose a suitable language model we need to know our VRAM (graphics card RAM) and RAM amounts; programs like Task Manager or Activity Monitor show them. Most computers today have 8GB VRAM and 16GB RAM. To be safe, the presentation used https://huggingface.co/unsloth/Qwen3-VL-4B-Instruct-GGUF as the base model; as mentioned in the quantization section, any quantization value above Q4_K_M can be chosen with confidence. Qwen3-VL is a multimodal large language model, meaning it has vision support, which makes it a good choice for the exercise. Alongside the model GGUF we need to download the MMPROJ file (mmproj-F16.gguf).

After downloading the model files and llama.cpp, we have two options for running them. llama-cli lets us interact through the command line, while llama-server allows a web interface and API connections:

- llama-cli -m qwen3-8b.Q8_0.gguf --system-prompt "You are an expert translator specializing in converting texts into Classical Japanese Kanbun-Kundoku style" -p "通義千問 is a very fine model" --temp 0.7 --top-p 0.8 --top-k 20
- llama-server -c 32768 -m Qwen3.5-27B.Q6_K.gguf --mmproj Qwen3.5-27B.mmproj-f16.gguf --chat-template-kwargs '{"enable_thinking": false}' --host 0.0.0.0

Notes on the arguments: --temp, --top-p and --top-k represent hyperparameters; these values are used to filter the probabilities produced by the softmax operation. --chat-template-kwargs directly affects the chat template and can be used to turn off reasoning in reasoning models. -c specifies the context amount (we kept it at 4096 for the exercise). -m specifies the location of the model file and --mmproj the location of the mmproj file. If we were using an MoE (Mixture of Experts) model and wanted to split it between GPU and CPU, we could use --n-cpu-moe with --ngl 99, or --fit on (--ngl specifies how many weight layers to load onto the model's GPU side; 99 is generally used to mean "all of it", since no open-weight model currently exceeds 99 layers). And in multi-letter arguments (like --mmproj), mind the double dash.

Visual AI training is easier than textual AI training. Thanks to today's technological developments, diffusion models can be trained effectively with as few as 20 images.

Before training we must set our goal: do we want to train a style? Or a character? After stating the goal, we need to compile the images that best express the subject — 20 images are enough for this exercise. The images should preferably follow this order of importance: 1. Same resolution, same aspect ratio; 2. Same aspect ratio. Modern models (like Qwen Image) and training libraries (like ostris/ai-toolkit) also support images at different resolutions and different aspect ratios.

For a normal model we need to write a caption describing each image, and the concept we want to teach must stay constant in the prompt (here, "skeskin person"). In edit models there is a control set (the image we have) and a dataset/target (the image we want to obtain) — in other words, edit models are taught by example, and the prompt usually stays fixed: a single caption like "Turn this photograph into a still from an animated film, similar to that of 君の名は。Anime." is written for all images.

Important practical notes: anyone under 18 who will use services like RunPod or Google Colab must do this exercise under parental supervision. If you have the means, the training should be done locally — I personally don't (AMD graphics card + MLX). The presentation used Qwen-Image-2512 and Qwen-Image-Edit as the base; since I don't expect anyone to run a model that consumes 40GB of VRAM in training locally, it was demonstrated over RunPod. Cloud GPU services generally work "pay-as-you-go" (like RunPod), while Google Colab charges a single fee and gives you monthly credits.

The training flow: we rented a graphics card on RunPod (RTX 6000 PRO), uploaded the dataset we prepared in ostris/ai-toolkit and placed the captions. We selected the Qwen Image Edit model and set the learning rate to 0.00002 for this trial (the default is 0.00001). We selected the dataset as Target and the control set as Control. We uploaded three sample images so we can watch the progress as it trains — over the 3000-iteration process the images will improve over time and follow the prompt more closely. A 3000-iteration training will take roughly 3-5 hours on this graphics card.

The goal of this exercise is to connect to llama-server through the command line or a UI and get output in any format we want.

If you don't know software, you can not only handle this part easily with AI, you also get to learn the RAG (Retrieval Augmented Generation) concept along the way — because llama-server develops so fast that the training databases will most likely contain outdated documentation. RAG is the name given to the process of computing the embeddings of a given query, comparing them with a document's embeddings, and retrieving the most relevant parts. Before the exercise, install Miniconda and/or a vibe-coding tool (careful: Miniconda, not Anaconda — Miniconda prevents Python dependency hell, version incompatibility and dependency chasing problems). This exercise is useful even if you already know software.

Important principles on Vibe Coding: Vibe Coding is the name given to the coding process a user carries out with Large Language Model assistance. Vibe Coding is not "order the computer around, let it write while I enjoy myself". It requires at least theoretical knowledge about the software and the language being used; the person must understand the code that was written and be able to debug it. If the person has theoretical and preferably practical knowledge, and vibe codes to speed up work and prototype, they can reach an effective result in a short time. From my own observation: most developers today no longer write software with traditional methods, and some developers dangerously vibe-code without knowing, looking at, or examining what they write. Not knowing what you are doing in vibe-coding is dangerous.

KV Cache: as the name suggests, it is the state of the message history being loaded in memory. To maintain it, there must be a message history (System, User, Assistant, User, Assistant...) and the cycle must be correct (user first, assistant after). Items in the message history must not be reprocessed after they have been sent and processed — otherwise the KV Cache is invalidated. The severity depends on where in the history the message sits.

Metadata: a user message can carry various metadata — time, RAG results, tool call results. All metadata must be wrapped in XML tags: <time>...</time>, <tool_call_results>...</tool_call_results>, or a combined <metadata> block. As for tool calls, in my experience the "tools" parameter (OpenAI API) should never be used (my experiences have always been bad, and some models don't support it). Instead it should be prompted: the model should follow that prompt and write the tool call itself inside XML or JSON. For this reason the jinja argument should be avoided with llama.cpp. This also makes streaming and tool calls usable at the same time. That output should then be parsed with a parser, and if the conditions are met, the tool call should be executed. Tools should conform to the Model Context Protocol; servers should be automatically discovered and connected accordingly, and they should be given to the model at the very start, in the system message, together with tool call usage instructions.

Activating the API: a command like llama-server --models-dir models --fit on -c 131072 -n -1 --host 0.0.0.0 serves the main models, and llama-server --models-dir embeddings --embedding --pooling rank --host 0.0.0.0 --port 8081 serves embedding/reranker models. Since --models-dir is a newly added command in llama-server, vibe-coding assistants may have trouble recognizing it — feed the documentation (https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) to the assistant. To load vision models with --models-dir, each model must sit under its own folder. If you don't specify a model directory you can load any model you want with -m.

In the application example we wrote a /chat endpoint with Flask: the user message is received, a RAG search is run and its hits are added as metadata inside a <relevant_documents> tag along with the time inside a <time> tag; the request is streamed to llama-server; tool calls in the output are parsed and their results are fed back to the model inside <tool_call_results>.

At the end of the presentation I left the audience five assignments:

1. On a local large language model (through LM Studio or llama-cli, for example), run the same input at three different temperature settings — what did you notice? Play with different parameters, experiment!
2. Using any Large Language Model, prepare a fine-tuning database of 20 entries on one subject in the HuggingFace JSON Chat Format. Then review and test whether this database would actually work with RAG as well.
3. We trained a LoRA in Exercise 1. Using ComfyUI (local) / Draw Things (iOS, macOS, local) / Fal.AI (cloud), can you generate an image with that LoRA over an API connection?
4. Prompt Engineering: can you get any local model to write a correct Kakari Musubi sentence by changing only the system message and providing the necessary content & documents?
5. Experiment with quantization: download a very low variant of the same model (like TQ1_0) and a high one (like Q8_0). What differences did you observe? Then download a lower quantization of a higher model and a higher quantization of a lower model (say Q8_0 at 4B versus Q6_K at 8B, or Q4_K_M at 14B) — what did you observe?