---
title: "CUDA-LLM: A GitHub Repository for Development"
url: https://stacklist.com/card/16639b59-945a-4bcc-918b-69432443258a
source_url: "https://github.com/MagicCoding2006/CUDA-LLM"
stack: https://stacklist.com/stack/4aae218c-38d7-4c05-b7ae-f2db0029a2e8
summary: "CUDA LLM is a GPU-optimized decoder-only language model implementation featuring custom C++/CUDA attention kernels, INT8/INT4 quantization, and a complete training and serving stack designed to run on laptop GPUs. The 496M parameter model achieves 230 tok/s on laptop INT4 decode and includes advanced features like sparse attention, RAG integration, and production-grade deployment capabilities."
tags: "cuda, llm, gpu-optimization, transformer, quantization, inference, flashattention"
key_entities: "CUDA (technology), PyTorch (technology), FlashAttention (technology), HuggingFace (technology), Modal (technology), Safetensors (technology), RoPE (technology), RMSNorm (technology), SwiGLU (technology), BPE tokenizer (technology), LLaMA (technology), quantization (concept), sparse-attention (concept), RAG (concept), instruction-fine-tuning (concept), laptop-gpu (location)"
classification: "reference"
content_hash: "sha256:d0d612a7e299f9e380f20479c2b6bd40d063344a91df1156a6537330c622dbae"
acp_version: "0.2"
token_counts_approximate: 9428
visibility: public
agent_accessible: true
status: "final"
---

# CUDA-LLM: A GitHub Repository for Development

GPU-Optimized Long-Context Transformer LLM + ML stack A small decoder-only language model written from scratch (no HuggingFace) in PyTorch, with custom C++/CUDA attention kernels (a FlashAttention-style tiled forward + backward and a sliding-window sparse variant), wrapped in a complete training + serving stack : from-scratch BPE tokenizer, config-driven trainer, instruction fine-tuning, and a streaming chat server. The 496M SFT deployment adds custom weight-only INT8/INT4 kernels, fused quantized projections, CUDA-graph decoding, an interactive quality/speed A/B workbench, RAG, production metrics, and reproducible laptop/Modal benchmarks. Everything is sized to train and run locally on a laptop GPU (developed/benchmarked on an RTX 3050 Laptop, 4 GB , compute capability 8.6). Quick navigation If you are reviewing this project, these sections provide the fastest tour: Section What it shows Headline results The strongest model, inference, long-context, and RAG results at a glance. What's in here A feature-level overview with direct links to the main implementations. Custom CUDA kernel How the tiled attention forward and backward kernels work. Training and serving stack The tokenizer, trainer, instruction-tuning, and serving architecture. Results and benchmarks Measured training, memory, throughput, and end-to-end performance. CUDA-graph decode Quantized INT8/INT4 inference and single-stream decode optimization. Deployment Modal GPU deployment, artifacts, scheduling, health checks, and production controls. RAG Retrieval-assisted factual evaluation without additional training. Observability Training metrics, GPU telemetry, serving latency, and dashboards. Setup and usage Installation and commands for training, testing, chatting, and serving. Project layout A concise map of the repository. Headline results Result Measurement Deployed SFT model 496.3M unique parameters , 1,024-token trained context Laptop INT4 decode 230 tok/s , 4.35 ms/token, 340 MB peak allocated VRAM Modal A10 INT4 serving 289.5 tok/s , 22 ms warm TTFT, 709 ms for 200 tokens Modal A10 INT8 serving 265.5 tok/s , 22 ms warm TTFT; same output as FP16 on the reported deterministic prompt CUDA graph gain on legacy FP16 path 3.7x on the ~500M checkpoint (31.3 to 117 tok/s) Long-context sparse attention ~48x faster prefill and ~10x lower peak memory than materialized attention at 8,192 tokens on the benchmark model Factual probe with retrieval 40% to 70% accuracy on the recorded small evaluation set Results use different harnesses and are labeled accordingly below; isolated decode, end-to-end serving, cold start, and retrieval latency are not conflated. What's in here Area Implementation Concurrent serving FIFO streaming generation scheduler accepts concurrent HTTP clients while serializing the single mutable CUDA graph/KV cache; includes disconnect cancellation, failure isolation, and per-variant queue statistics ; serve/scheduler.py Production controls Versioned Safetensors artifacts with SHA-256 verification, anonymous per-IP hourly quotas, bounded admission/deadlines, readiness/liveness/model health, request IDs, graceful scheduler shutdown, structured job logs, and Prometheus latency histograms Quantized inference Custom graph-safe INT8 and packed INT4 weight-only GEMV kernels; fused QKV and SwiGLU input projections reduce launches without changing quantized logits ; llmkit/quant.py , csrc/int8_gemv.cu , csrc/int4_gemv.cu A/B evaluation Interactive FP16/INT8/INT4 × eager/CUDA-graph comparison with selectable context, deterministic decoding, per-answer TTFT, latency, and tok/s Deployment Modal GPU deployment with persistent model volume, CUDA extension prebuild, graph warmup, RAG cache, observability, and scale-to-zero ; modal_app.py , DEPLOY.md Model LLaMA-style decoder: token embedding, RoPE , pre-norm RMSNorm , SwiGLU MLP, causal attention, weight tying ; all hand-written in model/ CUDA kernel Tiled FlashAttention forward + backward (online softmax, shared-memory K/V tiles, causal + sliding-window), differentiable via an autograd.Function ; csrc/flash_attention.cu , flash_ext/ Attention backends naive (baseline), sdpa (PyTorch fused), flash (custom CUDA), sparse (custom CUDA sliding window) ; model/backends.py Fast decode CUDA-graph single-stream decoding over a static KV cache ; captures the per-token forward once and replays it, erasing kernel-launch overhead ( 3–7× tok/s ) ; llmkit/fast_decode.py , bench_decode.py Tokenizers from-scratch char and byte-level BPE (with chat special tokens) ; data/ Training infra YAML experiment configs, Trainer with checkpoint/resume, grad-accumulation, AMP, cosine LR, JSONL metric logging, model registry ; llmkit/ Chat / SFT chat template + response-masked instruction fine-tuning ; llmkit/chat.py , llmkit/trainer.py Serving streaming KV-cached inference engine, CLI chat REPL , and a FastAPI server + web UI ; llmkit/engine.py , serve/ Benchmarks attention-op and full-model long-context sweeps ; benchmark.py , bench_model.py Serving validation Concurrent HTTP load testing plus deterministic quality/speed probes ; load_test.py , eval_serving.py Tests kernel fwd/bwd correctness (GPU) + model logic (CPU) ; tests/ Pipeline at a glance pretrain (base LM) instruction fine-tune serve ───────────────── ───────────────────── ───── run.py pretrain ──ckpt──▶ run.py sft ──ckpt──▶ run.py chat / serve (REPL + API + web) BPE tokenizer chat template, streaming, KV cache, custom CUDA attn response-masked loss swappable attn backend The custom CUDA kernel The core experiment is csrc/flash_attention.cu . It implements scaled-dot-product attention without ever materialising the (S × S) score matrix: Grid (ceil(S/64), n_heads, batch) , block of 64 threads ; one query row per thread. Streams over key/value tiles of 64 rows staged into shared memory . Maintains a running softmax per query row (max m , denominator l ) and an output accumulator, rescaling by exp(m - m_new) as new tiles arrive (the FlashAttention trick). A single window parameter switches between full-causal and sliding-window attention, so the same kernel serves both the flash and sparse backends. Generalised to S_q ≤ S_k so it is correct both for full prefill passes and for single-token cached decoding. A matching backward kernel computes dQ/dK/dV (recomputing the softmax probabilities from the saved log-sum-exp, atomic-accumulating dK/dV ), so the kernel is fully differentiable and the model can train on it ; verified against autograd to ~1e-6 ( tests/test_backward.py ). Because the score matrix is never written to global memory, peak memory grows linearly with context instead of quadratically ; which is exactly what lets longer contexts fit on a 4 GB card. Training + serving stack The model is wrapped in a small ML platform under llmkit/ , driven by YAML configs ( configs/ ) through one entry point ( run.py ): Tokenizer ; a from-scratch byte-level BPE ( ~2.5× fewer tokens than char-level) with reserved chat special tokens. Trainer ; checkpoint/resume, gradient accumulation, AMP, cosine LR, JSONL metric logs, and a checkpoint registry ; one class serves both pretraining and SFT. Instruction fine-tuning ; a chat template ( <|system|> / <|user|> / <|assistant|> / <|endofturn|> ) and response-masked loss (only the assistant tokens are supervised), turning the base model into a chat model. Serving ; a streaming, KV-cached inference engine behind a CLI REPL and a FastAPI server with a web chat UI; the attention backend (including the custom CUDA kernel) is swappable at load time. Maxing out a 4 GB GPU A 4 GB card holds far more model than a naive setup uses. The knobs (all in the YAML config) that let a bigger model fit and train: Technique Effect Where Gradient checkpointing recompute block activations in backward → big activation-memory savings, fit deeper/wider models model.grad_checkpoint: true bf16 autocast half-precision compute on Ampere with no loss-scaler run.amp_dtype: bf16 Gradient accumulation large effective batch from a small, low-memory micro-batch optim.grad_accum: N 8-bit AdamW ~½ the optimizer state (8→2 bytes/param) → ~50% more trainable params optim.optimizer: adamw8bit Streamed corpus pull an arbitrary slice of the multi-GB TinyStories train file without downloading it all data.max_chars Tied embeddings / memoised BPE smaller param + fast tokenisation of large corpora on by default Verified result: stacking gradient checkpointing + bf16 + 8-bit Adam, an 85M-param model trains in just 1.3 GB on the 4 GB card ; less VRAM than the 32M model uses with plain AdamW . There is still headroom for ~150–200M params (data, not memory, becomes the limit). Configs: python run.py pretrain --config configs/pretrain_max.yaml # ~33M, bf16 python run.py pretrain --config configs/pretrain_xl.yaml # ~85M, ckpt+8bit python run.py sft --config configs/sft_xl.yaml python run.py chat --ckpt checkpoints/sft_xl Knowledge & scale: Wikipedia, big tokenizers, and Colab The TinyStories models are fluent but clueless. Two things change that ; a knowledge-bearing corpus and a production data pipeline : Fast tokenizer ; tokenizer: hf_bpe uses HuggingFace tokenizers (Rust byte-level BPE). The from-scratch Python BPE is kept as the learning artifact; hf_bpe is what makes Wikipedia-scale tokenisation feasible. Memory-mapped data ; prepare_data.py tokenises a corpus to a .bin token file once; MemmapDataset ( llmkit/data.py ) np.memmap s it so billions of tokens cost ~no RAM. Wikipedia corpus ; dataset: wikipedia streams English Wikipedia via HF datasets ( data/corpora.py ). A 128M model trained on 150 MB of Wikipedia (42M tokens) on the 4 GB GPU ( grad_checkpoint + 8-bit Adam, 2.2 GB VRAM ) shifts the model into an encyclopedic register ; real-world entities, dates, institutional language ; versus TinyStories' children's-story voice: prompt: "The National Institute for Science and Technology" 128M -> "... was established in 1819 as a member of the ..." (Wikipedia-style) chat -> "A computer is a computer that ... uses the machine to use data. It can be used to generate output ... such as a network ..." Facts are still wrong ; 128M on 42M tokens is ~40× under Chinchilla-optimal, so it learns the style of knowledge, not the facts. Fixing that needs more params and more tokens, which is the Colab path. python prepare_data.py --dataset wikipedia --max-chars 150000000 \ --tokenizer hf_bpe --vocab-size 8000 --out data/wiki128 python run.py pretrain --config configs/pretrain_128m.yaml # ~128M on Wikipedia Measuring the knowledge gain. eval_facts.py is a small factual-recall eval that scores by likelihood (does the model rank "Paris" above "London" for "capital of France is ___"), so it gives signal even when a tiny model can't generate the exact answer. Cloze accuracy (random baseline 25%): model cloze geography 85M TinyStories 28% (≈ chance) 0/7 128M Wikipedia (SFT) 44% 3/7 The TinyStories model scores zero on geography (never saw a capital city); the Wikipedia model nearly doubles chance. The eval turns "feels smarter" into a number you can watch climb as you scale ( python eval_facts.py --ckpt <dir> ). Going further on Colab (300M → 1B) COLAB.md + colab/train_colab.ipynb scale the same code on Colab's GPUs, trained on full Wikipedia: ~300M on an A100 with the normal trainer (8-bit Adam + ckpt + bf16) ; finishes in ~a day, genuinely more knowledgeable. ( configs/pretrain_colab_300m.yaml ) ~1B via DeepSpeed ZeRO-3 + CPU offload ( colab/train_deepspeed.py ) on a single GPU ; fits, but Wikipedia (~3–4B tokens) is ~1 epoch for 1B, so it's an engineering milestone, not a quality win. Checkpoints to Google Drive and resumes across Colab's session limits. ( configs/pretrain_colab_1b.yaml ) The honest summary: memory was never the wall (1B fits a single GPU with the right tricks); tokens and training time are . A well-trained 300M beats an undertrained 1B. Setup # 1. CUDA-enabled PyTorch (CPU wheel won't use the GPU) pip install torch==2.3.1 --index-url https://download.pytorch.org/whl/cu121 pip install -r requirements.txt # 2. A C++ compiler is required to JIT-build the CUDA kernel: # Windows -> "Build Tools for Visual Studio" with the C++ workload # Linux -> gcc + the matching CUDA toolkit The CUDA extension is compiled just-in-time on first use and cached; on Windows the build environment ( vcvars64.bat ) is auto-discovered, so you can build from any shell. To pre-build explicitly: python build_ext.py Usage Kernel + benchmarks python -m tests.test_correctness # custom CUDA forward vs PyTorch reference python -m tests.test_backward # custom CUDA gradients vs autograd python benchmark.py --seqs 512,2048,8192,16384,65536 --backends naive,sdpa,flash,sparse python bench_model.py --seqs 512,1024,2048,4096,8192 --backends naive,sdpa,flash,sparse Full pipeline (BPE → pretrain → instruction-tune → chat) # 1. pretrain a base model (BPE tokenizer, custom attention selectable in YAML) python run.py pretrain --config configs/pretrain_bpe.yaml # 2. instruction fine-tune on top of the base checkpoint python run.py sft --config configs/sft.yaml --init-from checkpoints/pretrain_bpe # 3a. chat in the terminal (streaming, KV-cached) python run.py chat --ckpt checkpoints/sft # 3b. or serve a web UI + streaming API -> http://127.0.0.1:8000 python run.py serve --ckpt checkpoints/sft --port 8000 # swap in the custom CUDA kernel at inference time python run.py chat --ckpt checkpoints/sft --backend flash # benchmark CUDA-graph decode vs the eager loop (verifies identical output) python bench_decode.py --ckpt checkpoints/sft_500m # serve fused weight-only inference python run.py serve --ckpt checkpoints/sft_500m --quant int8 --fast-context 256 --port 8000 # interactive FP16/INT8/INT4 x eager/graph quality-speed comparison python run.py serve --ckpt checkpoints/sft_500m --quant int4 \ --fast-context 256 --compare --warmup --rag --port 8000 Quick char-level demo (no BPE/infra) python train.py --steps 2000 && python generate.py --prompt " ROMEO: " --tokens 500 Results Measured on an RTX 3050 Laptop (4 GB) , compute capability 8.6, CUDA 12.5 / PyTorch 2.3.1+cu121. Training A 3.21M-parameter model (dim 256, 4 layers, 4 heads), char-level TinyShakespeare, 2000 steps in ~90 s : step 0 | train 4.250 | val 4.216 step 500 | train 1.464 | val 1.622 step 1000 | train 1.265 | val 1.516 step 1500 | train 1.152 | val 1.471 step 2000 | train 1.096 | val 1.493 Sample ( generate.py --prompt "ROMEO:" ): ROMEO: Pardon me, here was spoken to the beard, And call I have strong me delay, A princely door foe to stand now, And those springs come red of that thought of her commission will not be. Take some good worship in the troth Against the story shall be with my heart. Text stays coherent within the 256-token training window and then degrades as RoPE extrapolates past it ; a small, visible illustration of the very long-context problem the kernels address. BPE pretrain → instruction tuning (chat) Pretrained on TinyStories (general, simple English, well suited to tiny models) then SFT'd on general Alpaca instructions, with a real held-out val split and best-checkpointing. Three scales, all trained on the same 4 GB GPU : model base data base val sft val train VRAM how it fits 10.6M 4.2M tok 2.00 2.73 1.4 GB fp16 AdamW 32.1M 22.6M tok 1.72 2.35 3.0 GB bf16 AdamW 85.0M 56.7M tok 1.70 2.21 1.3 GB bf16 + grad-ckpt + 8-bit Adam The 85M model is 8× the params in less VRAM than the 32M ; gradient checkpointing + 8-bit Adam more than pay for the extra weights. Val loss improves monotonically with scale; chat coherence improves too, though incrementally (the ceiling is knowledge/data, not the pipeline). Sample (85M, run.py chat with repetition penalty): USER: Hi, how are you doing? BOT : I'm OK, I was doing. How did you do to be okay? USER: Give me three tips for staying healthy. BOT : 1. Note a hypothesis and maintain the abilities 2. Get or rest for flexible health and relaxation 3. Eating an efficient ... The model is fluent and correctly formatted (numbered lists, short stories, conversational replies, stops at <|endofturn|> ) but not knowledgeable ; factual answers are confidently wrong. That is the expected behaviour for a ~10–30M-param model: the bottleneck is parameter count and SFT-data size, not the pipeline. The same run.py + YAML scales to a genuinely capable chat model on a larger GPU/corpus (see Maxing out a 4 GB GPU ). Benchmarks Attention-only sweep, B=1, H=6, head_dim=64 , float32, causal; the sparse backend uses a 256-token sliding window. Peak attention memory (MB) ; the baseline is quadratic, the tiled/sparse kernels are linear: seq naive (baseline) sdpa flash (custom CUDA) sparse (custom CUDA) 512 25.3 12.5 12.5 12.5 2048 229.8 24.2 24.2 24.2 4096 862.1 40.0 40.0 40.0 8192 3359.8 71.4 71.4 71.4 16384 OOM 125.8 125.8 125.8 32768 OOM 251.7 251.7 251.7 65536 OOM 503.3 503.3 503.3 At seq 8192 the tiled kernel uses ~47× less memory than the materialised baseline (71 MB vs 3.36 GB). The baseline OOMs at 16384; the tiled/sparse kernels reach 65536 tokens in ~0.5 GB on the same 4 GB card (a naive attention matrix at 64K would need ~200 GB). Throughput (tokens/s) ; the sparse (sliding-window) kernel wins at long context because its cost is O(S·window) instead of O(S²) : seq naive sdpa flash (custom) sparse (custom) 2048 286,887 1,049,449 823,880 1,825,312 4096 145,427 569,751 473,751 1,377,733 8192 9,150 289,507 247,952 767,688 16384 OOM 141,898 124,520 395,971 65536 OOM 33,227 28,063 106,419 Takeaways The custom tiled kernel reproduces FlashAttention's linear memory scaling, matching PyTorch's fused sdpa footprint while never materialising the score matrix ; enabling 8× longer context than the baseline on the same GPU. The custom sparse kernel is ~3–4× faster than full attention at long context (e.g. 106K vs 28K tok/s at 65536). The hand-written kernel is a touch slower than PyTorch's heavily-tuned sdpa on raw speed (expected ; it favours a clear algorithm over micro-optimisation) but matches it on the metric that decides whether long context fits at all: peak memory. End-to-end full model Running the whole 10.6M-param model ( bench_model.py ) ; prefill peak VRAM/latency and incremental decode throughput as context grows: seq backend prefill ms peak MB decode tok/s 4096 naive 225.6 979.6 145 4096 sparse 53.6 201.6 155 8192 naive 6313.4 3546.6 149 8192 sdpa 246.8 346.3 78 8192 flash 268.1 346.3 41 8192 sparse 132.1 346.3 146 At 8192 the baseline needs 6.3 s and 3.5 GB to prefill; the sparse kernel does it in 132 ms and 346 MB (~10× less full-model memory, ~48× faster prefill). Decode throughput stays flat (~150 tok/s) for sparse as context grows, while full-attention decode degrades (78/41 tok/s at 8192) ; the sliding-window KV cache makes per-token cost independent of context length. CUDA-graph decode (single-stream throughput) One decode step of a small model is only microseconds of GPU math per kernel, so launching hundreds of kernels from Python per token dominates. llmkit/fast_decode.py fixes this with a static KV cache (every step is shape-identical) and a CUDA graph (capture the step's kernels once, replay() per token). Sampling, repetition penalty and stop-token logic stay in Python; only the forward is graphed, and the greedy output is token-for-token identical to the eager path. Reproduce with bench_decode.py (RTX 3050, fp16): model eager tok/s CUDA-graph tok/s speedup 85M 60.7 408.4 6.7× 476M 31.3 117.0 3.7× Smaller models gain more (launch overhead is a bigger fraction of their tiny per-step math). The engine enables it automatically on CUDA and falls back to the eager path on CPU. The FastAPI serving layer queues concurrent generation so the single mutable graph/cache remains one-request-at-a-time. Weight-only optimization sweep (RTX 3050 Laptop) bench_optimizations.py runs each implementation in a fresh process and reports steady-state single-stream decode separately from prefill. The custom projections use symmetric per-output-channel quantization; INT4 packs two signed weights per byte. Results for the 496M-parameter SFT checkpoint, 128 generated tokens, a 16-token prompt, and a 256-token static cache: implementation tok/s ms/token peak allocated VRAM FP16 eager KV cache 34.2 29.2 ~947 MB weights INT8 eager 37.7 26.5 508 MB INT8 + fused projections + CUDA graph 202.5 4.94 562 MB INT8 + fused graph, device-resident greedy 205.5 4.87 562 MB INT4 + fused projections + CUDA graph 230.0 4.35 340 MB INT4 + fused graph, device-resident greedy 225.2 4.44 340 MB Quantizing the vocabulary head raised the synthetic INT4 ceiling to 236.6 tok/s, but changed generation much more severely, so it is not the recommended mode. The INT8 and INT4 graph paths both generated coherent text on a factual smoke prompt; full task-quality evaluation is still required before deployment. The quantized runtime fuses Q/K/V into one projection and the two SwiGLU input projections into one, removing 72 kernel launches per generated token across 24 layers. Fused and unfused logits were exactly equal for both INT8 and INT4 in the numerical check, and full-checkpoint greedy token checksums matched. Use --unfused with bench_optimizations.py to reproduce the baseline. python bench_optimizations.py --variant int8-graph --tokens 128 python bench_optimizations.py --variant int4-graph --tokens 128 python bench_optimizations.py --variant int4-device --tokens 128 *-graph includes the per-token device-to-host synchronization required for immediate token streaming. *-device keeps greedy selection on the GPU and is the kernel/compute ceiling for buffered output. These custom kernels establish a local baseline; mature fused AWQ/GPTQ kernels may perform better. To compare output quality and speed interactively, start the server in comparison mode. It loads one FP16, INT8, and INT4 engine while fast/eager variants share each engine's weights. The chat UI then shows checkboxes for all six variants and reports TTFT, decode throughput, and total latency under each answer: python run.py serve --ckpt checkpoints/sft_500m --quant int4 \ --fast-context 256 --compare --warmup --port 8000 --warmup compiles the custom kernels and captures CUDA graphs before accepting the first real request. Comparison models are loaded best-effort: if the Windows GPU memory budget cannot hold a precision, the UI marks it unavailable instead of failing startup. Selected variants run sequentially for repeatable single-stream measurements. The UI enables deterministic top-1 generation by default so differences reflect quantization rather than sampling randomness. Repeated RAG queries are cached in memory for the lifetime of the server. Cloud serving: T4 vs A10 The SFT checkpoint contains 496.3M unique parameters . A legacy display counter reports 475.9M because it subtracts the tied embedding/head a second time; benchmark tables use the actual unique parameter count. The web UI was used for an end-to-end warm-container comparison on Modal. Each row below is a representative 200-token response with deterministic top-1 decoding and a 256-token logical context selection; the deployed fast decoder was allocated with a 512-token static graph. Unlike the isolated kernel benchmark, these measurements include sampling, token-to-text conversion, Python streaming, and ASGI delivery after the first token. Modal GPU mode tok/s ms/token TTFT total latency T4 FP16 fast 178.5 5.60 524 ms* 1,639 ms T4 INT8 fast 243.3 4.11 23 ms 840 ms T4 INT4 fast 262.7 3.81 481 ms* 1,238 ms A10 FP16 fast 188.3 5.31 26 ms 1,083 ms A10 INT8 fast 265.5 3.77 22 ms 771 ms A10 INT4 fast 289.5 3.45 22 ms 709 ms * The two high T4 TTFT values are recorded outliers from representative UI runs and are not used as steady-state TTFT claims. Warm A10 measurements were internally consistent: for INT4, 200 / 289.5 = 691 ms of decode plus ~22 ms TTFT closely matches the reported 709 ms total. At this model size, A10 delivered only ~5-10% more single-stream throughput than T4 in these representative runs, despite substantially stronger hardware. That is a useful negative result: the current decoder is increasingly limited by scalar dequantization, small-kernel launch overhead, sampling/synchronization, and the FP16 vocabulary projection rather than raw VRAM bandwidth. In these runs, A10 won on absolute speed while T4 was the stronger cost/performance option. A larger jump requires Tensor-Core W4A16 kernels (for example, Marlin/AWQ/GPTQ or TensorRT-LLM), not simply a larger GPU. Quality/speed interpretation INT8 is the production default candidate. In one deterministic comparison, FP16 and INT8 produced the same 200-token response while INT8 was 41% faster on A10. This is prompt-level evidence, not a claim of global equivalence. INT4 maximizes speed and minimizes VRAM , but its response diverged from FP16/INT8. It remained coherent in smoke tests, while factual accuracy still requires task-level evaluation. Fusing QKV and the two SwiGLU input projections is quality-neutral relative to the corresponding unfused quantized model: logits were exactly equal in the numerical test and full-checkpoint greedy token checksums matched. Quantizing the vocabulary head gave only a small synthetic speed increase and visibly destabilized INT4 output, so serving keeps the tied head in FP16. All precisions can hallucinate. Quantization comparisons measure inference tradeoffs; they do not repair limitations of a 500M model or its training data. Context and TTFT findings The checkpoint was trained for a maximum 1,024-token context. A fast variant cannot exceed its server-startup --fast-context ; eager variants can use the trained 1,024-token limit. Selecting a smaller logical context truncates/caps prompt plus generation, but the current CUDA graph retains its startup allocation; restart with a smaller --fast-context to benchmark a physically smaller static cache. Context covers system instructions, RAG passages, conversation history, and generated output together. Smaller static graph contexts generally improve decode and reduce KV memory; longer prompts increase prefill and therefore TTFT. Warm non-RAG TTFT reached 22-26 ms in the A10 measurements. Network RAG, embedding initialization, cold container startup, and graph capture are intentionally reported separately because they can add hundreds of milliseconds or seconds. The metrics implementation separates first-token time from decode throughput: throughput is (tokens after the first) / decode seconds , aggregated by total tokens and total time so short responses cannot inflate the dashboard. Modal deployment modal_app.py builds from a CUDA development image, pins a CUDA-enabled PyTorch build, precompiles the INT8/INT4 extensions for T4, A100, A10, and L4/L40S architectures, bakes in the retrieval embedder, mounts the checkpoint from a persistent Modal Volume, warms graphs, and scales to zero. modal volume create llm-weights python export_inference.py --ckpt checkpoints/sft_500m \ --out artifacts/sft_500m_fp16 --version sft-500m-v1 modal volume put llm-weights artifacts/sft_500m_fp16 sft_500m_v1 modal serve modal_app.py # development URL modal deploy modal_app.py # production deployment The comparison deployment loads FP16, INT8, and INT4 once per container; eager and fast modes share each precision's weights. See DEPLOY.md for the complete workflow and operational caveats. Concurrent serving and scheduling Batch-one CUDA-graph decode uses mutable static KV buffers, so replaying one decoder concurrently would corrupt request state. The FastAPI layer therefore accepts concurrent connections but submits generation to a dedicated FIFO worker: HTTP clients -> FIFO queue -> one GPU generation worker -> per-client token streams This preserves the fast graph path instead of silently sending overlapping requests through eager inference. Client disconnects mark queued/running jobs for cancellation; exceptions are returned to the affected stream without stopping the worker. Chat and completion use the same scheduler, and each job records its variant, queue wait, active duration, and completion state. Live scheduling data is available in the chat sidebar, /dashboard , /api/scheduler , the nested scheduler object in /api/metrics , and Prometheus: queue depth and active job/variant; submitted, completed, cancelled, and failed jobs; recent average queue wait and generation duration; per-variant submitted/completed counts; per-response job ID and queue-wait time. Modal accepts up to four concurrent HTTP inputs per container, while the scheduler keeps GPU generation single-writer. This lets metrics/static requests and queued streams share a warm container instead of requiring one GPU container per HTTP connection. It optimizes correctness and predictable latency; it does not claim parallel single-stream generation or continuous batching. Production artifact and service controls Training checkpoints include optimizer state and use PyTorch serialization; production serving exports a smaller inference-only artifact: python export_inference.py --ckpt checkpoints/sft_500m \ --out artifacts/sft_500m_fp16 --version sft-500m-v1 The verified real-checkpoint export is 946.7 MB and contains 496,334,080 unique FP16 parameters in model.safetensors , config/tokenizer files, tied-weight aliases, model/Git versions, creation time, and SHA-256 checksums. The loader verifies the weights checksum before model construction; training checkpoints remain supported through restricted weights_only loading. Production endpoints and admission behavior: Control Behavior /health/live process liveness /health/ready scheduler/model readiness; returns 503 when unavailable /health/model model version, artifact format, Git commit, precision, context, variants Queue bound 32 waiting jobs per container Queue deadline five seconds before an unstarted job expires Saturation HTTP 429 with Retry-After Shutdown stop admission, cancel queued work, terminate scheduler worker Logs structured JSON lifecycle events keyed by job ID and variant Metrics queue/generation histograms plus bounded per-variant counters Request validation bounded messages, prompt/system length, output tokens, temperature, and top-k Authentication optional constant-time X-API-Key validation when LLM_API_KEY is set Public-demo quota 30 generation requests per IP per hour by default; HTTP 429 plus standard quota/reset headers Privacy raw IP addresses are not stored; the persistent key is a truncated HMAC-SHA256 digest Correlation incoming or generated X-Request-ID is returned to the client and attached to scheduler logs/results The browser demo does not require a login. REQUESTS_PER_HOUR configures the hourly allowance (default 30), and RATE_LIMIT_SALT should be supplied through a Modal Secret so pseudonymous client keys cannot be reproduced from a public source value. Each selected comparison variant is a separate generation request, so a three-model comparison consumes three requests. Every response exposes X-RateLimit-Limit , X-RateLimit-Remaining , and X-RateLimit-Reset ; rejected requests also include Retry-After . Modal is intentionally configured for four concurrent HTTP inputs in one GPU container. A named Modal Dict preserves quota counts across scale-to-zero, while the single-container ceiling gives that store one writer and establishes a hard demo cost ceiling. The FIFO scheduler safely queues concurrent visitors inside the container. A multi-replica product would replace this fixed-window update with an atomic distributed limiter before increasing max_containers . Load and quality validation Two standard-library clients exercise the deployed service rather than an isolated model loop. The load harness issues concurrent streaming requests and reports status counts, requests/second, and p50/p95/p99 client TTFT, latency, and queue wait. The evaluation harness runs the same deterministic factual probes across every available precision/mode and writes answer-level CSV evidence. python load_test.py --url https://YOUR-APP.modal.run \ --requests 30 --concurrency 4 --variant int4-fast --tokens 64 python eval_serving.py --url https://YOUR-APP.modal.run \ --variants fp16-fast,int8-fast,int4-fast --context 256 \ --out serving_eval.csv The public demo defaults to 30 requests/IP/hour, so raise REQUESTS_PER_HOUR temporarily for controlled load tests. Structured completion logs split request time into queue wait, generation TTFT, decode throughput, and total latency using the same X-Request-ID returned to the client. The bounded RAG cache exposes only entry/hit/miss counts through /api/metrics , /metrics , and /dashboard ; queries and retrieved passages are never exported. Arbitrary prompt-prefix caching and continuous batching are intentionally not claimed here. Both require request-isolated KV-cache slots; the current CUDA graph owns one mutable static cache and is protected by the FIFO single-writer scheduler. RAG: making a small model factual at inference (no training) A ~500M model is fluent but hallucinates specifics ("Paris has 578 million people"). Retrieval-augmented generation can improve factual answers without any training ; the same checkpoint, plus a retrieval step: search Wikipedia for the question, inject the article's intro as context, and answer from it. Recall (unreliable) becomes read-and-summarise (reliable). llmkit/rag.py ; dependency-free Wikipedia API lookup. --rag toggle on run.py chat / serve (and a per-request flag in the API) ; same model, retrieval on/off, so you can A/B compare. compare_rag.py ; runs factual probes with RAG off vs on, side-by-side, and scores the delta. Measured on the 496M checkpoint ( python compare_rag.py --ckpt checkpoints/sft_500m ): mode factual accuracy example no-RAG 40% "Who wrote 1984?" → Jane Austen ❌ RAG 70% → George Orwell ✅ +30 points with zero retraining. Honest caveat: RAG can also hurt when retrieval returns the wrong article (the model trusts bad context), so retrieval quality is the weak link ; a good area for the local-embedding upgrade (V2). python run.py chat --ckpt checkpoints/sft_500m --rag --temperature 0.3 # factual mode python compare_rag.py --ckpt checkpoints/sft_500m # document the delta Observability (production-style metrics) The stack is instrumented end-to-end via llmkit/monitor.py (NVML GPU telemetry + MFU math): Training ; every eval step logs tokens/sec, MFU (model-FLOPs-utilisation), grad-norm, GPU util & VRAM alongside loss/LR to the run's JSONL. plot_metrics.py renders the curves: python plot_metrics.py --metrics checkpoints/pretrain_500m Serving ; the API tracks per-request TTFT (time-to-first-token), latency, and decode throughput , plus live GPU stats, exposed three ways: /api/metrics ; JSON snapshot (rolling averages) /metrics ; Prometheus text format (scrape into Grafana if you want) /dashboard ; a live web dashboard (stat tiles, GPU util/VRAM meters, a throughput sparkline, and FIFO scheduler status) that polls metrics in real time python run.py serve --ckpt checkpoints/sft_500m --port 8000 # chat -> http://127.0.0.1:8000 # dash -> http://127.0.0.1:8000/dashboard # prom -> curl http://127.0.0.1:8000/metrics Example serving snapshot: { "requests" : 42 , "tok_s" : 18.4 , "ttft_ms" : 63.2 , "latency_ms" : 410.5 , "mfu" : 9.1 , "gpu_util" : 78 , "gpu_mem_used_gb" : 2.1 , "gpu_mem_total_gb" : 4.3 , "gpu_temp_c" : 71 , "gpu_power_w" : 58.4 } Is it a chatbot? (honest scope) Out of the box the pretrained model only continues text. It becomes interactive after instruction fine-tuning ( run.py sft ), which teaches the chat format via the response-masked loss. The serving stack (REPL, API, web UI) then lets you talk to it. The caveat is scale and data, not plumbing : the tiny Shakespeare demo learns chat shape but little knowledge. The deployed 496M Wikipedia/UltraChat model is substantially more capable, yet still hallucinates facts across FP16, INT8, and INT4. RAG and evaluation make that limitation measurable; they do not turn a 500M checkpoint into a frontier model. Design notes / caveats The custom attention kernel is float32 ; under fp16 autocast the backends cast around it. It favours a clear, correct tiling/online-softmax demonstration over peak FLOP efficiency ; not a drop-in replacement for production FlashAttention. The INT8/INT4 decode kernels are custom scalar weight-only GEMVs. They establish a reproducible systems baseline but do not yet use Tensor-Core W4A16 layouts; this is why A10 scales only modestly over T4. Everything is intentionally small so it runs on 4 GB. Scale the model via the YAML configs for larger GPUs. Layout model/ from-scratch transformer (config, rope, rmsnorm/swiglu, attention, backends) csrc/ custom CUDA: attention fwd/bwd + INT8/packed-INT4 decode GEMVs flash_ext/ JIT builder/loader + differentiable autograd.Function data/ char + Python-BPE + fast hf_bpe tokenizers; tinystories/wikipedia corpora llmkit/ training/serving infra (config, trainer, checkpoint, data, chat, engine, logger) configs/ YAML configs: pretrain/sft, _max (33M), _xl (85M), _128m, _colab_{300m,1b} serve/ FastAPI server + web chat UI colab/ Colab notebook + DeepSpeed (ZeRO-3) trainer for 300M-1B run.py unified CLI: pretrain | sft | chat | complete | serve prepare_data.py tokenise a corpus to memory-mapped .bin (Wikipedia-scale) benchmark.py / bench_model.py / bench_optimizations.py operator, model, decode sweeps tests/ kernel-correctness (GPU) + model-logic/tokenizer (CPU) tests COLAB.md scaling on Colab: honest compute math + resume strategy modal_app.py / DEPLOY.md CUDA-image Modal deployment + persistent model volume