---
title: "Building a Custom ChatGPT Model from Scratch"
url: https://stacklist.com/card/0cbe0b53-0559-4f29-a5f6-523c257d6b46
source_url: "https://www.linkedin.com/posts/alex-lotkov-0b0948243_llm-aiengineering-aiinfrastructure-ugcPost-7483362836237160449-G2re/?utm_source=share&utm_medium=member_ios&rcm=ACoAAAI21ZsBNnZPaKuTab7nquKLCveUW7o-1DE"
stack: https://stacklist.com/stack/4aae218c-38d7-4c05-b7ae-f2db0029a2e8
summary: "AI Engineering project where Alex Lotkov built a 496M-parameter ChatGPT-like model from scratch in 24 hours for $20, achieving 230-340 tokens per second on a 4GB laptop GPU through aggressive memory optimization and custom CUDA kernels. The project demonstrates that model building, memory fitting, training, and efficient serving are distinct engineering challenges, with INT4 quantization providing optimal speed and memory usage."
tags: "llm-inference, gpu-optimization, cuda-kernels, quantization, ai-engineering, model-training, inference-optimization"
key_entities: "Alex Lotkov (person), CUDA (technology), FlashAttention (technology), PyTorch (technology), Transformer (technology), INT4 quantization (technology), RTX 3050 (technology), Modal A10 (technology), GPU memory optimization (concept), inference optimization (concept), RAG (concept)"
classification: "analysis"
content_hash: "sha256:f7978eaa1c2da5bcc08408f819876ffcdf262dde9eb385d61f95a2ecadc4643e"
acp_version: "0.2"
token_counts_approximate: 1058
visibility: public
agent_accessible: true
status: "final"
---

# Building a Custom ChatGPT Model from Scratch

Alex Lotkov 23h Report this post I built my own ChatGPT from scratch and trained the 496M-parameter model in 24 hours for $20 by aggressively optimizing GPU memory and inference. No fine-tuning of someone else’s model. No pretrained weights. No LLM APIs. No step-by-step tutorial. I started from random weights and built the entire stack: → Transformer architecture and tokenizer → Pretraining and instruction training → Custom C++ and CUDA attention kernels → FlashAttention-style forward and backward passes → Sliding-window sparse attention → Custom INT8 and packed INT4 inference kernels → CUDA graph decoding and KV caching → RAG, streaming deployment, and production metrics Key results: • 230 tokens per second on an RTX 3050 laptop GPU with only 4 GB of VRAM • 340 MB peak allocated VRAM with INT4 inference • 289.5 tokens per second on a Modal A10 • 3.7× faster FP16 decoding using CUDA graphs • 48× faster prefill and 10× lower peak memory at an 8,192-token context • Custom attention kernels scaled to 65,536 tokens on the 4 GB GPU • Factual accuracy improved from 40% to 70% after adding retrieval • Up to 299.5 tokens per second with INT4-fast inference on a Modal A10, with coherent output in live testing One of the most surprising findings was that stronger hardware did not automatically create proportionally faster inference. The A10 was only modestly faster than the T4 because the bottleneck had shifted to dequantization, synchronization, kernel launches, and the vocabulary projection. INT4 produced the highest speed and lowest memory usage, while INT8 offered a better balance between performance and output stability. This project taught me that building the model, fitting it into memory, training it effectively, and serving it efficiently are four different engineering problems. The full project is open source: GitHub: https://lnkd.in/dErX-wPr Live demo: https://lnkd.in/dBRRZ5e7 (The website and GPU may take 2 to 30 seconds to start.) I’ll be sharing deeper breakdowns of the CUDA kernels, quantization, long-context attention, GPU memory optimization, inference benchmarks, and deployment architecture. #LLM #AIEngineering #AIInfrastructure #MLSystems #MachineLearning #DeepLearning #CUDA #GPUComputing #GPUOptimization #InferenceOptimization #LLMInference #PyTorch #Quantization #FlashAttention #MLOps #OpenSource …more 64 22 Comments Like Comment Share Copy LinkedIn Facebook X Kevin Marley 14h Report this comment Nice work. Out of curiosity, what optimizer did you use? Also, roughly how many training tokens did you use, what was the final validation loss perplexity, and how many GPU hours did the pretraining take? That's a long list of achievements out of curiosity did you build the inference engine from scratch or modify a preexisting engine? Like Reply 1 Reaction 2 Reactions Samuel Alberts 16h Report this comment Very interesting. I’m always curious how much VRAM someone needs for training models. My guess is always 24 gigs for decent sized models, but even 12 gigs works with medium to large sized projects. Impressive 4 gigs on a laptop! No Google collab and a locally trained model. Well done! Like Reply 1 Reaction Mike Boscia 1m Report this comment This is perfect and is exactly what I am looking for. Can’t wait to check out the GitHub repos. Like Reply 1 Reaction Steven Fisher 5h Report this comment I guess you will soon be the proud recipient of a $$$ sign-on bonus & job offer 😃 Like Reply 1 Reaction 2 Reactions Rajagopal Chellapilla 23h Report this comment Absolutely incredible work. Building the entire stack from random weights and then optimizing it to run this efficiently on just 4 GB of VRAM is world-class engineering. Excited to see the deeper technical breakdowns Like Reply 1 Reaction Steve Harris 12h Report this comment Alex Lotkov - impressive!! Like Reply 1 Reaction 2 Reactions Josh Lindley 2h Report this comment That is amazing! Like Reply 1 Reaction Florin Anesia 15h Report this comment Bravo! 👏👌🙌 Like Reply 1 Reaction Owen Borcherding 11h Report this comment Keep up the strong AI work! Nice job Alex! Like Reply 1 Reaction See more comments To view or add a comment, sign in