---
title: "AI Memory System Ingestion: Beyond Compute Challenges"
url: https://stacklist.com/card/7c470cbb-4623-431f-8474-f640c6cf815a
source_url: "https://www.linkedin.com/posts/pauliusztin_i-used-to-think-ingesting-1000000-documents-share-7477286521130315776-xGxf/?utm_source=share&utm_medium=member_ios&rcm=ACoAAAI21ZsBNnZPaKuTab7nquKLCveUW7o-1DE"
stack: https://stacklist.com/stack/2b72dea2-1800-44ce-a6b3-7a1b4186801a
summary: "Ingesting 1,000,000 documents into an AI memory system is fundamentally an orchestration problem requiring two-level parallelism and independent work pools rather than just additional compute resources. The architecture separates data ingestion and memory transformation into independently scalable stages, with each bottleneck (LLM extraction, embeddings, database I/O) optimized separately using tools like Prefect for workflow coordination."
tags: "ai-memory, document-ingestion, system-architecture, parallelism, orchestration, workflow-optimization, scalability"
key_entities: "Paul Iusztin (person), Decoding AI (organization), Prefect (technology), Dask (technology), Ray (technology), vLLM (technology), pipeline-parallelism (concept), task-parallelism (concept), knowledge-graph (concept)"
classification: "analysis"
content_hash: "sha256:d3b9d6140b9cce9c0f97d1c6236f96da53480fe0d7ca1205f1509587c1371905"
acp_version: "0.2"
token_counts_approximate: 2086
visibility: public
agent_accessible: true
status: "final"
---

# AI Memory System Ingestion: Beyond Compute Challenges

Paul Iusztin Senior AI Engineer • Founder @ Decoding AI • Author @ LLM Engineer’s Handbook ~ I ship AI products and teach you about the process. 1d Report this post I used to think ingesting 1,000,000 documents into an AI memory system was mostly a compute problem. But I've been proven wrong... This is an orchestration problem. Throwing more GPUs at the pipeline won't help much if your architecture still processes everything sequentially. The system I've been designing separates ingestion into two independent work pools. The first turns raw data into documents. The second turns those documents into memory. Here's the high-level architecture: 𝟭/ 𝗧𝘄𝗼 𝗹𝗲𝘃𝗲𝗹𝘀 𝗼𝗳 𝗽𝗮𝗿𝗮𝗹𝗹𝗲𝗶𝘀𝗺 Imagine ingesting 1,000,000 documents. A Prefect workflow shards them into 1,000-document jobs. Each worker then processes those jobs in batches of 100. That gives you two levels of parallelism: 1. Pipeline parallelism → distribute shards across workers. 2. Task parallelism → batch expensive operations inside each worker. Need more throughput for LLMs or embeddings? Swap in a Dask or Ray cluster without changing the architecture. 𝟮/ 𝗗𝗮𝘁𝗮 𝘄𝗼𝗿𝗸 𝗽𝗼𝗼𝗹 The pipeline starts with: • Web URLs • RSS feeds A Prefect workflow flattens every source into URLs, shards them into jobs, and pushes them into a queue. Workers continuously pull jobs as capacity becomes available, making it easy to absorb traffic spikes by simply adding more workers. Each worker: • Flattens URLs • Scrapes 100 URLs concurrently • Transforms content • Batch-loads documents into storage This stage is mostly network and database I/O. Batching matters far more than compute. 𝟯/ 𝗠𝗲𝗺𝗼𝗿𝘆 𝘄𝗼𝗿𝗸 𝗽𝗼𝗼𝗹 Once documents reach the warehouse, a second Prefect workflow repeats the pattern. Each worker runs: • Chunking • Batched LLM extraction • Entity normalization • Batched embeddings • Batch-loads Knowledge graph objects into storage For example, 1,000 documents might become 10,000 chunks, processed in batches of 100. Every stage has different bottlenecks. • LLM extraction → vLLM • Entity normalization → database I/O • Embeddings → often CPU Treating them all the same leaves performance on the table. Here's the gist: Scaling AI memory isn't about adding GPUs. It's about designing an architecture where every bottleneck scales independently. This is why I chose Prefect . It orchestrates both pipelines with sharding, queues, retries, scheduling, and durable execution, making it practical to scale from thousands to millions of documents. P.S. What would stop your pipeline from ingesting one million documents today? 832 56 Comments Like Comment Share Copy LinkedIn Facebook X Md Rashedul Hasan 21h Report this comment This is a strong reminder that scalability in AI systems is rarely a single-resource problem; it is an architectural one. What stands out here is the shift from “more compute” to better decomposition: separating ingestion, transformation, and memory construction into independently scalable stages. That design choice is what turns throughput from a bottleneck into a controllable property of the system. I also appreciate the emphasis on parallelism at both the pipeline and task levels, because that is where many real systems fail—not in the model layer, but in the orchestration layer. In that sense, Prefect is not just a workflow tool here; it becomes a coordination fabric for heterogeneous bottlenecks. The broader lesson is that durable AI memory requires systems thinking as much as model quality. Like Reply 2 Reactions 3 Reactions Vitalii Serbyn 1d Report this comment the bottleneck that always surfaces first is the knowledge graph write pattern - most pipelines chunk and embed fine, then serialize on upserts because they're merging entity relationships one document at a time. batching the graph writes (grouping entities by type, then bulk-merging edges) usually matters more than parallelizing the LLM calls upstream. Like Reply 6 Reactions 7 Reactions David Emery 1d Report this comment Concurrency is always hard. Reasoning about what can be done in parallel, what needs to be serialized/synchronized, and then about what can go wrong, takes both training and experience. It starts with having a common language to describe concurrency and synchronization problems. Like Reply 7 Reactions 8 Reactions Jono Herrington 1d Report this comment The part of this that bites people in production is what happens when the two pools get out of sync. If the data pool ingests faster than the memory pool can chunk, extract, and embed, you end up with a queue that grows without bound until something falls over, usually storage costs or a database connection pool. I've seen teams build exactly this kind of two-pool architecture and skip building backpressure between them because it never shows up in a demo. It only shows up three weeks into production when someone dumps a bulk import job and the ingestion pool cheerfully keeps accepting work the memory pool has no chance of keeping up with. The fix is boring, cap the queue depth and let the ingestion side slow down or shed load, but almost nobody builds it until they've been paged for the first outage it causes. Like Reply 5 Reactions 6 Reactions Guillaume Belisle 22h Report this comment The separation between data work and memory work makes a lot of sense. Raw data ingestion is mostly I/O, retries, scraping, transforms, and storage. Memory construction is chunking, extraction, entity normalization, embeddings, and graph writes. Different failure modes. Different scaling patterns. Different monitoring needs. Collapsing them into one pipeline hides the bottleneck. Like Reply 2 Reactions 3 Reactions Aymane MAGHOUTI 1d Report this comment Solid design, the part I can't figure out: with Prefect's durable execution and retries, what happens when a worker fails mid-batch during LLM extraction, say 60 of 100 chunks already got entities written, then it crashes? Do retries reprocess the whole batch (risking duplicate entities before normalization catches them), or is there checkpointing at the chunk level? Batch size is great for throughput, but it's also your unit of retry, and those two pull in opposite directions. Like Reply 1 Reaction 2 Reactions Jhanvi Dattani 17h Report this comment Thank you sharing this Paul Iusztin I have small question why do we need 2 entities in both pools: Orchestrator and sharding & Fan Out. Like can it be simplified having just one service whose work is to distribute the docs in chunk of say 100, add this chunks into queue and worker picks up this 100 chunks from queue. Curious to know what would be reason to have 2 entities: Orchestrator and sharding & Fan Out Like Reply 1 Reaction 2 Reactions Chat Data 21h Report this comment The struggle of herding a million documents into an AI memory system is very real—sounds less like running a GPU farm and more like trying to coordinate cats who’ve read Kafka! Orchestration really does become the main event, as anyone who’s juggled inputs and chunking nightmares knows. That’s where https://www.chat-data.com / shines bright. It supports everything from structured .csv files to wild unstructured PDFs, and its workflow builder lets you automate multi-step imports and context handling. All the heavy orchestration is handled while you sip your coffee—no herding required. Like Reply 1 Reaction Nilesh Chavan 21h Report this comment Nice demo! I’m curious about the incremental update strategy. In production, we rarely reindex everything. Usually, only a few documents—or even a few chunks within a document—change. detect those changes and update only the affected embeddings instead of rebuilding the entire index?, as efficient incremental indexing is critical for keeping ingested systems scalable and cost-effective. Like Reply 1 Reaction 2 Reactions Moti Atedgi 1d Report this comment Scale always exposes the infrastructure!! The moment people realize that LLM context limits don't solve the ingestion pipeline problem, they understand why we need proper distributed system patterns. This is the difference from play with AI, to make it production ready Like Reply 4 Reactions 5 Reactions See more comments To view or add a comment, sign in