GenAI App Architecture Explained (Part 2: Completing the Big Picture)

Méoc

 In the first article, we outlined the high-level architecture of a modern GenAI application: the orchestrator, the embeddings model, the vector database, and the external actions or tools.




Today, we complete that picture by adding the supporting components that make the system resilient, observable, and reliable.


The often-overlooked building blocks


In the diagram above, these gray blocks are the hidden backbone of any production-grade GenAI app. Let's focus on the 3 remaining ones.



LLM Cache

This is where previously generated model responses are stored so they can be reused later. The goal: reduce latency and avoid unnecessary calls to expensive models.

Typical tools: Redis, SQLite, GPTCache.

Caching is crucial for both cost control and responsiveness. When a user asks the same or a very similar question, the system can serve a cached response instead of re-querying the model.

But caching in the LLM context is not as straightforward as caching a static API response. Queries are often semantically similar but not identical. To handle this, advanced caches (like GPTCache) compute embeddings for both the incoming query and the previously cached queries, then use a vector similarity search (cosine or dot-product similarity) to find the most relevant cached result.

That means even if two questions differ slightly (e.g., “What is GenAI?” vs “Can you explain generative AI?”), the cache might still return the same stored answer — if their embeddings are close enough.

Important caveats:

  • Caching must be tuned carefully: too broad a match leads to irrelevant answers, too strict means little reuse.

  • Cached outputs can become outdated when models or context data change.

  • Sensitive data should never be stored unencrypted in cache.


Logging / LLMOps

This layer records everything that happens between the user, the orchestrator, and the model. It’s what allows you to debug, analyze usage, measure latency, and improve prompts over time.

Core tools: Weights & Biases, MLflow, PromptLayer, Helicone.

But LLMOps doesn’t stop at tracing the LLM itself. You also need infrastructure-level observability. Tools such as Grafana, Datadog, Prometheus, and OpenTelemetry help you monitor the entire stack — from API throughput and GPU load to database latency and network health.
They provide dashboards, alerts, and traces that ensure your app remains reliable and cost-efficient even as traffic scales.

This cross-layer observability (LLM + infra + orchestration) is what transforms a prototype chatbot into a production system.


Validation

Validation layers act as the final filter before a response is sent to the user. Their role: ensure outputs are safe, consistent, and useful.

Typical frameworks: Guardrails, Rebuff, Guidance, LMQL.

They implement a mix of techniques:

  • Rule-based checks: regex or structured constraints (e.g., JSON schema validation).

  • LLM-based validation: another model reviews or critiques the first model’s output.

  • Classifier-based checks: toxicity, bias, or factuality classifiers using fine-tuned models.

Reliability varies:

  • Guardrails (from NVIDIA) offers a flexible framework to define validation logic, but effectiveness depends on rule coverage.

  • Rebuff leverages anomaly detection to catch unexpected responses, but requires careful tuning.

  • LMQL allows “programming” the model output with constraints embedded in the generation process — powerful but harder to scale.

Validation should be seen as a probabilistic defense: it reduces risks but can’t fully eliminate hallucinations or unsafe outputs.


What’s next

In this second part, we’ve completed the software architecture — from orchestration to caching, validation, and observability.

But the foundation still lies beneath: the Hardware Layer — GPUs, RAM, storage, and networking.
That’s where the next article will take us: Part 3 – The Hardware Layer: Powering GenAI at Scale.

most viewed articles

From Chat to Action: The New Gen AI Revolution

Méoc
Image
Artificial intelligence has already impressed us with its ability to generate text, answer questions, translate, code, and summarize. But all of that… remained passive. Today, AI is taking a new leap. It moves into action. It no longer just talks to you. It can do: publish an article, send an email, schedule a meeting, even run an entire blog. And that changes everything. Real-world examples of AI in action When I talk about generative AI that acts, I am not talking about just producing text or images. I'm talking about executing concrete, autonomous, targeted tasks. Here are a few examples from the field: 1. Fully automated blogging (personal case) An AI agent can: Generate a blog post from an idea. Publish it on a platform like Blogger. Create a cover image or visual summary. Share it automatically on social media. Update a tracking sheet or project management board. This isn’t a dream. I already have it running, and I’ll share the how-t...
READ MORE

GenAI App Architecture Explained (Part 1: The Big Picture)

Méoc
Image
 The first time you interact with an app like ChatGPT, all you see is the chat box. It feels magical — you type, it answers. But for anyone in Ops or engineering, one question immediately comes up:  what’s really happening behind the scenes? At the heart of every GenAI app lies an LLM (Large Language Model), but the LLM alone is just one piece of the puzzle. Around it sits an entire stack: orchestration layers, data pipelines, embeddings, vector databases, plugins, caches, observability tools, and guardrails. Together, they transform a “highly clever text generator” into a production-grade GenAI application. In this post, we’ll start with the big picture: the main components and flows that make an app like ChatGPT work. In the next parts, we’ll drill down into each layer (RAG, Ops monitoring, validation, etc.), highlighting both how they work and what usually breaks in production. Picture of what it feels like the first time we interact with a LLM: Which remind me of this famo...
READ MORE

How to Pick the Best LLM for Programming Cost vs Capability in 2025

Méoc
Image
In 2025, dozens of language models can now write, debug, or refactor code. But which ones are actually good — and which give the best value for money? Let's try to make a quick recap for your future programming session. Disclaimer: this is a moving market and each day can see tremendous changes so this is a picture in time and a methodology example for your own review  🧱 How We Calculate Token Costs The 80 / 20 blended formula Most IDE copilots send a large prompt (project context, previous messages, code files) and receive a short answer (patch, explanation). Field logs from Cline, GitHub Copilot, Cursor and Sourcegraph Cody show that roughly 80 % of tokens are input and 20 % are output . To give a single price you can reason about, we compute: Blended Cost ($/M) = 0.80 × Price_in  +  0.20 × Price_out Example : Claude 4 Opus costs $15 /M input and $75 /M output → 0.80×15 + 0.20×...
READ MORE

Can Natural Language Really Replace Code? The Revolution Is Already Underway

Méoc
Image
Natural language processing  has been floating around the tech world for yearsbut what about Natural Language programming?  What about programming whole application directly in Natural language? The tools are here, the agents are live, and people who’ve never written a line of code are already building automations, querying data, and triggering real-world actions … just by describing what they want. Before we get ahead of ourselves, let’s unpack what this term actually means—and why it’s already changing the way we create. Natural language programming isn’t one thing — it splits into two complementary approaches. Depending on the goal, you might want to generate code from a plain-English prompt, or trigger actions directly without code ever surfacing. These two paths are reshaping what “programming” even means. Text-to-code Tools like Codex, Copilot, Cline, or Windsurf translate prose into Python, JavaScript, and more. Text-to-action Autonomous agents interpret i...
READ MORE

Internet Is Nasty? Really? How?

Méoc
Image
Many years ago I connected a server directly to the Internet and watched the incoming traffic. Within minutes, connection attempts targeted every open port on the machine. Most were automated bots scanning the entire address space, looking for weak spots to steal data or take control of the host. That was nasty — and it happened twenty years ago. Fast-forward to today. I wanted to show my kids that the Internet is more than YouTube and Netflix; it is also full of tireless robots poking anything left unprotected. Normally I would drop an idea like this because of the effort involved, but I also wanted to try a new “code vibing” tool called Windsurf. Code vibing? Think of it as ultra-fast, AI-assisted development: you describe what you want, and the tool either codes it for you or constantly boosts your productivity. Here’s a video showing how I changed a small part of the interface by simply… asking for it. Building a Live Attack Dashboard in 6 Hours with AI To create my app, I ...
READ MORE

Demystifying GPU Sizing for LLMs

Méoc
Image
  The Context Large Open Source (OSS) language models are springing up everywhere – Mistrals, Llama, surprise releases like Kimi K2 and, this month, OpenAI OSS models. Each of them promises open‑source autonomy and fine‑tuned intelligence, but there’s a very practical question lurking in the background: how much hardware does it take to run them? The answer isn’t just about having “some GPUs”; it hinges on understanding the interplay between model size, memory bandwidth and inference throughput. One of my hobby projects is a GPU infrastructure estimator that addresses exactly this question. Initially built as an unwieldy spreadsheet, it has evolved into a simple web tool designed to answer a surprisingly common question: “Can I run the latest open‑source models on‑prem for tens or hundreds of thousands of euros?” . What the tool does At its core, the estimator calculates three things: memory requirements , compute throughput , and latency . To make the experience approachable, it...
READ MORE

GenAI App Architecture Explained (Part 3: The Hardware stack)

Méoc
Image
 In  Part 1  and  Part 2  of this series, we explored the high-level components of a modern GenAI application, from the user interface to the RAG pipeline. Now, let's get down to the metal. What type of resources do we need to make it all run? What Resources Do We Need? Like any application, LLMs rely on fundamental hardware resources to function: compute, memory, storage, and network. And sometimes, they need a  lot  of them. Compute LLM architectures, particularly Transformers, are heavy consumers of compute and memory. This is inherent to their design, which must process massive amounts of calculations to predict subsequent tokens. Think about it: each new token generated is the result of a complex calculation. All tokens from the prompt are turned into mathematical representations (vectors) to understand the intricate relationships between them. Based on this analysis, the model performs billions of calculations to predict the most likely next toke...
READ MORE