GenAI App Architecture Explained (Part 5: Full-Stack Observability with LLASTAKS)

Méoc

 From Kubernetes Health to End-to-End RAG Tracing with Grafana Cloud.

end to end tracing

From "It's Built" to "It Works"

In this series, we've journeyed from the high-level architecture of GenAI apps (Part 1) and their reliability components (Part 2), down to the hardware that powers them (Part 3).

In Part 4, we finally got our hands dirty by deploying LLASTAKS, our complete GenAI playground on Kubernetes (EKS). Our stack is running, complete with vLLM, a FAISS vector store, and a RAG chatbot.

But now, the real Ops work begins.

A RAG application is a complex, distributed system. If a user says "the chatbot is slow," what does that mean? Is it the Kubernetes cluster? The FAISS search? The vLLM token generation?

Without data, you're just guessing.

In this article, we'll implement the 005-observability stage of LLASTAKS. Our goal is to get full visibility across three critical layers:

  1. Infrastructure (Kubernetes): The health of our cluster, nodes, and pods.

  2. Services: Key metrics for each service, such as vLLM latency and FAISS search counts.

  3. Application: End-to-end traceability, from the user's request to the final response.

To do this, we will connect our EKS cluster to Grafana Cloud and its LGTM stack (Loki, Grafana, Tempo, Mimir).


1. The Foundation: Infrastructure Metrics (Kubernetes)

Kubernetes overview

Before blaming the AI, we must check the plumbing.

Using Terraform, LLASTAKS deploys the Grafana Alloy Agents as a DaemonSet (one per node) on our cluster. These agents immediately begin collecting and sending basic Kubernetes metrics to Mimir, which is our Prometheus-compatible metrics storage.

In just a few minutes, we get this "Kubernetes Overview" dashboard.

We can see the health of our nodes, CPU and memory usage, and pod status at a glance. If a vllm pod is in a CrashLoopBackOff state due to a GPU memory shortage, this is the first place we'll see it.


2. The Core: Service Metrics

Chatbot metrics

Faiss metrics


vLLM metrics



Knowing the cluster is healthy is good. Knowing what our application services are doing is better.

The next step is to collect custom metrics for each microservice in our LLM App stack. The Alloy agents are configured to automatically "scrape" any pod that exposes a /metrics endpoint and has the correct annotation (e.g., k8s.grafana.com/scrape: "true").

We have instrumented all our services to expose these critical metrics:

  • vLLM: Exposes vllm_request_duration_seconds, vllm_tokens_generated_total, and vllm_queue_size.

  • FAISS-wrap: Exposes faiss_wrap_request_latency_seconds, faiss_wrap_search_total, and faiss_wrap_index_size.

  • chatbot-rag (The Orchestrator): Exposes chatbot_requests_total, chatbot_vllm_requests_total (to track calls to the LLM), and chatbot_rag_chunks_sent (to see how much context we are sending).

By placing these key metrics on a single dashboard, we get a "single pane of glass" for our entire application.

Now, we can correlate events. If we see a latency spike on chatbot_request_duration_seconds, we can instantly compare it to vllm_request_duration_seconds and faiss_wrap_request_latency_seconds to find the culprit.


3. The "Holy Grail": Tracing a RAG Query End-to-End

Metrics tell us what is slow. Traces tell us why.

This is the most critical part of GenAI observability. Using OpenTelemetry, we instrumented the code for our chatbot-rag and faiss-wrap services. Every user request now generates a trace that is sent to Grafana Tempo.

chatbot-rag POST /api/chat


Log view for each trace


UX POV


Let's look at the anatomy of a single RAG query in Grafana.

What we see is the complete "waterfall" of the request, showing how one operation leads to the next:

  1. POST /api/chat (Parent Span): This is the total time the user experienced for the entire request.

  2. rag.retrieve_context: This is a manual span I added, which in turn calls the next span.

  3. POST /search (HTTP): This span shows the orchestrator's call to the FAISS service.

  4. faiss.search: This is another manual span, which contains the following child spans.

  5. faiss.embed_query and faiss.index_search: These spans clearly separate the time spent creating the query embedding from the time spent on the pure vector search.

  6. vllm.generate: Back in the orchestrator, this span measures the HTTP call to vLLM.

The real magic, however, lies in the span attributes. By clicking on any span, I can see rich business context:

  • On the chatbot-rag span, I can see attributes like rag.query, rag.chunks_sent, and llm.tokens_prompt.

  • On the faiss-wrap span, I can see faiss.results_count and faiss.score_avg.

I can now answer complex questions. "Is this request slow because we are sending too many chunks (rag.chunks_sent)?" or "Is the relevance poor because the faiss.score_avg is low?"

We are no longer just debugging performance; we are debugging the effectiveness of our RAG pipeline.


Conclusion: From Black Box to Glass Box

In Part 4, we built the LLASTAKS playground. In this post, we've lit it up.

Observability is not an "add-on" for GenAI applications; it is a fundamental requirement. Without it, you are flying blind.

Feel free to implement it on your own, it's a lot of fun!

You can find all the Terraform code, annotations, and OpenTelemetry instrumentation in the LLASTAKS GitHub repository.

most viewed articles

From Chat to Action: The New Gen AI Revolution

Méoc
Image
Artificial intelligence has already impressed us with its ability to generate text, answer questions, translate, code, and summarize. But all of that… remained passive. Today, AI is taking a new leap. It moves into action. It no longer just talks to you. It can do: publish an article, send an email, schedule a meeting, even run an entire blog. And that changes everything. Real-world examples of AI in action When I talk about generative AI that acts, I am not talking about just producing text or images. I'm talking about executing concrete, autonomous, targeted tasks. Here are a few examples from the field: 1. Fully automated blogging (personal case) An AI agent can: Generate a blog post from an idea. Publish it on a platform like Blogger. Create a cover image or visual summary. Share it automatically on social media. Update a tracking sheet or project management board. This isn’t a dream. I already have it running, and I’ll share the how-t...
READ MORE

GenAI App Architecture Explained (Part 1: The Big Picture)

Méoc
Image
 The first time you interact with an app like ChatGPT, all you see is the chat box. It feels magical — you type, it answers. But for anyone in Ops or engineering, one question immediately comes up:  what’s really happening behind the scenes? At the heart of every GenAI app lies an LLM (Large Language Model), but the LLM alone is just one piece of the puzzle. Around it sits an entire stack: orchestration layers, data pipelines, embeddings, vector databases, plugins, caches, observability tools, and guardrails. Together, they transform a “highly clever text generator” into a production-grade GenAI application. In this post, we’ll start with the big picture: the main components and flows that make an app like ChatGPT work. In the next parts, we’ll drill down into each layer (RAG, Ops monitoring, validation, etc.), highlighting both how they work and what usually breaks in production. Picture of what it feels like the first time we interact with a LLM: Which remind me of this famo...
READ MORE

How to Pick the Best LLM for Programming Cost vs Capability in 2025

Méoc
Image
In 2025, dozens of language models can now write, debug, or refactor code. But which ones are actually good — and which give the best value for money? Let's try to make a quick recap for your future programming session. Disclaimer: this is a moving market and each day can see tremendous changes so this is a picture in time and a methodology example for your own review  🧱 How We Calculate Token Costs The 80 / 20 blended formula Most IDE copilots send a large prompt (project context, previous messages, code files) and receive a short answer (patch, explanation). Field logs from Cline, GitHub Copilot, Cursor and Sourcegraph Cody show that roughly 80 % of tokens are input and 20 % are output . To give a single price you can reason about, we compute: Blended Cost ($/M) = 0.80 × Price_in  +  0.20 × Price_out Example : Claude 4 Opus costs $15 /M input and $75 /M output → 0.80×15 + 0.20×...
READ MORE

GenAI App Architecture Explained (Part 2: Completing the Big Picture)

Méoc
Image
 In the first article , we outlined the high-level architecture of a modern GenAI application: the orchestrator, the embeddings model, the vector database, and the external actions or tools. Today, we complete that picture by adding the supporting components that make the system resilient, observable, and reliable . The often-overlooked building blocks In the diagram above, these gray blocks are the hidden backbone of any production-grade GenAI app. Let's focus on the 3 remaining ones. LLM Cache This is where previously generated model responses are stored so they can be reused later. The goal: reduce latency and avoid unnecessary calls to expensive models. Typical tools: Redis, SQLite, GPTCache. Caching is crucial for both cost control and responsiveness . When a user asks the same or a very similar question, the system can serve a cached response instead of re-querying the model. But caching in the LLM context is not as straightforward as caching a static API response. Queries ...
READ MORE

Can Natural Language Really Replace Code? The Revolution Is Already Underway

Méoc
Image
Natural language processing  has been floating around the tech world for yearsbut what about Natural Language programming?  What about programming whole application directly in Natural language? The tools are here, the agents are live, and people who’ve never written a line of code are already building automations, querying data, and triggering real-world actions … just by describing what they want. Before we get ahead of ourselves, let’s unpack what this term actually means—and why it’s already changing the way we create. Natural language programming isn’t one thing — it splits into two complementary approaches. Depending on the goal, you might want to generate code from a plain-English prompt, or trigger actions directly without code ever surfacing. These two paths are reshaping what “programming” even means. Text-to-code Tools like Codex, Copilot, Cline, or Windsurf translate prose into Python, JavaScript, and more. Text-to-action Autonomous agents interpret i...
READ MORE

Internet Is Nasty? Really? How?

Méoc
Image
Many years ago I connected a server directly to the Internet and watched the incoming traffic. Within minutes, connection attempts targeted every open port on the machine. Most were automated bots scanning the entire address space, looking for weak spots to steal data or take control of the host. That was nasty — and it happened twenty years ago. Fast-forward to today. I wanted to show my kids that the Internet is more than YouTube and Netflix; it is also full of tireless robots poking anything left unprotected. Normally I would drop an idea like this because of the effort involved, but I also wanted to try a new “code vibing” tool called Windsurf. Code vibing? Think of it as ultra-fast, AI-assisted development: you describe what you want, and the tool either codes it for you or constantly boosts your productivity. Here’s a video showing how I changed a small part of the interface by simply… asking for it. Building a Live Attack Dashboard in 6 Hours with AI To create my app, I ...
READ MORE

Demystifying GPU Sizing for LLMs

Méoc
Image
  The Context Large Open Source (OSS) language models are springing up everywhere – Mistrals, Llama, surprise releases like Kimi K2 and, this month, OpenAI OSS models. Each of them promises open‑source autonomy and fine‑tuned intelligence, but there’s a very practical question lurking in the background: how much hardware does it take to run them? The answer isn’t just about having “some GPUs”; it hinges on understanding the interplay between model size, memory bandwidth and inference throughput. One of my hobby projects is a GPU infrastructure estimator that addresses exactly this question. Initially built as an unwieldy spreadsheet, it has evolved into a simple web tool designed to answer a surprisingly common question: “Can I run the latest open‑source models on‑prem for tens or hundreds of thousands of euros?” . What the tool does At its core, the estimator calculates three things: memory requirements , compute throughput , and latency . To make the experience approachable, it...
READ MORE

GenAI App Architecture Explained (Part 3: The Hardware stack)

Méoc
Image
 In  Part 1  and  Part 2  of this series, we explored the high-level components of a modern GenAI application, from the user interface to the RAG pipeline. Now, let's get down to the metal. What type of resources do we need to make it all run? What Resources Do We Need? Like any application, LLMs rely on fundamental hardware resources to function: compute, memory, storage, and network. And sometimes, they need a  lot  of them. Compute LLM architectures, particularly Transformers, are heavy consumers of compute and memory. This is inherent to their design, which must process massive amounts of calculations to predict subsequent tokens. Think about it: each new token generated is the result of a complex calculation. All tokens from the prompt are turned into mathematical representations (vectors) to understand the intricate relationships between them. Based on this analysis, the model performs billions of calculations to predict the most likely next toke...
READ MORE