GenAI App Architecture Explained (Part 4: LLASTAKS — A Full LLM App Playground on Kubernetes)

Méoc

A practical guide to spin up a complete, test-friendly GenAI stack (LLM + RAG + app + observability) on AWS EKS.

TL;DR

LLASTAKS (LLM App STAck on Kubernetes as a Service) is a reproducible, low-touch blueprint to deploy an end-to-end GenAI playground—not just a model endpoint. In ~one command you provision AWS EKS with a GPU node, run vLLM behind an OpenAI-compatible API, add a chatbot frontend, a RAG pipeline (ingestion → FAISS search → RAG chatbot), and wire observability (metrics/logs/traces). Use it to prototype features (e.g., function calling, LoRA fine-tuning) and to learn real-world constraints before building production systems.

Overview of LLASTAKS

Why this project?

Most “playgrounds” stop at the LLM API. Real applications need more: storage for weights, a serving layer, an app surface, retrieval, orchestration, and instrumentation. LLASTAKS gives you the whole app loop so you can:

  • Experiment quickly with local + cluster deployments
  • Validate architectural choices (EKS + GPU, EBS, internal DNS, service wiring)
  • Benchmark UX & quality with RAG
  • Measure reality with Prometheus/Grafana (and optional Loki/Tempo)
  • Iterate safely (IaC + scripts) and clean up when you’re done
  • Work safely: you connect to the app through port forward, no direct public Internet access.

I used a Qwen3:8B (quantized on 4bit) which is intelligent enough for various scenarios and, at the same time, lite enough to reduce the GPU need and the price. Feel free to use your own model and change the EC2 size to modify the price. With Qwen3-8b-INT4, the price is roughly 1€/H for the whole platform.

AWS provide new accounts with 100$ credits so it gives you around 80H of playground for free per new accounts.

What you get (capabilities)

  • EKS cluster with a dedicated GPU node for inference (e.g., g5.xlarge) and a CPU node for app/RAG jobs
  • vLLM serving an OpenAI-compatible API at http://vllm.llasta.cluster.local:8000
  • Chatbot (FastAPI backend + minimal HTML/JS) for quick, realistic testing
  • RAG pipeline: data ingestion → FAISS wrapper service → RAG chatbot with tunables (top-k, context size)
  • Observability: Prometheus metrics out of the box; optional Grafana Cloud dashboards; tracing with OpenTelemetry (Tempo)
  • Automation: end-to-end Deploy.sh (Terraform + K8s manifests) and clean structure per component

The repository folders provide a step by step structure: 000-K8 deployment/, 001-Copy weights to ebs/, 002-vLLM deployment/, 003-chatbot/, 004-RAG/, 005-observability/. Each stage contain a Stage_readme.md.

Prerequisites

  • AWS account with GPU quota for G-instances
  • Local tools: awscli, kubectl, terraform, docker, python3
  • Basic familiarity with Kubernetes contexts and port-forwarding

Cost note: GPU instances are not free; spin up only when needed and destroy when done.

Quickstart — from zero to working LLM



  1. Clone the repository, follow the Readme.md instruction. 

This will:

  • Provision EKS with the right node groups (GPU/CPU)
  • Prepare EBS + PVCs for model weights and FAISS index
  • Deploy vLLM and expose vllm.llasta.cluster.local:8000
  1. Verify the cluster and service
kubectl get nodes
kubectl -n llasta get pods,svc
  1. Test vLLM
kubectl -n llasta port-forward svc/vllm 8000:8000
curl -s http://localhost:8000/v1/models | jq
curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3-8b-…",
    "messages": [{"role":"user","content":"Give me one design goal of LLASTAKS."}]
  }' | jq
  1. Run the Chatbot
  • In-cluster: kubectl apply -f 003-chatbot/chatbot.yaml then port-forward svc/chatbot to http://localhost:8080/
  • Local dev: install deps from 003-chatbot/backend/requirements.txt and run FastAPI

You'll find all the information the Stage_Readme.md of the different parts 000-K8 deployment/, 001-Copy weights to ebs/, 002-vLLM deployment/, 003-chatbot/

Stage RAG — Objective & Scope



Build a simple, reliable, low-cost RAG V1 on EKS with vLLM (Qwen3-8B INT4) for generation, FAISS for retrieval, and a reranker for quality. The goal is to understand end-to-end behavior—from ingest to retrieval—its limitations, and how the LLM’s reasoning complements retrieval.

What to do

  • Deploy faiss-wrap + chatbot-RAG.
  • Run ingest.py twice on the sample sets (Hard-to-read vs Easy-to-read) and reflect on outcomes.
  • Ingest Easy-to-read into FAISS and test the RAG; compare with a large hosted LLM (e.g., GPT-5) to understand differences.
  • Reset FAISS and ingest Build your own electric car; reflect again.

Follow 0004-RAG/Stage_Readme.md to set up this part.

005-Observability — simplified preview




Observability is a core pillar of LLASTAKS, but we’ll dive deep into it in an upcoming article. This section just gives a quick preview:

  • Metrics: Prometheus collects system and app metrics (requests, latency, FAISS search time, GPU usage). You can plug it into Grafana Cloud later.
  • Logs: Loki can centralize logs from all pods for debugging.
  • Traces: Tempo and OpenTelemetry trace requests from chatbot → FAISS → vLLM.

The full setup, dashboards, and alerting best practices will be detailed in Part 5 — Observability and Debugging GenAI Apps.

Lessons learned

  1. LLM are hungry GPU beast and Quantization matters: Using an INT4-quantized Qwen3-8B made the model fit GPU memory and start reliably on smaller accelerators—while keeping reasonable quality. KV cache can still dominate; adjust --max-model-len and utilization.
  2. RAG ≠ math: Vector search over PDFs with many numbers underperforms for analytics questions. For structured data (transactions), prefer tool-use / function calling to a SQL/Parquet store (PAL-style workflows) and let the LLM plan/format queries.
  3. Chunking quality drives retrieval: Page-sized chunks are too coarse. Favor semantic chunking (by section/paragraph), enrich with metadata (chapter/section/type), and track sources precisely.
  4. IaC keeps you sane: Terraform is a great tool to quickly spin up and clean up this platform.

The current stack

  • Model server: vLLM (OpenAI-compatible API)
  • Base model: Qwen3-8B (INT4 variant for memory efficiency)
  • Embedding: BGE-M3 (CPU OK)
  • Vector DB: FAISS (local index on PVC/EBS via wrapper service)
  • Apps: Chatbot : act as the orchestrator
  • Observability: Prometheus, Grafana Cloud, Loki, Tempo

Conclusion & Next Steps

LLASTAKS isn’t meant to be a polished production system—it’s a living playground. Its purpose is to let you experience how a real GenAI app behaves: from the model serving layer to retrieval and orchestration, all the way to the metrics that reveal where the bottlenecks lie. By deploying it yourself, you start to understand how LLMs, retrieval systems, and observability tools fit together in a practical, reproducible stack.

Running it on EKS gives you full control: every network rule, storage choice, and node behavior is visible and adjustable. You can swap any component—vLLM for another model server, FAISS for another vector DB, or Prometheus for another observability tool—and see instantly how your stack evolves. And if you don’t have GPUs handy, smaller or quantized models can make experimentation affordable, while local tools like Ollama keep things simple.

As you move forward, remember that LLASTAKS is just the foundation. Part 5 of this series will dive deeper into observability—showing how to trace requests, analyze performance, and optimize the LLM–RAG loop in real time. Until then, treat this as your sandbox: break it, fix it, and learn from it.

When you’re done exploring, destroy your EKS cluster to avoid unnecessary GPU costs, and keep snapshots only if you plan to reuse weights. And finally, if this project sparks ideas or helps you in your journey, ⭐ star the repository and share your feedback—every insight helps improve the stack for the community.

most viewed articles

From Chat to Action: The New Gen AI Revolution

Méoc
Image
Artificial intelligence has already impressed us with its ability to generate text, answer questions, translate, code, and summarize. But all of that… remained passive. Today, AI is taking a new leap. It moves into action. It no longer just talks to you. It can do: publish an article, send an email, schedule a meeting, even run an entire blog. And that changes everything. Real-world examples of AI in action When I talk about generative AI that acts, I am not talking about just producing text or images. I'm talking about executing concrete, autonomous, targeted tasks. Here are a few examples from the field: 1. Fully automated blogging (personal case) An AI agent can: Generate a blog post from an idea. Publish it on a platform like Blogger. Create a cover image or visual summary. Share it automatically on social media. Update a tracking sheet or project management board. This isn’t a dream. I already have it running, and I’ll share the how-t...
READ MORE

GenAI App Architecture Explained (Part 1: The Big Picture)

Méoc
Image
 The first time you interact with an app like ChatGPT, all you see is the chat box. It feels magical — you type, it answers. But for anyone in Ops or engineering, one question immediately comes up:  what’s really happening behind the scenes? At the heart of every GenAI app lies an LLM (Large Language Model), but the LLM alone is just one piece of the puzzle. Around it sits an entire stack: orchestration layers, data pipelines, embeddings, vector databases, plugins, caches, observability tools, and guardrails. Together, they transform a “highly clever text generator” into a production-grade GenAI application. In this post, we’ll start with the big picture: the main components and flows that make an app like ChatGPT work. In the next parts, we’ll drill down into each layer (RAG, Ops monitoring, validation, etc.), highlighting both how they work and what usually breaks in production. Picture of what it feels like the first time we interact with a LLM: Which remind me of this famo...
READ MORE

How to Pick the Best LLM for Programming Cost vs Capability in 2025

Méoc
Image
In 2025, dozens of language models can now write, debug, or refactor code. But which ones are actually good — and which give the best value for money? Let's try to make a quick recap for your future programming session. Disclaimer: this is a moving market and each day can see tremendous changes so this is a picture in time and a methodology example for your own review  🧱 How We Calculate Token Costs The 80 / 20 blended formula Most IDE copilots send a large prompt (project context, previous messages, code files) and receive a short answer (patch, explanation). Field logs from Cline, GitHub Copilot, Cursor and Sourcegraph Cody show that roughly 80 % of tokens are input and 20 % are output . To give a single price you can reason about, we compute: Blended Cost ($/M) = 0.80 × Price_in  +  0.20 × Price_out Example : Claude 4 Opus costs $15 /M input and $75 /M output → 0.80×15 + 0.20×...
READ MORE

GenAI App Architecture Explained (Part 2: Completing the Big Picture)

Méoc
Image
 In the first article , we outlined the high-level architecture of a modern GenAI application: the orchestrator, the embeddings model, the vector database, and the external actions or tools. Today, we complete that picture by adding the supporting components that make the system resilient, observable, and reliable . The often-overlooked building blocks In the diagram above, these gray blocks are the hidden backbone of any production-grade GenAI app. Let's focus on the 3 remaining ones. LLM Cache This is where previously generated model responses are stored so they can be reused later. The goal: reduce latency and avoid unnecessary calls to expensive models. Typical tools: Redis, SQLite, GPTCache. Caching is crucial for both cost control and responsiveness . When a user asks the same or a very similar question, the system can serve a cached response instead of re-querying the model. But caching in the LLM context is not as straightforward as caching a static API response. Queries ...
READ MORE

Can Natural Language Really Replace Code? The Revolution Is Already Underway

Méoc
Image
Natural language processing  has been floating around the tech world for yearsbut what about Natural Language programming?  What about programming whole application directly in Natural language? The tools are here, the agents are live, and people who’ve never written a line of code are already building automations, querying data, and triggering real-world actions … just by describing what they want. Before we get ahead of ourselves, let’s unpack what this term actually means—and why it’s already changing the way we create. Natural language programming isn’t one thing — it splits into two complementary approaches. Depending on the goal, you might want to generate code from a plain-English prompt, or trigger actions directly without code ever surfacing. These two paths are reshaping what “programming” even means. Text-to-code Tools like Codex, Copilot, Cline, or Windsurf translate prose into Python, JavaScript, and more. Text-to-action Autonomous agents interpret i...
READ MORE

Internet Is Nasty? Really? How?

Méoc
Image
Many years ago I connected a server directly to the Internet and watched the incoming traffic. Within minutes, connection attempts targeted every open port on the machine. Most were automated bots scanning the entire address space, looking for weak spots to steal data or take control of the host. That was nasty — and it happened twenty years ago. Fast-forward to today. I wanted to show my kids that the Internet is more than YouTube and Netflix; it is also full of tireless robots poking anything left unprotected. Normally I would drop an idea like this because of the effort involved, but I also wanted to try a new “code vibing” tool called Windsurf. Code vibing? Think of it as ultra-fast, AI-assisted development: you describe what you want, and the tool either codes it for you or constantly boosts your productivity. Here’s a video showing how I changed a small part of the interface by simply… asking for it. Building a Live Attack Dashboard in 6 Hours with AI To create my app, I ...
READ MORE

Demystifying GPU Sizing for LLMs

Méoc
Image
  The Context Large Open Source (OSS) language models are springing up everywhere – Mistrals, Llama, surprise releases like Kimi K2 and, this month, OpenAI OSS models. Each of them promises open‑source autonomy and fine‑tuned intelligence, but there’s a very practical question lurking in the background: how much hardware does it take to run them? The answer isn’t just about having “some GPUs”; it hinges on understanding the interplay between model size, memory bandwidth and inference throughput. One of my hobby projects is a GPU infrastructure estimator that addresses exactly this question. Initially built as an unwieldy spreadsheet, it has evolved into a simple web tool designed to answer a surprisingly common question: “Can I run the latest open‑source models on‑prem for tens or hundreds of thousands of euros?” . What the tool does At its core, the estimator calculates three things: memory requirements , compute throughput , and latency . To make the experience approachable, it...
READ MORE

GenAI App Architecture Explained (Part 3: The Hardware stack)

Méoc
Image
 In  Part 1  and  Part 2  of this series, we explored the high-level components of a modern GenAI application, from the user interface to the RAG pipeline. Now, let's get down to the metal. What type of resources do we need to make it all run? What Resources Do We Need? Like any application, LLMs rely on fundamental hardware resources to function: compute, memory, storage, and network. And sometimes, they need a  lot  of them. Compute LLM architectures, particularly Transformers, are heavy consumers of compute and memory. This is inherent to their design, which must process massive amounts of calculations to predict subsequent tokens. Think about it: each new token generated is the result of a complex calculation. All tokens from the prompt are turned into mathematical representations (vectors) to understand the intricate relationships between them. Based on this analysis, the model performs billions of calculations to predict the most likely next toke...
READ MORE