LLM App Engineering

What is LLM Application Engineering ?

LLM Application Engineering is about taking a language model and turning it into a real, working system. This includes handling memory, injecting domain knowledge, integrating tools, and ensuring the model behaves reliably at scale.

Let’s explore the essential building blocks used in modern LLM-powered applications.

How LLMs remember ?

LLMs do not have any built-in memory. Every request send to an LLM model is independent and stateless. So, the model doesn’t remember previous conversations unless it’s send to them again along with the query in the same request.

To simulate memory, applications maintain a context window, which stores the conversation or important information and passes it along with each new query.

The context window is a fixed-size limit. If the conversation/message history grows too long, we must either remove older messages or summarize them.

This is how chatbot maintain “long-term conversation” without the model ever storing anything internally.

One simple approach is a Rolling Window, where only the recent messages are kept. While a more efficient approach is summarized memory, where older parts of the conversation are compressed into short summaries, to reduce token usage and keeps the conversation relevant.

A typical LLM request flow and memory management via context passing

• User Input goes to the Conversation Manager.
• If history exceeds the model’s context window, older messages are summarized/compressed.
• Summaries are stored and reused.
• The system builds a prompt containing system instructions, recent history (or summaries), retrieved documents (RAG), and the new user input.
• The LLM generates a response, which is appended back to recent history.
• Important facts can be persisted in a selective memory store (KV store or vector DB) for later retrieval.
• For RAG, the user query is embedded and used to retrieve relevant chunks which are injected into the prompt before the model call.

flowchart TD
  A[User Input] --> B[Conversation Manager]
  B --> C{Is history > context window?}
  C -- No --> D[Build Prompt\n System + Recent History + User Input]
  C -- Yes --> E[Compress / Summarize Old Messages]
  E --> F[Summaries Store]
  F --> D
  D --> G[LLM Model ➜ Generate Response]
  G --> H[Assistant Response]
  H --> I[Append to Recent History]
  I --> J[Trim / Maintain Rolling Window]
  J --> K[Persist Important Facts\n Selective Memory Store]
  K --> L[Vector DB / Key-Value Store]
  L --> B

  subgraph Notes [ ]
    direction LR
    N1[System messages and retrieved docs are \nincluded in the prompt when needed]
    N2[Optional RAG: \nembed user query → search Vector DB \n→ inject top-k chunks into prompt]
  end

  %% Optional RAG flow
  A --> M[Query Encoder → Embedding]
  M --> L
  L --> O[Retrieve Top-k Chunks]
  O --> D

Customization Strategies

To make an LLM behave like an expert, it needs customization. Following are two popular methods to customize it:

• RAG (Retrieval-Augmented Generation): Add external knowledge at query time.
• Fine-Tuning: Train the model on your domain data so the knowledge becomes internal.

RAG

RAG (Retrieval-Augmented Generation) is often the easiest approach because it doesn’t modify the model. Instead, we store documents in a vector database and fetch the most relevant chunks whenever the model needs them.

This keeps the system fresh and if the documents change, the model instantly uses the updated information.

RAG is the better starting point for begineers because it avoids model training and still provides excellent accuracy for document-based queries.

Fine-Tuning

Fine-tuning on the other hand, helps when we need the model to speak in a particular style or follow specific behaviors consistently.

It also reduces prompt size because the model already knows your domain after training.

Insight:
→ Google’s Notebook LLM is a classic example of RAG.

Ecosystem Tools

Developers rarely build everything from scratch.

Tools like LangChain and LlamaIndex help structure prompts, manage retrieval, connect models, and integrate tools easily. They provide building blocks such as chains, agents, histories, and retrieval modules that simplify development.

Vector databases such as Pinecone, Weaviate, Milvus or even simple local embeddings work alongside these frameworks. They store text as numerical vectors so the system can “search by meaning,” not just keywords.

These tools are not required for small projects, but they become essential once you want a scalable or production-grade system.

Agentic AI

Agentic AI - when the LLMs start taking actions. Your application plays the role of an orchestrator, ensuring safety and controlling which tools the model can access and the agent decides which tool to use and when to use it.

An LLM agent is a system where the model is allowed to use tools, not just generate text but to:

• search the web
• run code
• query a database
• call APIs
• perform multi-step plans

This approach is powerful but requires strict guardrails, because you don’t want the model performing unsafe actions or making uncontrolled decisions.

A simple agent workflow looks like this:

1. User gives a complex task.
2. LLM thinks about what needs to be done.
3. It selects one or more tools.
4. Your app executes those tools.
5. The LLM receives the tool results and produces a final response.

Common System Patterns

Most LLM apps fall into a few common patterns. These patterns are the foundation of nearly all modern LLM-powered applications.

Monitoring and Reliability Basics

Reliable AI systems require continuous monitoring - without it, quality may degrade unnoticed.

To keep the system healthy, we must monitor:

• latency
• token usage
• error rates
• hallucination risks
• costs

Most platforms provide dashboards for these metrics. As your system grows, you may add evaluation prompts, A/B testing, and safety filters.