(A) What is RAG (Retrieval-Augmented Generation)?

Retrieval-Augmented Generation (RAG) is a technique that allows an AI agent to retrieve relevant information from external data sources before generating a response. Instead of relying entirely on the information learned during model training or sending huge amounts of data directly into the context window, the system searches for relevant information and provides only the necessary context to the LLM.

Typically, documents or other data are split into smaller chunks, converted into numerical embeddings, and stored in a vector database such as PGVector. When a user asks a question, the query is converted into an embedding, and similarity search retrieves the most relevant chunks.

Where can we use RAG?

• Document Retrieval: Search through large PDFs, technical documentation, research papers, and uploaded files.

• Conversation Memory: Retrieve relevant information from previous conversations instead of loading the entire chat history into the context window.

• AI Coding Agents: Index large tool responses, source code files, and repository documentation to retrieve relevant functions, classes, or code snippets when needed.

• Customer Support: Retrieve answers from company FAQs, product documentation, and internal knowledge bases.

• Enterprise Knowledge Management: Search across internal documents, reports, policies, and organizational data.

• Large Tool Outputs: Store and index massive API responses, logs, or tool execution results, allowing an agent to retrieve only the relevant information instead of passing megabytes of data into its context window.

• Research Agents: Retrieve relevant information from multiple sources to support research, analysis, and report generation.

Core idea: RAG separates information storage from information retrieval. Rather than continuously feeding everything to an LLM, we store information externally and retrieve only what is relevant to the current task.

(B) What is PGVector?

PGVector is a PostgreSQL extension that enables a relational database to store and search vector embeddings. It allows us to combine traditional database capabilities with vector similarity search, making PostgreSQL useful for building RAG-based applications.

When we use PGVector with LangChain, it organizes the stored information into collections. Each collection contains multiple document chunks, their corresponding embeddings, and metadata. When a query arrives, PGVector compares its embedding against the stored vectors and retrieves the most relevant chunks.

What does PGVector store?

• Embeddings: Numerical vector representations of text generated by an embedding model. These vectors help identify semantically similar content.

• Documents: The actual text chunks extracted from uploaded documents, PDFs, or other data sources.

• Metadata: Additional information about each chunk, such as its source filename, page number, title, or other attributes.

• Collections: Logical groups used to organize related document embeddings.

Understanding LangChain PGVector Tables

When using LangChain's traditional PGVector implementation, two primary tables are involved:

1. langchain_pg_collection

This table maintains the collection information. Think of a collection as a logical container that groups related document chunks.

For example, suppose we create a collection named suraj-collection.

The table contains: name - (you can use your jwt token to pass a unique name like collection_1), and a uuid col which holds unique id to your collection name .

2. langchain_pg_embedding

This table stores the actual document chunks and their vector representations, and other things like Actual text content of the document chunkcmetadataAdditional information such as source, page number, and title are also stored .

For example, if we upload a PDF and split it into 100 chunks, the embedding table can contain 100 separate records, each containing its own document text, embedding, and metadata. All 100 records can reference the same collection UUID.

Understanding the Two Tables

Table 1 – langchain_pg_embedding

This displays the actual embedding records. Here multiple rows have the same collection_id (uuid):

c118868f-8be9-4e13-b10...

This means those document chunks belong to the same collection. Each row has its own ID, vector embedding, document content, and metadata.

Table 2 – langchain_pg_collection

This displays the collection record. like collection name i.e (user_jwt_token) and the uuid and this uuid points to the records of pg_embeddings .

How Can We Use a Collection?

Suppose we want to create a collection called suraj-collection.

Using LangChain, we can initialize a PGVector store with that collection name:

vector_store = PGVector(
    embeddings=embedding_model,
    collection_name="suraj-collection",
    connection=DATABASE_URL,
)

Now we can insert documents:

vector_store.add_documents(documents)

LangChain processes the documents, generates embeddings, and stores the corresponding chunks in the embedding table under the selected collection.

When we want to retrieve relevant information:

results = vector_store.similarity_search(
    "Explain the authentication architecture",
    k=3
)

The query is converted into an embedding, and PGVector searches for the three most semantically similar document chunks within the selected collection.

(C) Designing Persistent AI Agent Backend .

1. The Problem: Why Does an AI Agent Need Persistent Memory?

When we interact with an AI agent, it maintains a conversation consisting of user messages, assistant responses, and sometimes tool calls and their results.

However, an LLM has a limited context window. This means it can only process a certain amount of information at a time. As the conversation grows, we eventually reach a point where sending the entire conversation to the LLM becomes inefficient or impossible.

For example, imagine having a conversation with an AI coding agent for several hours. During that time:

• We ask it to build an authentication system.

• It creates multiple files.

• It reads existing source code.

• It executes commands and receives tool responses.

• We ask it to modify the database schema.

• We discuss several implementation decisions.

If we continuously send every previous message and every tool response to the LLM, the context window will eventually fill up.

One solution is context compaction, but simply summarizing the entire conversation is not enough. Important technical decisions, file paths, code snippets, and user preferences might disappear during summarization.

So we need an architecture that separates persistent storage from the LLM's active context.

2. First Principle: Store Every Conversation in the Database

Before implementing compaction or RAG, we need to establish one important rule:

Never treat the LLM's context window as our permanent storage.

Every conversation message should be saved in our backend database.

For example, suppose a user sends:

"Build an authentication system using JWT and PostgreSQL."

Our backend should first save this message in the database and then send it to the LLM.

When the LLM responds:

"I will create the authentication routes, middleware, and user model."

We save that response as well.

The same applies to subsequent messages, tool calls, and tool results.

We can maintain a conversation_messages table containing:

• Message ID

• Thread ID

• Role (user, assistant, tool)

• Message content

• Token count (optional)

• Created timestamp

• Additional metadata

Every message belongs to a particular conversation thread.

For example, if a user has three different conversations, each conversation has its own thread_id. This allows us to retrieve messages belonging to a particular thread without mixing different conversations.

The important thing is that even if our backend restarts, the original conversation remains available in PostgreSQL.

The database becomes our source of truth, not the context window.

3. Understanding Context Compaction

Now that we are saving every conversation, let's understand what happens when the context window starts filling up.

Suppose our agent has a context window of 100,000 tokens.

After several hours of interaction, the conversation history reaches approximately 85,000 tokens.

We cannot keep sending everything indefinitely. We need to reduce the amount of information supplied to the LLM.

This is where context compaction comes in.

Compaction is the process of reducing the active conversation context while preserving the important information required for future interactions.

However, we should not simply summarize everything and delete the original messages.

Instead, we can divide our conversation history into three parts:

A. Recent Messages

These are the latest messages that are still useful for understanding the current task.

We keep them directly inside the active context window.

For example:

• The user's latest request.

• The assistant's latest response.

• Recent tool calls and their results.

B. Rolling Summary

Older conversation messages can be compressed into a summary.

For example, imagine the user and agent discussed authentication for 50 messages.

Instead of sending all 50 messages again, we can maintain a summary such as:

"The user is building a FastAPI backend with JWT authentication and PostgreSQL. Authentication routes and middleware have been implemented. Passwords are hashed before storage. The user wants refresh-token support next."

This summary preserves the important progress and decisions without requiring the original conversation to be loaded into the context window.

We can store this summary in a separate conversation_summaries table.

It should also track which messages have already been summarized, so that we don't repeatedly summarize the same conversation history.

C. Original Conversation Archive

This is the most important part.

Even after summarization, we never delete the original conversation messages just because they are no longer needed in the active context.

They remain safely stored in conversation_messages.

If the user later asks:

"What was the exact authentication approach we discussed earlier?"

We can retrieve the original messages using database queries or RAG rather than relying entirely on the summary.

This gives us three layers:

• Recent messages for immediate context.

• Rolling summary for conversational continuity.

• Original messages for historical retrieval.

4. Why Do We Need RAG for Conversation History?

Now comes an interesting problem.

Suppose a user has been interacting with an agent for six months. The database contains thousands of messages across multiple conversations.

Should we retrieve all those messages whenever the user asks something?

Absolutely not.

We need a mechanism to find only the information relevant to the current question.

This is where RAG becomes useful.

Whenever we save conversation messages, we can selectively process meaningful messages and convert them into embeddings.

For example:

User message:
"I decided to use PostgreSQL instead of MongoDB because I need relational queries."

This message can be converted into an embedding and stored in PGVector along with its original text and metadata.

The metadata can contain:

• User ID

• Thread ID

• Message ID

• Role

• Source type

Later, the user asks:

"Which database did I decide to use?"

The system converts this question into an embedding and searches the relevant vector records.

It retrieves the original message and passes it to the LLM.

The LLM can then answer:

"You decided to use PostgreSQL because you needed relational queries."

Notice that we did not need to load the entire six-month conversation.

We retrieved only the relevant information.

An important distinction: not every message needs an embedding. Messages such as "Hi", "Okay", or "Thanks" usually provide little value for semantic retrieval. We can prioritize meaningful messages and use SQL for exact lookups.

5. Handling Massive Tool Responses Using RAG

Now let's consider another important problem, especially when building AI coding agents.

Suppose we ask an agent:

"Understand the authentication implementation in my repository."

The agent calls a tool that reads multiple files.

The tool returns:

• auth.py

• middleware.py

• routes.py

• models.py

• database.py

• config.py

Imagine the combined response is around 2–3 MB.

If we directly pass this entire response to the LLM, it consumes a huge amount of context tokens.

It also makes the agent slower and more expensive.

Instead, we can apply a similar architecture using artifact storage and RAG.

Step 1: Execute the Tool

The agent calls a tool to inspect the repository.

The tool returns the contents of multiple files.

At this point, instead of directly forwarding the entire response to the LLM, we persist the complete output in our backend.

Step 2: Store the Original Tool Response

We can create a table called tool_invocations to record information about each tool execution.

It can contain:

• Tool invocation ID

• Thread ID

• Tool name

• Tool arguments

• Execution status

• Created timestamp

Next, we can store the actual tool response in an artifacts table or external object storage such as AWS S3 or Azure Blob Storage.

The artifact record can contain:

• Artifact ID

• Tool invocation ID

• Thread ID

• Storage location

• File type

• File size

• Checksum

• Indexing status

Why do we need a separate artifact?

Because the complete tool response might be several megabytes. We don't necessarily want to place huge payloads inside our conversation messages or LangGraph state.

We can keep the original response safely stored and use its artifact ID whenever we need it again.

Step 3: Split the Tool Response into Smaller Chunks

Now that the original response is stored, we can process it for retrieval.

Suppose the tool returned six source files.

We should not treat the entire 2–3 MB response as one document.

Instead, we split it into meaningful chunks.

For normal documents, we can split text using a character or token-based text splitter.

However, when dealing with source code, we should use code-aware chunking.

For example:

• Split Python code by functions and classes.

• Split JavaScript or TypeScript code by functions, classes, and components.

• Preserve file paths and line numbers.

• Keep related code together whenever possible.

Suppose auth.py contains five functions.

We can create separate chunks for each function instead of treating the entire file as one large text block.

This makes retrieval much more useful.

Step 4: Generate Embeddings and Store Them in PGVector

After splitting the content, we generate embeddings for each meaningful chunk.

These embeddings and their original text are stored in our PGVector collection.

Each chunk should maintain useful metadata, such as:

• User ID

• Thread ID

• Project ID (if applicable)

• Artifact ID

• File path

• Function or class name

• Starting and ending line numbers

• Source type

For example, if a chunk belongs to auth.py, its metadata should identify that file.

This allows us to retrieve the exact source location later.

We should also track indexing status and handle embedding failures so that an artifact is not incorrectly considered ready before its chunks have been indexed.

Step 5: Return a Small Summary to the Agent

Now comes the interesting part.

Instead of sending the entire 2–3 MB tool response to the LLM, we send a small summary.

For example:

"The repository inspection is complete. Six files were processed and indexed. The authentication implementation appears to be distributed across auth.py, middleware.py, and routes.py. The complete tool response is stored as artifact A123 and can be searched when needed."

The agent now knows that the information exists and can retrieve it when necessary.

This significantly reduces the amount of information occupying the active context window.

Step 6: Retrieve Relevant Code When Needed

Suppose the user now asks:

"How does the authentication middleware validate the JWT?"

The agent does not need to read all six files again.

Instead:

1. The question is converted into an embedding.

2. The retrieval system searches the relevant PGVector collection.

3. It applies the appropriate user, thread, or project filters.

4. It retrieves the most relevant code chunks.

5. It fetches the original content associated with those chunks.

6. The relevant code is supplied to the LLM.

The agent can now explain the JWT validation logic without loading the entire repository inspection result.

One important detail: if we need an entire file or exact current source code, we should fetch that file directly from the original artifact or repository. RAG is primarily for finding relevant information, not reconstructing entire source files from partial chunks.

6. How Should We Organize the Database?

We can maintain a common PostgreSQL database with separate tables for different responsibilities.

1. users

Stores user account information.

2. chat_threads

Stores conversation thread information and associates each thread with its owner.

3. conversation_messages

Stores every user message, assistant response, and relevant tool message.

4. conversation_summaries

Stores compressed summaries of older conversation history, including the message boundary covered by each summary.

5. tool_invocations

Stores tool execution information, including tool name, arguments, status, and timestamps.

6. artifacts

Stores references to complete tool responses, uploaded documents, and other large outputs. The actual content can be stored in PostgreSQL or external object storage.

7. structured_memories

Stores important extracted facts in a structured format.

For example:

• User prefers PostgreSQL.

• Project uses FastAPI.

• Authentication is handled using JWT.

These facts can be retrieved using normal SQL queries rather than always relying on vector search.

8. rag_chunks

Stores the individual text chunks, their embeddings, and associated metadata.

This can act as a common retrieval layer for conversation history, uploaded documents, tool responses, and project files.

We can distinguish these sources using a source_type field.

For example:

• conversation

• tool_response

• uploaded_document

• code_file

• rolling_summary

This avoids building a completely separate RAG infrastructure for every type of data.

7. How Does Everything Work Together?

Let's put everything into one practical workflow.

Imagine a user starts a new conversation with an AI coding agent.

Initially:

The user sends a message. The backend saves it in conversation_messages and passes it to the agent.

The assistant responds. Its response is also saved.

When the agent calls a tool:

The tool execution is recorded in tool_invocations.

The complete tool output is stored as an artifact.

If the output is suitable for indexing, it is split into meaningful chunks, converted into embeddings, and stored in rag_chunks.

The agent receives a short summary and an artifact reference instead of the entire output.

When the conversation becomes large:

The compaction layer identifies older messages that should no longer remain in the active context.

It creates or updates the rolling summary.

The original messages remain in the database.

Recent messages and the latest summary are retained in the active context.

When the user asks about something discussed earlier:

The retrieval system searches the conversation archive and other relevant sources.

It retrieves the required information and supplies it to the agent.

The agent can continue the conversation without loading the entire historical context.

8. Important Design Principles

There are a few things worth remembering when designing this architecture.

• Never delete original conversation messages just because compaction has happened.

• Never depend on the LLM's context window as permanent storage.

• Store large tool responses as artifacts instead of keeping megabytes of data inside the active graph state.

• Use RAG to retrieve relevant information rather than loading every stored document.

• Use SQL for exact queries and structured facts; use vector search for semantic retrieval.

• Apply user, thread, and project filters to prevent unrelated information from being mixed.

• Preserve source references, file paths, and line numbers so that retrieved information can be traced back to its original source.

• Perform embedding generation asynchronously when appropriate, so storing a message or artifact does not have to wait for the entire indexing pipeline.

• Keep recent conversation context, rolling summaries, structured memories, and the original archive as separate responsibilities.

9. Final Understanding

The main idea behind this architecture is to separate what the AI agent needs right now from everything it has ever seen.

PostgreSQL acts as the persistent storage layer. PGVector provides semantic retrieval. The compaction layer manages the active context window. Structured memory preserves important facts, while artifact storage handles massive tool outputs.

Together, these components allow us to build AI agents that can maintain conversation history, retrieve information from previous interactions, work with large codebases and tool responses, and continue operating even after context-window limits or server restarts.

The context window is the agent's working memory. The database is its permanent record. RAG is how it finds what it needs. Compaction is how it keeps its working memory manageable.