← Back to projects

AI / LLM FeaturedCommercial (anonymized)

Local Chat + RAG platform

A web chat with a local LLM, a vector database and a dedicated RAG service — user accounts, document indexing, retrieval.

Problem

A team needed to chat with their own documents using an LLM, but without sending content to a public cloud. A plain chat without retrieval “hallucinated”, and data sensitivity ruled out off-the-shelf SaaS.

Context

The solution was designed for local / self-hosted deployment, separating the application layer from the compute layer (model, embeddings). Data control and the ability to run on a local model were the priorities.

Requirements

  • Chat with conversation history and user roles.
  • Uploading documents and building a RAG index.
  • Retrieval from a vector database + answers from a local model.
  • The ability to split deployment (application vs. compute).

Role

I designed the architecture and implemented everything: the frontend and API (Next.js), the auth and persistence layer, a dedicated RAG service (FastAPI), and the integration with the vector database and the local model.

Architecture

Technologies

Next.js 14 + React 18 (UI and API), NextAuth + Prisma + PostgreSQL (auth, conversations, document metadata), Qdrant (vector database), FastAPI (RAG service), Ollama (local model), and an optional embedding/rerank service.

Design decisions

  • A separate RAG service instead of retrieval logic in the web app — a clean separation of concerns and independent scaling.
  • Relational + vector databases side by side: PostgreSQL for metadata and conversations, Qdrant for retrieval.
  • A local model (Ollama) as the default, with the option to swap in a remote one.

Trade-offs

  • A local model means full data control, at the cost of performance and higher hardware requirements compared to the cloud.
  • More services (Next.js + FastAPI + Qdrant + Ollama) means deployment flexibility at the cost of operational complexity.

Technical challenges

  • Answer quality depends on chunking and retrieval quality.
  • Splitting the application and compute layers required clear contracts between services.
  • Managing conversation state and its links to documents/index.

Solutions

  • A dedicated RAG service with a clear API (indexing, queries) and optional rerank.
  • Metadata and history in PostgreSQL, vectors in Qdrant — each tool for its job.
  • Deployment modes separating the app from compute (option for separate hosts).

Result

A full chat + RAG platform running locally: auth, UI, document indexing and retrieval, all while keeping data under control. The architecture lets you scale or swap the compute layer independently of the application.

Lessons learned

  • The retrieval pipeline (chunking, embeddings, rerank) has the biggest impact on quality — not the model itself.
  • Separating services from the start makes later scaling and debugging easier.

What I’d do differently today

  • Add answer-quality evaluation (test sets, metrics) and retrieval observability.
  • Consider a cache layer for the most frequent queries and embeddings.

Interview talking points

  • Why PostgreSQL and Qdrant side by side rather than a single database.
  • How the app / RAG-service split affects scaling and data security.
  • The local model vs. remote service trade-off in the context of privacy.