Manshu RAG Knowledge Base and Chat Backend#

#RAG; #FastAPI

Repository: Manshu RAG Knowledge Base and Chat Backend

The flowchart below summarizes the two main paths: “document upload and parsing” and “user question and streaming answer”.

graph TD
    A[User uploads document] --> B[Chunked upload to MinIO]
    B --> C[Track chunk progress in Redis]
    C --> D{All chunks done?}
    D -->|No| B
    D -->|Yes| E[Merge file]
    E --> F[Write metadata to MySQL]
    F --> G[Publish Kafka message]
    G --> H[Kafka consumer]
    H --> I[Parse document with Tika]
    I --> J[Split text into chunks]
    J --> K[Call OpenAI Embedding]
    K --> L[Write to Elasticsearch]
    L --> M[Write to DocumentVector table]
    
    N[User asks a question] --> O[Embed the question]
    O --> P[Elasticsearch hybrid search]
    P --> Q[Get Top-K results]
    Q --> R[Assemble prompt]
    R --> S[OpenAI streaming generation]
    S --> T[Push via WebSocket]

Project Overview#

The Manshu backend covers the full pipeline of “upload documents → parse and index → semantic/keyword retrieval → chat with an LLM grounded in the retrieved results”. I was responsible for the AI and backend development on this project.

Tech stack: Python + FastAPI as the main framework, MySQL for business data and metadata, Redis for sessions, rate limiting and CAPTCHAs, MinIO for files and raw documents, Elasticsearch for hybrid vector and full-text search, and Kafka to split parsing and embedding into async jobs. Both chat and embeddings use OpenAI (embedding defaults to text-embedding-3-small, chat defaults to gpt-3.5-turbo; both are configurable). Deployment uses Docker + Uvicorn. All APIs share the /api/v1 prefix; chat runs over WebSocket at /api/v1/chat, with the JWT passed via the token query parameter.

CategoryTechnologyPurpose
Backend frameworkPython + FastAPIREST API and WebSocket service
DatabaseMySQLBusiness data and metadata storage
CacheRedisSession management, rate limiting, CAPTCHAs
Object storageMinIOFile and raw document storage
Search engineElasticsearchHybrid vector and full-text search
Message queueKafkaAsync document parsing and embedding jobs
LLMOpenAIEmbedding (text-embedding-3-small) and Chat (gpt-3.5-turbo)
Document parsingApache TikaMulti-format document parsing
DeploymentDocker + UvicornContainerized deployment and ASGI server

Background#

As the volume of internal documents grew, plain keyword search could no longer meet the need for “semantic retrieval + LLM-generated answers grounded in retrieved results”. Documents also come in many formats (PDF, Word, Excel, PPT, HTML, Markdown, etc.), requiring a unified parsing and ingestion flow. This project chains upload, parsing, retrieval and chat into a configurable, observable pipeline, with multi-tenant isolation based on permissions and organization tags. Health checks and structured logging make operations and troubleshooting easier.

Health checks expose /health (fast liveness probe) and /health/detailed (deep probe); the latter verifies connectivity and connection pool status for MySQL, Redis, Elasticsearch, MinIO and Kafka. On startup the app connects to these components in dependency order. If the Kafka consumer fails to start, only a warning is logged and the app keeps serving requests; only document parsing and embedding are temporarily unavailable.


Architecture and Technology Choices#

The system uses a decoupled frontend/backend architecture: the frontend talks to the backend over HTTP and WebSocket, and the backend provides REST APIs and a WebSocket chat endpoint. Requests first pass JWT verification and Redis rate limiting before reaching business logic. Uploaded files are stored in MinIO, metadata is written to MySQL, and a parsing job is published to the Kafka document_parse topic. On the consumer side, Apache Tika parses the document, the text is split into chunks (chunk size 500, overlap 50), OpenAI Embedding is called, and vectors and metadata are written to Elasticsearch and MySQL respectively. When a user asks a question, the question is embedded first, then Elasticsearch runs vector search (dense_vector, 1536 dimensions, cosine similarity) and IK full-text search (ik_max_word for indexing, ik_smart for querying) in parallel. The two result sets are score-normalized and fused with configurable weights (default: vector 0.7, full-text 0.3). The Top-K chunks are inserted into the prompt, OpenAI generates a streaming response, and it is pushed to the frontend over the same WebSocket.

Rationale: Elasticsearch supports both vector and full-text search in the same index, so dense_vector plus IK tokenization is enough to implement hybrid search with tunable weights. Parsing and embedding are slow, so processing them asynchronously through Kafka avoids blocking the upload API and makes it easy to add more consumers and retry policies later. Tika covers common formats such as PDF, Office, HTML and Markdown, and falls back to UTF-8 decoding for plain text when parsing fails. FastAPI has solid async and WebSocket support and auto-generates OpenAPI docs, which simplifies frontend integration and API contract management.

Key Technology Rationale#

  • Elasticsearch

    Supports both semantic (vector) and keyword (full-text) retrieval in the same index. dense_vector combined with IK tokenization enables hybrid search with adjustable weights, making it easy to balance “semantic similarity” against “keyword matches”.

  • Kafka

    Parsing and embedding take a long time; running them synchronously would easily cause upload timeouts. Async processing through Kafka lets uploads respond quickly and makes it easy to add more consumers, retries and dead-letter handling later.

  • Apache Tika

    Covers common formats such as PDF, Office, HTML and Markdown. When parsing fails, it falls back to UTF-8 decoding for plain text, reducing the impact of a single failed file on the overall pipeline.


Features#

Health Checks#

The system provides two levels of health checks for load balancer integration and monitoring/alerting:

  • Basic check: /health, a fast liveness probe used by load balancer health checks
  • Detailed check: /health/detailed, a deep probe that verifies connectivity and connection pool status for MySQL, Redis, Elasticsearch, MinIO and Kafka

On startup the app connects to these components in dependency order. If the Kafka consumer fails to start, only a warning is logged and the app continues to serve everything except document parsing.

Authentication and Authorization#

Provides registration, login, image CAPTCHAs and email verification codes, plus JWT issuance and verification. Expiration and length of CAPTCHAs and email codes are configurable. Redis-based per-endpoint rate limiting gives image CAPTCHA, email code and registration endpoints their own windows and limits (e.g. CAPTCHA requests per IP per minute) to prevent brute-force attempts and abuse. A JWT is returned on successful login and sent on subsequent requests via the Authorization header or the token query parameter.

Core features:

  • Image CAPTCHA generation and verification (stored in Redis, configurable expiration)
  • Email verification code delivery (SMTP config, configurable length and expiration)
  • JWT issuance and verification (via Authorization header or query parameter)
  • Redis-based per-endpoint rate limiting (separate windows for image CAPTCHA, email code and registration endpoints)

Rate limiting policy:

  • Limit on CAPTCHA requests per IP per minute
  • Limit on email code requests per IP per minute
  • Limit on registration requests per IP per minute

File and Document Management#

Uploads support chunking: each chunk is first uploaded to a temporary MinIO directory, with chunk progress tracked in Redis. Once all chunks are done, the merge API produces the final file and writes it to MySQL (file metadata, user, organization tag, public flag), then publishes a parsing job to the Kafka topic document_parse. The consumer fetches the file from MinIO, parses it with Tika, chunks it, calls Embedding per chunk, and writes vectors to Elasticsearch and metadata to the DocumentVector table. Supported formats depend on Tika and include PDF, Word, Excel, PPT, HTML, Markdown, TXT, RTF and more. On parse failure the error is logged; if needed, only metadata is persisted and the same message is not consumed again, so a single failure cannot block the queue. Document listing, deletion and related APIs are tied to permissions and organization tags, so users can only operate on data they are authorized to access.

  1. Chunked upload: each chunk is first uploaded to a temporary MinIO directory, with progress tracked in Redis
  2. File merge: once all chunks are done, the merge API produces the final file and writes it to MySQL
  3. Job dispatch: a parsing job is published to the Kafka topic document_parse
  4. Async processing: the consumer fetches the file from MinIO, parses it with Tika, chunks it, calls Embedding per chunk and writes to Elasticsearch

Supported formats: PDF, Word, Excel, PPT, HTML, Markdown, TXT, RTF and more (via Tika)

Fault tolerance: On parse failure the error is logged; if needed, only metadata is persisted and the same message is not consumed again, avoiding queue blockage.

The query text is embedded with the same Embedding model, then Elasticsearch runs vector search and IK full-text search in parallel. The two scores are normalized and merged with configurable weights, and results are isolated by user_id, org_tag and is_public so only chunks the current user may access are returned. Index fields include file_md5, chunk_id, text_content, vector, user_id, org_tag, is_public, file_name and model_version, supporting access control and result traceability.

Retrieval flow:

  1. Embed the question (text-embedding-3-small, 1536 dimensions)
  2. Run two queries in parallel:
    • Vector search: dense_vector field, cosine similarity
    • Full-text search: IK tokenization (ik_max_word for indexing, ik_smart for querying)
  3. Normalize scores and merge with configurable weights (default: vector 0.7, full-text 0.3)
  4. Isolate data by user_id, org_tag and is_public

Index fields: file_md5, chunk_id, text_content, vector, user_id, org_tag, is_public, file_name, model_version

Streaming Chat#

The WebSocket path is /api/v1/chat. On connect, the client passes token (JWT) in the query string and optionally conversation_id (a new conversation is created if omitted). The flow is: receive user message → hybrid search for context → assemble prompt → call OpenAI streaming generation → push the response over the same WebSocket. Conversation history is persisted to MySQL; each conversation keeps at most the latest N messages (CONVERSATION_MAX_MESSAGES, default 20) and has an expiration (CONVERSATION_TTL_DAYS, default 7 days). Generation can be stopped: a stop token with a TTL is stored in Redis, and the frontend includes it when sending a stop command. On the WebSocket side, connections are capped per user (default 10) and per instance (default 1000), with an idle timeout (default 3600 seconds) and periodic heartbeat cleanup (inactive connections cleared every 300 seconds by default) so connections are not held indefinitely.

Chat flow:

Receive user message → Hybrid search for context → Assemble prompt → 
Call OpenAI streaming generation → Push response over the same WebSocket

Conversation management:

  • Conversation history persisted to MySQL
  • Each conversation keeps at most the latest N messages (CONVERSATION_MAX_MESSAGES, default 20)
  • Conversations expire (CONVERSATION_TTL_DAYS, default 7 days)
  • Stop generation supported via a Redis-stored stop token with TTL

Connection management:

  • Max connections per user: 10
  • Max connections per instance: 1000
  • Idle timeout: 3600 seconds
  • Periodic heartbeat cleanup: inactive connections cleared every 300 seconds

Administration#

A dedicated admin router (admin_router), separated from regular user permissions, handles user management, system configuration and more, and can be extended with auditing, operation logs and other features as needed.


Technical Challenges and Solutions#

Asynchronous Document Processing#

Challenge: Running parsing and embedding synchronously inside the upload API makes large files prone to timeouts and blocks the endpoint, affecting other requests.

Solution: The upload API only writes to MinIO, persists metadata and publishes a Kafka message. The consumer parses with Tika, chunks the text (chunk size 500, overlap 50), calls Embedding and writes to Elasticsearch. Parse failures are logged and degraded gracefully (if Tika fails, UTF-8 decoding is attempted; if that also fails, only metadata is stored and the message is not consumed again), preventing endless retries of the same message from blocking the queue. The upload API returns quickly while processing completes in the background.

Hybrid Search#

Challenge: Vector search alone is insensitive to exact matches such as proper nouns and IDs; full-text search alone lacks semantic generalization. Neither balances “semantic similarity” and “keyword matches” on its own.

Solution: For the same set of document chunks, Elasticsearch stores both a vector field (1536 dimensions, cosine similarity) and an IK-tokenized text field. At query time both searches run in parallel, scores are normalized and fused with configurable weights (SEARCH_VECTOR_WEIGHT / SEARCH_TEXT_WEIGHT), which makes later tuning easy. In practice, both proper-noun and semantic questions get more relevant results, and RAG answers are more consistent with the knowledge base.

For the same set of document chunks, Elasticsearch stores both:

  • Vector field: 1536 dimensions, cosine similarity
  • Text field: IK tokenization (ik_max_word / ik_smart)

At query time both searches run in parallel, and scores are normalized and fused with configurable weights (SEARCH_VECTOR_WEIGHT / SEARCH_TEXT_WEIGHT).


Summary#

The full RAG pipeline is in place, from document ingestion, parsing and embedding to hybrid search and streaming chat. Upload and parsing are decoupled asynchronously, and multi-tenant isolation via permissions and organization tags is implemented as required. Future directions include multi-model support (other Embedding and Chat models), finer-grained permissions and auditing, and iterative improvements to retrieval ranking and re-ranking.