Solution

Approach

After analyzing the knowledge management process, it was determined that critical documentation resides in Confluence and Jira. A RAG-based LLM application should be developed to:

• Ingest documentation from Confluence and Jira and store it for RAG lookups.
• Utilize a locally hosted LLM to answer employee queries with context from the RAG system.
• Periodically poll Confluence and Jira to keep the RAG vector store up to date.
• Host both the embedding model for RAG and the LLM locally to ensure data security.

Key Features

The proposed application includes:

• Locally hosted models to ensure sensitive data remains within controlled boundaries.
• LLM-powered assistant to reduce time spent searching for information.
• RAG-based responses to provide accurate, contextually relevant answers based on the latest documentation without hallucinations.
• Periodic polling to keep the knowledge base current with documentation updates.
• Exportable query logs in CSV format for auditing and tracking usage.

Architecture

Implementation

Project Overview

• Project Name: Knowledge Base RAG Assistant
• Duration: 12 weeks
• Team Composition: 3 developers, 1 QA engineer, 1 project manager
• Technologies Used:
  • Python
  • Ollama
  • Redis
  • ChromaDB
  • Open WebUI
  • Postgres

Phase 1: Discovery

The product owner collaborated with stakeholders to identify key pain points. A market study evaluated off-the-shelf tools or plugins for knowledge management. After initial findings, the team opted for a custom RAG-based LLM application. A feasibility study was conducted to:

• Identify open-source embedding models.
• Evaluate chunking strategies and their trade-offs.
• Select suitable LLM models.
• Estimate compute resource requirements.
• Identify libraries or frameworks for building a self-hosted LLM application.

Phase 2: Prototyping and Specification

Based on Phase 1 research, the engineering team defined high-level specifications and shortlisted open-source embedding and LLM models. A prototype was built for technical validation, and its findings informed detailed engineering specifications. After stakeholder discussions, specifications were finalized, and development began. Regular check-ins ensured alignment with project objectives.

Phase 3: Testing

Once core components were implemented, the QA team validated the application. Test scenarios were developed, and test data was prepared. Feedback was shared with the development team, and fixes were validated iteratively.

Phase 4: Deployment

The application was deployed to a private AWS VPC with robust security measures to prevent external access and attacks.

Results

• Significantly reduced time spent searching for internal knowledge.
• Improved employee productivity through faster access to accurate, up-to-date information.
• Ensured data security by keeping sensitive information within the company’s controlled environment.

Conclusion

The Musikaar engineering team leveraged its expertise in LLMs and RAG to develop a secure, efficient assistant for the internal knowledge base. This solution significantly improved knowledge accessibility and employee productivity while maintaining strict data security standards.