RBI & Financial Notifications RAG System ## Overview This application monitors financial regulatory websites such as the Reserve Bank of India (RBI) notifications and other relevant financial websites. Users can query the latest financial regulations, banking rules, and policy updates in a conversational manner. The system leverages vector stores and embeddings to perform semantic search and retrieval-augmented generation (RAG) to provide contextual answers. --- ## Workflow & Steps ### 1. Data Collection / Scraping - Identify relevant financial websites to scrape, e.g., RBI notifications, SEBI circulars, banking regulatory updates. - Use web scraping tools (requests + BeautifulSoup, Selenium, or Apify actors) to extract notifications. - Extract key information: - Notification title - Date of publication - Full content (HTML or plain text) - Unique identifiers or URLs - Save collected data locally or in a database (JSON, CSV, or SQL). ### 2. Data Preprocessing - Clean extracted content: - Remove HTML tags, scripts, and unnecessary formatting. - Normalize text (remove extra whitespace, special characters). - Split large documents into chunks or paragraphs: - Each chunk should be small enough for embeddings and LLMs (~400–500 tokens). - Maintain metadata such as file_name, publication_date, source_url, chunk_index. ### 3. Embeddings & Vector Store - Convert each chunk into numeric vectors using embedding models (e.g., sentence-transformers or Groq embeddings). - Store vectors in a vector database (FAISS, Chroma, Pinecone) with metadata. - Enables semantic similarity search for user queries. ### 4. Query Interface (RAG) - Accept user queries regarding financial regulations or banking rules. - Convert the query into an embedding vector. - Retrieve top-k relevant chunks from the vector store using similarity search. - Construct a context-aware prompt by concatenating retrieved chunks. - Use a Large Language Model (LLM) like Groq, GPT, or LLaMA to generate an answer based on context. - Return the answer along with references to the source documents for transparency. ### 5. Example Use Cases - Ask about RBI policies like liquidity adjustment facility rates or repo rate changes. - Ask about banking regulations, e.g., minimum balance rules, checkbook issuance, penalties. - Ask about labor-related financial laws, taxation rules, or other regulatory notifications. - Users receive accurate, context-rich responses grounded in source documents. ### 6. Deployment & Automation - Wrap the Python script as an Apify actor or cloud service: - Accept input queries via API. - Schedule periodic scraping to keep the vector store updated. - Store API keys (Groq, etc.) securely via environment variables. - Optionally, build a web or chat interface for real-time interaction. ### 7. Best Practices - Limit chunk size to avoid exceeding LLM token limits. - Use metadata to trace answers to original sources. - Implement error handling for web scraping and API calls. - Maintain a versioned vector store for audit and update purposes. --- ## Future Enhancements - Personalized Notifications: Fetch and push relevant information based on user profile: - Normal stock market investor - CEO of a large organization - Regulatory professional - Improved Retrieval: Enhance semantic search for more precise and context-aware results. - Continuous Updates: Auto-refresh vector store with latest notifications and documents. - Advanced Summarization: Summarize multiple notifications to provide digestible insights. - Multi-Source Integration: Add more financial regulatory websites and circulars for broader coverage. ---