README.md

---
title: WebIQ Backend
emoji: 🌍
colorFrom: purple
colorTo: blue
sdk: docker
pinned: false
---

Check out the configuration reference at https://huggingface.co/docs/hub/spaces-config-reference

# WebIQ – Boosts your web intelligence with AI-powered insights

## Overview
WebIQ is a powerful **web scraping** and **question-answering (QA)** chatbot that follows the **Retrieval-Augmented Generation (RAG)** pipeline. It extracts and retrieves key insights from any website and generates **AI-powered** responses based on the extracted data. WebIQ leverages **FAISS** for efficient similarity search, **LangChain** for retrieval orchestration, and state-of-the-art **LLMs** for response generation.

## Features
- **Automated Web Scraping**: Extracts text data from webpages, caches it locally, and supports both targeted and full-site scraping.
- **Vector Embeddings**: Uses FAISS to store and retrieve information efficiently.
- **LLM Integration**: Supports OpenAI (GPT-4) and Hugging Face (Llama-2, Mistral, etc.).
- **Chunking for Optimization**: Splits documents into meaningful chunks to enhance retrieval quality.
- **Asynchronous Processing**: Uses `asyncio` for efficient execution.
- **Caching Mechanism**: Ensures previously processed webpages are not reprocessed.
- **Batch Processing**: Processes large numbers of URLs efficiently.
- **Memory Usage Logging**: Tracks memory consumption before and after each batch for efficiency monitoring.
- **Multi-Page Scraping**: Seamlessly scrapes content from multiple webpages and aggregates insights.

## Installation

1. Clone the repository:
    ```sh
    git clone https://github.com/Siddharth-Chandel/WebIQ.git
    cd WebIQ
    ```

2. Create a virtual environment and activate it:
    ```sh
    python -m venv venv
    source venv/bin/activate  # On Windows use `venv\Scripts\activate`
    ```

3. Install the required dependencies:
    ```sh
    pip install -r requirements.txt
    ```

4. Set up environment variables by creating a `.env` file:
    ```sh
    HUGGINGFACEHUB_API_TOKEN=your_huggingface_token
    OPENAI_API_KEY=your_openai_api_key  # If using OpenAI
    ```

## Usage

1. Run the chatbot script:
    ```sh
    python chatbot.py
    ```
2. Enter a **URL** when prompted (e.g., `https://playwright.dev`).
3. Enter your **query** (e.g., `Describe Playwright and its benefits`).
4. The chatbot will scrape the webpage, process the data, and return an AI-generated response.

## Example Output
```
====================* Answer *====================
Playwright is an end-to-end testing framework that provides...

=================* Source Documents *=================
Source 1:
file: cache/playwright-dev/pages/page_1.txt
Content: Playwright is a Node.js library that automates browsers.
```
## Practical Use Cases
- **Research Assistance**: Quickly extract and summarize information from research papers, blogs, or documentation.
- **Competitive Analysis**: Monitor competitors' websites and extract relevant insights for business strategy.
- **Customer Support**: Enhance chatbot capabilities by integrating real-time website data retrieval.
- **Market Intelligence**: Gather structured data from news sites, product pages, or financial reports for analysis.
- **SEO Optimization**: Analyze webpage content for better keyword targeting and content strategy.

## Technologies Used
- **RAG** (Providing better context)
- **LangChain** (Retrieval-based QA system)
- **FAISS** (Efficient similarity search)
- **Hugging Face Transformers** (LLMs & embeddings)
- **OpenAI GPT-4** (Optional for LLM-based response generation)
- **Crawl4AI** (An LLM-based web-scraper)
- **AsyncIO** (Increment the processing speed)
- **Rich** (For colorful CLI outputs)

## Future Enhancements
- Develop an interactive web UI using Streamlit or FastAPI for a seamless user experience.
- Enhance retrieval quality with advanced RAG tuning and improved embeddings.

## License
This project is licensed under the **MIT License**.

## Author
Siddharth Chandel - Developed as part of NLP & AI research.
Let's connect on [LinkedIn](https://www.linkedin.com/in/siddharth-chandel-001097245/) !!!

---
_Contributions are welcome! Feel free to fork and enhance._ 🚀