- Added docstring for the whole project

Dockerfile

@@ -8,7 +8,6 @@ COPY requirements.txt .
 RUN pip install --break-system-packages -r requirements.txt
 RUN python -m playwright install --with-deps chromium
 
-# RUN pip install watchfiles
 
 COPY . .
 

README.md

@@ -7,15 +7,159 @@ sdk: docker
 app_port: 7860
 ---
 
-## LLM Web Scraper
+# LLM Web Scraper (🕸️ → 🤖 → 🧠 → ❓ → 📄)
 
-This application uses Docker for deployment on Hugging Face Spaces.
+Scrape any web page, ask questions, and get structured answers powered by LangChain, FireCrawl, and leading LLMs from NVIDIA and Google—all wrapped in a clean Gradio interface.
 
-It combines web scraping with the power of Large Language Models (LLMs) to extract specific information from web pages.
+🔗 **Live Demo**: https://huggingface.co/spaces/frkhan/llm-web-scrapper
 
-### How to Use
-1. **Enter a URL**: Provide the URL of the web page you want to analyze.
-2. **Define Your Query**: Specify the exact information you're looking for.
-3. **Scrape the Web Page**: Choose a scraper and extract the page content.
-4. **Select Model & Provider**: Choose an LLM for information extraction.
-5. **Extract Info by LLM**: Get a structured answer based on your query.
+---
+
+### 🚀 Features
+
+-   🕸️ **Multi-Backend Scraping**: Choose between `FireCrawl` for robust, API-driven scraping and `Crawl4AI` for local, Playwright-based scraping.
+-   🧠 **Intelligent Extraction**: Use powerful LLMs (NVIDIA or Google Gemini) to understand your query and extract specific information from scraped content.
+-   📊 **Structured Output**: Get answers in markdown tables, JSON, or plain text, as requested.
+-   📈 **Full Observability**: Integrated with `Langfuse` to trace both scraping and LLM-extraction steps.
+-   ✨ **Interactive UI**: A clean and simple interface built with `Gradio`.
+-   🐳 **Docker-Ready**: Comes with `Dockerfile` and `docker-compose` configurations for easy local and production deployment.
+
+---
+
+### 🛠️ Tech Stack
+
+| Component | Purpose |
+| :--- | :--- |
+| **LangChain** | Orchestration of LLM calls |
+| **FireCrawl / Crawl4AI** | Web scraping backends |
+| **NVIDIA / Gemini** | LLM APIs for information extraction |
+| **Langfuse** | Tracing and observability for all operations |
+| **Gradio** | Interactive web UI |
+| **Docker** | Containerized deployment |
+
+---
+
+## 📦 Installation
+
+### Option 1: Run Locally
+
+1.  **Clone the repository:**
+    ```bash
+    git clone https://github.com/KI-IAN/llm-web-scrapper.git
+    cd llm-web-scrapper
+    ```
+
+2.  **Install dependencies:**
+    ```bash
+    pip install -r requirements.txt
+    ```
+
+3.  **Install Playwright browsers (for Crawl4AI):**
+    ```bash
+    playwright install
+    ```
+
+4.  **Create a `.env` file** in the root directory with your API keys:
+    ```env
+    GOOGLE_API_KEY=your_google_api_key
+    NVIDIA_API_KEY=your_nvidia_api_key
+    FIRECRAWL_API_KEY=your_firecrawl_api_key
+    
+    # Optional: For Langfuse tracing
+    LANGFUSE_PUBLIC_KEY=pk-lf-...
+    LANGFUSE_SECRET_KEY=sk-lf-...
+    LANGFUSE_HOST=https://cloud.langfuse.com
+    ```
+
+5.  **Run the application:**
+    ```bash
+    python app.py
+    ```
+
+---
+
+### Option 2: Run with Docker
+
+1.  **For Production:**
+    This uses the standard `docker-compose.yml`.
+    ```bash
+    docker compose up --build
+    ```
+
+2.  **For Local Development (with live code reload):**
+    This uses `docker-compose.dev.yml` to mount your local code into the container.
+    ```bash
+    docker compose -f docker-compose.dev.yml up --build
+    ```
+
+Access the app at http://localhost:12200.
+
+---
+
+## 🔑 Getting API Keys
+
+To use this app, you'll need API keys for **Google Gemini**, **NVIDIA NIM**, and **FireCrawl**. For full observability, you'll also need keys for **Langfuse**.
+
+-   **Google Gemini API Key**:
+    1.  Visit the Google AI Studio.
+    2.  Click **"Create API Key"** and copy the key.
+
+-   **NVIDIA NIM API Key**:
+    1.  Go to the NVIDIA API Catalog.
+    2.  Choose a model, go to the "API" tab, and click **"Get API Key"**.
+
+-   **FireCrawl API Key**:
+    1.  Sign up at FireCrawl.dev.
+    2.  Find your API key in the dashboard.
+
+-   **Langfuse API Keys (Optional)**:
+    1.  Sign up or log in at [Langfuse Cloud](https://cloud.langfuse.com/).
+    2.  Navigate to your project settings and then to the "API Keys" tab.
+    3.  Create a new key pair to get your `LANGFUSE_PUBLIC_KEY` (starts with `pk-lf-...`) and `LANGFUSE_SECRET_KEY` (starts with `sk-lf-...`).
+    4.  Add these to your `.env` file to enable tracing.
+
+---
+
+## 🧪 How to Use
+
+1.  **Enter a URL**: Provide the URL of the web page you want to analyze.
+2.  **Define Your Query**: Specify what you want to extract (e.g., "product name, price, and rating" or "summarize this article").
+3.  **Scrape the Web Page**: Choose a scraper (`Crawl4AI` or `FireCrawl`) and click **"Scrape Website"**.
+4.  **Select Model & Provider**: Choose an LLM to process the scraped content.
+5.  **Extract Info**: Click **"Extract Info by LLM"** to get a structured answer.
+
+---
+
+### 📁 File Structure
+
+```
+llm-web-scrapper/
+├── .env                  # Local environment variables (not tracked by git)
+├── .github/              # GitHub Actions workflows
+├── .gitignore
+├── docker-compose.yml    # Production Docker configuration
+├── docker-compose.dev.yml# Development Docker configuration
+├── Dockerfile
+├── requirements.txt
+├── app.py                # Gradio UI and application logic
+├── config.py             # Environment variable loading
+├── crawl4ai_client.py    # Client for Crawl4AI scraper
+├── firecrawl_client.py   # Client for FireCrawl scraper
+└── llm_inference_service.py # Logic for LLM calls
+```
+
+---
+
+## 📜 License
+
+This project is open-source and distributed under the **MIT License**. Feel free to use, modify, and distribute it.
+
+---
+
+## 🤝 Acknowledgements
+
+-   LangChain for orchestrating LLM interactions.
+-   FireCrawl & Crawl4AI for providing powerful scraping backends.
+-   NVIDIA AI Endpoints & Google Gemini for their state-of-the-art LLMs.
+-   Langfuse for providing excellent observability tools.
+-   Gradio for making UI creation simple and elegant.

app.py

@@ -1,3 +1,11 @@
+"""
+This module sets up and runs the Gradio web interface for the LLM Web Scraper application.
+
+It orchestrates the UI components, event handling for scraping and LLM extraction,
+and integrates with backend services for scraping (FireCrawl, Crawl4AI) and
+LLM inference. It also initializes and uses Langfuse for tracing application performance.
+"""
+
 import gradio as gr
 import firecrawl_client
 import crawl4ai_client
@@ -16,7 +24,20 @@ if LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY:
     langfuse = get_client()
 
 def parse_model_provider(selection):
-    # Expected format: "<model_name> (<provider>)"
+    """
+    Parses a model and provider from a selection string.
+
+    The expected format is "<model_name> (<provider>)".
+
+    Args:
+        selection (str): The string to parse.
+
+    Returns:
+        tuple[str, str]: A tuple containing the model name and provider.
+
+    Raises:
+        ValueError: If the selection string is not in the expected format.
+    """
     if "(" in selection and ")" in selection:
         model = selection.split(" (")[0].strip()
         provider = selection.split(" (")[1].replace(")", "").strip()
@@ -24,6 +45,21 @@ def parse_model_provider(selection):
     raise ValueError(f"Invalid selection format: {selection}")
     
 def llm_response_wrapper(query, scrape_result, model_provider_selection, progress=gr.Progress(track_tqdm=True)):
+    """
+    A generator function that wraps the LLM inference call for the Gradio UI.
+
+    It yields an initial status message, calls the LLM service to extract information,
+    and then yields the final result or an error message.
+
+    Args:
+        query (str): The user's query for information extraction.
+        scrape_result (str): The scraped markdown content from the website.
+        model_provider_selection (str): The selected model and provider string.
+        progress (gr.Progress, optional): Gradio progress tracker. Defaults to gr.Progress(track_tqdm=True).
+
+    Yields:
+        str: Status messages and the final LLM response as a markdown string.
+    """
     yield "⏳ Generating response... Please wait."
     
     model, provider = parse_model_provider(model_provider_selection)
@@ -33,9 +69,19 @@ def llm_response_wrapper(query, scrape_result, model_provider_selection, progres
     yield result
 
 async def scrape_website(url, scraper_selection, progress=gr.Progress(track_tqdm=True)):
-    """
-    Performs the scraping and yields Gradio component updates directly.
-    This generator pattern is the most reliable way to handle sequential UI updates.
+    """An async generator that scrapes a website based on user selection for the Gradio UI.
+
+    This function yields an initial status message, then performs the web scraping
+    using the selected tool (FireCrawl or Crawl4AI). If Langfuse is configured,
+    it wraps the scraping operation in a trace for observability.
+
+    Args:
+        url (str): The URL of the website to scrape.
+        scraper_selection (str): The scraping tool selected by the user.
+        progress (gr.Progress, optional): Gradio progress tracker. Defaults to gr.Progress(track_tqdm=True).
+
+    Yields:
+        str: A status message, followed by the scraped markdown content or an error message.
     """
     # 1. First, yield an update to show the loading state and hide the old image.
     yield "⏳ Scraping website... Please wait."
@@ -69,6 +115,7 @@ async def scrape_website(url, scraper_selection, progress=gr.Progress(track_tqdm
         yield markdown
 
 #Gradio UI
+# This block defines the entire Gradio user interface, including layout and component interactions.
 with gr.Blocks() as gradio_ui:
     gr.HTML("""
     <div style="display: flex; align-items: center; gap: 20px; flex-wrap: wrap; margin-bottom: 20px;">
@@ -127,14 +174,13 @@ with gr.Blocks() as gradio_ui:
     
     with gr.Column():
         url_input = gr.Textbox(label="Enter URL to scrape", placeholder="https://example.com/query?search=cat+food", lines=1)
-        # search_query_input = gr.Textbox(label="Enter your query", placeholder="Paw paw fish adult cat food", lines=1)
-        query_input = gr.Textbox(label="What information do you want to find?", placeholder="Find product name, price, rating", lines=1)
+        query_input = gr.Textbox(label="What information do you want to find?", placeholder="Find product name, price, rating etc. / Summarize the content of this page", lines=2)
         
         with gr.Row():
             scraper_dropdown = gr.Dropdown(
                 label="Select Scraper",
-                choices=["Scrape with FireCrawl", "Scrape with Crawl4AI"],
-                value="Scrape with FireCrawl"
+                choices=["Scrape with Crawl4AI", "Scrape with FireCrawl"],
+                value="Scrape with Crawl4AI"
             )
             scrape_btn = gr.Button("Scrape Website")
             clear_btn = gr.Button("Clear")

crawl4ai_client.py

@@ -1,8 +1,24 @@
-import asyncio
+"""
+This module provides a client for interacting with the Crawl4AI library.
+
+It encapsulates the logic for scraping a website using Crawl4AI and extracting
+its content as a markdown string, handling potential errors during the process.
+"""
+
 from crawl4ai import AsyncWebCrawler
 
 
 async def scrape_and_get_markdown_with_crawl4ai(url: str) -> str:
+    """
+    Asynchronously scrapes a given URL using Crawl4AI and returns its content as markdown.
+
+    Args:
+        url (str): The URL of the website to scrape.
+
+    Returns:
+        str: The scraped content in markdown format. If scraping fails or returns
+             no content, a formatted error message string is returned.
+    """
     try:
         async with AsyncWebCrawler() as crawler:
             result = await crawler.arun(url=url) 

docker-compose.dev.yml

@@ -12,9 +12,9 @@ services:
       - NVIDIA_API_KEY=${NVIDIA_API_KEY}  # Load this key from .env in local/dev environment
       - GOOGLE_API_KEY=${GOOGLE_API_KEY}  # Load this key from .env in local/dev environment
       - FIRECRAWL_API_KEY=${FIRECRAWL_API_KEY}  # Load this key from .env in local/dev environment
-      - LANGFUSE_PUBLIC_KEY=${LANGFUSE_PUBLIC_KEY}
-      - LANGFUSE_SECRET_KEY=${LANGFUSE_SECRET_KEY}
-      - LANGFUSE_HOST=${LANGFUSE_HOST}
+      - LANGFUSE_PUBLIC_KEY=${LANGFUSE_PUBLIC_KEY}  # Load this key from .env in local/dev environment
+      - LANGFUSE_SECRET_KEY=${LANGFUSE_SECRET_KEY}  # Load this key from .env in local/dev environment
+      - LANGFUSE_HOST=${LANGFUSE_HOST}  # Load this key from .env in local/dev environment
     volumes:
       - .:/app:rw           # This is for local development. Docker reads the code from the host machine. Changes on the host are reflected in the container.
     restart: unless-stopped

docker-compose.yml

@@ -12,7 +12,7 @@ services:
       - NVIDIA_API_KEY=${NVIDIA_API_KEY}  # Load this key from .env or manually add the secret
       - GOOGLE_API_KEY=${GOOGLE_API_KEY}  # Load this key from .env or manually add the secret
       - FIRECRAWL_API_KEY=${FIRECRAWL_API_KEY}  # Load this key from .env in local/dev environment
-      - LANGFUSE_PUBLIC_KEY=${LANGFUSE_PUBLIC_KEY}
-      - LANGFUSE_SECRET_KEY=${LANGFUSE_SECRET_KEY}
-      - LANGFUSE_HOST=${LANGFUSE_HOST}
+      - LANGFUSE_PUBLIC_KEY=${LANGFUSE_PUBLIC_KEY} # Load this key from .env or manually add the secret
+      - LANGFUSE_SECRET_KEY=${LANGFUSE_SECRET_KEY} # Load this key from .env or manually add the secret
+      - LANGFUSE_HOST=${LANGFUSE_HOST} # Load this key from .env or manually add the secret
     restart: unless-stopped

firecrawl_client.py

@@ -1,10 +1,27 @@
+"""
+This module provides a client for interacting with the FireCrawl service.
+
+It encapsulates the logic for scraping a website using the FireCrawlLoader from
+LangChain, converting the scraped documents into a single markdown string, and
+handling potential errors during the process.
+"""
+
 from langchain_community.document_loaders import FireCrawlLoader
 from langchain_core.documents import Document
 from config import FIRE_CRAWL_API_KEY
 
 
 def scrape_with_firecrawl(url: str) -> list[Document]:
-    
+    """
+    Scrapes a given URL using FireCrawl and returns the content as a list of Documents.
+
+    Args:
+        url (str): The URL of the website to scrape.
+
+    Returns:
+        list[Document]: A list of LangChain Document objects, where each document
+                        represents a scraped page.
+    """
     loader = FireCrawlLoader(url=url,
                              api_key=FIRE_CRAWL_API_KEY,
                              mode='scrape')
@@ -17,6 +34,17 @@ def scrape_with_firecrawl(url: str) -> list[Document]:
     return pages
 
 def get_markdown_from_documents(docs: list[Document]) -> str:
+    """
+    Converts a list of LangChain Documents into a single markdown string.
+
+    Each document's content is appended, separated by a horizontal rule.
+
+    Args:
+        docs (list[Document]): A list of Document objects to process.
+
+    Returns:
+        str: A string containing the combined content in markdown format.
+    """
     markdown_content = ""
     for i, doc in enumerate(docs):
         markdown_content += f"### Page {i+1}\n"
@@ -25,6 +53,18 @@ def get_markdown_from_documents(docs: list[Document]) -> str:
 
 
 def scrape_and_get_markdown_with_firecrawl(url: str) -> str:
+    """
+    Orchestrates the scraping of a URL with FireCrawl and returns the content as markdown.
+
+    This is the main entry point function for this module. It handles the full
+    process of scraping, content conversion, and error handling.
+
+    Args:
+        url (str): The URL of the website to scrape.
+
+    Returns:
+        str: The scraped content in markdown format, or a formatted error message string if an issue occurs.
+    """
     try:
         docs = scrape_with_firecrawl(url)
         if not docs:

llm_inference_service.py

@@ -1,3 +1,11 @@
+"""
+This module provides the service for interacting with Large Language Models (LLMs).
+
+It is responsible for initializing the Langfuse callback handler for tracing,
+constructing the appropriate prompt for information extraction, initializing the
+selected chat model, and invoking the model to get a response.
+"""
+
 from langchain.chat_models import init_chat_model
 from langfuse.langchain import CallbackHandler
 from langfuse import Langfuse
@@ -5,24 +13,41 @@ from langfuse import Langfuse
 from config import LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, LANGFUSE_HOST
 
 # Initialize Langfuse client
-# It is safe to do this even if keys are not set, as the handler will only be used if keys are present.
+# This block sets up the Langfuse callback handler for LangChain.
+# It initializes the Langfuse client and creates a CallbackHandler instance
+# only if the required API keys are available. The handler is then added to
+# a list of callbacks that can be passed to LLM invocations for tracing.
 langfuse_callback_handler = None
 callbacks = []
 
 if LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY:
-    langfuse = Langfuse(
+    Langfuse(
         public_key=LANGFUSE_PUBLIC_KEY,
         secret_key=LANGFUSE_SECRET_KEY,
         host=LANGFUSE_HOST,
     )
-    
     langfuse_callback_handler = CallbackHandler()
-    
     callbacks.append(langfuse_callback_handler)
 
 
 
 def extract_page_info_by_llm(user_query: str, scraped_markdown_content: str, model_name: str, model_provider: str) -> str:
+    """
+    Extracts information from scraped content using a specified Large Language Model.
+
+    This function constructs a detailed prompt, initializes the selected chat model,
+    and invokes it with the scraped content and user query. If Langfuse is configured,
+    it uses a callback handler to trace the LLM interaction.
+
+    Args:
+        user_query (str): The user's query specifying what information to extract.
+        scraped_markdown_content (str): The markdown content from the scraped web page.
+        model_name (str): The name of the LLM to use for extraction.
+        model_provider (str): The provider of the LLM (e.g., 'google_genai', 'nvidia').
+
+    Returns:
+        str: The content of the LLM's response.
+    """
     
     if not scraped_markdown_content:
         return "No relevant information found to answer your question."