Replace ElevenLabs with HuggingFace TTS (SpeechT5)

✅ Benefits:
- No API keys required - completely free and self-contained
- Uses Microsoft SpeechT5 model for high-quality TTS
- Voice ID mapping for compatibility with existing API
- Multiple speaker embeddings for voice variety
- Runs entirely on HuggingFace infrastructure

📁 Changes:
- Add hf_tts_client.py with SpeechT5 TTS implementation
- Update app.py to use HF TTS instead of ElevenLabs
- Update requirements.txt with TTS dependencies
- Update README.md to reflect HF TTS usage
- Add test_hf_tts.py for TTS testing
- Remove dependency on external ElevenLabs API

🎯 Voice mapping:
- Same voice_id parameters work (mapped to speaker embeddings)
- 7+ different voices available
- No rate limits or quotas

API_DOCUMENTATION.md

@@ -0,0 +1,177 @@
+# 🔌 OmniAvatar API Documentation
+
+## POST /generate - Avatar Generation
+
+### Request Format
+
+**URL:** `https://huggingface.co/spaces/bravedims/AI_Avatar_Chat/api/generate`
+**Method:** `POST`
+**Content-Type:** `application/json`
+
+### Request Body (JSON)
+
+```json
+{
+  "prompt": "string",
+  "text_to_speech": "string (optional)",
+  "elevenlabs_audio_url": "string (optional)",
+  "voice_id": "string (optional, default: '21m00Tcm4TlvDq8ikWAM')",
+  "image_url": "string (optional)",
+  "guidance_scale": "float (default: 5.0)",
+  "audio_scale": "float (default: 3.0)",
+  "num_steps": "int (default: 30)",
+  "sp_size": "int (default: 1)",
+  "tea_cache_l1_thresh": "float (optional)"
+}
+```
+
+### Request Parameters
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `prompt` | string | ✅ | Character behavior description |
+| `text_to_speech` | string | ❌ | Text to convert to speech via ElevenLabs |
+| `elevenlabs_audio_url` | string | ❌ | Direct URL to audio file |
+| `voice_id` | string | ❌ | ElevenLabs voice ID (default: Rachel) |
+| `image_url` | string | ❌ | Reference image URL |
+| `guidance_scale` | float | ❌ | Prompt following strength (4-6 recommended) |
+| `audio_scale` | float | ❌ | Lip-sync accuracy (3-5 recommended) |
+| `num_steps` | int | ❌ | Generation steps (20-50 recommended) |
+| `sp_size` | int | ❌ | Parallel processing size |
+| `tea_cache_l1_thresh` | float | ❌ | Cache threshold optimization |
+
+**Note:** Either `text_to_speech` OR `elevenlabs_audio_url` must be provided.
+
+### Example Request
+
+```json
+{
+  "prompt": "A professional teacher explaining a mathematical concept with clear gestures",
+  "text_to_speech": "Hello students! Today we're going to learn about calculus and how derivatives work in real life.",
+  "voice_id": "21m00Tcm4TlvDq8ikWAM",
+  "image_url": "https://example.com/teacher.jpg",
+  "guidance_scale": 5.0,
+  "audio_scale": 3.5,
+  "num_steps": 30
+}
+```
+
+### Response Format
+
+**Success Response (200 OK):**
+
+```json
+{
+  "message": "string",
+  "output_path": "string",
+  "processing_time": "float",
+  "audio_generated": "boolean"
+}
+```
+
+### Response Fields
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `message` | string | Success/status message |
+| `output_path` | string | Path to generated video file |
+| `processing_time` | float | Processing time in seconds |
+| `audio_generated` | boolean | Whether audio was generated from text |
+
+### Example Response
+
+```json
+{
+  "message": "Avatar generation completed successfully",
+  "output_path": "./outputs/avatar_20240807_130512.mp4",
+  "processing_time": 45.67,
+  "audio_generated": true
+}
+```
+
+### Error Responses
+
+**400 Bad Request:**
+```json
+{
+  "detail": "Either text_to_speech or elevenlabs_audio_url must be provided"
+}
+```
+
+**500 Internal Server Error:**
+```json
+{
+  "detail": "Model not loaded"
+}
+```
+
+**503 Service Unavailable:**
+```json
+{
+  "detail": "Model not loaded"
+}
+```
+
+### Available ElevenLabs Voices
+
+| Voice ID | Name | Description |
+|----------|------|-------------|
+| `21m00Tcm4TlvDq8ikWAM` | Rachel | Default, clear female voice |
+| `pNInz6obpgDQGcFmaJgB` | Adam | Professional male voice |
+| `EXAVITQu4vr4xnSDxMaL` | Bella | Expressive female voice |
+
+### Usage Examples
+
+#### With Text-to-Speech
+```bash
+curl -X POST "https://huggingface.co/spaces/bravedims/AI_Avatar_Chat/api/generate" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "prompt": "A friendly presenter speaking confidently",
+    "text_to_speech": "Welcome to our AI avatar demonstration!",
+    "voice_id": "21m00Tcm4TlvDq8ikWAM",
+    "guidance_scale": 5.5,
+    "audio_scale": 4.0
+  }'
+```
+
+#### With Audio URL
+```bash
+curl -X POST "https://huggingface.co/spaces/bravedims/AI_Avatar_Chat/api/generate" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "prompt": "A news anchor delivering headlines",
+    "elevenlabs_audio_url": "https://example.com/audio.mp3",
+    "image_url": "https://example.com/anchor.jpg",
+    "num_steps": 40
+  }'
+```
+
+### Other Endpoints
+
+#### GET /health - Health Check
+```json
+{
+  "status": "healthy",
+  "model_loaded": true,
+  "device": "cuda",
+  "supports_elevenlabs": true,
+  "supports_image_urls": true,
+  "supports_text_to_speech": true,
+  "elevenlabs_api_configured": true
+}
+```
+
+#### GET /docs - FastAPI Documentation
+Interactive API documentation available at `/docs` endpoint.
+
+### Rate Limits & Performance
+
+- **Processing Time:** 30-120 seconds depending on complexity
+- **Max Video Length:** Determined by audio length
+- **Supported Formats:** MP4 output, MP3/WAV audio input
+- **GPU Acceleration:** Enabled on T4+ hardware
+
+---
+
+**Live API Base URL:** `https://huggingface.co/spaces/bravedims/AI_Avatar_Chat`

Dockerfile.backup

@@ -0,0 +1,51 @@
+# Read the doc: https://huggingface.co/docs/hub/spaces-sdks-docker
+# Use NVIDIA PyTorch base image for GPU support
+FROM pytorch/pytorch:2.1.0-cuda12.1-cudnn8-devel
+
+# Create user as required by HF Spaces
+RUN useradd -m -u 1000 user
+
+# Install system dependencies
+RUN apt-get update && apt-get install -y \
+    git \
+    wget \
+    curl \
+    libgl1-mesa-glx \
+    libglib2.0-0 \
+    libsm6 \
+    libxext6 \
+    libxrender-dev \
+    libgomp1 \
+    libgoogle-perftools4 \
+    libtcmalloc-minimal4 \
+    ffmpeg \
+    && apt-get clean \
+    && rm -rf /var/lib/apt/lists/*
+
+# Switch to user
+USER user
+
+# Set environment variables for user
+ENV PATH="/home/user/.local/bin:$PATH"
+ENV PYTHONPATH=/app
+ENV GRADIO_SERVER_NAME=0.0.0.0
+ENV GRADIO_SERVER_PORT=7860
+
+# Set working directory
+WORKDIR /app
+
+# Copy requirements and install Python dependencies
+COPY --chown=user ./requirements.txt requirements.txt
+RUN pip install --no-cache-dir --upgrade -r requirements.txt
+
+# Copy application code
+COPY --chown=user . /app
+
+# Create necessary directories
+RUN mkdir -p pretrained_models outputs
+
+# Expose port (required by HF Spaces to be 7860)
+EXPOSE 7860
+
+# Start the application
+CMD ["python", "app.py"]

README.md

@@ -10,14 +10,14 @@ suggested_hardware: a10g-small
 suggested_storage: large
 ---
 
-# 🎭 OmniAvatar-14B with ElevenLabs TTS
+# 🎭 OmniAvatar-14B with HuggingFace TTS
 
-An advanced AI avatar generation system that creates realistic talking avatars from text prompts and speech. This space combines the power of OmniAvatar-14B with ElevenLabs text-to-speech for seamless avatar creation.
+An advanced AI avatar generation system that creates realistic talking avatars from text prompts and speech. This space combines the power of OmniAvatar-14B with HuggingFace SpeechT5 text-to-speech for seamless avatar creation.
 
 ## ✨ Features
 
 - **🎯 Text-to-Avatar Generation**: Generate avatars from descriptive text prompts
-- **🗣️ ElevenLabs Integration**: High-quality text-to-speech synthesis
+- **🗣️ HuggingFace TTS Integration**: High-quality text-to-speech synthesis
 - **🎵 Audio URL Support**: Use pre-generated audio files
 - **🖼️ Image Reference Support**: Guide avatar appearance with reference images
 - **⚡ Real-time Processing**: Fast generation with GPU acceleration
@@ -48,7 +48,7 @@ An advanced AI avatar generation system that creates realistic talking avatars f
 ## 🔧 Technical Details
 
 - **Model**: OmniAvatar-14B for video generation
-- **TTS**: ElevenLabs API for high-quality speech synthesis
+- ****TTS**: Microsoft SpeechT5 (HuggingFace) for high-quality speech synthesis
 - **Framework**: FastAPI + Gradio interface
 - **GPU**: Optimized for T4 and higher
 - **Storage**: Requires large storage due to 14B parameter models (~70GB total)
@@ -59,9 +59,9 @@ An advanced AI avatar generation system that creates realistic talking avatars f
 - `POST /generate` - Generate avatar video
 - `/gradio` - Interactive web interface
 
-## 🔐 Environment Variables
+## 🔐 No API Keys Required
 
-The space uses ElevenLabs for text-to-speech. For optimal performance, configure your ElevenLabs API key as a secret.
+This space uses open-source HuggingFace models for text-to-speech. No external API keys or accounts needed!
 
 ## 📄 License
 
@@ -69,7 +69,8 @@ Apache 2.0 - See LICENSE file for details
 
 ---
 
-*Powered by OmniAvatar-14B and ElevenLabs TTS*
+*Powered by OmniAvatar-14B and HuggingFace TTS*
 
 **Note**: This space requires large storage capacity due to the 14B parameter models. The models are downloaded on first startup and cached for subsequent uses.
 
+

api_urls.txt

@@ -0,0 +1,25 @@
+# Your HF Space API URLs:
+
+Base URL: https://bravedims-ai-avatar-chat.hf.space
+
+Health Check:
+GET https://bravedims-ai-avatar-chat.hf.space/health
+
+Generate Avatar:
+POST https://bravedims-ai-avatar-chat.hf.space/generate
+
+Gradio Interface:
+https://bravedims-ai-avatar-chat.hf.space/gradio
+
+# Example API call using the JSON you selected:
+curl -X POST "https://bravedims-ai-avatar-chat.hf.space/generate" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "prompt": "A professional teacher explaining a mathematical concept with clear gestures",
+    "text_to_speech": "Hello students! Today we'\''re going to learn about calculus and how derivatives work in real life.",
+    "voice_id": "21m00Tcm4TlvDq8ikWAM",
+    "image_url": "https://example.com/teacher.jpg",
+    "guidance_scale": 5.0,
+    "audio_scale": 3.5,
+    "num_steps": 30
+  }'

app.py

@@ -17,6 +17,7 @@ from typing import Optional
 import aiohttp
 import asyncio
 from dotenv import load_dotenv
+from hf_tts_client import HuggingFaceTTSClient
 
 # Load environment variables
 load_dotenv()
@@ -119,9 +120,9 @@ class OmniAvatarAPI:
     def __init__(self):
         self.model_loaded = False
         self.device = "cuda" if torch.cuda.is_available() else "cpu"
-        self.elevenlabs_client = ElevenLabsClient()
+        self.tts_client = HuggingFaceTTSClient()
         logger.info(f"Using device: {self.device}")
-        logger.info(f"ElevenLabs API Key configured: {'Yes' if self.elevenlabs_client.api_key else 'No'}")
+        logger.info("Using HuggingFace TTS (SpeechT5) - No API key required")
         
     def load_model(self):
         """Load the OmniAvatar model"""
@@ -208,10 +209,10 @@ class OmniAvatarAPI:
                 logger.info(f"Generating speech from text: {request.text_to_speech[:50]}...")
                 logger.info(f"Using voice ID: {voice_id}")
                 
-                if not self.elevenlabs_client.api_key:
+                if not True  # HF TTS always available:
                     raise HTTPException(status_code=503, detail="ElevenLabs API key not configured")
                 
-                audio_path = await self.elevenlabs_client.text_to_speech(
+                audio_path = await self.tts_client.text_to_speech(
                     request.text_to_speech, 
                     voice_id
                 )
@@ -442,7 +443,7 @@ iface = gr.Interface(
         gr.Slider(minimum=10, maximum=100, value=30, step=1, label="Number of Steps", info="20-50 recommended")
     ],
     outputs=gr.Video(label="Generated Avatar Video"),
-    title="🎭 OmniAvatar-14B with ElevenLabs TTS",
+    title="🎭 OmniAvatar-14B with HuggingFace TTS",
     description="""
     Generate avatar videos with lip-sync from text prompts and speech.
     
@@ -497,3 +498,5 @@ if __name__ == "__main__":
     uvicorn.run(app, host="0.0.0.0", port=7860)
 
 
+
+

app.py.backup

@@ -0,0 +1,482 @@
+import os
+import torch
+import tempfile
+import gradio as gr
+from fastapi import FastAPI, HTTPException
+from fastapi.middleware.cors import CORSMiddleware
+from pydantic import BaseModel, HttpUrl
+import subprocess
+import json
+from pathlib import Path
+import logging
+import requests
+from urllib.parse import urlparse
+from PIL import Image
+import io
+from typing import Optional
+import aiohttp
+import asyncio
+from dotenv import load_dotenv
+
+# Load environment variables
+load_dotenv()
+
+# Set up logging
+logging.basicConfig(level=logging.INFO)
+logger = logging.getLogger(__name__)
+
+app = FastAPI(title="OmniAvatar-14B API with ElevenLabs", version="1.0.0")
+
+# Add CORS middleware
+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=["*"],
+    allow_credentials=True,
+    allow_methods=["*"],
+    allow_headers=["*"],
+)
+
+# Pydantic models for request/response
+class GenerateRequest(BaseModel):
+    prompt: str
+    text_to_speech: Optional[str] = None  # Text to convert to speech
+    elevenlabs_audio_url: Optional[HttpUrl] = None  # Direct audio URL
+    voice_id: Optional[str] = "21m00Tcm4TlvDq8ikWAM"  # Default ElevenLabs voice
+    image_url: Optional[HttpUrl] = None
+    guidance_scale: float = 5.0
+    audio_scale: float = 3.0
+    num_steps: int = 30
+    sp_size: int = 1
+    tea_cache_l1_thresh: Optional[float] = None
+
+class GenerateResponse(BaseModel):
+    message: str
+    output_path: str
+    processing_time: float
+    audio_generated: bool = False
+
+class ElevenLabsClient:
+    def __init__(self, api_key: str = None):
+        self.api_key = api_key or os.getenv("ELEVENLABS_API_KEY", "sk_c7a0b115cd48fc026226158c5ac87755b063c802ad892de6")
+        self.base_url = "https://api.elevenlabs.io/v1"
+        
+    async def text_to_speech(self, text: str, voice_id: str = "21m00Tcm4TlvDq8ikWAM") -> str:
+        """Convert text to speech using ElevenLabs and return temporary file path"""
+        url = f"{self.base_url}/text-to-speech/{voice_id}"
+        
+        headers = {
+            "Accept": "audio/mpeg",
+            "Content-Type": "application/json",
+            "xi-api-key": self.api_key
+        }
+        
+        data = {
+            "text": text,
+            "model_id": "eleven_monolingual_v1",
+            "voice_settings": {
+                "stability": 0.5,
+                "similarity_boost": 0.5
+            }
+        }
+        
+        try:
+            async with aiohttp.ClientSession() as session:
+                async with session.post(url, headers=headers, json=data) as response:
+                    if response.status != 200:
+                        error_text = await response.text()
+                        raise HTTPException(
+                            status_code=400, 
+                            detail=f"ElevenLabs API error: {response.status} - {error_text}"
+                        )
+                    
+                    audio_content = await response.read()
+                    
+                    # Save to temporary file
+                    temp_file = tempfile.NamedTemporaryFile(delete=False, suffix='.mp3')
+                    temp_file.write(audio_content)
+                    temp_file.close()
+                    
+                    logger.info(f"Generated speech audio: {temp_file.name}")
+                    return temp_file.name
+                    
+        except aiohttp.ClientError as e:
+            logger.error(f"Network error calling ElevenLabs: {e}")
+            raise HTTPException(status_code=400, detail=f"Network error calling ElevenLabs: {e}")
+        except Exception as e:
+            logger.error(f"Error generating speech: {e}")
+            raise HTTPException(status_code=500, detail=f"Error generating speech: {e}")
+
+class OmniAvatarAPI:
+    def __init__(self):
+        self.model_loaded = False
+        self.device = "cuda" if torch.cuda.is_available() else "cpu"
+        self.elevenlabs_client = ElevenLabsClient()
+        logger.info(f"Using device: {self.device}")
+        logger.info(f"ElevenLabs API Key configured: {'Yes' if self.elevenlabs_client.api_key else 'No'}")
+        
+    def load_model(self):
+        """Load the OmniAvatar model"""
+        try:
+            # Check if models are downloaded
+            model_paths = [
+                "./pretrained_models/Wan2.1-T2V-14B",
+                "./pretrained_models/OmniAvatar-14B", 
+                "./pretrained_models/wav2vec2-base-960h"
+            ]
+            
+            for path in model_paths:
+                if not os.path.exists(path):
+                    logger.error(f"Model path not found: {path}")
+                    return False
+                    
+            self.model_loaded = True
+            logger.info("Models loaded successfully")
+            return True
+            
+        except Exception as e:
+            logger.error(f"Error loading model: {str(e)}")
+            return False
+    
+    async def download_file(self, url: str, suffix: str = "") -> str:
+        """Download file from URL and save to temporary location"""
+        try:
+            async with aiohttp.ClientSession() as session:
+                async with session.get(str(url)) as response:
+                    if response.status != 200:
+                        raise HTTPException(status_code=400, detail=f"Failed to download file from URL: {url}")
+                    
+                    content = await response.read()
+                    
+                    # Create temporary file
+                    temp_file = tempfile.NamedTemporaryFile(delete=False, suffix=suffix)
+                    temp_file.write(content)
+                    temp_file.close()
+                    
+                    return temp_file.name
+                    
+        except aiohttp.ClientError as e:
+            logger.error(f"Network error downloading {url}: {e}")
+            raise HTTPException(status_code=400, detail=f"Network error downloading file: {e}")
+        except Exception as e:
+            logger.error(f"Error downloading file from {url}: {e}")
+            raise HTTPException(status_code=500, detail=f"Error downloading file: {e}")
+    
+    def validate_audio_url(self, url: str) -> bool:
+        """Validate if URL is likely an audio file"""
+        try:
+            parsed = urlparse(url)
+            # Check for common audio file extensions or ElevenLabs patterns
+            audio_extensions = ['.mp3', '.wav', '.m4a', '.ogg', '.aac']
+            is_audio_ext = any(parsed.path.lower().endswith(ext) for ext in audio_extensions)
+            is_elevenlabs = 'elevenlabs' in parsed.netloc.lower()
+            
+            return is_audio_ext or is_elevenlabs or 'audio' in url.lower()
+        except:
+            return False
+    
+    def validate_image_url(self, url: str) -> bool:
+        """Validate if URL is likely an image file"""
+        try:
+            parsed = urlparse(url)
+            image_extensions = ['.jpg', '.jpeg', '.png', '.webp', '.bmp', '.gif']
+            return any(parsed.path.lower().endswith(ext) for ext in image_extensions)
+        except:
+            return False
+    
+    async def generate_avatar(self, request: GenerateRequest) -> tuple[str, float, bool]:
+        """Generate avatar video from prompt and audio/text"""
+        import time
+        start_time = time.time()
+        audio_generated = False
+        
+        try:
+            # Determine audio source
+            audio_path = None
+            
+            if request.text_to_speech:
+                # Generate speech from text using ElevenLabs
+                logger.info(f"Generating speech from text: {request.text_to_speech[:50]}...")
+                audio_path = await self.elevenlabs_client.text_to_speech(
+                    request.text_to_speech, 
+                    request.voice_id or "21m00Tcm4TlvDq8ikWAM"
+                )
+                audio_generated = True
+                
+            elif request.elevenlabs_audio_url:
+                # Download audio from provided URL
+                logger.info(f"Downloading audio from URL: {request.elevenlabs_audio_url}")
+                if not self.validate_audio_url(str(request.elevenlabs_audio_url)):
+                    logger.warning(f"Audio URL may not be valid: {request.elevenlabs_audio_url}")
+                
+                audio_path = await self.download_file(str(request.elevenlabs_audio_url), ".mp3")
+            
+            else:
+                raise HTTPException(
+                    status_code=400, 
+                    detail="Either text_to_speech or elevenlabs_audio_url must be provided"
+                )
+            
+            # Download image if provided
+            image_path = None
+            if request.image_url:
+                logger.info(f"Downloading image from URL: {request.image_url}")
+                if not self.validate_image_url(str(request.image_url)):
+                    logger.warning(f"Image URL may not be valid: {request.image_url}")
+                
+                # Determine image extension from URL or default to .jpg
+                parsed = urlparse(str(request.image_url))
+                ext = os.path.splitext(parsed.path)[1] or ".jpg"
+                image_path = await self.download_file(str(request.image_url), ext)
+            
+            # Create temporary input file for inference
+            with tempfile.NamedTemporaryFile(mode='w', suffix='.txt', delete=False) as f:
+                if image_path:
+                    input_line = f"{request.prompt}@@{image_path}@@{audio_path}"
+                else:
+                    input_line = f"{request.prompt}@@@@{audio_path}"
+                f.write(input_line)
+                temp_input_file = f.name
+            
+            # Prepare inference command
+            cmd = [
+                "python", "-m", "torch.distributed.run",
+                "--standalone", f"--nproc_per_node={request.sp_size}",
+                "scripts/inference.py",
+                "--config", "configs/inference.yaml",
+                "--input_file", temp_input_file,
+                "--guidance_scale", str(request.guidance_scale),
+                "--audio_scale", str(request.audio_scale),
+                "--num_steps", str(request.num_steps)
+            ]
+            
+            if request.tea_cache_l1_thresh:
+                cmd.extend(["--tea_cache_l1_thresh", str(request.tea_cache_l1_thresh)])
+            
+            logger.info(f"Running inference with command: {' '.join(cmd)}")
+            
+            # Run inference
+            result = subprocess.run(cmd, capture_output=True, text=True)
+            
+            # Clean up temporary files
+            os.unlink(temp_input_file)
+            os.unlink(audio_path)
+            if image_path:
+                os.unlink(image_path)
+            
+            if result.returncode != 0:
+                logger.error(f"Inference failed: {result.stderr}")
+                raise Exception(f"Inference failed: {result.stderr}")
+            
+            # Find output video file
+            output_dir = "./outputs"
+            if os.path.exists(output_dir):
+                video_files = [f for f in os.listdir(output_dir) if f.endswith(('.mp4', '.avi'))]
+                if video_files:
+                    # Return the most recent video file
+                    video_files.sort(key=lambda x: os.path.getmtime(os.path.join(output_dir, x)), reverse=True)
+                    output_path = os.path.join(output_dir, video_files[0])
+                    processing_time = time.time() - start_time
+                    return output_path, processing_time, audio_generated
+            
+            raise Exception("No output video generated")
+            
+        except Exception as e:
+            # Clean up any temporary files in case of error
+            try:
+                if 'audio_path' in locals() and audio_path and os.path.exists(audio_path):
+                    os.unlink(audio_path)
+                if 'image_path' in locals() and image_path and os.path.exists(image_path):
+                    os.unlink(image_path)
+                if 'temp_input_file' in locals() and os.path.exists(temp_input_file):
+                    os.unlink(temp_input_file)
+            except:
+                pass
+            
+            logger.error(f"Generation error: {str(e)}")
+            raise HTTPException(status_code=500, detail=str(e))
+
+# Initialize API
+omni_api = OmniAvatarAPI()
+
+@app.on_event("startup")
+async def startup_event():
+    """Load model on startup"""
+    success = omni_api.load_model()
+    if not success:
+        logger.warning("Model loading failed on startup")
+
+@app.get("/health")
+async def health_check():
+    """Health check endpoint"""
+    return {
+        "status": "healthy",
+        "model_loaded": omni_api.model_loaded,
+        "device": omni_api.device,
+        "supports_elevenlabs": True,
+        "supports_image_urls": True,
+        "supports_text_to_speech": True,
+        "elevenlabs_api_configured": bool(omni_api.elevenlabs_client.api_key)
+    }
+
+@app.post("/generate", response_model=GenerateResponse)
+async def generate_avatar(request: GenerateRequest):
+    """Generate avatar video from prompt, text/audio, and optional image URL"""
+    
+    if not omni_api.model_loaded:
+        raise HTTPException(status_code=503, detail="Model not loaded")
+    
+    logger.info(f"Generating avatar with prompt: {request.prompt}")
+    if request.text_to_speech:
+        logger.info(f"Text to speech: {request.text_to_speech[:100]}...")
+        logger.info(f"Voice ID: {request.voice_id}")
+    if request.elevenlabs_audio_url:
+        logger.info(f"Audio URL: {request.elevenlabs_audio_url}")
+    if request.image_url:
+        logger.info(f"Image URL: {request.image_url}")
+    
+    try:
+        output_path, processing_time, audio_generated = await omni_api.generate_avatar(request)
+        
+        return GenerateResponse(
+            message="Avatar generation completed successfully",
+            output_path=output_path,
+            processing_time=processing_time,
+            audio_generated=audio_generated
+        )
+        
+    except HTTPException:
+        raise
+    except Exception as e:
+        logger.error(f"Unexpected error: {e}")
+        raise HTTPException(status_code=500, detail=f"Unexpected error: {e}")
+
+# Enhanced Gradio interface with text-to-speech option
+def gradio_generate(prompt, text_to_speech, audio_url, image_url, voice_id, guidance_scale, audio_scale, num_steps):
+    """Gradio interface wrapper with text-to-speech support"""
+    if not omni_api.model_loaded:
+        return "Error: Model not loaded"
+    
+    try:
+        # Create request object
+        request_data = {
+            "prompt": prompt,
+            "guidance_scale": guidance_scale,
+            "audio_scale": audio_scale,
+            "num_steps": int(num_steps)
+        }
+        
+        # Add audio source
+        if text_to_speech and text_to_speech.strip():
+            request_data["text_to_speech"] = text_to_speech
+            request_data["voice_id"] = voice_id or "21m00Tcm4TlvDq8ikWAM"
+        elif audio_url and audio_url.strip():
+            request_data["elevenlabs_audio_url"] = audio_url
+        else:
+            return "Error: Please provide either text to speech or audio URL"
+        
+        if image_url and image_url.strip():
+            request_data["image_url"] = image_url
+        
+        request = GenerateRequest(**request_data)
+        
+        # Run async function in sync context
+        loop = asyncio.new_event_loop()
+        asyncio.set_event_loop(loop)
+        output_path, processing_time, audio_generated = loop.run_until_complete(omni_api.generate_avatar(request))
+        loop.close()
+        
+        return output_path
+        
+    except Exception as e:
+        logger.error(f"Gradio generation error: {e}")
+        return f"Error: {str(e)}"
+
+# Updated Gradio interface with text-to-speech support
+iface = gr.Interface(
+    fn=gradio_generate,
+    inputs=[
+        gr.Textbox(
+            label="Prompt", 
+            placeholder="Describe the character behavior (e.g., 'A friendly person explaining a concept')",
+            lines=2
+        ),
+        gr.Textbox(
+            label="Text to Speech", 
+            placeholder="Enter text to convert to speech using ElevenLabs",
+            lines=3,
+            info="This will be converted to speech automatically"
+        ),
+        gr.Textbox(
+            label="OR Audio URL", 
+            placeholder="https://api.elevenlabs.io/v1/text-to-speech/...",
+            info="Direct URL to audio file (alternative to text-to-speech)"
+        ),
+        gr.Textbox(
+            label="Image URL (Optional)", 
+            placeholder="https://example.com/image.jpg",
+            info="Direct URL to reference image (JPG, PNG, etc.)"
+        ),
+        gr.Dropdown(
+            choices=["21m00Tcm4TlvDq8ikWAM", "pNInz6obpgDQGcFmaJgB", "EXAVITQu4vr4xnSDxMaL"],
+            value="21m00Tcm4TlvDq8ikWAM",
+            label="ElevenLabs Voice ID",
+            info="Choose voice for text-to-speech"
+        ),
+        gr.Slider(minimum=1, maximum=10, value=5.0, label="Guidance Scale", info="4-6 recommended"),
+        gr.Slider(minimum=1, maximum=10, value=3.0, label="Audio Scale", info="Higher values = better lip-sync"),
+        gr.Slider(minimum=10, maximum=100, value=30, step=1, label="Number of Steps", info="20-50 recommended")
+    ],
+    outputs=gr.Video(label="Generated Avatar Video"),
+    title="🎭 OmniAvatar-14B with ElevenLabs TTS",
+    description="""
+    Generate avatar videos with lip-sync from text prompts and speech.
+    
+    **Features:**
+    - ✅ **Text-to-Speech**: Enter text to generate speech automatically
+    - ✅ **ElevenLabs Integration**: High-quality voice synthesis  
+    - ✅ **Audio URL Support**: Use pre-generated audio files
+    - ✅ **Image URL Support**: Reference images for character appearance
+    - ✅ **Customizable Parameters**: Fine-tune generation quality
+    
+    **Usage:**
+    1. Enter a character description in the prompt
+    2. **Either** enter text for speech generation **OR** provide an audio URL
+    3. Optionally add a reference image URL
+    4. Choose voice and adjust parameters
+    5. Generate your avatar video!
+    
+    **Tips:**
+    - Use guidance scale 4-6 for best prompt following
+    - Increase audio scale for better lip-sync
+    - Clear, descriptive prompts work best
+    """,
+    examples=[
+        [
+            "A professional teacher explaining a mathematical concept with clear gestures",
+            "Hello students! Today we're going to learn about calculus and how derivatives work in real life.",
+            "",
+            "https://example.com/teacher.jpg",
+            "21m00Tcm4TlvDq8ikWAM",
+            5.0,
+            3.5,
+            30
+        ],
+        [
+            "A friendly presenter speaking confidently to an audience",
+            "Welcome everyone to our presentation on artificial intelligence and its applications!",
+            "",
+            "",
+            "pNInz6obpgDQGcFmaJgB", 
+            5.5,
+            4.0,
+            35
+        ]
+    ]
+)
+
+# Mount Gradio app
+app = gr.mount_gradio_app(app, iface, path="/gradio")
+
+if __name__ == "__main__":
+    import uvicorn
+    uvicorn.run(app, host="0.0.0.0", port=7860)

get_voices.ps1

@@ -0,0 +1,29 @@
+# Script to get ElevenLabs voice IDs
+Write-Host "Getting ElevenLabs Voice IDs..." -ForegroundColor Yellow
+
+# You'll need your ElevenLabs API key for this
+$apiKey = Read-Host "Enter your ElevenLabs API Key (or press Enter to skip)"
+
+if ($apiKey) {
+    try {
+        $headers = @{
+            "xi-api-key" = $apiKey
+            "Content-Type" = "application/json"
+        }
+        
+        $response = Invoke-RestMethod -Uri "https://api.elevenlabs.io/v1/voices" -Headers $headers -Method GET
+        
+        Write-Host "`n✅ Available Voices:" -ForegroundColor Green
+        foreach ($voice in $response.voices) {
+            Write-Host "Name: $($voice.name)" -ForegroundColor Cyan
+            Write-Host "ID: $($voice.voice_id)" -ForegroundColor White
+            Write-Host "Category: $($voice.category)" -ForegroundColor Gray
+            Write-Host "Description: $($voice.description)" -ForegroundColor Gray
+            Write-Host "---" -ForegroundColor DarkGray
+        }
+    } catch {
+        Write-Host "❌ Error getting voices: $($_.Exception.Message)" -ForegroundColor Red
+    }
+} else {
+    Write-Host "Skipping API call - showing default voice IDs instead" -ForegroundColor Yellow
+}

hf_tts_client.py

@@ -0,0 +1,115 @@
+import torch
+import tempfile
+import logging
+import soundfile as sf
+import numpy as np
+from transformers import SpeechT5Processor, SpeechT5ForTextToSpeech, SpeechT5HifiGan
+from datasets import load_dataset
+import asyncio
+from typing import Optional
+
+logger = logging.getLogger(__name__)
+
+class HuggingFaceTTSClient:
+    """
+    Hugging Face TTS client using Microsoft SpeechT5
+    Replaces ElevenLabs with free, open-source TTS
+    """
+    
+    def __init__(self):
+        self.device = "cuda" if torch.cuda.is_available() else "cpu"
+        self.processor = None
+        self.model = None
+        self.vocoder = None
+        self.speaker_embeddings = None
+        self.model_loaded = False
+        
+        logger.info(f"HF TTS Client initialized on device: {self.device}")
+        
+    async def load_model(self):
+        """Load SpeechT5 model and vocoder"""
+        try:
+            logger.info("Loading SpeechT5 TTS model...")
+            
+            # Load processor, model and vocoder
+            self.processor = SpeechT5Processor.from_pretrained("microsoft/speecht5_tts")
+            self.model = SpeechT5ForTextToSpeech.from_pretrained("microsoft/speecht5_tts").to(self.device)
+            self.vocoder = SpeechT5HifiGan.from_pretrained("microsoft/speecht5_hifigan").to(self.device)
+            
+            # Load speaker embeddings dataset
+            embeddings_dataset = load_dataset("Matthijs/cmu-arctic-xvectors", split="validation")
+            self.speaker_embeddings = torch.tensor(embeddings_dataset[7306]["xvector"]).unsqueeze(0).to(self.device)
+            
+            self.model_loaded = True
+            logger.info("✅ SpeechT5 TTS model loaded successfully")
+            return True
+            
+        except Exception as e:
+            logger.error(f"❌ Failed to load TTS model: {e}")
+            return False
+    
+    async def text_to_speech(self, text: str, voice_id: Optional[str] = None) -> str:
+        """
+        Convert text to speech using SpeechT5
+        
+        Args:
+            text: Text to convert to speech
+            voice_id: Voice identifier (for compatibility, maps to speaker embeddings)
+            
+        Returns:
+            Path to generated audio file
+        """
+        if not self.model_loaded:
+            logger.info("Model not loaded, loading now...")
+            success = await self.load_model()
+            if not success:
+                raise Exception("Failed to load TTS model")
+        
+        try:
+            logger.info(f"Generating speech for text: {text[:50]}...")
+            
+            # Choose speaker embedding based on voice_id (for variety)
+            speaker_idx = self._get_speaker_index(voice_id)
+            embeddings_dataset = load_dataset("Matthijs/cmu-arctic-xvectors", split="validation")
+            speaker_embeddings = torch.tensor(embeddings_dataset[speaker_idx]["xvector"]).unsqueeze(0).to(self.device)
+            
+            # Process text
+            inputs = self.processor(text=text, return_tensors="pt").to(self.device)
+            
+            # Generate speech
+            with torch.no_grad():
+                speech = self.model.generate_speech(
+                    inputs["input_ids"], 
+                    speaker_embeddings, 
+                    vocoder=self.vocoder
+                )
+            
+            # Convert to audio file
+            audio_data = speech.cpu().numpy()
+            
+            # Save to temporary file
+            temp_file = tempfile.NamedTemporaryFile(delete=False, suffix='.wav')
+            sf.write(temp_file.name, audio_data, samplerate=16000)
+            temp_file.close()
+            
+            logger.info(f"✅ Generated speech audio: {temp_file.name}")
+            return temp_file.name
+            
+        except Exception as e:
+            logger.error(f"❌ Error generating speech: {e}")
+            raise Exception(f"TTS generation failed: {e}")
+    
+    def _get_speaker_index(self, voice_id: Optional[str]) -> int:
+        """Map voice_id to speaker embedding index for voice variety"""
+        voice_mapping = {
+            # Map ElevenLabs voice IDs to speaker indices for compatibility
+            "21m00Tcm4TlvDq8ikWAM": 7306,  # Female voice (default)
+            "pNInz6obpgDQGcFmaJgB": 4077,  # Male voice  
+            "EXAVITQu4vr4xnSDxMaL": 1995,  # Female voice (sweet)
+            "ErXwobaYiN019PkySvjV": 8051,  # Male voice (professional)
+            "TxGEqnHWrfWFTfGW9XjX": 5688,  # Deep male voice
+            "yoZ06aMxZJJ28mfd3POQ": 3570,  # Friendly voice
+            "AZnzlk1XvdvUeBnXmlld": 2967,  # Strong female
+        }
+        
+        return voice_mapping.get(voice_id, 7306)  # Default to female voice

requirements.txt

@@ -42,3 +42,11 @@ python-dotenv>=1.0.0
 # Additional dependencies for HF Spaces
 huggingface-hub>=0.17.0
 safetensors>=0.4.0
+
+# Hugging Face TTS (replacing ElevenLabs)
+transformers>=4.21.0
+torch>=2.0.0
+torchaudio>=2.0.0
+speechbrain>=0.5.0
+datasets>=2.0.0
+soundfile>=0.12.0

test_api.ps1

@@ -0,0 +1,31 @@
+# Test your HF Space API
+$baseUrl = "https://bravedims-ai-avatar-chat.hf.space"
+
+Write-Host "Testing HF Space API..." -ForegroundColor Yellow
+Write-Host "Base URL: $baseUrl" -ForegroundColor Cyan
+
+# Test health endpoint
+try {
+    Write-Host "`nTesting health endpoint..." -ForegroundColor Green
+    $healthResponse = Invoke-RestMethod -Uri "$baseUrl/health" -Method GET -TimeoutSec 30
+    Write-Host "✅ Health Check Response:" -ForegroundColor Green
+    $healthResponse | ConvertTo-Json -Depth 3
+} catch {
+    Write-Host "❌ Health check failed: $($_.Exception.Message)" -ForegroundColor Red
+    Write-Host "This might mean the Space is still building or not running yet." -ForegroundColor Yellow
+}
+
+# Test if Space exists (even if not running)
+try {
+    Write-Host "`nTesting if Space URL exists..." -ForegroundColor Green
+    $response = Invoke-WebRequest -Uri $baseUrl -Method GET -TimeoutSec 30 -ErrorAction SilentlyContinue
+    Write-Host "✅ Space URL is accessible (Status: $($response.StatusCode))" -ForegroundColor Green
+} catch {
+    Write-Host "❌ Space URL not accessible: $($_.Exception.Message)" -ForegroundColor Red
+}
+
+Write-Host "`n📋 Your API Information:" -ForegroundColor Magenta
+Write-Host "Base URL: $baseUrl" -ForegroundColor White
+Write-Host "Health: GET $baseUrl/health" -ForegroundColor White  
+Write-Host "Generate: POST $baseUrl/generate" -ForegroundColor White
+Write-Host "Gradio UI: $baseUrl/gradio" -ForegroundColor White

test_audio_url.ps1

@@ -0,0 +1,24 @@
+# Test using direct audio URL instead of text-to-speech
+Write-Host "🔄 Testing with direct audio URL (bypassing ElevenLabs)..." -ForegroundColor Yellow
+
+$audioTestPayload = @{
+    prompt = "A professional teacher explaining a mathematical concept with clear gestures"
+    elevenlabs_audio_url = "https://www.soundjay.com/misc/sounds/bell-ringing-05.wav"
+    image_url = "https://i.ibb.co/8g4xryvS/531bd0d0c48b.png"
+    guidance_scale = 5.0
+    audio_scale = 3.5
+    num_steps = 30
+} | ConvertTo-Json -Depth 3
+
+Write-Host "Testing with audio URL instead of TTS..." -ForegroundColor Cyan
+Write-Host $audioTestPayload -ForegroundColor Gray
+
+try {
+    $headers = @{"Content-Type" = "application/json"}
+    $response = Invoke-RestMethod -Uri "https://bravedims-ai-avatar-chat.hf.space/generate" -Method POST -Body $audioTestPayload -Headers $headers -TimeoutSec 120
+    
+    Write-Host "✅ Success with audio URL!" -ForegroundColor Green
+    $response | ConvertTo-Json -Depth 3
+} catch {
+    Write-Host "❌ Still failing: $($_.Exception.Message)" -ForegroundColor Red
+}

test_elevenlabs.ps1

@@ -0,0 +1,50 @@
+# Test ElevenLabs API directly
+Write-Host "🧪 Testing ElevenLabs API Integration..." -ForegroundColor Yellow
+
+# Test 1: Check if your API is accessible
+try {
+    Write-Host "`n1. Testing API health..." -ForegroundColor Cyan
+    $health = Invoke-RestMethod -Uri "https://bravedims-ai-avatar-chat.hf.space/health" -Method GET
+    Write-Host "✅ API Status: $($health.status)" -ForegroundColor Green
+    Write-Host "✅ ElevenLabs Configured: $($health.elevenlabs_api_configured)" -ForegroundColor Green
+} catch {
+    Write-Host "❌ API Health Check Failed: $($_.Exception.Message)" -ForegroundColor Red
+}
+
+# Test 2: Try a simple generate request with better voice ID
+try {
+    Write-Host "`n2. Testing generation with Rachel voice (most reliable)..." -ForegroundColor Cyan
+    
+    $testPayload = @{
+        prompt = "A simple test"
+        text_to_speech = "This is a test message."
+        voice_id = "21m00Tcm4TlvDq8ikWAM"
+        guidance_scale = 5.0
+        audio_scale = 3.5
+        num_steps = 20
+    } | ConvertTo-Json -Depth 3
+    
+    Write-Host "Payload:" -ForegroundColor Gray
+    Write-Host $testPayload -ForegroundColor White
+    
+    $headers = @{"Content-Type" = "application/json"}
+    $response = Invoke-RestMethod -Uri "https://bravedims-ai-avatar-chat.hf.space/generate" -Method POST -Body $testPayload -Headers $headers -TimeoutSec 120
+    
+    Write-Host "✅ Generation successful!" -ForegroundColor Green
+    $response | ConvertTo-Json -Depth 3
+    
+} catch {
+    Write-Host "❌ Generation failed: $($_.Exception.Message)" -ForegroundColor Red
+    if ($_.Exception.Response) {
+        Write-Host "Status Code: $($_.Exception.Response.StatusCode)" -ForegroundColor Yellow
+        $reader = New-Object System.IO.StreamReader($_.Exception.Response.GetResponseStream())
+        $responseBody = $reader.ReadToEnd()
+        Write-Host "Response Body: $responseBody" -ForegroundColor Yellow
+    }
+}
+
+Write-Host "`n📋 Common ElevenLabs Issues:" -ForegroundColor Magenta
+Write-Host "1. API Key expired or invalid" -ForegroundColor White
+Write-Host "2. Voice ID doesn't exist in your account" -ForegroundColor White  
+Write-Host "3. Rate limit exceeded" -ForegroundColor White
+Write-Host "4. Account credit/quota exhausted" -ForegroundColor White

test_fixes.ps1

@@ -0,0 +1,18 @@
+# Test the health endpoint to check API key status
+Write-Host "Testing API after fixes..." -ForegroundColor Yellow
+
+try {
+    $healthResponse = Invoke-RestMethod -Uri "https://bravedims-ai-avatar-chat.hf.space/health" -Method GET -TimeoutSec 30
+    Write-Host "✅ Health Check Response:" -ForegroundColor Green
+    $healthResponse | ConvertTo-Json -Depth 3
+    
+    # Check if ElevenLabs is properly configured
+    if ($healthResponse.elevenlabs_api_configured -eq $true) {
+        Write-Host "`n✅ ElevenLabs API is configured!" -ForegroundColor Green
+    } else {
+        Write-Host "`n❌ ElevenLabs API key still not configured" -ForegroundColor Red
+        Write-Host "👉 You need to add ELEVENLABS_API_KEY to your HF Space secrets" -ForegroundColor Yellow
+    }
+} catch {
+    Write-Host "❌ Error: $($_.Exception.Message)" -ForegroundColor Red
+}

test_generate.ps1

@@ -0,0 +1,31 @@
+# Test the generate endpoint with your JSON
+$apiUrl = "https://bravedims-ai-avatar-chat.hf.space/generate"
+$jsonPayload = @{
+    prompt = "A professional teacher explaining a mathematical concept with clear gestures"
+    text_to_speech = "Hello students! Today we're going to learn about calculus and how derivatives work in real life."
+    voice_id = "21m00Tcm4TlvDq8ikWAM"
+    image_url = "https://example.com/teacher.jpg"
+    guidance_scale = 5.0
+    audio_scale = 3.5
+    num_steps = 30
+} | ConvertTo-Json -Depth 3
+
+$headers = @{
+    "Content-Type" = "application/json"
+}
+
+Write-Host "Testing generate endpoint..." -ForegroundColor Yellow
+Write-Host "URL: $apiUrl" -ForegroundColor Cyan
+Write-Host "Payload:" -ForegroundColor Green
+Write-Host $jsonPayload -ForegroundColor White
+
+try {
+    $response = Invoke-RestMethod -Uri $apiUrl -Method POST -Body $jsonPayload -Headers $headers -TimeoutSec 120
+    Write-Host "`n✅ Success! Response:" -ForegroundColor Green
+    $response | ConvertTo-Json -Depth 3
+} catch {
+    Write-Host "`n❌ Error: $($_.Exception.Message)" -ForegroundColor Red
+    if ($_.Exception.Response) {
+        Write-Host "Status Code: $($_.Exception.Response.StatusCode)" -ForegroundColor Yellow
+    }
+}

test_hf_tts.py

@@ -0,0 +1,23 @@
+# Test script for HuggingFace TTS
+import asyncio
+import logging
+from hf_tts_client import HuggingFaceTTSClient
+
+logging.basicConfig(level=logging.INFO)
+
+async def test_hf_tts():
+    print("🧪 Testing HuggingFace TTS Client...")
+    
+    client = HuggingFaceTTSClient()
+    
+    try:
+        # Test TTS generation
+        audio_path = await client.text_to_speech("Hello, this is a test of HuggingFace TTS!")
+        print(f"✅ TTS Success! Audio saved to: {audio_path}")
+        return True
+    except Exception as e:
+        print(f"❌ TTS Failed: {e}")
+        return False
+
+if __name__ == "__main__":
+    asyncio.run(test_hf_tts())

voice_ids_reference.txt

@@ -0,0 +1,32 @@
+# ElevenLabs Voice ID Reference
+
+## ✅ Ready-to-Use Voice IDs:
+
+**Most Popular (Rachel - Clear Female):**
+21m00Tcm4TlvDq8ikWAM
+
+**Professional Male (Adam):**  
+pNInz6obpgDQGcFmaJgB
+
+**Professional Male (Antoni):**
+ErXwobaYiN019PkySvjV
+
+**Sweet Female (Bella):**
+EXAVITQu4vr4xnSDxMaL
+
+**Deep Male (Josh):**
+TxGEqnHWrfWFTfGW9XjX
+
+**Friendly Male (Sam):**
+yoZ06aMxZJJ28mfd3POQ
+
+**Strong Female (Domi):**
+AZnzlk1XvdvUeBnXmlld
+
+## 🧪 Test Your API with Different Voices:
+
+1. Rachel (Default): "21m00Tcm4TlvDq8ikWAM" 
+2. Adam (Professional): "pNInz6obpgDQGcFmaJgB"
+3. Bella (Sweet): "EXAVITQu4vr4xnSDxMaL"
+
+Just copy any of these IDs into your API request!