Spaces:
Running
Running
File size: 337 Bytes
8bce833 6cca3b1 8bce833 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
User-agent: * Allow: / # Block access to API routes and private areas Disallow: /api/ Disallow: /auth/ Disallow: /_next/ Disallow: /assets/ # Allow specific public API endpoints if needed Allow: /api/public/ # Sitemap location Sitemap: https://huggingface.co/deepsite/sitemap.xml # Crawl-delay for respectful crawling Crawl-delay: 1 |