Log In Sign Up

super-desh.com
robots.txt

Robots Exclusion Standard data for super-desh.com

Resource Scan

Scan Details

Site Domain super-desh.com
Base Domain super-desh.com
Scan Status Ok
Last Scan2025-05-26T09:30:14+00:00
Next Scan 2025-06-25T09:30:14+00:00

Last Scan

Scanned2025-05-26T09:30:14+00:00
URL https://super-desh.com/robots.txt
Domain IPs 18.156.88.174, 3.124.161.162, 3.74.190.245
Response IP 3.74.190.245
Found Yes
Hash 07f0b403a7be4e56e88ab57d17ce17d96f681ecffba3882bd67114b756e56d02
SimHash 38151a0363eb

Groups

*

Rule Path
Allow /
Disallow */login
Disallow */registration

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

semrushbot
semrushbot-sa
semrushbot-ba

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

petalbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

serpstatbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

oai-searchbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

facebookbot
facebookbot

Rule Path
Disallow /

googlebot
googlebot-image
googlebot-mobile
bingbot
applebot
yandexbot
duckduckbot

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://super-desh.com/sitemap.xml
Back to top

Comments

  • High-volume SEO crawlers - Rate limit
  • AI Bot Control - Restrict training on content
  • Social Media Bots
  • Always welcome Google, Bing, and other mainstream search engines
  • Sitemap URL for super-desh.com
Back to top