Log In Sign Up

comprehensive-urology.com
robots.txt

Robots Exclusion Standard data for comprehensive-urology.com

Resource Scan

Scan Details

Site Domain comprehensive-urology.com
Base Domain comprehensive-urology.com
Scan Status Ok
Last Scan2025-10-30T02:33:19+00:00
Next Scan 2025-11-29T02:33:19+00:00

Last Scan

Scanned2025-10-30T02:33:19+00:00
URL https://comprehensive-urology.com/robots.txt
Domain IPs 141.193.213.10, 141.193.213.11
Response IP 141.193.213.10
Found Yes
Hash 77cdc65815baf5860aa6b8134358609b9d34547799b1a0b78ddada0f3f69d772
SimHash 4884d621e0b7

Groups

googlebot

Rule Path
Disallow

googlebot-mobile

Rule Path
Disallow

googlebot-image

Rule Path
Disallow

googlebot-news

Rule Path
Disallow

googlebot-video

Rule Path
Disallow

alexacrawler

Rule Path
Disallow

bingbot

Rule Path
Disallow

bingpreview

Rule Path
Disallow

duckduckbot

Rule Path
Disallow

yahoo! slurp

Rule Path
Disallow

semrushbot

Rule Path
Disallow

screaming frog seo spider

Rule Path
Disallow

searchatlas

Rule Path
Disallow

otto

Rule Path
Disallow

koalalinks

Rule Path
Disallow

facebot

Rule Path
Disallow

facebook external hit

Rule Path
Disallow

linkedinbot

Rule Path
Disallow

chatgpt

Rule Path
Disallow

gptbot

Rule Path
Disallow

chatpgpt-user

Rule Path
Disallow

openai

Rule Path
Disallow

applebot

Rule Path
Disallow

claudebot

Rule Path
Disallow

pinterestbot

Rule Path
Disallow

instagram

Rule Path
Disallow

msnbot

Rule Path
Disallow

*

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://comprehensive-urology.com/sitemap_index.xml
Back to top

Comments

  • Allow whitelisted bots
  • Block all others and apply crawl delay
  • Sitemap
Back to top