Log In Sign Up

towingdepot.com
robots.txt

Robots Exclusion Standard data for towingdepot.com

Resource Scan

Scan Details

Site Domain towingdepot.com
Base Domain towingdepot.com
Scan Status Ok
Last Scan2026-01-15T19:42:44+00:00
Next Scan 2026-02-14T19:42:44+00:00

Last Scan

Scanned2026-01-15T19:42:44+00:00
URL https://towingdepot.com/robots.txt
Domain IPs 104.21.48.242, 172.67.157.4, 2606:4700:3030::6815:30f2, 2606:4700:3037::ac43:9d04
Response IP 172.67.157.4
Found Yes
Hash 5f5662e8c85a4d4779d9a696c88d2b8e1f2b2ee45fd1468d40aa064d94aabe9b
SimHash 681ada23f6e6

Groups

*

Rule Path
Disallow /dealer

googlebot
googlebot-image
googlebot-video
googlebot-news

Rule Path
Disallow /dealer

bingbot
bingpreview

Rule Path
Disallow /dealer

duckduckbot

Rule Path
Disallow /dealer

baiduspider

Rule Path
Disallow /dealer

yandexbot

Rule Path
Disallow /dealer

petalbot

Rule Path
Disallow /dealer

sogou

Rule Path
Disallow /dealer

seznambot

Rule Path
Disallow /dealer

exabot

Rule Path
Disallow /dealer

coccocbot

Rule Path
Disallow /dealer

gptbot

Product Comment
gptbot OpenAI
Rule Path
Disallow /

chatgpt-user

Product Comment
chatgpt-user OpenAI browsing helper UA
Rule Path
Disallow /

google-extended

Product Comment
google-extended Google AI training extension
Rule Path
Disallow /

applebot-extended

Product Comment
applebot-extended Apple AI training extension
Rule Path
Disallow /

anthropic-ai

Product Comment
anthropic-ai Anthropic
Rule Path
Disallow /

claudebot

Product Comment
claudebot Anthropic alt UA
Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

linkedinbot

Rule Path
Disallow /

youbot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

ai2bot

Rule Path
Disallow /

phindbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

siteauditbot

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

scribbrbot

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

nimbostratus-bot

Rule Path
Disallow /

python-requests

Rule Path
Disallow /

python-urllib

Rule Path
Disallow /

curl

Rule Path
Disallow /

wget

Rule Path
Disallow /

libwww-perl

Rule Path
Disallow /

go-http-client

Rule Path
Disallow /

java

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

crawler4j

Rule Path
Disallow /

httpx

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://towingdepot.com/sitemap.xml
Back to top

Comments

  • ----------------------------
  • Sitemaps
  • ----------------------------
  • ----------------------------
  • Global rule: never crawl /dealer
  • ----------------------------
  • ----------------------------
  • Allow mainstream search engines (except /dealer above)
  • ----------------------------
  • ----------------------------
  • Explicit AI/data crawlers: BLOCK ALL
  • ----------------------------
Back to top