Log In Sign Up

rvtravel.com
robots.txt

Robots Exclusion Standard data for rvtravel.com

Resource Scan

Scan Details

Site Domain rvtravel.com
Base Domain rvtravel.com
Scan Status Ok
Last Scan2024-06-05T02:35:28+00:00
Next Scan 2024-07-05T02:35:28+00:00

Last Scan

Scanned2024-06-05T02:35:28+00:00
URL https://rvtravel.com/robots.txt
Redirect https://www.rvtravel.com/robots.txt
Redirect Domain www.rvtravel.com
Redirect Base rvtravel.com
Domain IPs 104.26.0.54, 104.26.1.54, 172.67.71.54, 2606:4700:20::681a:136, 2606:4700:20::681a:36, 2606:4700:20::ac43:4736
Redirect IPs 104.26.0.54, 104.26.1.54, 172.67.71.54, 2606:4700:20::681a:136, 2606:4700:20::681a:36, 2606:4700:20::ac43:4736
Response IP 104.26.1.54
Found Yes
Hash f4182ed3a54822832f17ba0e2f1ce69fa78489b3e01d296ac7e25f9a85594438
SimHash 06284965c722

Groups

*

Rule Path
Disallow
Disallow /cgi-bin/
Disallow /wp-admin/

siteauditbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

semrushbot-si

Rule Path
Allow /

msnbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 1

bingbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 1

ahrefsbot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 1

grapeshot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 1

comscore

Rule Path
Disallow

Other Records

Field Value
crawl-delay 1

baiduspider

Rule Path
Disallow /

baiduspider-image

Rule Path
Disallow /

baiduspider-mobile

Rule Path
Disallow /

baiduspider-news

Rule Path
Disallow /

baiduspider-video

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yandexmetrika

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

twitterbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

youbot

Rule Path
Disallow /

piplbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.rvtravel.com/sitemap_index.xml
Back to top

Comments

  • robots.txt for rvtravel.com
  • we are allowing most bots in
  • we are asking them to not scan out back-end
  • we are letting semrush in
  • we are slowing down some bots
  • we are blocking Chinese and Russian bots
  • we are blocking AI scrapers
Back to top