Log In Sign Up

roadstr.fr
robots.txt

Robots Exclusion Standard data for roadstr.fr

Resource Scan

Scan Details

Site Domain roadstr.fr
Base Domain roadstr.fr
Scan Status Ok
Last Scan2024-09-03T06:14:03+00:00
Next Scan 2024-10-03T06:14:03+00:00

Last Scan

Scanned2024-09-03T06:14:03+00:00
URL https://roadstr.fr/robots.txt
Redirect https://www.roadstr.fr/robots.txt
Redirect Domain www.roadstr.fr
Redirect Base roadstr.fr
Domain IPs 2001:41d0:1:1b00:213:186:33:87, 213.186.33.87
Redirect IPs 108.128.72.146, 54.216.252.255, 54.73.26.109
Response IP 54.73.26.109
Found Yes
Hash 7e7242fabb9d3bd646af42b2bf0b30fa8a1274198c3f52e013a4209b500ccc33
SimHash 428c28c7be74

Groups

*

Rule Path
Disallow /*?*
Disallow *?more_reviews*
Allow /users/sign_up
Allow /users/sign_in
Allow /users/sign_up?from=rent_button
Allow /*?page
Allow /voitures?brand_name=

amazonbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

friendlycrawler

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

img2dataset

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

omgili

Rule Path
Disallow /

openai

Rule Path
Disallow /

spawning-ai

Rule Path
Disallow /

the knowledge ai

Rule Path
Disallow /

timpibot

Rule Path
Disallow /

webzio

Rule Path
Disallow /

youbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.roadstr.fr/sitemap.xml
sitemap https://www.roadstr.fr/blog/sitemap.xml
Back to top

Comments

  • See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
  • To ban all spiders from the entire site uncomment the next two lines:
Back to top