Log In Sign Up

volkswagen-abu-dhabi.com
robots.txt

Robots Exclusion Standard data for volkswagen-abu-dhabi.com

Resource Scan

Scan Details

Site Domain volkswagen-abu-dhabi.com
Base Domain volkswagen-abu-dhabi.com
Scan Status Ok
Last Scan2024-11-10T12:36:54+00:00
Next Scan 2024-12-10T12:36:54+00:00

Last Scan

Scanned2024-11-10T12:36:54+00:00
URL https://www.volkswagen-abu-dhabi.com/robots.txt
Domain IPs 108.157.254.129, 108.157.254.22, 108.157.254.58, 108.157.254.99, 2600:9000:2753:6800:e:3b11:e240:93a1, 2600:9000:2753:9800:e:3b11:e240:93a1, 2600:9000:2753:9e00:e:3b11:e240:93a1, 2600:9000:2753:a000:e:3b11:e240:93a1, 2600:9000:2753:a600:e:3b11:e240:93a1, 2600:9000:2753:ba00:e:3b11:e240:93a1, 2600:9000:2753:dc00:e:3b11:e240:93a1, 2600:9000:2753:e400:e:3b11:e240:93a1
Response IP 108.157.254.99
Found Yes
Hash be2f315865d8827a66a523c1e605b8f9a5ee413e44aaa9d7bdc02bfed3eebfe3
SimHash 29609bc4cdb5

Groups

googlebot

Rule Path
Allow /app/
Disallow /*.integrator-includes.txt
Disallow /*.integrator-includes.json

slurp

Rule Path
Allow /app/

bingbot

Rule Path
Allow /app/
Disallow /*.integrator-includes.txt
Disallow /*.integrator-includes.json

msnbot

Rule Path
Allow /app/
Disallow /*.integrator-includes.txt
Disallow /*.integrator-includes.json

*

Rule Path
Disallow /app/
Disallow /*notification
Disallow /*.integrator-includes.txt
Disallow /*.integrator-includes.json
Back to top

Other Records

Field Value
sitemap https://www.volkswagen-abu-dhabi.com/en.sitemap.xml
sitemap https://www.volkswagen-abu-dhabi.com/ar.sitemap.xml
Back to top

Comments

  • ROBOTS TXT
  • whitelist user agents / pages to visit for app-paths
  • default block
  • sitemap
Back to top