Log In Sign Up

mesaderegalos.liverpool.com.mx
robots.txt

Robots Exclusion Standard data for mesaderegalos.liverpool.com.mx

Resource Scan

Scan Details

Site Domain mesaderegalos.liverpool.com.mx
Base Domain liverpool.com.mx
Scan Status Ok
Last Scan2024-11-03T08:57:30+00:00
Next Scan 2024-12-03T08:57:30+00:00

Last Scan

Scanned2024-11-03T08:57:30+00:00
URL https://mesaderegalos.liverpool.com.mx/robots.txt
Domain IPs 104.83.196.185
Response IP 184.25.221.89
Found Yes
Hash 4f01665d78c3394c0d5e9b576e461c3c3bde8847f2b863060dffb4c22160c13b
SimHash 3442e7b26ff6

Groups

*

Rule Path
Allow /

googlebot

Rule Path
Disallow

adsbot-google

Rule Path
Disallow

googlebot-image

Rule Path
Disallow

googlebot-image

Rule Path
Disallow

googlebot-video

Rule Path
Disallow

googlebot-mobile

Rule Path
Disallow

mediapartners-google

Rule Path
Disallow
Disallow

cazoodlebot
mj12bot
dotbot/1.0
gigabot/2.0

No rules defined. All paths allowed.

Back to top

Comments

  • Version 2023.01.27
  • Mesa de regalos - liverpool.com.mx
  • For all robots
  • Allow specific Google robots
  • Block access to specific groups of pages (Landings)
  • Block access to specific groups of pages (Categories)
  • Block CazoodleBot as it does not present correct accept content headers
  • Block MJ12bot as it is just noise (Majestic)
  • Block dotbot as it cannot parse base urls properly (SEOMoz)
  • Block Gigabot (internal search engine bot)
Back to top

Warnings

  • `user agent` is not a known field.
Back to top