forum.lu
robots.txt

Robots Exclusion Standard data for forum.lu

Resource Scan

Scan Details

Site Domain forum.lu
Base Domain forum.lu
Scan Status Ok
Last Scan2024-09-25T22:43:36+00:00
Next Scan 2024-10-25T22:43:36+00:00

Last Scan

Scanned2024-09-25T22:43:36+00:00
URL https://forum.lu/robots.txt
Domain IPs 95.217.192.177
Response IP 95.217.192.177
Found Yes
Hash be95fcdf3b547534735760f05a181bb18bd6fb58a7c73d2fb60892a419f75204
SimHash 6040d9c54675

Groups

*

Rule Path
Disallow /

googlebot
googlebot-image
mediapartners-google
msnbot
msnbot-media
yahoo-blogs
yahoo-mmcrawler

Rule Path
Disallow /wp-login.php

Other Records

Field Value
crawl-delay 20

bingbot
bing

Rule Path
Disallow /xmlrpc.php
Disallow /wp-login.php

Other Records

Field Value
crawl-delay 30

slurp

Rule Path
Disallow /wp-login.php

Other Records

Field Value
crawl-delay 10

Comments

  • disallow all
  • but allow only important bots