Log In Sign Up

listminut.be
robots.txt

Robots Exclusion Standard data for listminut.be

Resource Scan

Scan Details

Site Domain listminut.be
Base Domain listminut.be
Scan Status Ok
Last Scan2024-05-22T16:27:08+00:00
Next Scan 2024-06-21T16:27:08+00:00

Last Scan

Scanned2024-05-22T16:27:08+00:00
URL https://listminut.be/robots.txt
Redirect https://listminut-docs.s3.eu-west-1.amazonaws.com/seo/robots.txt
Redirect Domain listminut-docs.s3.eu-west-1.amazonaws.com
Redirect Base amazonaws.com
Domain IPs 172.66.40.152, 172.66.43.104, 2606:4700:3108::ac42:2898, 2606:4700:3108::ac42:2b68
Redirect IPs 3.5.68.159, 3.5.68.6, 3.5.69.153, 3.5.69.66, 3.5.71.63, 52.218.100.240, 52.218.109.144, 52.92.20.186
Response IP 3.5.70.181
Found Yes
Hash 6080078e8c735724f8969ff0b2d0fbbd8b40c4a76d409606222ca6ac276c4792
SimHash 4802d9118231

Groups

*

Rule Path
Disallow */tasks/
Disallow */listworkers/
Disallow */experiment/
Disallow /*?noindex=1*

baiduspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

yahoo! slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

rogerbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

yandexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10
Back to top

Comments

  • Disallows
  • Block and delay
Back to top