anderlecht-online.be
robots.txt

Robots Exclusion Standard data for anderlecht-online.be

Resource Scan

Scan Details

Site Domain anderlecht-online.be
Base Domain anderlecht-online.be
Scan Status Ok
Last Scan2024-11-14T18:59:30+00:00
Next Scan 2024-11-21T18:59:30+00:00

Last Scan

Scanned2024-11-14T18:59:30+00:00
URL https://anderlecht-online.be/robots.txt
Redirect https://www.anderlecht-online.be/robots.txt
Redirect Domain www.anderlecht-online.be
Redirect Base anderlecht-online.be
Domain IPs 104.26.2.81, 104.26.3.81, 172.67.70.142, 2606:4700:20::681a:251, 2606:4700:20::681a:351, 2606:4700:20::ac43:468e
Redirect IPs 104.26.2.81, 104.26.3.81, 172.67.70.142, 2606:4700:20::681a:251, 2606:4700:20::681a:351, 2606:4700:20::ac43:468e
Response IP 104.26.2.81
Found Yes
Hash 3419854445ab326419bcdb6fe2a6307ed0942fad99b8c2eed0362af4534b0aaa
SimHash 3d5d8d840fd1

Groups

*

Rule Path
Disallow /admin/
Disallow /mailchimp/
Disallow /memcache/
Disallow /newlayout/
Disallow /Old_layout/
Disallow /phpFlickr-3.1/
Disallow /PHPMailer-master/
Disallow /members/
Disallow /bulletproof-master/
Disallow /nl/articlecatcher/203069

Comments

  • For domain: http://www.anderlecht-online.be - robots.txt
  • All robots will spider the domain, besides on the following directories