blog-nouvelles-technologies.fr
robots.txt

Robots Exclusion Standard data for blog-nouvelles-technologies.fr

Resource Scan

Scan Details

Site Domain blog-nouvelles-technologies.fr
Base Domain blog-nouvelles-technologies.fr
Scan Status Ok
Last Scan2024-05-26T18:47:15+00:00
Next Scan 2024-06-02T18:47:15+00:00

Last Scan

Scanned2024-05-26T18:47:15+00:00
URL https://blog-nouvelles-technologies.fr/robots.txt
Domain IPs 104.26.8.99, 104.26.9.99, 172.67.72.129, 2606:4700:20::681a:863, 2606:4700:20::681a:963, 2606:4700:20::ac43:4881
Response IP 104.26.8.99
Found Yes
Hash 498eec8a27d5964a528c8a7a9c2e0945ad03451a963073253e638a8e74b994cc
SimHash e90019966730

Groups

scrapy

Rule Path
Allow /

*

Rule Path
Disallow /wp-login.php
Disallow */trackback
Disallow /*/feed
Disallow /*/comments
Disallow /cgi-bin
Disallow /*.php$
Disallow /*.inc$
Disallow /*.gz
Disallow /*.cgi