scriptgirl.nl
robots.txt

Robots Exclusion Standard data for scriptgirl.nl

Resource Scan

Scan Details

Site Domain scriptgirl.nl
Base Domain scriptgirl.nl
Scan Status Ok
Last Scan2024-11-11T01:01:23+00:00
Next Scan 2024-11-18T01:01:23+00:00

Last Scan

Scanned2024-11-11T01:01:23+00:00
URL https://scriptgirl.nl/robots.txt
Domain IPs 213.249.67.15, 2a01:448:2001::15
Response IP 213.249.67.15
Found Yes
Hash 17bb50be456aca9772a03ccc6150bc5729c49d4b43d6ab078e9efa22ab12e598
SimHash 6710d8176720

Groups

scrapy

Rule Path
Allow /

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10