matome-plus.com
robots.txt

Robots Exclusion Standard data for matome-plus.com

Resource Scan

Scan Details

Site Domain matome-plus.com
Base Domain matome-plus.com
Scan Status Ok
Last Scan2024-05-25T09:57:42+00:00
Next Scan 2024-06-24T09:57:42+00:00

Last Scan

Scanned2024-05-25T09:57:42+00:00
URL https://matome-plus.com/robots.txt
Domain IPs 49.212.120.135
Response IP 49.212.120.135
Found Yes
Hash 5d4911f48f4625aa7c7e0fc590239f4ed09200931ca204dac158cdc2ab930ad3
SimHash 2614d8928213

Groups

baiduspider

Rule Path
Disallow /

shim-crawler

Rule Path
Disallow /

steeler

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

safaribookmarkchecker

Rule Path
Disallow /

becomebot

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 15

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 900

teoma

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 900

ask jeeves

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 900

yetibot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 900