faperoni.com
robots.txt

Robots Exclusion Standard data for faperoni.com

Resource Scan

Scan Details

Site Domain faperoni.com
Base Domain faperoni.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-05-20T01:24:22+00:00
Next Scan 2024-06-03T01:24:22+00:00

Last Successful Scan

Scanned2024-04-12T00:07:07+00:00
URL https://faperoni.com/robots.txt
Domain IPs 104.27.201.89, 104.27.202.89, 2606:4700:21::681b:c959, 2606:4700:21::681b:ca59
Response IP 104.27.201.89
Found Yes
Hash ef8c323fb0858c2e531fec90e93c33136fcbb296fcfc276f53854cd75876864d
SimHash 4a54de23e325

Groups

*

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

googlebot

Rule Path
Allow /
Disallow /join/
Disallow /open/
Disallow /pbw/

bingbot

Rule Path
Allow /
Disallow /join/
Disallow /open/
Disallow /pbw/

slurp

Rule Path
Allow /
Disallow /join/
Disallow /open/
Disallow /pbw/

yandex

Rule Path
Allow /
Disallow /join/
Disallow /open/
Disallow /pbw/

baiduspider

Rule Path
Allow /
Disallow /join/
Disallow /open/
Disallow /pbw/

twitterbot

Rule Path
Allow /
Disallow /join/
Disallow /open/

facebookexternalhit

Rule Path
Allow /
Disallow /join/
Disallow /open/

facebot

Rule Path
Allow /
Disallow /join/
Disallow /open/

ahrefsbot

Rule Path
Allow /
Disallow /join/
Disallow /open/

Other Records

Field Value
crawl-delay 10

ahrefssiteaudit

Rule Path
Allow /
Disallow /join/
Disallow /open/

Warnings

  • `host` is not a known field.