sffworld.com
robots.txt

Robots Exclusion Standard data for sffworld.com

Resource Scan

Scan Details

Site Domain sffworld.com
Base Domain sffworld.com
Scan Status Ok
Last Scan2024-10-04T01:08:19+00:00
Next Scan 2024-10-11T01:08:19+00:00

Last Scan

Scanned2024-10-04T01:08:19+00:00
URL https://sffworld.com/robots.txt
Domain IPs 209.59.164.57
Response IP 209.59.164.57
Found Yes
Hash 99ff6c697431c848f5d49218d4d0c5c69d623ec9d41acf3efde98bd9af316949
SimHash 526ed0324da9

Groups

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 80

blekkobot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 80

applebot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

awariosmartbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

senutobot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

mojeekbot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

bidswitchbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

hubspot links crawler 1.0

Rule Path
Disallow /

hubspot webcrawler

Rule Path
Disallow /

proximic

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

hubspot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

sogou blog

Rule Path
Disallow /

sogou inst spider

Rule Path
Disallow /

sogou news spider

Rule Path
Disallow /

sogou orion spider

Rule Path
Disallow /

sogou spider2

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

boardreader

Rule Path
Disallow /

googlebot-news

Rule Path
Disallow /forum/
Disallow /forums/
Disallow /community/
Disallow /authors/

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 80

baiduspider

Rule Path
Disallow /

voilabot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 80

boardtracker

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 80

*

Rule Path
Disallow /forum/find-new/
Disallow /forum/members/
Disallow /forum/attachments/
Disallow /forum/login/
Allow /