Log In Sign Up

winnfreenet.com
robots.txt

Robots Exclusion Standard data for winnfreenet.com

Resource Scan

Scan Details

Site Domain winnfreenet.com
Base Domain winnfreenet.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-06-05T07:59:46+00:00
Next Scan 2025-06-12T07:59:46+00:00

Last Successful Scan

Scanned2025-05-28T07:57:09+00:00
URL https://winnfreenet.com/robots.txt
Domain IPs 209.222.98.191
Response IP 209.222.98.191
Found Yes
Hash 2ef72ce8b523efc6b30013de4b396c444ca42f666dc7f24bd1306cd37447824c
SimHash 60309f41e881

Groups

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow

ccbot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

catchbot

Rule Path
Disallow /

twengabot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

spbot

Rule Path
Disallow /

itsapic.com_crawler

Rule Path
Disallow /

botmobi

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

exooba crawler

Rule Path
Disallow /

lapozzbot

Rule Path
Disallow /

cuill

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

twiceler

Rule Path
Disallow /

werelate

Rule Path
Disallow /

accelobot

Rule Path
Disallow /

iconsurf

Rule Path
Disallow /

steeler

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

volabot

Rule Path
Disallow /

voyager

Rule Path
Disallow /

voyager/1.0

Rule Path
Disallow /

yandex

Rule Path
Disallow /

irlbot

Rule Path
Disallow /

szukacz

Rule Path
Disallow /

findlinks

Rule Path
Disallow /

surveybot

Rule Path
Disallow /

http://www.almaden.ibm.com/cs/crawler

Rule Path
Disallow /

www.adressendeutschland.de

Rule Path
Disallow /

sbider

Rule Path
Disallow /

redkernel

Rule Path
Disallow /

redkernel www-spider

Rule Path
Disallow /

netresearchserver

Rule Path
Disallow /

nimblecrawler

Rule Path
Disallow /

ocelli

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

nextgensearchbot

Rule Path
Disallow /

psbot

Rule Path
Disallow /

plukkie

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 6

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

*

Rule Path
Allow /
Allow /shop/
Allow /wp/
Disallow /cgi-bin/
Disallow /contact.php
Disallow /submit.php
Disallow /submit2.php
Disallow /register.php
Disallow /wp/wp-admin/
Disallow /wp/wp-content/
Back to top

Other Records

Field Value
sitemap http://www.winnfreenet.com/wp/sitemap.xml.gz
Back to top

Comments

  • BEGIN XML-SITEMAP-PLUGIN
  • END XML-SITEMAP-PLUGIN
Back to top