Log In Sign Up

pc-freak.net
robots.txt

Robots Exclusion Standard data for pc-freak.net

Resource Scan

Scan Details

Site Domain pc-freak.net
Base Domain pc-freak.net
Scan Status Ok
Last Scan2024-09-28T05:10:25+00:00
Next Scan 2024-10-05T05:10:25+00:00

Last Scan

Scanned2024-09-28T05:10:25+00:00
URL https://pc-freak.net/robots.txt
Domain IPs 109.104.212.130, 213.91.190.233
Response IP 213.91.190.233
Found Yes
Hash 987af42a4575c295967dbb875a14cb6974e65c32be69cee02c1b0f2b51b0ed2e
SimHash 4098a96181f2

Groups

*

Rule Path
Disallow

wget
webzip
webmirror
webcopy
netants
getright
webcapture

Rule Path
Disallow /pictures/

megaindex.ru

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

youbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.pc-freak.net/sitemap.xml
Back to top

Comments

  • Allow all
  • robots.txt for pc-freak.net
  • Slow down bots
  • User-agent: *
  • mass download
  • User-agent: Libwww-perl
  • Disallow: /files/
  • https://megaindex.com/crawler
Back to top