Log In Sign Up

sobchak.blog
robots.txt

Robots Exclusion Standard data for sobchak.blog

Resource Scan

Scan Details

Site Domain sobchak.blog
Base Domain sobchak.blog
Scan Status Ok
Last Scan2024-09-20T11:20:53+00:00
Next Scan 2024-10-20T11:20:53+00:00

Last Scan

Scanned2024-09-20T11:20:53+00:00
URL https://sobchak.blog/robots.txt
Domain IPs 2a05:f480:1400:26e7:5400:4ff:fe90:5295, 95.179.128.143
Response IP 95.179.128.143
Found Yes
Hash 72c823347699c587787b056add3a7a63a7eace1681a35288a8fe82e21fbd7db8
SimHash 4ed3d423bfb0

Groups

siteauditbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 1

semrushbot-si

Rule Path
Allow /

*

Rule Path
Disallow /wp-json/
Disallow /?s=*
Disallow /search/*
Disallow */?bbp_reply_to
Disallow */?_escaped_fragment_=
Disallow /?bbp_reply_to
Disallow */?filter_by
Disallow /forums/reply/
Disallow /dveri
Disallow */?share
Disallow */?customize_theme
Disallow */nb
Disallow /forums/users/

nuclei
wikido
riddler
petalbot
zoominfobot
go-http-client
node/simplecrawler
cazoodlebot
dotbot/1.0
gigabot
barkrowler
blexbot
magpie-crawler

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://sobchak.blog/sitemap_index.xml
sitemap https://sobchak.blog/post-sitemap.xml
sitemap https://sobchak.blog/topic-sitemap.xml
Back to top

Comments

  • Global rules
  • -----------------
  • We're experimenting with blocking search results to prevent search result spam
  • Sitemap
  • -----------------
  • Ban bots that don't benefit us.
  • --------------------------------
Back to top