softddl.org
robots.txt

Robots Exclusion Standard data for softddl.org

Resource Scan

Scan Details

Site Domain softddl.org
Base Domain softddl.org
Scan Status Ok
Last Scan2025-08-11T19:28:58+00:00
Next Scan 2025-09-10T19:28:58+00:00

Last Scan

Scanned2025-08-11T19:28:58+00:00
URL https://softddl.org/robots.txt
Domain IPs 104.21.9.132, 172.67.160.12, 2606:4700:3031::6815:984, 2606:4700:3036::ac43:a00c
Response IP 172.67.160.12
Found Yes
Hash e26a49f19b3db4dcff724e9d437fd1f86d9e0e9bfb3dc2d52b316998465ea8f4
SimHash 590cd4f02730

Groups

*

Rule Path
Disallow /user/
Disallow /newposts/
Disallow /statistics.html
Disallow /*subaction%3Duserinfo
Disallow /*subaction%3Dnewposts
Disallow /*do%3Dlastcomments
Disallow /*do%3Dfeedback
Disallow /*do%3Dregister
Disallow /*do%3Dlostpassword
Disallow /*do%3Daddnews
Disallow /*do%3Dstats
Disallow /*do%3Dpm
Disallow /*do%3Dsearch
Disallow /*do%3Ddownload

bingbot

Rule Path
Allow /

Other Records

Field Value
crawl-delay 5

yandex

Rule Path
Allow /

Other Records

Field Value
crawl-delay 10

googlebot

Rule Path
Allow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

pinterest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

seokicks-robot

Rule Path
Disallow /

sistrix crawler

Rule Path
Disallow /

uptimerobot/2.0

Rule Path
Disallow /

ezooms robot

Rule Path
Disallow /

perl lwp

Rule Path
Disallow /

netestate ne crawler

Rule Path
Disallow /

wiseguys robot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://softddl.org/sitemap.xml

Comments

  • ALL BOTS
  • Crawlers Setup
  • Block SEOkicks
  • Block SISTRIX
  • Block UptimeRobot
  • Block Ezooms Robot
  • Block Perl LWP
  • Block netEstate NE Crawler
  • Block WiseGuys Robot

Warnings

  • 2 invalid lines.