Log In Sign Up

dromen.site
robots.txt

Robots Exclusion Standard data for dromen.site

Resource Scan

Scan Details

Site Domain dromen.site
Base Domain dromen.site
Scan Status Ok
Last Scan2024-09-16T04:05:48+00:00
Next Scan 2024-09-23T04:05:48+00:00

Last Scan

Scanned2024-09-16T04:05:48+00:00
URL https://dromen.site/robots.txt
Domain IPs 104.16.150.108, 104.16.151.108, 2606:4700::6810:966c, 2606:4700::6810:976c
Response IP 104.16.151.108
Found Yes
Hash 443b09a0ef79c88684433f4201eed3f1ae3d94705b719589be35b5df63225eb8
SimHash 48b149f24eb0

Groups

*

Rule Path
Disallow /cgi-bin
Disallow /wp-
Disallow /?s=
Disallow *%26s%3D
Disallow /search
Disallow /author/
Disallow *?attachment_id=
Disallow */feed
Disallow */rss
Disallow */embed
Allow /wp-content/uploads/
Allow /wp-content/themes/
Allow /*/*.js
Allow /*/*.css
Allow /wp-*.png
Allow /wp-*.jpg
Allow /wp-*.jpeg
Allow /wp-*.gif
Allow /wp-*.svg
Allow /wp-*.pdf
Allow /wp-admin/admin-ajax.php

ahrefs
ahrefsbot
alphabot
alphaseobot
alphaseobot-sa
alipesnewsbot
barkrowler
blexbot
bomborabot
careerbot
dotbot
go 1.1 package http
gumgum
gumgum-bot
linguee
ip-web-crawler.com
petalbot
semrushbot
semrushbot-ba
semrushbot-sa
seznambot
turnitinbot
trendictionbot
weborama
weborama-fetcher
yisouspider
zoominfobot

Rule Path
Disallow /

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10
Back to top

Other Records

Field Value
sitemap https://dromen.site/sitemap_index.xml
Back to top

Comments

  • -----------------------------------------------------------------------------
  • Banned crawlers
  • -----------------------------------------------------------------------------
  • -----------------------------------------------------------------------------
  • All other crawlers
  • -----------------------------------------------------------------------------
Back to top

Warnings

  • 1 invalid line.
Back to top