Log In Sign Up

rubylane.com
robots.txt

Robots Exclusion Standard data for rubylane.com

Resource Scan

Scan Details

Site Domain rubylane.com
Base Domain rubylane.com
Scan Status Ok
Last Scan2024-10-29T22:32:16+00:00
Next Scan 2024-11-28T22:32:16+00:00

Last Scan

Scanned2024-10-29T22:32:16+00:00
URL https://rubylane.com/robots.txt
Redirect https://www.rubylane.com/robots.txt
Redirect Domain www.rubylane.com
Redirect Base rubylane.com
Domain IPs 34.235.3.167, 44.199.80.12
Redirect IPs 34.235.3.167, 44.199.80.12
Response IP 34.235.3.167
Found Yes
Hash 48da6f4a311700b607b231437a53b3a0fef8acbdc4af0f169213a6f899d70e6e
SimHash 5a1e5506dac8

Groups

ahrefsbot
psbot
spbot
linksmanager
redcarpet
omniexplorer_bot
turnitinbot
twiceler
grub-client
naverbot
e-societyrobot
pcnbot
aipbot
pgbot
mj12bot
voilabot
dealozbot
pimeyes.com crawler
petalbot
semrushbot
seokicks
the knowledge ai
siteauditbot
semrushbot-ba
semrushbot-si
semrushbot-swa
semrushbot-ct
semrushbot-bm
splitsignalbot
semrushbot-coub
dataforseobot
dotbot
yandex
yandexbot
sogou
imagesiftbot
claudebot

Rule Path
Disallow /

pinterestbot

Rule Path
Disallow /cgi-bin
Disallow /clickv1
Disallow /too
Disallow /en/
Disallow /ni/
Disallow /_hp

Other Records

Field Value
crawl-delay 2

bingbot
applebot

Rule Path
Disallow /_pod/
Disallow /_podl/
Disallow /cgi-bin
Disallow /clickv1
Disallow /too
Disallow /en/
Disallow /ni/
Disallow /_hp

Other Records

Field Value
crawl-delay 2

*

Rule Path
Disallow /cgi-bin
Disallow /clickv1
Disallow /too
Disallow /en/
Disallow /ni/
Disallow /_hp
Back to top

Other Records

Field Value
sitemap https://www.rubylane.com/sitemaps/rubylaneSitemap.xml
sitemap https://www.rubylane.com/sitemaps/rubylaneSitemapindex_listing.xml
Back to top

Comments

  • specification: https://www.robotstxt.org/norobots-rfc.txt
Back to top

Warnings

  • 1 invalid line.
Back to top