specifiedby.com
robots.txt
Robots Exclusion Standard data for specifiedby.com
Resource Scan
Scan Details
| Site Domain | specifiedby.com |
| Base Domain | specifiedby.com |
| Scan Status | Ok |
| Last Scan | 2024-09-22T22:52:32+00:00 |
| Next Scan | 2024-10-22T22:52:32+00:00 |
Last Scan
| Scanned | 2024-09-22T22:52:32+00:00 |
| URL | https://specifiedby.com/robots.txt |
| Redirect | https://www.specifiedby.com/robots.txt |
| Redirect Domain | www.specifiedby.com |
| Redirect Base | specifiedby.com |
| Domain IPs | 104.26.8.22, 104.26.9.22, 172.67.68.182, 2606:4700:20::681a:816, 2606:4700:20::681a:916, 2606:4700:20::ac43:44b6 |
| Redirect IPs | 104.26.8.22, 104.26.9.22, 172.67.68.182, 2606:4700:20::681a:816, 2606:4700:20::681a:916, 2606:4700:20::ac43:44b6 |
| Response IP | 104.26.8.22 |
| Found | Yes |
| Hash | 01954c9b7a55f76adc513b7e32f04f353377d09f2985d212fa07667eade0441c |
| SimHash | 401e51d8e6de |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /api/* |
| Disallow | /dashboard/* |
| Disallow | /behaviour/* |
| Disallow | /management/* |
| Disallow | /messages/* |
| Disallow | /load-message-modal |
| Disallow | /timber/* |
| Disallow | /community/* |
| Disallow | /search/autocomplete |
| Disallow | /clear-search-history |
| Disallow | /users/sendConfirmationCode |
| Disallow | /css/components |
| Disallow | /specifiedby/cpds |
mj12bot
linkdexbot
teoma
seznambot
bubing
aolbuild
baidu
baiduspider
semrushbot
dotbot
dotbot
dotbot/1.2
spbot
garlikcrawler
coccocbot-web
barkrowler
ccbot
seekport crawler
go-http-client
blexbot
zoominfobo
cliqzbot
blexbot
mail.ru_bot
megaindex.ru
megaindex.ru/2.0
yandexbot
linguee bot
liebaofast
mb2345browser
micromessenger
petalbot
applebot
sogou web spider/4.0
yisouspider
bytespider
sogou web spider
sogou inst spider
awariosmartbot/1.0
zoombot
sogou web spider/4.0
dataforseobot
serendeputybot/0.8.6
http://omgili.com
omgili
help@dataminr.com
http://tweetedtimes.com
trendsmapresolver/0.1
spark/828
cfnetwork/1240.0.4
darwin/20.5.0
infotigerbot/1.9
megaindex.ru/2.0
seekport crawler
mojeekbot
brightedge crawler/1.0
amazonbot/0.1
awariorssbot
awariosmartbot
inetdex-bot
startmebot
neevabot
petalbot
semrushbot-ba
cfnetwork
my-tiny-bot
| Rule | Path |
|---|---|
| Disallow | / |
*
No rules defined. All paths allowed.
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
Warnings
- 1 invalid line.