effectsdatabase.com
robots.txt
Robots Exclusion Standard data for effectsdatabase.com
Resource Scan
Scan Details
| Site Domain | effectsdatabase.com |
| Base Domain | effectsdatabase.com |
| Scan Status | Ok |
| Last Scan | 2024-06-16T14:11:38+00:00 |
| Next Scan | 2024-07-16T14:11:38+00:00 |
Last Scan
| Scanned | 2024-06-16T14:11:38+00:00 |
| URL | https://effectsdatabase.com/robots.txt |
| Domain IPs | 109.68.161.56 |
| Response IP | 109.68.161.56 |
| Found | Yes |
| Hash | e04bc5f7bcbabe0eb236fe4ff6c653999ec38e4120648c8a1b7bd71365a65268 |
| SimHash | 5cd556d25490 |
Groups
a6-indexer
adsbot
ahrefsbot
aihitbot
alphaseobot
alphaseobot-sa
awariosmartbot
bdcbot
bleriot
blexbot
baiduspider
barkrowler
bingpreview
buck
bytespider
ccbot
clickagy intelligence bot
clickagy intelligence bot v2
cliqzbot
coccocbot-web
dataforseobot
daum
domaincrawler
dotbot
extlinksbot
gigabot
grapeshotcrawler
hybridbot
indeedbot
istellabot
keybot
linespider
linkpadbot
lumtelbot
ltx71
magpie-crawler
mail.ru_bot
mauibot
maxpointcrawler
mj12bot
mojeekbot
mtrobot
naver
naverbot
neevabot
netseer
nikto
nutch
petalbot
plukkie
proximic
qwantify
qwant-news
rytebot
rogerbot
scrapy
seekport crawler
semrush
semrushbot
seokicks
siteexplorer
slurp
smtbot
spbot
spiderbot
spiderbot/nutch-1.7
the knowledge ai
toutiaospider
yahoo ad monitoring
yandex
dataprovider
| Rule | Path |
|---|---|
| Disallow | / |
adsbot-google
applebot
bingbot
google
googlebot
google-bot
googlebot-mobile
mozilla/5.0 (compatible;contxbot/1.0)
mediapartners
mediapartners-google
msnbot
msnbot-media
pinterestbot
seznambot
twitterbot
| Rule | Path |
|---|---|
| Disallow | /_cron/* |
| Disallow | /admin/* |
| Disallow | /ads/* |
| Disallow | /random/* |
| Disallow | /feed/* |
| Disallow | /files/* |
| Disallow | /find/* |
| Disallow | /related/* |
| Disallow | /contact |
| Disallow | /logout |
| Disallow | /user |
| Disallow | /404 |
Other Records
| Field | Value |
|---|---|
| crawl-delay | 5 |
*
| Rule | Path |
|---|---|
| Disallow | / |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.effectsdatabase.com/sitemap.xml |
| sitemap | https://www.effectsdatabase.com/sitemap_categories.xml |
| sitemap | https://www.effectsdatabase.com/sitemap_categories2.xml |
Warnings
- 1 invalid line.
- `request-rate` is not a known field.
Comments