roller.com
robots.txt
Robots Exclusion Standard data for roller.com
Resource Scan
Scan Details
| Site Domain | roller.com |
| Base Domain | roller.com |
| Scan Status | Ok |
| Last Scan | 2024-10-08T02:27:58+00:00 |
| Next Scan | 2024-11-07T02:27:58+00:00 |
Last Scan
| Scanned | 2024-10-08T02:27:58+00:00 |
| URL | https://roller.com/robots.txt |
| Domain IPs | 23.88.64.173 |
| Response IP | 23.88.64.173 |
| Found | Yes |
| Hash | d62f0cb81eddff1ae8299153f149fe9d29483d8b7fa4d9a54a20457b3f5f0295 |
| SimHash | 8f07dab8a793 |
Groups
lcc
velenpublicwebcrawler
mozilla/5.0 (compatible; velenpublicwebcrawler/1.0; +https://velen.io)
neevabot
infotigerbot
mozilla/5.0 (compatible; infotigerbot/1.9; +https://infotiger.com/bot)
blexbot
seekportbot
mozilla/5.0 (compatible; seekportbot; +https://bot.seekport.com)
dataforseobot*
dataforseobot
mozilla/5.0 (compatible; dataforseobot; +https://dataforseo.com/dataforseo-bot)
mozilla/5.0 (compatible; dataforseobot/1.0; +https://dataforseo.com/dataforseo-bot)
mozilla/5.0 (compatible; censysinspect/1.1; +https://about.censys.io/)
adsbot
mozilla/5.0 (compatible; adsbot/3.1; +https://seostar.co/robot/)
adsbot/3.1 (+https://seostar.co/robot/)
mozilla/5.0 (compatible; seekport crawler; http://seekport.com/)
mozilla/5.0 (compatible; mojeekbot/0.10; +https://www.mojeek.com/bot.html)
mojeekbot/0.10
mojeekbot*
test user agent
megaindex*
auskunftbot*
serpstatbot/2.0 beta (advanced backlink tracking bot; http://serpstatbot.com/; abuse@serpstatbot.com)
serpstatbot*
melantech/nutch-1.16-snapshot
melantech*
python-requests/2.24.0
python-requests*
node-fetch/1.0 (+https://github.com/bitinn/node-fetch)
node-fetch*
piorbo*
adsbot/3.1
adsbot*
vagabond*
apache-httpclient/4.5.2 (java/1.8.0_232)
apache-httpclient*
checkmarknetwork*
vsusearchspider/1.0
implisensebot 1.1
netestate ne crawler (+http://www.website-datenbank.de/)
netestat*
crawler4j (https://github.com/yasserg/crawler4j/)
crawler4*
lcc
velenpublicwebcrawler*
velenpublicwebcrawler/1.0; +https://velen.io
photon/1.0
spider
curl*
cliqzbot*
ccbot*
ccbot/2.0 (https://commoncrawl.org/faq/)
go-http-client/2.0
zoominfobot
domaincrawler
domaincrawler*
info@domaincrawler.com; http://www.domaincrawler.com/www.le-service.de
qwantify/1.0
seekport crawler
mozilla/5.0 (compatible;clcrawler/1.0)
mozilla/5.0 (compatible; seekport crawler; http://seekport.com/)
mozilla/5.0 (compatible; qwantify/vr; https://www.qwant.com/)
mozilla/5.0 (compatible; qwantify/2.0n; https://www.qwant.com/)/*
mozilla/5.0 (compatible; qwantify/2.1n; https://www.qwant.com/)/*
mozilla/5.0 (compatible; qwantify/bleriot/1.1; +https://help.qwant.com/bot)
mozilla/5.0 (compatible; qwantify/bleriot/1.1; https://help.qwant.com/bot)
mozilla/5.0 (compatible; qwantify/mermoz/0.1; https://www.qwant.com/; https://www.github.com/qwantresearch/mermoz)
mozilla/5.0 (compatible; qwantify/2.4w; +https://www.qwant.com/)/2.4w
mozilla/5.0 (compatible; qwantify/2.4w; https://www.qwant.com/)/2.4w
ahrefsbot
dotbot
mj12bot
seoscanners.net
mozilla/5.0 (compatible; seokicks; +https://www.seokicks.de/robot.html)
seokicks-robot
seokicks
yandexbot/3.0
yandexbot
yandex*
| Rule | Path |
|---|---|
| Disallow | / |
*
| Rule | Path |
|---|---|
| Disallow | /en/ |
*
| Rule | Path |
|---|---|
| Allow | / |
Warnings
- 2 invalid lines.