Log In Sign Up

crucigramar.com
robots.txt

Robots Exclusion Standard data for crucigramar.com

Resource Scan

Scan Details

Site Domain crucigramar.com
Base Domain crucigramar.com
Scan Status Ok
Last Scan2024-09-29T07:07:30+00:00
Next Scan 2024-10-29T07:07:30+00:00

Last Scan

Scanned2024-09-29T07:07:30+00:00
URL https://crucigramar.com/robots.txt
Redirect https://www.crucigramar.com/robots.txt
Redirect Domain www.crucigramar.com
Redirect Base crucigramar.com
Domain IPs 104.21.45.33, 172.67.208.91, 2606:4700:3033::6815:2d21, 2606:4700:3035::ac43:d05b
Redirect IPs 104.21.45.33, 172.67.208.91, 2606:4700:3033::6815:2d21, 2606:4700:3035::ac43:d05b
Response IP 104.21.45.33
Found Yes
Hash 6367e81bc62fbf2c067dedb5d50d9fc23c0b726e4c55bbe11d9086d8d4727e27
SimHash c06c9450e2aa

Groups

applebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 50

mozilla/5.0+(compatible;+barkrowler/0.9;++https://babbar.tech/crawler)

Rule Path
Disallow /

mozilla/5.0+(macintosh;+intel+mac+os+x+10_10_1)+applewebkit/600.2.5+(khtml,+like+gecko)+version/8.0.2+safari/600.2.5+(amazonbot/0.1;++https://developer.amazon.com/support/amazonbot)

Rule Path
Disallow /

twitterbot

Rule Path
Disallow /

facebookexternalhit

Rule Path
Disallow /

rollcrawl

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

junglee bot

Rule Path
Disallow /

pinterestbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

semrushbot-bm

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mozilla/5.0 (compatible; megaindex.ru/2.0; +http://megaindex.com/crawler)

Rule Path
Disallow /

mozilla/5.0+(compatible;+megaindex.ru/2.0;++http://megaindex.com/crawler)

Rule Path
Disallow /

megaindex.ru/2.0

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

pinterest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 200

pinterestbot/1.0

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 200

mozilla/5.0 (compatible; pinterestbot/1.0; +https://www.pinterest.com/bot.html)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 300

mozilla/5.0+(compatible;+pinterestbot/1.0;++http://www.pinterest.com/bot.html)

Rule Path
Disallow /

mozilla/5.0 (compatible; pinterestbot/1.0; http://www.pinterest.com/bot.html)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 300

mozilla/5.0+(compatible;+adsbot/3.1)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mozilla/5.0+(compatible;+dotbot/1.1;+http://www.opensiteexplorer.org/dotbot,+help@moz.com)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 200

mozilla/5.0+(compatible;+blexbot/1.0;++http://webmeup-crawler.com/)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

blexbot/1.0

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 200

ccbot/2.0 (http://commoncrawl.org/faq/)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 200

ccbot/2.0

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 200

mozilla/5.0 (compatible; megaindex.ru/2.0; +http://megaindex.com/crawler)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 200

mozilla/5.0+(compatible;+dataforseobot/1.0;++https://dataforseo.com/dataforseo-bot)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2000
Back to top

Comments

  • Block MegaIndex.ru
  • User-agent: *
  • Sitemap: https://www.crucigramar.com/sitemap.xml
Back to top