Log In Sign Up

thedope.news
robots.txt

Robots Exclusion Standard data for thedope.news

Resource Scan

Scan Details

Site Domain thedope.news
Base Domain thedope.news
Scan Status Ok
Last Scan2024-09-17T13:03:52+00:00
Next Scan 2024-10-17T13:03:52+00:00

Last Scan

Scanned2024-09-17T13:03:52+00:00
URL http://thedope.news/robots.txt
Domain IPs 3.111.239.174
Response IP 3.111.239.174
Found Yes
Hash 637f9cffa0063d0df5a2392fc1cb843709948514479be7c6123889aef05b8aed
SimHash 6134c880f7df

Groups

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin/
Disallow /cgi-bin/
Disallow /wp-admin/
Disallow /linkout/
Disallow /recommended/
Disallow /comments/feed/
Disallow /trackback/
Disallow /index.php
Disallow /xmlrpc.php
Disallow /tag/
Disallow /category/
Disallow /author/

rogerbot

Rule Path
Disallow /tag/

Other Records

Field Value
crawl-delay 30

pinterest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

pinterestbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

pinterestbot/1.0

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

pinterest/0.2 (+https://www.pinterest.com/bot.html)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mozilla/5.0 (compatible; pinterestbot/1.0; +https://www.pinterest.com/bot.html)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mozilla/5.0 (linux; android 6.0.1; nexus 5x build/mmb29p) applewebkit/537.36 (khtml, like gecko) chrome/41.0.2272.96 mobile safari/537.36 (compatible; pinterestbot/1.0; +https://www.pinterest.com/bot.html)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 60

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

duckduckbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

ia_archiver

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

amazonbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

ninjabot

Rule Path
Disallow /

mediapartners-google*

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

adsbot-google

Rule Path
Allow /

googlebot-mobile

Rule Path
Allow /

seekport crawler

Rule Path
Disallow /

seekbot

Rule Path
Disallow /

mozilla/5.0 (compatible; yak/1.0; http://linkfluence.com/; bot@linkfluence.com)

Rule Path
Disallow /

buck/2.2; (+https://app.hypefactors.com/media-monitoring/about.html)

Rule Path
Disallow /

mozilla/5.0 (compatible; daum/4.1; +http://cs.daum.net/faq/15/4118.html?faqid=28966)

Rule Path
Disallow /

yandex

Rule Path
Disallow /admin/*

yandex

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mozilla/5.0 (compatible; semanticbot/1.0; +http://sempi.tech/bot.html)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

baiduspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mozilla/5.0 (compatible; mj12bot/v1.4.8; http://mj12bot.com/)

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

mozilla/5.0 (linux; android 7.0;) applewebkit/537.36 (khtml, like

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://thedope.news/sitemap_index.xml
Back to top

Comments

  • Disallow Seekbot - http://www.seekport.co.uk/seekbot/
  • Disallow Seekbot - http://www.seekport.co.uk/seekbot/
  • new bots info added
Back to top

Warnings

  • 1 invalid line.
  • `petalbot;+http` is not a known field.
Back to top