Log In Sign Up

toprpsites.com
robots.txt

Robots Exclusion Standard data for toprpsites.com

Resource Scan

Scan Details

Site Domain toprpsites.com
Base Domain toprpsites.com
Scan Status Ok
Last Scan2024-09-20T23:30:47+00:00
Next Scan 2024-09-27T23:30:47+00:00

Last Scan

Scanned2024-09-20T23:30:47+00:00
URL https://toprpsites.com/robots.txt
Redirect https://www.toprpsites.com/robots.txt
Redirect Domain www.toprpsites.com
Redirect Base toprpsites.com
Domain IPs 104.21.28.46, 172.67.144.60, 2606:4700:3036::ac43:903c, 2606:4700:3037::6815:1c2e
Redirect IPs 104.21.28.46, 172.67.144.60, 2606:4700:3036::ac43:903c, 2606:4700:3037::6815:1c2e
Response IP 172.67.144.60
Found Yes
Hash 85bcd300bee34f5ace3559f69fcc800ee037b98c5a4544ed3dd9fede3aed814d
SimHash 3affa46212b2

Groups

mediapartners-google

Rule Path
Disallow

googlebot-image

Rule Path
Disallow /

*

Rule Path
Disallow *a%3Duser_cpl
Disallow */user_cpl/
Disallow /user_cpl/

genieo

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

yandex

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

baiduspider baiduspider-video baiduspider-image baiduspider-video baiduspider-news baiduspider-favo baiduspider-cpro baiduspider-ads

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

spinn3r

Rule Path
Disallow /

sogou

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

discobot

Rule Path
Disallow /

uptimebot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

acoon

Rule Path
Disallow /

mlbot

Rule Path
Disallow /

sosospider

Rule Path
Disallow /
Back to top

Comments

  • http://riskyinternet.com/what-is/web-robot/jikespider-6996/
  • http://riskyinternet.com/what-is/web-robot/yandex-media-6835/
  • http://riskyinternet.com/what-is/web-robot/yandex-std-crawler-6821/
  • http://riskyinternet.com/what-is/web-robot/yandex-images-6828/
  • http://riskyinternet.com/what-is/web-robot/baiduspider-5925/
  • http://riskyinternet.com/what-is/web-robot/naverbot-crawler-6443/
  • http://help.naver.com/robots
  • http://riskyinternet.com/what-is/web-robot/spinn3rspinner-aka-tailrank-inc-7073/
  • http://spinn3r.com/robot
  • http://riskyinternet.com/what-is/web-robot/new-sogou-spider-6562/
  • http://riskyinternet.com/what-is/web-robot/internet-archiver-lift-all-internet-content-over-and-over-3517/
  • http://archive.org/abot/exclude.php
  • http://www.alexa.com/help/webmasters
  • http://riskyinternet.com/what-is/web-robot/ahrefscom-bot-6002/
  • https://ahrefs.com/robot/
  • http://riskyinternet.com/what-is/web-robot/mj12-bot-4994/
  • http://www.majestic12.co.uk/projects/dsearch/mj12bot.php
  • http://riskyinternet.com/what-is/web-robot/discoveryengine-robot-6142/
  • http://discoveryengine.com/discobot.html
  • http://uptimerobot.com
  • http://fulltext.sblog.cz/category/robot
  • http://riskyinternet.com/what-is/web-robot/dotnetdotcom-3573/
  • http://www.acoon.de/robot.asp
  • http://riskyinternet.com/what-is/web-robot/mlbot-3559/
  • http://www.metadatalabs.com/mlbot
Back to top