asianroleplay.com
robots.txt

Robots Exclusion Standard data for asianroleplay.com

Resource Scan

Scan Details

Site Domain asianroleplay.com
Base Domain asianroleplay.com
Scan Status Ok
Last Scan2024-06-10T23:23:05+00:00
Next Scan 2024-06-17T23:23:05+00:00

Last Scan

Scanned2024-06-10T23:23:05+00:00
URL https://asianroleplay.com/robots.txt
Redirect https://www.asianroleplay.com/robots.txt
Redirect Domain www.asianroleplay.com
Redirect Base asianroleplay.com
Domain IPs 104.21.52.191, 172.67.203.53, 2606:4700:3031::ac43:cb35, 2606:4700:3036::6815:34bf
Redirect IPs 104.21.52.191, 172.67.203.53, 2606:4700:3031::ac43:cb35, 2606:4700:3036::6815:34bf
Response IP 104.21.52.191
Found Yes
Hash 2e78ccf587453d260ff3dbeb5f5c0cd5b4fff20bcc3855d3017e623c1e7218dd
SimHash 4a5fe66696ba

Groups

mediapartners-google

Rule Path
Disallow

googlebot-image

Rule Path
Disallow /

genieo

Rule Path
Disallow /

jikespider

Rule Path
Disallow /

yandex

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

baiduspider baiduspider-video baiduspider-image baiduspider-video baiduspider-news baiduspider-favo baiduspider-cpro baiduspider-ads

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

spinn3r

Rule Path
Disallow /

sogou

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

discobot

Rule Path
Disallow /

uptimebot

Rule Path
Disallow /

seznambot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

acoon

Rule Path
Disallow /

mlbot

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

*

Rule Path
Disallow /add_bookmark.php
Disallow /add_comment.php
Disallow /add_group.php
Disallow /add_testimonial.php
Disallow /advertise.php
Disallow /albums.php
Disallow /approve_friend.php
Disallow /approve_test.php
Disallow /blocked.php
Disallow /block_user.php
Disallow /blognotifications.php
Disallow /bulletinnotifications.php
Disallow /commentnotifications.php
Disallow /delete_test.php
Disallow /edit_comments.php
Disallow /flag.php
Disallow /forgot_password.php
Disallow /forward_friend.php
Disallow /friendnotifications.php
Disallow /gallery.php
Disallow /gallery/
Disallow /groupnotifications.php
Disallow /group_images/
Disallow /home.php
Disallow /images/
Disallow /invite_friend.php
Disallow /join_me.php
Disallow /login.php
Disallow /m/
Disallow /m/images/
Disallow /m/old_dirs/
Disallow /m/old_files/
Disallow /m/photos/
Disallow /m/rss_blog.php
Disallow /m/songs/
Disallow /m/xspf/
Disallow /make_default.php
Disallow /mobile.php
Disallow /new_friends.php
Disallow /notice.php
Disallow /old_dirs/
Disallow /old_files/
Disallow /online_friends.php
Disallow /photos/
Disallow /picturenotifications.php
Disallow /profile_setting.php
Disallow /rank_band.php
Disallow /reply.php
Disallow /rss_blog.php
Disallow /search_profiles.php
Disallow /send_message.php
Disallow /send_message1.php
Disallow /shared_layouts.php
Disallow /songs/
Disallow /statusnotifications.php
Disallow /status_stream.php
Disallow /subscribe_blog.php
Disallow /user_images/
Disallow /view_blogs_category.php
Disallow /view_blog_layout.php
Disallow /view_comments.php
Disallow /view_friends.php
Disallow /view_group.php
Disallow /view_layout.php
Disallow /view_message.php
Disallow /view_messages.php
Disallow /view_photo.php
Disallow /view_post.php
Disallow /view_testimonials.php
Disallow /xspf/

Other Records

Field Value
crawl-delay 10

Comments

  • http://riskyinternet.com/what-is/web-robot/jikespider-6996/
  • http://riskyinternet.com/what-is/web-robot/yandex-media-6835/
  • http://riskyinternet.com/what-is/web-robot/yandex-std-crawler-6821/
  • http://riskyinternet.com/what-is/web-robot/yandex-images-6828/
  • http://riskyinternet.com/what-is/web-robot/baiduspider-5925/
  • http://riskyinternet.com/what-is/web-robot/naverbot-crawler-6443/
  • http://help.naver.com/robots
  • http://riskyinternet.com/what-is/web-robot/spinn3rspinner-aka-tailrank-inc-7073/
  • http://spinn3r.com/robot
  • http://riskyinternet.com/what-is/web-robot/new-sogou-spider-6562/
  • http://riskyinternet.com/what-is/web-robot/internet-archiver-lift-all-internet-content-over-and-over-3517/
  • http://archive.org/abot/exclude.php
  • http://www.alexa.com/help/webmasters
  • http://riskyinternet.com/what-is/web-robot/ahrefscom-bot-6002/
  • https://ahrefs.com/robot/
  • http://riskyinternet.com/what-is/web-robot/mj12-bot-4994/
  • http://www.majestic12.co.uk/projects/dsearch/mj12bot.php
  • http://riskyinternet.com/what-is/web-robot/discoveryengine-robot-6142/
  • http://discoveryengine.com/discobot.html
  • http://uptimerobot.com
  • http://fulltext.sblog.cz/category/robot
  • http://riskyinternet.com/what-is/web-robot/dotnetdotcom-3573/
  • http://www.acoon.de/robot.asp
  • http://riskyinternet.com/what-is/web-robot/mlbot-3559/
  • http://www.metadatalabs.com/mlbot