Log In Sign Up

chinatalk.net
robots.txt

Robots Exclusion Standard data for chinatalk.net

Resource Scan

Scan Details

Site Domain chinatalk.net
Base Domain chinatalk.net
Scan Status Ok
Last Scan2024-10-12T04:14:23+00:00
Next Scan 2024-10-19T04:14:23+00:00

Last Scan

Scanned2024-10-12T04:14:23+00:00
URL https://chinatalk.net/robots.txt
Redirect https://www.chinatalk.net/robots.txt
Redirect Domain www.chinatalk.net
Redirect Base chinatalk.net
Domain IPs 54.65.28.72
Redirect IPs 54.65.28.72
Response IP 54.65.28.72
Found Yes
Hash 22e788382e4ec4dbffd309502ad3977afa605a7c6a87dcd2cf7a7b98cddc866b
SimHash 733cd3452fbb

Groups

*

Rule Path
Disallow /assets/
Disallow /assets-frontpage/
Disallow /contents/

test-bot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

applebot

Rule Path
Disallow /

bingbot

Rule Path
Disallow /

bidswitchbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

boardreader

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

coccocbot-web

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

facebookexternalhit

Rule Path
Disallow /

femtosearchbot

Rule Path
Disallow /

feedly

Rule Path
Disallow /

feedburner

Rule Path
Disallow /

getintent crawler

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

hybridbot

Rule Path
Disallow /

ias_crawler

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

netseer

Rule Path
Disallow /

owler

Rule Path
Disallow /

pinterestbot

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

safednsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

semantic-visions.com

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

tracemyfile

Rule Path
Disallow /

theoldreader.com

Rule Path
Disallow /

weborama-fetcher

Rule Path
Disallow /
Back to top

Comments

  • http://www.apple.com/go/applebot
  • http://www.bing.com/bingbot.htm
  • http://www.opensiteexplorer.org/dotbot
  • https://www.facebook.com/externalhit_uatext.php
  • http://getintent.com/bot.html
  • http://www.grapeshot.co.uk/crawler.php
  • http://mj12bot.com
  • http://www.netseer.com/crawler.html
  • http://www.pinterest.com/bot.html
  • http://www.semrush.com/bot.html
  • http://www.semrush.com/bot.html
  • http://www.trendiction.com/en/publisher/bot
Back to top