Log In Sign Up

chinesestandard.net
robots.txt

Robots Exclusion Standard data for chinesestandard.net

Resource Scan

Scan Details

Site Domain chinesestandard.net
Base Domain chinesestandard.net
Scan Status Ok
Last Scan2025-03-30T16:45:17+00:00
Next Scan 2025-04-29T16:45:17+00:00

Last Scan

Scanned2025-03-30T16:45:17+00:00
URL https://chinesestandard.net/robots.txt
Domain IPs 199.233.255.72
Response IP 199.233.255.72
Found Yes
Hash a70a4c78c01501b1f49ad82c8d8186d40a5f14d68ec5f4fa21bca26b341a013d
SimHash 1a52d82765d2

Groups

googlebot

Rule Path Comment
Disallow /Sitemap/* All use /SitemapBing/ <-- Goodle sitemap == /Sitemap/; Bing sitemap == /SitemapBing/
Disallow /Cart/* -
Disallow /PDF.aspx?PDF=* Prevent duplicate
Disallow /PDFOpenLib/* -
Disallow /PDFValidLib/* -
Disallow /PDFObsoLib/* -
Disallow /PDFmain/* -
Disallow /PDF/Sample.aspx/* -
Disallow /CN/* Moved to /CHN/
Disallow /CHN/* 2018-04
Disallow /X10/* 2018-05
Disallow /X10 2018-05
Disallow /PDF-English-Translation-302/* Backup folder
Disallow /PDF-English-Translation-GTran_HTML/* Backup folder
Disallow /CUSTOMER-DOWNLOAD/* Temp folder
Disallow /TEST/* Test folder
Disallow /Cart/* -
Disallow /CN/Cart/* -
Disallow /FlexPaper/* -
Disallow /FancyBox/* -
Disallow /Cookie/* -
Disallow /Public* -
Disallow /AD* -
Disallow /Part* -
Disallow /PDF-English-Translat/* Obsolete folder
Disallow /PDF-Translated-English/* Obsolete folder
Disallow /List-PDF/GBT2.asp* -
Disallow /China/* Purchase folder ## robots=noindex
Disallow /CHINA/* Purchase folder ## robots=noindex
Disallow /*.vb$ Disallow: *.vb ## This might block ALL
Disallow /Default.aspx?PDF-English-ID=*.html$ To remedy the sitemap mistake
Disallow /*.aspx?StdID=* -

bingbot

Rule Path Comment
Disallow /SitemapGonly/* -
Disallow /Cart/* -
Disallow /Sitemap/* All use /SitemapBing/ <-- Goodle sitemap == /Sitemap/; Bing sitemap == /SitemapBing/
Disallow /CHN/* Google /CHN/
Disallow /CUSTOMER-DOWNLOAD/* Temp folder
Disallow /PDF-English-Translation-302/* Backup folder
Disallow /PDF-English-Translation-GTran_HTML/* Backup folder
Disallow /TEST/* Test folder
Disallow /CN/Cart/* -
Disallow /FlexPaper/* -
Disallow /FancyBox/* -
Disallow /Cookie/* -
Disallow /Public* -
Disallow /AD* -
Disallow /Part* -
Disallow /English-Translation-PDF/* Obsolete folder
Disallow /PDF-English-Translat/* Obsolete folder
Disallow /PDF-Translated-English/* Obsolete folder
Disallow /List-PDF/GBT2.asp* -
Disallow /China/* Purchase folder ## robots=noinex
Disallow /CHINA/* Purchase folder ## robots=noinex
Disallow /TEST/* -
Disallow /*.vb$ Disallow: *.vb ## This might block ALL
Disallow /Default.aspx?PDF-English-ID=*.html$ To remedy the sitemap mistake
Disallow /*.aspx?StdID=* -
Disallow /*.aspx?RelatedID=* -
Disallow /AMP/* -
Disallow /Mobile/* -
Disallow /PDFObsoLib/* -
Disallow /PDFValidLib/* -
Disallow /PDFmain/* -

baiduspider

Rule Path Comment
Disallow /SitemapGonly/* -
Disallow /Cart/* -
Disallow /Default.aspx?* Ban Baidu to crawle English pages
Disallow /List-PDF/* Ban Baidu to crawle English pages
Disallow /CN/* -
Disallow /CHN/* Use /China/
Disallow /CUSTOMER-DOWNLOAD/* Temp folder
Disallow /PDF-English-Translation-302/* Backup folder
Disallow /PDF-English-Translation-GTran_HTML/* Backup folder
Disallow /TEST/* Test folder
Disallow /CN/Cart/* -
Disallow /FlexPaper/* -
Disallow /FancyBox/* -
Disallow /Cookie/* -
Disallow /Public* -
Disallow /AD* -
Disallow /Part* -
Disallow /English-Translation-PDF/* Obsolete folder
Disallow /PDF-English-Translat/* Obsolete folder
Disallow /PDF-Translated-English/* Obsolete folder
Disallow /List-PDF/GBT2.asp* -
Disallow /TEST/* -
Disallow /*.vb$ Disallow: *.vb ## This might block ALL
Disallow /Default.aspx?PDF-English-ID=*.html$ To remedy the sitemap mistake
Disallow /*.aspx?StdID=* -
Disallow /*.aspx?RelatedID=* -
Disallow /PDF-Excerpt -
Disallow /PDF-Excerpt/ -
Disallow /AMP/* -
Disallow /Mobile/* -

amazonbot

Product Comment
amazonbot Other Amazon bots not following rules -- blocked by IP.
Rule Path Comment
Disallow / disallow all

twitterbot

Rule Path
Disallow /

facebot

Product Comment
facebot Facebook
Rule Path Comment
Disallow / disallow all

facebookexternalhit

Product Comment
facebookexternalhit Facebook
Rule Path Comment
Disallow / disallow all

facebookexternalhit1.1

Product Comment
facebookexternalhit1.1 Facebook
Rule Path Comment
Disallow / disallow all

ahrefsbot

Rule Path
Disallow /

prerender

Rule Path
Disallow /

moz dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

ahrefs site audit

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

*

Product Comment
* Important Line !!
Rule Path Comment
Disallow /SitemapGonly/* -
Disallow /CHN/* Use /China/
Disallow /PDF-English-Translation-302/* Backup folder
Disallow /PDF-English-Translation-GTran_HTML/* Backup folder
Disallow /CUSTOMER-DOWNLOAD/* Temp folder
Disallow /TEST/* Test folder
Disallow /Cart/* -
Disallow /CN/Cart/* -
Disallow /FlexPaper/* -
Disallow /FancyBox/* -
Disallow /Cookie/* -
Disallow /Public* -
Disallow /AD* -
Disallow /Part* -
Disallow /English-Translation-PDF/* Obsolete folder
Disallow /PDF-English-Translat/* Obsolete folder
Disallow /PDF-Translated-English/* Obsolete folder
Disallow /List-PDF/GBT2.asp* -
Disallow /China/* Purchase folder ## robots=noindex
Disallow /TEST/* -
Disallow /*.vb$ Disallow: *.vb ## This might block ALL
Disallow /Default.aspx?PDF-English-ID=*.html$ To remedy the sitemap mistake
Disallow /*.aspx?StdID=* -
Disallow /*.aspx?RelatedID=* -
Disallow /PDF-Excerpt -
Disallow /PDF-Excerpt/ -
Back to top

Comments

  • Remark. "*": Wild card. "$": End (End with previous letters)
  • SECTION 1: ============== PART 1: Google ===============
  • Disallow: /PDF-Excerpt/ShowPDFExcerpt.aspx*
  • Disallow: /PDF-English-Translation/* ## Managed by TagX
  • Disallow: /PDF/mEnglish.aspx/* ## Split for Mobile
  • Disallow: /PDF/Sample.aspx/* ## Managed by TagX
  • Disallow: /PDF/BOOK.aspx/* ## Managed by TagX
  • Disallow: /BUY.aspx/* ## Managed by TagX
  • Disallow: /AMP/*
  • Disallow: /Mobile/*
  • Disallow: /M/*
  • Disallow: /Img/* ## Open to prevent img error
  • Disallow: /*.aspx?RelatedID=*
  • SECTION 2: ============== PART 2: Bing ===============
  • Disallow: /PDF-English-Translation/*
  • Disallow: /Img/*
  • SECTION 3: ============ PART 3: Baidu ==================
  • Disallow: /Sitemap/* ## All use /SitemapBing/ <-- Goodle sitemap == /Sitemap/; Bing sitemap == /SitemapBing/
  • Disallow: /PDF-English-Translation/*
  • Disallow: /Img/*
  • Disallow: /China/* ## Purchase folder ## robots=noindex
  • SECTION 4: ============== PART 4: Banned bots ===============
  • User-agent: Alibaba Bot -- Not following rules, Blocked by IP
  • Disallow: /
  • SECTION 5: ALL spiders ============== PART 5: Other spiders: ALL ===============
  • Disallow: /PDF-English-Translation/*
  • Disallow: /Img/*
  • Disallow:/AMP/*
  • Disallow:/Mobile/*
  • SECTION 6: China Spiders (memo) ============ PART 6 ==================
  • User-agent: sogou spider
  • Disallow: /Default.asp*
  • Disallow: /PDF-English-Translat/
  • Disallow: /List-PDF/
  • User-agent: sohu agent
  • Disallow: /Default.asp*
  • Disallow: /PDF-English-Translat/
  • Disallow: /List-PDF/
  • User-agent: iaskspider
  • Disallow: /Default.asp*
  • Disallow: /PDF-English-Translat/
  • Disallow: /List-PDF/
  • User-agent: YodaoBot
  • Disallow: /Default.asp*
  • Disallow: /PDF-English-Translat/
  • Disallow: /List-PDF/
  • User-agent: Sogou Orion spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
  • Disallow: /
  • User-agent: Sogou web spider/3.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
  • Disallow: /
  • User-agent: Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)
  • Disallow: /
Back to top