compsource.com
robots.txt

Robots Exclusion Standard data for compsource.com

Resource Scan

Scan Details

Site Domain compsource.com
Base Domain compsource.com
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2025-07-27T08:47:21+00:00
Next Scan 2025-10-25T08:47:21+00:00

Last Successful Scan

Scanned2024-09-30T13:17:42+00:00
URL https://compsource.com/robots.txt
Redirect https://www.compsource.com/robots.txt
Redirect Domain www.compsource.com
Redirect Base compsource.com
Domain IPs 52.66.101.246
Redirect IPs 64.185.181.238
Response IP 64.185.181.238
Found Yes
Hash 89ae3ff41e6ce5ce320fafeb02d68f5deb286e72acb82cd2be9f5e5bff94d3b5
SimHash 495d19546fb9

Groups

*

Rule Path
Disallow /shopping/
Disallow /login/
Disallow /*?keyword=*
Disallow /*?keyword_this_page=*
Disallow */frvcvdywtbwba*

webcrawler

Rule Path
Disallow /shopping/

yandex

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

bingbot

Rule Path
Allow /

googlebot

Rule Path
Allow /

ravencrawler

Rule Path
Allow /

Other Records

Field Value
sitemap https://www.compsource.com/sitemaps/siteindex1.xml

Comments

  • Disallow: /qanda.asp?mode=Q
  • Disallow: /*&sort=*
  • Disallow indexing directories for specific web crawlers
  • Sitemap files

Warnings

  • 2 invalid lines.