Log In Sign Up

amegroups.com
robots.txt

Robots Exclusion Standard data for amegroups.com

Resource Scan

Scan Details

Site Domain amegroups.com
Base Domain amegroups.com
Scan Status Ok
Last Scan2024-09-25T04:02:19+00:00
Next Scan 2024-10-25T04:02:19+00:00

Last Scan

Scanned2024-09-25T04:02:19+00:00
URL https://amegroups.com/robots.txt
Redirect https://www.amegroups.com/robots.txt
Redirect Domain www.amegroups.com
Redirect Base amegroups.com
Domain IPs 47.74.40.41
Redirect IPs 47.74.40.41
Response IP 47.74.40.41
Found Yes
Hash afafce968aedeae07db3028cdeb59931f0db27eee602c1132f399cbbd416220b
SimHash 904455528378

Groups

dotbot

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

mozilla/4.0 (compatible; msie 9.0; windows nt 6.1

Rule Path
Disallow /

*

Rule Path
Disallow /js/
Disallow /styles/
Disallow /css/
Disallow /statics/
Disallow /vendor/
Disallow /template
Disallow /favicon.ico
Back to top

Comments

  • -----------------------------------------------------------------------------
  • 禁止爬虫爬取无效URL，提升网站核心静态资源抓取及索引效率。
  • 无效URLåŒ
  • 等各种无需被SE收录的URL。
  • -----------------------------------------------------------------------------
Back to top

Warnings

  • 1 invalid line.
  • `含` is not a known field.
Back to top