crem.coffee
robots.txt
Robots Exclusion Standard data for crem.coffee
Resource Scan
Scan Details
Site Domain | crem.coffee |
Base Domain | crem.coffee |
Scan Status | Ok |
Last Scan | 2024-09-11T18:52:17+00:00 |
Next Scan | 2024-10-11T18:52:17+00:00 |
Last Scan
Scanned | 2024-09-11T18:52:17+00:00 |
URL | https://crem.coffee/robots.txt |
Redirect | https://www.crem.coffee/robots.txt |
Redirect Domain | www.crem.coffee |
Redirect Base | crem.coffee |
Domain IPs | 64.25.118.88 |
Redirect IPs | 13.107.246.59, 2620:1ec:bdf::59 |
Response IP | 13.107.246.59 |
Found | Yes |
Hash | 99d1e1ef2b48f70e4748cb132b41613d1d9d9eef6baca544ab732c7636b16397 |
SimHash | 6a4cdb5444cb |
Groups
google
Rule | Path |
---|---|
Disallow | /search |
Disallow | /Global-Tools |
Disallow | /Special-Pages |
Disallow | /Pricebook |
Disallow | /api |
Other Records
Field | Value |
---|---|
crawl-delay | 10 |
hubspot links crawler 1.0 http://www.hubspot.com/
Rule | Path |
---|---|
Disallow | /search |
Disallow | /Global-Tools |
Disallow | /Special-Pages |
Disallow | /Pricebook |
Disallow | /api |
Other Records
Field | Value |
---|---|
crawl-delay | 10 |
hubspot crawler 1.0 http://www.hubspot.com/
Rule | Path |
---|---|
Disallow | /search |
Disallow | /Global-Tools |
Disallow | /Special-Pages |
Disallow | /Pricebook |
Disallow | /api |
Other Records
Field | Value |
---|---|
crawl-delay | 10 |
hubspot crawler
Rule | Path |
---|---|
Disallow | /search |
Disallow | /Global-Tools |
Disallow | /Special-Pages |
Disallow | /Pricebook |
Disallow | /api |
Other Records
Field | Value |
---|---|
crawl-delay | 10 |
baiduspider
Rule | Path |
---|---|
Disallow | /search |
Disallow | /Global-Tools |
Disallow | /Special-Pages |
Disallow | /Pricebook |
Disallow | /api |
Other Records
Field | Value |
---|---|
crawl-delay | 10 |
bingbot
Rule | Path |
---|---|
Disallow | /search |
Disallow | /Global-Tools |
Disallow | /Special-Pages |
Disallow | /Pricebook |
Disallow | /api |
Other Records
Field | Value |
---|---|
crawl-delay | 10 |
googlebot
Rule | Path |
---|---|
Disallow | /search |
Disallow | /Global-Tools |
Disallow | /Special-Pages |
Disallow | /Pricebook |
Disallow | /api |
Other Records
Field | Value |
---|---|
crawl-delay | 10 |
googlebot-image
Rule | Path |
---|---|
Disallow | /search |
Disallow | /Global-Tools |
Disallow | /Special-Pages |
Disallow | /Pricebook |
Disallow | /api |
Other Records
Field | Value |
---|---|
crawl-delay | 10 |
iaskspider
Rule | Path |
---|---|
Disallow | /search |
Disallow | /Global-Tools |
Disallow | /Special-Pages |
Disallow | /Pricebook |
Disallow | /api |
Other Records
Field | Value |
---|---|
crawl-delay | 10 |
msnbot
Rule | Path |
---|---|
Disallow | /search |
Disallow | /Global-Tools |
Disallow | /Special-Pages |
Disallow | /Pricebook |
Disallow | /api |
Other Records
Field | Value |
---|---|
crawl-delay | 10 |
sogou spider
Rule | Path |
---|---|
Disallow | /search |
Disallow | /Global-Tools |
Disallow | /Special-Pages |
Disallow | /Pricebook |
Disallow | /api |
Other Records
Field | Value |
---|---|
crawl-delay | 10 |
sosospider
Rule | Path |
---|---|
Disallow | /search |
Disallow | /Global-Tools |
Disallow | /Special-Pages |
Disallow | /Pricebook |
Disallow | /api |
Other Records
Field | Value |
---|---|
crawl-delay | 10 |
teoma
Rule | Path |
---|---|
Disallow | /search |
Disallow | /Global-Tools |
Disallow | /Special-Pages |
Disallow | /Pricebook |
Disallow | /api |
Other Records
Field | Value |
---|---|
crawl-delay | 10 |
yahoo! slurp
Rule | Path |
---|---|
Disallow | /search |
Disallow | /Global-Tools |
Disallow | /Special-Pages |
Disallow | /Pricebook |
Disallow | /api |
Other Records
Field | Value |
---|---|
crawl-delay | 10 |
yahoo! slurp china
Rule | Path |
---|---|
Disallow | /search |
Disallow | /Global-Tools |
Disallow | /Special-Pages |
Disallow | /Pricebook |
Disallow | /api |
Other Records
Field | Value |
---|---|
crawl-delay | 10 |
yahooseeker
Rule | Path |
---|---|
Disallow | /search |
Disallow | /Global-Tools |
Disallow | /Special-Pages |
Disallow | /Pricebook |
Disallow | /api |
Other Records
Field | Value |
---|---|
crawl-delay | 10 |
yandexbot
Rule | Path |
---|---|
Disallow | /search |
Disallow | /Global-Tools |
Disallow | /Special-Pages |
Disallow | /Pricebook |
Disallow | /api |
Other Records
Field | Value |
---|---|
crawl-delay | 10 |
yandeximages
Rule | Path |
---|---|
Disallow | /search |
Disallow | /Global-Tools |
Disallow | /Special-Pages |
Disallow | /Pricebook |
Disallow | /api |
Other Records
Field | Value |
---|---|
crawl-delay | 10 |
sebot-wa
Rule | Path |
---|---|
Disallow | /search |
Disallow | /Global-Tools |
Disallow | /Special-Pages |
Disallow | /Pricebook |
Disallow | /api |
Other Records
Field | Value |
---|---|
crawl-delay | 10 |
*
Rule | Path |
---|---|
Disallow | / |
Other Records
Field | Value |
---|---|
sitemap | https://www.crem.coffee/sitemap.xml |
sitemap | https://www.crem.coffee/sitemap.xml |
sitemap | https://www.crem.coffee/sitemap.xml |
sitemap | https://www.crem.coffee/sitemap.xml |
sitemap | https://www.crem.coffee/sitemap.xml |
sitemap | https://www.crem.coffee/sitemap.xml |
sitemap | https://www.crem.coffee/sitemap.xml |
sitemap | https://www.crem.coffee/sitemap.xml |
sitemap | https://www.crem.coffee/sitemap.xml |
sitemap | https://www.crem.coffee/sitemap.xml |
sitemap | https://www.crem.coffee/sitemap.xml |
sitemap | https://www.crem.coffee/sitemap.xml |
sitemap | https://www.crem.coffee/sitemap.xml |
sitemap | https://www.crem.coffee/sitemap.xml |
sitemap | https://www.crem.coffee/sitemap.xml |
sitemap | https://www.crem.coffee/sitemap.xml |
sitemap | https://www.crem.coffee/sitemap.xml |
sitemap | https://www.crem.coffee/sitemap.xml |
sitemap | https://www.crem.coffee/sitemap.xml |