deal-mac.com
robots.txt

Robots Exclusion Standard data for deal-mac.com

Resource Scan

Scan Details

Site Domain deal-mac.com
Base Domain deal-mac.com
Scan Status Ok
Last Scan2024-11-11T21:43:00+00:00
Next Scan 2024-11-18T21:43:00+00:00

Last Scan

Scanned2024-11-11T21:43:00+00:00
URL https://deal-mac.com/robots.txt
Redirect https://www.dealnews.com/robots.txt
Redirect Domain www.dealnews.com
Redirect Base dealnews.com
Domain IPs 108.156.133.11, 108.156.133.116, 108.156.133.3, 108.156.133.97
Redirect IPs 151.101.130.49, 151.101.194.49, 151.101.2.49, 151.101.66.49
Response IP 199.232.46.49
Found Yes
Hash 682fa45f8a8d4f27002c5e0664a9c6404bfda8f414efac5fea130aa1894dd4ae
SimHash 4d45dd354451

Groups

*

Rule Path
Disallow /dead-end/

Other Records

Field Value
crawl-delay 2

Other Records

Field Value
sitemap https://www.dealnews.com/sitemap_index.xml
sitemap https://www.dealnews.com/sitemap-articles-01.xml.gz
sitemap https://www.dealnews.com/sitemap-black-friday-01.xml.gz
sitemap https://www.dealnews.com/sitemap-brands-01.xml.gz
sitemap https://www.dealnews.com/sitemap-categories-01.xml.gz
sitemap https://www.dealnews.com/sitemap-facets-01.xml.gz
sitemap https://www.dealnews.com/sitemap-other-01.xml.gz
sitemap https://www.dealnews.com/sitemap_news-features-01.xml.gz

Comments

  • /robots.txt for http://dealnews.com/