dealnews.com
robots.txt

Robots Exclusion Standard data for dealnews.com

Resource Scan

Scan Details

Site Domain dealnews.com
Base Domain dealnews.com
Scan Status Ok
Last Scan2024-06-05T15:40:39+00:00
Next Scan 2024-06-12T15:40:39+00:00

Last Scan

Scanned2024-06-05T15:40:39+00:00
URL https://dealnews.com/robots.txt
Redirect https://www.dealnews.com:443/robots.txt
Redirect Domain www.dealnews.com
Redirect Base dealnews.com
Domain IPs 3.220.94.20, 52.202.136.160, 52.45.51.144
Redirect IPs 151.101.130.49, 151.101.194.49, 151.101.2.49, 151.101.66.49
Response IP 199.232.46.49
Found Yes
Hash 682fa45f8a8d4f27002c5e0664a9c6404bfda8f414efac5fea130aa1894dd4ae
SimHash 4d45dd354451

Groups

*

Rule Path
Disallow /dead-end/

Other Records

Field Value
crawl-delay 2

Other Records

Field Value
sitemap https://www.dealnews.com/sitemap_index.xml
sitemap https://www.dealnews.com/sitemap-articles-01.xml.gz
sitemap https://www.dealnews.com/sitemap-black-friday-01.xml.gz
sitemap https://www.dealnews.com/sitemap-brands-01.xml.gz
sitemap https://www.dealnews.com/sitemap-categories-01.xml.gz
sitemap https://www.dealnews.com/sitemap-facets-01.xml.gz
sitemap https://www.dealnews.com/sitemap-other-01.xml.gz
sitemap https://www.dealnews.com/sitemap_news-features-01.xml.gz

Comments

  • /robots.txt for http://dealnews.com/