dlnws.com
robots.txt

Robots Exclusion Standard data for dlnws.com

Resource Scan

Scan Details

Site Domain dlnws.com
Base Domain dlnws.com
Scan Status Ok
Last Scan2024-06-06T17:12:28+00:00
Next Scan 2024-06-13T17:12:28+00:00

Last Scan

Scanned2024-06-06T17:12:28+00:00
URL https://dlnws.com/robots.txt
Redirect https://www.dealnews.com/robots.txt
Redirect Domain www.dealnews.com
Redirect Base dealnews.com
Domain IPs 13.33.88.120, 13.33.88.24, 13.33.88.50, 13.33.88.99
Redirect IPs 151.101.130.49, 151.101.194.49, 151.101.2.49, 151.101.66.49
Response IP 199.232.46.49
Found Yes
Hash 682fa45f8a8d4f27002c5e0664a9c6404bfda8f414efac5fea130aa1894dd4ae
SimHash 4d45dd354451

Groups

*

Rule Path
Disallow /dead-end/

Other Records

Field Value
crawl-delay 2

Other Records

Field Value
sitemap https://www.dealnews.com/sitemap_index.xml
sitemap https://www.dealnews.com/sitemap-articles-01.xml.gz
sitemap https://www.dealnews.com/sitemap-black-friday-01.xml.gz
sitemap https://www.dealnews.com/sitemap-brands-01.xml.gz
sitemap https://www.dealnews.com/sitemap-categories-01.xml.gz
sitemap https://www.dealnews.com/sitemap-facets-01.xml.gz
sitemap https://www.dealnews.com/sitemap-other-01.xml.gz
sitemap https://www.dealnews.com/sitemap_news-features-01.xml.gz

Comments

  • /robots.txt for http://dealnews.com/