dexerto.com
robots.txt

Robots Exclusion Standard data for dexerto.com

Resource Scan

Scan Details

Site Domain dexerto.com
Base Domain dexerto.com
Scan Status Ok
Last Scan2024-11-09T09:53:09+00:00
Next Scan 2024-11-16T09:53:09+00:00

Last Scan

Scanned2024-11-09T09:53:09+00:00
URL https://dexerto.com/robots.txt
Redirect https://www.dexerto.com/robots.txt
Redirect Domain www.dexerto.com
Redirect Base dexerto.com
Domain IPs 104.18.6.51, 104.18.7.51, 2606:4700::6812:633, 2606:4700::6812:733
Redirect IPs 104.18.6.51, 104.18.7.51, 2606:4700::6812:633, 2606:4700::6812:733
Response IP 104.18.6.51
Found Yes
Hash 2bae4602f59e385c3033b4cb4ca45704abed806ef9af5e1f0f3ba0fefc9eb6bd
SimHash 49151452c513

Groups

*

Rule Path
Disallow /search/
Disallow /cdn-cgi/
Allow /cdn-cgi/image/

Other Records

Field Value
sitemap https://www.dexerto.com/sitemap_index.xml
sitemap https://www.dexerto.com/news-sitemap.xml