internetretailer.com
robots.txt

Robots Exclusion Standard data for internetretailer.com

Resource Scan

Scan Details

Site Domain internetretailer.com
Base Domain internetretailer.com
Scan Status Ok
Last Scan2024-05-25T05:23:31+00:00
Next Scan 2024-06-01T05:23:31+00:00

Last Scan

Scanned2024-05-25T05:23:31+00:00
URL https://www.internetretailer.com/robots.txt
Domain IPs 141.193.213.20, 141.193.213.21
Response IP 141.193.213.20
Found Yes
Hash 5ea285159ca4608bc8370230a3fc4a1b6b85b3b4be617518b71e13ad0b974ef3
SimHash 40205b3223b0

Groups

scrapy

Rule Path
Allow /

*

Rule Path
Disallow /wp-admin/
Disallow /cart/
Disallow /?wc-ajax=get_refreshed_fragments
Disallow /xmlrpc.php
Allow /wp-admin/admin-ajax.php
Allow /wp-content/uploads/

googlebot

Rule Path
Disallow /wp-admin/
Disallow /hs/manage-preferences/unsubscribe-simple
Allow /wp-admin/admin-ajax.php
Allow /wp-content/uploads/
Allow .js
Allow .css