websiteclosers.com
robots.txt

Robots Exclusion Standard data for websiteclosers.com

Resource Scan

Scan Details

Site Domain websiteclosers.com
Base Domain websiteclosers.com
Scan Status Ok
Last Scan2026-03-16T01:41:38+00:00
Next Scan 2026-04-15T01:41:38+00:00

Last Scan

Scanned2026-03-16T01:41:38+00:00
URL https://websiteclosers.com/robots.txt
Domain IPs 104.21.75.161, 172.67.178.187, 2606:4700:3033::6815:4ba1, 2606:4700:3035::ac43:b2bb
Response IP 104.21.75.161
Found Yes
Hash bbecfd56e46a7053c59f9478e0aa185cbbe9b21c131b38d63fa9aa3dec758024
SimHash cca201b64be1

Groups

scrapy

Rule Path
Allow /

*

Rule Path
Disallow /cgi-bin
Disallow /wp-
Disallow /wp/
Disallow /?
Disallow */comments
Disallow /users/
Disallow */trackback
Disallow */feed
Disallow */rss
Disallow */embed*
Allow /wp-admin/admin-ajax.php
Allow /wp-includes/css/dist/
Allow /wp-content/cache/min/1/
Allow /wp-content/plugins/wp-rocket/
Allow /wp-content/themes/closers/fonts/
Allow /wp-content/uploads/
Allow */
Disallow /terms-conditions
Disallow /privacy-policy
Disallow /thank-you
Disallow /tag/
Disallow /category/
Disallow /resource-type/
Disallow /testimonials-category/
Disallow /wp-contet/uploads/*/*/*.pdf
Disallow /wp-contet/uploads/*/*/*.doc
Disallow /wp-contet/uploads/*/*/*.docx
Disallow /*.pdf
Disallow /*.doc
Disallow /*.docx
Disallow /*gclid%3D*
Disallow /*fbclid%3D*
Disallow /*?utm_source=
Disallow /*utm_medium%3D
Disallow /*utm_campaign%3D
Disallow /*utm_content%3D
Disallow /*utm_term%3D
Allow */favicon/*.png

Other Records

Field Value
sitemap https://www.websiteclosers.com/sitemap_index.xml