techgropse.com
robots.txt

Robots Exclusion Standard data for techgropse.com

Resource Scan

Scan Details

Site Domain techgropse.com
Base Domain techgropse.com
Scan Status Ok
Last Scan2025-10-17T00:43:44+00:00
Next Scan 2025-11-16T00:43:44+00:00

Last Scan

Scanned2025-10-17T00:43:44+00:00
URL https://techgropse.com/robots.txt
Redirect https://www.techgropse.com/robots.txt
Redirect Domain www.techgropse.com
Redirect Base techgropse.com
Domain IPs 104.21.14.15, 172.67.133.191, 2606:4700:3030::ac43:85bf, 2606:4700:3032::6815:e0f
Redirect IPs 104.21.14.15, 172.67.133.191, 2606:4700:3030::ac43:85bf, 2606:4700:3032::6815:e0f
Response IP 172.67.133.191
Found Yes
Hash a1d135db83afc71b5bcd8c372818e6bfcca9165d57bebbb106e56ed78d8acad2
SimHash 71154253cbc1

Groups

*

Rule Path
Allow /wp-admin/admin-ajax.php

gptbot

Rule Path
Allow /

oai-searchbot

Rule Path
Allow /

ccbot

Rule Path
Allow /

bingbot

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

perplexitybot

Rule Path
Allow /

claudebot

Rule Path
Allow /

amazonbot

Rule Path
Allow /

applebot

Rule Path
Allow /

bytespider

Rule Path
Allow /

claudebot

Rule Path
Allow /

duckassistbot

Rule Path
Allow /

google-cloudvertexbot

Rule Path
Allow /

googleother

Rule Path
Allow /

gptbot

Rule Path
Allow /

meta-externalagent

Rule Path
Allow /

petalbot

Rule Path
Allow /

tiktokspider

Rule Path
Allow /

ccbot

Rule Path
Allow /

*

Rule Path
Disallow /cgi-bin/
Disallow /*.phtml$
Disallow /*.htm$
Disallow /blog/wp-admin/
Disallow /thank-you/

angeltales
michellereneesurrogate
idena-apps

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.techgropse.com/blog/post-sitemap.xml
sitemap https://www.techgropse.com/sitemap.xml
sitemap https://www.techgropse.com/blog/sitemap_index.xml

Comments

  • Allow AI/data collection bots full access to public content