astrotak.com
robots.txt

Robots Exclusion Standard data for astrotak.com

Resource Scan

Scan Details

Site Domain astrotak.com
Base Domain astrotak.com
Scan Status Ok
Last Scan2024-05-18T09:44:11+00:00
Next Scan 2024-06-17T09:44:11+00:00

Last Scan

Scanned2024-05-18T09:44:11+00:00
URL https://astrotak.com/robots.txt
Redirect https://www.astrotak.com:443/robots.txt
Redirect Domain www.astrotak.com
Redirect Base astrotak.com
Domain IPs 13.127.112.82, 15.207.175.211, 3.110.6.210
Redirect IPs 108.157.254.105, 108.157.254.12, 108.157.254.37, 108.157.254.46, 2600:9000:2753:1800:9:71ec:8280:93a1, 2600:9000:2753:1e00:9:71ec:8280:93a1, 2600:9000:2753:5000:9:71ec:8280:93a1, 2600:9000:2753:5800:9:71ec:8280:93a1, 2600:9000:2753:8e00:9:71ec:8280:93a1, 2600:9000:2753:a00:9:71ec:8280:93a1, 2600:9000:2753:aa00:9:71ec:8280:93a1, 2600:9000:2753:dc00:9:71ec:8280:93a1
Response IP 108.157.254.105
Found Yes
Hash 51e1a5b9869f995367f9660d946696e9cf869b045fa8d1a4ff92d1119b47d929
SimHash 418cd440c7d7

Groups

*

Rule Path
Allow /
Disallow /?*
Disallow /static.astrotak.com/*
Disallow /m.astrotak.com/*

freefind

Rule Path
Disallow /

becomebot

Rule Path
Disallow /

chatgpt

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

nutch

Rule Path
Disallow /

jetbot/1.0

Rule Path
Disallow /

jetbot

Rule Path
Disallow /

webzip

Rule Path
Disallow /

netmechanic

Rule Path
Disallow /

url_spider_pro

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

prowebwalker

Rule Path
Disallow /

cheesebot

Rule Path
Disallow /

lnspiderguy

Rule Path
Disallow /

alexibot

Rule Path
Disallow /

webzip/4.0

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

websauger

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

netants

Rule Path
Disallow /

mister pix

Rule Path
Disallow /

webauto

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.astrotak.com/sitemap/sitemap.xml