Log In Sign Up

magzter.com
robots.txt

Robots Exclusion Standard data for magzter.com

Resource Scan

Scan Details

Site Domain magzter.com
Base Domain magzter.com
Scan Status Ok
Last Scan2024-09-27T05:10:46+00:00
Next Scan 2024-10-04T05:10:46+00:00

Last Scan

Scanned2024-09-27T05:10:46+00:00
URL https://magzter.com/robots.txt
Redirect https://www.magzter.com/robots.txt
Redirect Domain www.magzter.com
Redirect Base magzter.com
Domain IPs 18.155.68.14, 18.155.68.52, 18.155.68.86, 18.155.68.95, 2600:9000:23d2:3400:4:2c53:6200:93a1, 2600:9000:23d2:3800:4:2c53:6200:93a1, 2600:9000:23d2:4200:4:2c53:6200:93a1, 2600:9000:23d2:7a00:4:2c53:6200:93a1, 2600:9000:23d2:aa00:4:2c53:6200:93a1, 2600:9000:23d2:ac00:4:2c53:6200:93a1, 2600:9000:23d2:b200:4:2c53:6200:93a1, 2600:9000:23d2:f200:4:2c53:6200:93a1
Redirect IPs 108.157.254.108, 108.157.254.13, 108.157.254.21, 108.157.254.36, 2600:9000:2753:2400:d:565c:f280:93a1, 2600:9000:2753:3400:d:565c:f280:93a1, 2600:9000:2753:4600:d:565c:f280:93a1, 2600:9000:2753:8800:d:565c:f280:93a1, 2600:9000:2753:9000:d:565c:f280:93a1, 2600:9000:2753:b000:d:565c:f280:93a1, 2600:9000:2753:d800:d:565c:f280:93a1, 2600:9000:2753:f400:d:565c:f280:93a1
Response IP 108.157.254.108
Found Yes
Hash 72f24492875ba123a3e1eaaf7ac1e0aaa8678e4bf42102910879ab07cf7df374
SimHash 76149861c7b1

Groups

*

Rule Path
Disallow /android/
Disallow /share/
Disallow /mailer/
Disallow /offers/parents_day/
Disallow /preview/
Disallow /reader/
Disallow /login/
Disallow /login_modal/
Disallow /connect/
Disallow /irctc/

anthropic-ai

Rule Path
Disallow /stories/newspaper/Toronto-Star/

claude-web

Rule Path
Disallow /stories/newspaper/Toronto-Star/

applebot-extended

Rule Path
Disallow /stories/newspaper/Toronto-Star/

bytespider

Rule Path
Disallow /stories/newspaper/Toronto-Star/

ccbot

Rule Path
Disallow /stories/newspaper/Toronto-Star/

chatgpt-user

Rule Path
Disallow /stories/newspaper/Toronto-Star/

cohere-ai

Rule Path
Disallow /stories/newspaper/Toronto-Star/

diffbot

Rule Path
Disallow /stories/newspaper/Toronto-Star/

facebookbot

Rule Path
Disallow /stories/newspaper/Toronto-Star/

googleother

Rule Path
Disallow /stories/newspaper/Toronto-Star/

google-extended

Rule Path
Disallow /stories/newspaper/Toronto-Star/

gptbot

Rule Path
Disallow /stories/newspaper/Toronto-Star/

imagesiftbot

Rule Path
Disallow /stories/newspaper/Toronto-Star/

perplexitybot

Rule Path
Disallow /stories/newspaper/Toronto-Star/

omigilibot

Rule Path
Disallow /stories/newspaper/Toronto-Star/

omigili

Rule Path
Disallow /stories/newspaper/Toronto-Star/

Other Records

Field Value
crawl-delay 1
Back to top

Other Records

Field Value
sitemap https://www.magzter.com/sitemapxml/index.xml
Back to top