magzter.com
robots.txt

Robots Exclusion Standard data for magzter.com

Resource Scan

Scan Details

Site Domain magzter.com
Base Domain magzter.com
Scan Status Ok
Last Scan2024-11-15T11:08:27+00:00
Next Scan 2024-11-22T11:08:27+00:00

Last Scan

Scanned2024-11-15T11:08:27+00:00
URL https://magzter.com/robots.txt
Redirect https://www.magzter.com/robots.txt
Redirect Domain www.magzter.com
Redirect Base magzter.com
Domain IPs 18.155.68.14, 18.155.68.52, 18.155.68.86, 18.155.68.95, 2600:9000:23d2:2400:4:2c53:6200:93a1, 2600:9000:23d2:4a00:4:2c53:6200:93a1, 2600:9000:23d2:5000:4:2c53:6200:93a1, 2600:9000:23d2:6800:4:2c53:6200:93a1, 2600:9000:23d2:7c00:4:2c53:6200:93a1, 2600:9000:23d2:a400:4:2c53:6200:93a1, 2600:9000:23d2:b000:4:2c53:6200:93a1, 2600:9000:23d2:f600:4:2c53:6200:93a1
Redirect IPs 108.157.254.108, 108.157.254.13, 108.157.254.21, 108.157.254.36, 2600:9000:2753:2200:d:565c:f280:93a1, 2600:9000:2753:2600:d:565c:f280:93a1, 2600:9000:2753:3600:d:565c:f280:93a1, 2600:9000:2753:5600:d:565c:f280:93a1, 2600:9000:2753:8e00:d:565c:f280:93a1, 2600:9000:2753:b400:d:565c:f280:93a1, 2600:9000:2753:e000:d:565c:f280:93a1, 2600:9000:2753:ea00:d:565c:f280:93a1
Response IP 108.157.254.108
Found Yes
Hash 88ded7c10a3562b13ebd47511f5031a2b3c67640a506bcdc2f903ec25b9f66b1
SimHash 7b149861c5b2

Groups

*

Rule Path
Disallow /android/
Disallow /share/
Disallow /mailer/
Disallow /offers/parents_day/
Disallow /preview/
Disallow /reader/
Disallow /login/
Disallow /login_modal/
Disallow /connect/
Disallow /irctc/
Disallow /stories/business/Fortune-India/
Disallow /stories/business/3703/Fortune-India-Magazine-Magazine%20%20%20Business/
Disallow /stories/Newspaper/Hindustan-Times/
Disallow /stories/newspaper/16993/Hindustan-Times-Magazine-Magazine%20%20%20HT/
Disallow /stories/newspaper/17006/Mint-Mumbai-Magazine-Magazine%20%20%20HT/
Disallow /stories/newspaper/Mint-Mumbai/
Disallow /stories/newspaper/The-Washington-Post/
Disallow /stories/newspaper/31495/The-Washington-Post-Magazine-Magazine%20%20%20HT/

anthropic-ai

Rule Path
Disallow /stories/newspaper/Toronto-Star/

claude-web

Rule Path
Disallow /stories/newspaper/Toronto-Star/

applebot-extended

Rule Path
Disallow /stories/newspaper/Toronto-Star/

bytespider

Rule Path
Disallow /stories/newspaper/Toronto-Star/

ccbot

Rule Path
Disallow /stories/newspaper/Toronto-Star/

chatgpt-user

Rule Path
Disallow /stories/newspaper/Toronto-Star/

cohere-ai

Rule Path
Disallow /stories/newspaper/Toronto-Star/

diffbot

Rule Path
Disallow /stories/newspaper/Toronto-Star/

facebookbot

Rule Path
Disallow /stories/newspaper/Toronto-Star/

googleother

Rule Path
Disallow /stories/newspaper/Toronto-Star/

google-extended

Rule Path
Disallow /stories/newspaper/Toronto-Star/

gptbot

Rule Path
Disallow /stories/newspaper/Toronto-Star/

imagesiftbot

Rule Path
Disallow /stories/newspaper/Toronto-Star/

perplexitybot

Rule Path
Disallow /stories/newspaper/Toronto-Star/

omigilibot

Rule Path
Disallow /stories/newspaper/Toronto-Star/

omigili

Rule Path
Disallow /stories/newspaper/Toronto-Star/

Other Records

Field Value
crawl-delay 1

Other Records

Field Value
sitemap https://www.magzter.com/sitemapxml/index.xml