Log In Sign Up

medicalnewstoday.com
robots.txt

Robots Exclusion Standard data for medicalnewstoday.com

Resource Scan

Scan Details

Site Domain medicalnewstoday.com
Base Domain medicalnewstoday.com
Scan Status Ok
Last Scan2024-05-09T19:01:41+00:00
Next Scan 2024-05-16T19:01:41+00:00

Last Scan

Scanned2024-05-09T19:01:41+00:00
URL https://medicalnewstoday.com/robots.txt
Redirect https://www.medicalnewstoday.com/robots.txt
Redirect Domain www.medicalnewstoday.com
Redirect Base medicalnewstoday.com
Domain IPs 108.138.246.115, 108.138.246.119, 108.138.246.123, 108.138.246.84
Redirect IPs 13.225.4.127, 13.225.4.27, 13.225.4.54, 13.225.4.74, 2600:9000:21b4:1000:0:db4d:7d40:93a1, 2600:9000:21b4:4e00:0:db4d:7d40:93a1, 2600:9000:21b4:7e00:0:db4d:7d40:93a1, 2600:9000:21b4:8e00:0:db4d:7d40:93a1, 2600:9000:21b4:a800:0:db4d:7d40:93a1, 2600:9000:21b4:b800:0:db4d:7d40:93a1, 2600:9000:21b4:da00:0:db4d:7d40:93a1, 2600:9000:21b4:f800:0:db4d:7d40:93a1
Response IP 13.225.4.27
Found Yes
Hash 1e58a4f7e3e517f96db44d32414395f6fbe064a9f9aca58cce4d4219ec5ee2ad
SimHash 2b0158e4f395

Groups

mediapartners-google*

Rule Path
Disallow

nutch

Rule Path
Disallow

Other Records

Field Value
crawl-delay 5

slurp

Rule Path
Disallow /*.gif$
Disallow /*.jpg$

*

Rule Path
Disallow /linkfwd.php
Disallow /counters.php
Disallow /articles/mnt-*
Disallow /program/mnt-*
Disallow /api/*
Disallow */null$
Disallow */inline$

Other Records

Field Value
crawl-delay 5

gptbot

Rule Path
Disallow /
Disallow /articles/widget-sampler
Disallow /test/
Back to top

Other Records

Field Value
sitemap https://www.medicalnewstoday.com/sitemap.xml
Back to top

Comments

  • Wordpress Previews
  • API Routes
  • Invalid URLs
  • Sitemaps
  • Widget Sampler
  • Static Test Articles
Back to top