Log In Sign Up

mghazinews.org
robots.txt

Robots Exclusion Standard data for mghazinews.org

Resource Scan

Scan Details

Site Domain mghazinews.org
Base Domain mghazinews.org
Scan Status Ok
Last Scan2024-06-29T12:17:13+00:00
Next Scan 2024-07-06T12:17:13+00:00

Last Scan

Scanned2024-06-29T12:17:13+00:00
URL https://mghazinews.org/robots.txt
Redirect https://www.mghazinews.org/robots.txt
Redirect Domain www.mghazinews.org
Redirect Base mghazinews.org
Domain IPs 104.21.3.98, 172.67.153.126, 2606:4700:3030::ac43:997e, 2606:4700:3037::6815:362
Redirect IPs 104.21.3.98, 172.67.153.126, 2606:4700:3030::ac43:997e, 2606:4700:3037::6815:362
Response IP 172.67.153.126
Found Yes
Hash 9081c88dd389c1ed97a911175f46bc84b5a920b1edce511e270b5a62e37be990
SimHash 65358541f63a

Groups

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin/
Disallow /cgi-bin/
Disallow /wp-admin/
Disallow /linkout/
Disallow /recommended/
Disallow /comments/feed/
Disallow /trackback/
Disallow /index.php
Disallow /xmlrpc.php
Disallow /tag/
Disallow /page/

rogerbot

Rule Path
Disallow /tag/
Disallow /quizs*

pinterest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

pinterestbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

pinterestbot/1.0

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

pinterest/0.2 (+https://www.pinterest.com/bot.html)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mozilla/5.0 (compatible; pinterestbot/1.0; +https://www.pinterest.com/bot.html)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mozilla/5.0 (linux; android 6.0.1; nexus 5x build/mmb29p) applewebkit/537.36 (khtml, like gecko) chrome/41.0.2272.96 mobile safari/537.36 (compatible; pinterestbot/1.0; +https://www.pinterest.com/bot.html)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

semrushbot-sa

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mediapartners-google*

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

adsbot-google

Rule Path
Allow /

googlebot-mobile

Rule Path
Allow /

seekport crawler

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

seekbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

yandex

Rule Path
Disallow /admin/*

mozilla/5.0 (compatible; mj12bot/v1.4.8; https://mj12bot.com/)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

facebookexternalhit/1.1

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

facebookexternalhit/1.1 (+https://www.facebook.com/externalhit_uatext.php)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

blexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

mozilla/5.0 (macintosh; intel mac os x 10_9) applewebkit/537.71 (khtml, like gecko) version/7.0 safari/537.71 (rival iq, rivaliq.com)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mozilla/5.0 (macintosh; intel mac os x 10_11_6) applewebkit/537.36 (khtml, like gecko) chrome/52.0.2743.116 safari/537.36 (rival iq, rivaliq.com)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mozilla/5.0(macintosh; intel mac os x 10_11_6)applewebkit/537.36(khtml, like gecko)chrome/52.0.2743.116safari/537.36(rival iq, rivaliq.com)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mozilla/5.0 (macintosh; intel mac os x 10_11_6) applewebkit/537.36 (khtml, like gecko) chrome/52.0.2743.116 safari/537.36 (rival iq, rivaliq.com)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mozilla/5.0 (macintosh; intel mac os x 10_9) applewebkit/537.71 (khtml, like gecko) version/7.0 safari/537.71 (rival iq, rivaliq.com)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

haosouspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mozilla/5.0+(compatible; uptimerobot/2.0; https://www.uptimerobot.com/)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mozilla/5.0 (compatible; bingbot/2.0; +https://www.bing.com/bingbot.htm)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

mozilla/5.0 applewebkit/537.36 (khtml, like gecko; compatible; bingbot/2.0; +https://www.bing.com/bingbot.htm) chrome/w.x.y.z safari/537.36 edg/w.x.y.z

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

mozilla/5.0 (linux; android 6.0.1; nexus 5x build/mmb29p) applewebkit/537.36 (khtml, like gecko) chrome/w.x.y.z mobile safari/537.36 edg/w.x.y.z (compatible; bingbot/2.0; +https://www.bing.com/bingbot.htm)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

mozilla/5.0 (compatible; adidxbot/2.0; +https://www.bing.com/bingbot.htm)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

mozilla/5.0 (iphone; cpu iphone os 7_0 like mac os x) applewebkit/537.51.1 (khtml, like gecko) version/7.0 mobile/11a465 safari/9537.53 (compatible; adidxbot/2.0; +https://www.bing.com/bingbot.htm)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

mozilla/5.0 (windows phone 8.1; arm; trident/7.0; touch; rv:11.0; iemobile/11.0; nokia; lumia 530) like gecko (compatible; adidxbot/2.0; +https://www.bing.com/bingbot.htm)

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

mozilla/5.0 (windows nt 6.1; wow64) applewebkit/534+ (khtml, like gecko) bingpreview/1.0b

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

mozilla/5.0 (windows phone 8.1; arm; trident/7.0; touch; rv:11.0; iemobile/11.0; nokia; lumia 530) like gecko bingpreview/1.0b

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

msnbot-media

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2
Back to top

Other Records

Field Value
sitemap https://www.mghazinews.com/sitemap_index.xml
Back to top

Comments

  • Crawl-delay: 30
  • User-agent: Slurp
  • Crawl-delay: 10
  • User-agent: DuckDuckBot
  • Crawl-delay: 10
  • User-agent: ia_archiver
  • Crawl-delay: 10
  • User-agent: amazonbot
  • Crawl-delay: 10
  • User-agent: NinjaBot
  • Disallow: /
  • Allow Seekbot with delay - https://www.seekport.co.uk/seekbot/
  • Allow Seekbot with delay - https://www.seekport.co.uk/seekbot/
  • User-agent: Mozilla/5.0 (compatible; YaK/1.0; https://linkfluence.com/; bot@linkfluence.com)
  • Disallow: /
  • User-agent: Buck/2.2; (+https://app.hypefactors.com/media-monitoring/about.html)
  • Disallow: /
  • User-agent: Mozilla/5.0 (compatible; Daum/4.1; +https://cs.daum.net/faq/15/4118.html?faqId=28966)
  • Disallow: /
  • new bots info added
  • User-agent: Yandex
  • Crawl-delay: 10
  • User-agent: Mozilla/5.0 (compatible; Semanticbot/1.0; +https://sempi.tech/bot.html)
  • Crawl-delay: 10
  • User-agent: Baiduspider
  • Crawl-delay: 10
  • User-agent: PetalBot
  • Disallow: /
  • User-agent: Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://aspiegel.com/petalbot)
  • Disallow: /
  • Source: https://www.rocketrental.co.nz/robots.txt
  • Header: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider
  • Header: Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0); 360Spider(compatible; HaosouSpider; https://www.haosou.com/help/help_3_2.html)
  • Source: https://developers.whatismybrowser.com/useragents/parse/1109735-uptime-robot
  • bing other agents delay goes here. Source: https://www.bing.com/webmaster/help/which-crawlers-does-bing-use-8c184ec0
  • and siteground.com/kb/how_to_decrease_the_crawl_rate_of_bings_search_engine_bot/
Back to top

Warnings

  • 2 invalid lines.
Back to top