Log In Sign Up

markaba.news
robots.txt

Robots Exclusion Standard data for markaba.news

Resource Scan

Scan Details

Site Domain markaba.news
Base Domain markaba.news
Scan Status Ok
Last Scan2026-02-07T03:09:01+00:00
Next Scan 2026-03-09T03:09:01+00:00

Last Scan

Scanned2026-02-07T03:09:01+00:00
URL https://markaba.news/robots.txt
Redirect https://www.markaba.news/robots.txt
Redirect Domain www.markaba.news
Redirect Base markaba.news
Domain IPs 104.21.57.102, 172.67.162.241, 2606:4700:3033::6815:3966, 2606:4700:3036::ac43:a2f1
Redirect IPs 104.21.57.102, 172.67.162.241, 2606:4700:3033::6815:3966, 2606:4700:3036::ac43:a2f1
Response IP 172.67.162.241
Found Yes
Hash 195a107257750c80b7c6eba1516c574af03b3b5ff007e08075ff1dc068bc9781
SimHash 391c26704da8

Groups

*

Rule Path
Allow /
Disallow /admin/
Disallow /api/
Disallow /auth/
Disallow /login
Disallow /_next/
Disallow /static/
Disallow /404
Disallow /500
Disallow /search?*

ahrefsbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.markaba.news/api/sitemap-index.xml
Back to top

Comments

  • Robots.txt for Markaba News
  • This file tells search engine crawlers which pages or files the crawler can or can't request from your site.
  • Main site crawling rules
  • Disallow access to admin and private areas
  • Disallow access to error pages
  • Disallow access to search results with parameters to avoid duplicate content
  • Sitemap locations
  • Block malicious bots
Back to top