Log In Sign Up

maateen.me
robots.txt

Robots Exclusion Standard data for maateen.me

Resource Scan

Scan Details

Site Domain maateen.me
Base Domain maateen.me
Scan Status Ok
Last Scan2025-10-28T09:39:57+00:00
Next Scan 2025-11-27T09:39:57+00:00

Last Scan

Scanned2025-10-28T09:39:57+00:00
URL https://maateen.me/robots.txt
Domain IPs 104.21.4.87, 172.67.131.223, 2606:4700:3034::6815:457, 2606:4700:3034::ac43:83df
Response IP 104.21.4.87
Found Yes
Hash 7bbbc4853e66f47966aaa5e00305bf050fda35012d452716787e72351aa2a9bb
SimHash 6484ba336492

Groups

*

Rule Path
Allow /
Disallow /admin/
Disallow /*.json$
Disallow /*_print$
Disallow /*?print$
Allow /static/
Allow /assets/
Allow /img/

Other Records

Field Value
crawl-delay 1

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

slurp

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

baiduspider

Rule Path
Allow /

yandexbot

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://maateen.me/sitemap.xml
Back to top

Comments

  • Disallow specific paths that shouldn't be indexed
  • Allow important directories
  • Sitemap
  • Crawl delay for respectful crawling
  • Allow all major search engines
Back to top