Log In Sign Up

weia.world
robots.txt

Robots Exclusion Standard data for weia.world

Resource Scan

Scan Details

Site Domain weia.world
Base Domain weia.world
Scan Status Ok
Last Scan2025-12-03T17:18:59+00:00
Next Scan 2026-01-02T17:18:59+00:00

Last Scan

Scanned2025-12-03T17:18:59+00:00
URL https://weia.world/robots.txt
Domain IPs 104.26.0.165, 104.26.1.165, 172.67.74.73, 2606:4700:20::681a:1a5, 2606:4700:20::681a:a5, 2606:4700:20::ac43:4a49
Response IP 104.26.1.165
Found Yes
Hash 0a5f5881a5e4eeca01c5a3a3267bb3688d99cea4dc5b4ac1e9c8ff689ce233d4
SimHash 3855d461e700

Groups

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

yandexbot

Rule Path
Allow /

ahrefsbot

Rule Path
Allow /

semrushbot

Rule Path
Allow /

jetseo

Rule Path
Allow /

seokicks-robot

Rule Path
Allow /

mj12bot

Rule Path
Disallow /

scrapybot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

xenforo

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

netsystemsresearch

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

*

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 10

*

Rule Path
Disallow *.pdf$
Disallow *.doc$
Disallow *.docx$
Disallow *.xls$
Disallow *.xlsx$

nikto

Rule Path
Disallow /

sqlmap

Rule Path
Disallow /

nutch

Rule Path
Disallow /

pythonbot

Rule Path
Disallow /

webbandit

Rule Path
Disallow /

zmeu

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://weia.world/sitemap.xml
Back to top

Comments

  • Allow Googlebot
  • Allow Bingbot
  • Allow Yandexbot
  • Allow AhrefsBot
  • Allow SemrushBot
  • Allow JetSEO
  • Allow SEOkicks
  • Disallow MJ12bot
  • Disallow ScrapyBot
  • Disallow rogerbot
  • Disallow Baiduspider
  • Disallow PetalBot
  • Disallow DotBot
  • Disallow XenForo
  • Disallow AspiegelBot
  • Disallow NetSystemsResearch
  • Disallow DataForSeoBot
  • Disallow the rest
  • Protect Special Files
  • Disallow nikto
  • Disallow sqlmap
  • Disallow nutch
  • Disallow pythonbot
  • Disallow webbandit
  • Disallow ZmEu
Back to top