Log In Sign Up

nebi.com
robots.txt

Robots Exclusion Standard data for nebi.com

Resource Scan

Scan Details

Site Domain nebi.com
Base Domain nebi.com
Scan Status Ok
Last Scan2025-10-22T23:24:42+00:00
Next Scan 2025-11-21T23:24:42+00:00

Last Scan

Scanned2025-10-22T23:24:42+00:00
URL https://nebi.com/robots.txt
Domain IPs 104.21.67.125, 172.67.174.199, 2606:4700:3030::ac43:aec7, 2606:4700:3037::6815:437d
Response IP 172.67.174.199
Found Yes
Hash 0aa4b380c85e2fbd8fa0b63795dfedaf8717cf47c9c74a0add3d7542c3b638e1
SimHash 4d14da75d7b0

Groups

*

Rule Path
Allow /
Disallow /api/
Disallow /admin/
Disallow /private/

Other Records

Field Value
crawl-delay 10

googlebot

Rule Path
Allow /
Disallow /api/
Disallow /admin/
Disallow /private/

bingbot

Rule Path
Allow /
Disallow /api/
Disallow /admin/
Disallow /private/

slurp

Rule Path
Allow /
Disallow /api/
Disallow /admin/
Disallow /private/
Back to top

Other Records

Field Value
sitemap https://nebi.com/sitemap.xml
Back to top

Comments

  • *
  • Host
  • Sitemaps
  • Crawl-delay
  • Google Bot
  • Bing Bot
  • Yahoo Bot
Back to top

Warnings

  • `host` is not a known field.
Back to top