nyburs.com
robots.txt

Robots Exclusion Standard data for nyburs.com

Resource Scan

Scan Details

Site Domain nyburs.com
Base Domain nyburs.com
Scan Status Ok
Last Scan2026-01-31T17:28:39+00:00
Next Scan 2026-02-14T17:28:39+00:00

Last Scan

Scanned2026-01-31T17:28:39+00:00
URL https://nyburs.com/robots.txt
Redirect https://www.nyburs.com/robots.txt
Redirect Domain www.nyburs.com
Redirect Base nyburs.com
Domain IPs 76.76.21.21
Redirect IPs 104.26.6.60, 104.26.7.60, 172.67.71.93, 2606:4700:20::681a:63c, 2606:4700:20::681a:73c, 2606:4700:20::ac43:475d
Response IP 104.26.7.60
Found Yes
Hash 6b1c412aa7017c026cb07f5fd4b95d7f3817f7103a0c678547a281d30778c835
SimHash 2c865605c4e2

Groups

*

Rule Path
Allow /
Allow /landing
Allow /blogs
Allow /blogs/*
Allow /privacy
Allow /term-and-condition
Allow /why-nyburs
Allow /community-guidelines
Allow /cookie-policy
Allow /grievance-policy
Disallow /chat
Disallow /notification
Disallow /settings
Disallow /profile/edit
Disallow /api/

facebookexternalhit

Rule Path
Allow /

twitterbot

Rule Path
Allow /

linkedinbot

Rule Path
Allow /

whatsapp

Rule Path
Allow /

telegrambot

Rule Path
Allow /

slackbot

Rule Path
Allow /

discordbot

Rule Path
Allow /

pinterestbot

Rule Path
Allow /

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

Other Records

Field Value
sitemap https://www.nyburs.com/sitemap.xml

Comments

  • Robots.txt for Nyburs
  • Allow all search engines to crawl the site
  • Allow crawling of public pages
  • Disallow crawling of private/authenticated pages
  • Disallow API endpoints
  • Allow social media crawlers specifically
  • Sitemap location