Log In Sign Up

newagebd.net
robots.txt

Robots Exclusion Standard data for newagebd.net

Resource Scan

Scan Details

Site Domain newagebd.net
Base Domain newagebd.net
Scan Status Ok
Last Scan2024-05-15T01:12:56+00:00
Next Scan 2024-05-22T01:12:56+00:00

Last Scan

Scanned2024-05-15T01:12:56+00:00
URL https://newagebd.net/robots.txt
Domain IPs 104.26.14.170, 104.26.15.170, 172.67.69.148, 2606:4700:20::681a:eaa, 2606:4700:20::681a:faa, 2606:4700:20::ac43:4594
Response IP 104.26.14.170
Found Yes
Hash a9cf185521605e99167279d33308834d0d4429caa2632853cc39fa037dd32929
SimHash 6b2cd290c371

Groups

*

Rule Path
Disallow /

googlebot

Rule Path
Disallow
Allow /

mediapartners-google

Rule Path
Disallow
Allow /

googlebot-news

Rule Path
Disallow
Allow /

twitterbotf

Rule Path
Disallow
Allow /

bingbot

Rule Path
Disallow
Allow /

msnbot

Rule Path
Disallow
Allow /

facebookexternalhit

Rule Path
Disallow
Allow /
Back to top

Comments

  • Crawl-delay: 1
  • Crawl-delay: 5
  • Crawl-delay: 5
  • Crawl-delay: 1
  • Crawl-delay: 1
  • Crawl-delay: 1
  • Crawl-delay: 5
  • User-agent: *
  • Disallow:
Back to top