Log In Sign Up

narcotics.com
robots.txt

Robots Exclusion Standard data for narcotics.com

Resource Scan

Scan Details

Site Domain narcotics.com
Base Domain narcotics.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-09-17T01:37:15+00:00
Next Scan 2025-12-16T01:37:15+00:00

Last Successful Scan

Scanned2025-02-20T00:26:54+00:00
URL https://narcotics.com/robots.txt
Domain IPs 104.21.81.56, 172.67.157.41, 2606:4700:3030::6815:5138, 2606:4700:3035::ac43:9d29
Response IP 172.67.157.41
Found Yes
Hash 98a3c4652537226f12599e06a64b779918cc0a88d810d31199820d958d180a82
SimHash e864eacac52f

Groups

bytespider

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

youbot

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

*

Rule Path
Disallow
Back to top

Other Records

Field Value
sitemap https://www.narcotics.com/sitemap_index.xml
Back to top

Comments

  • START YOAST BLOCK
  • ---------------------------
  • Block Bytespider, used by various services for crawling.
  • Block Diffbot, a bot for extracting structured data from websites.
  • Block ImagesiftBot, likely related to image processing or crawling.
  • Block Omgili, a bot related to web crawling by Omgili.
  • Block Omgilibot, a bot used for crawling by Omgili.
  • Block YouBot, another general bot.
  • ---------------------------
  • END YOAST BLOCK
Back to top