startupbuffer.com
robots.txt

Robots Exclusion Standard data for startupbuffer.com

Resource Scan

Scan Details

Site Domain startupbuffer.com
Base Domain startupbuffer.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-11-18T12:37:56+00:00
Next Scan 2024-11-19T12:37:56+00:00

Last Successful Scan

Scanned2024-11-11T12:37:47+00:00
URL https://startupbuffer.com/robots.txt
Domain IPs 104.21.74.22, 172.67.195.124, 2606:4700:3030::6815:4a16, 2606:4700:3035::ac43:c37c
Response IP 104.21.74.22
Found Yes
Hash e730e25c99864e9c50c47d7e88df7f25217d3bd0df20dea4302d50e69d9aec60
SimHash 555f1503cbc8

Groups

yandexbot
ahrefsbot
semrushbot
dataforseobot
dotbot
blexbot
mj12bot
scraptastico
amazonbot
bytespider
applebot
awariobot
scrapy
barkrowler
bingbot
claudebot
facebookexternalhit
bot

Rule Path
Disallow /

*

Rule Path
Allow /

Other Records

Field Value
sitemap http://startupbuffer.com/sitemap.xml
sitemap http://startupbuffer.com/resources/sitemap_index.xml
sitemap http://startupbuffer.com/site/compare_sitemap_all

Comments

  • Sitemap: https://startupbuffer.com/mpg-sitemap-world-cities.xml