Log In Sign Up

bgp.tools
robots.txt

Robots Exclusion Standard data for bgp.tools

Resource Scan

Scan Details

Site Domain bgp.tools
Base Domain bgp.tools
Scan Status Ok
Last Scan2024-09-19T19:05:40+00:00
Next Scan 2024-10-19T19:05:40+00:00

Last Scan

Scanned2024-09-19T19:05:40+00:00
URL https://bgp.tools/robots.txt
Domain IPs 185.230.223.150, 2a0c:2f07:9459:443::247
Response IP 185.230.223.150
Found Yes
Hash e564f22392ad1de472369f5b5c4e96e6c0f5e2374eb97b136b2e2d7d7758b4d8
SimHash e21cbd38ce3e

Groups

*

Rule Path
Disallow /edit/*
Disallow /fdns*
Disallow /fdns/*
Disallow /rir-owner/
Disallow /rir-owner/*
Disallow /search
Disallow /search?q=*
Disallow /pathimg/
Disallow /ixp-nexthop/
Disallow /dns/
Disallow /communities/*?show-prefixes=*
Disallow /bad-packets/
Disallow /internal/

bingbot

Rule Path
Disallow /pathimg/
Disallow /edit/*
Disallow /fdns*
Disallow /fdns/*
Disallow /rir-owner/
Disallow /rir-owner/*
Disallow /search
Disallow /search?q=*
Disallow /ixp-nexthop/
Disallow /dns/
Disallow /communities/*?show-prefixes=*
Disallow /bad-packets/
Disallow /internal/

Other Records

Field Value
crawl-delay 2

amazonbot

Rule Path
Disallow /

googlebot

Rule Path
Disallow /pathimg/
Disallow /edit/*
Disallow /fdns*
Disallow /fdns/*
Disallow /rir-owner/
Disallow /rir-owner/*
Disallow /search
Disallow /search?q=*
Disallow /ixp-nexthop/
Disallow /dns/
Disallow /bad-packets/
Disallow /internal/

yandex

Rule Path
Disallow /pathimg/
Disallow /prefix/
Disallow /as-set/
Disallow /edit/*
Disallow /fdns*
Disallow /fdns/*
Disallow /rir-owner/
Disallow /rir-owner/*
Disallow /search
Disallow /search?q=*
Disallow /ixp-nexthop/
Disallow /dns/
Disallow /communities/*?show-prefixes=*
Disallow /internal/
Disallow /bad-packets/

Other Records

Field Value
crawl-delay 2

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

awariorssbot

Rule Path
Disallow /

awariosmartbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

facebookexternalhit

Rule Path
Disallow /

qwantbot

Rule Path
Disallow /
Back to top

Comments

  • Bing's crawling rate as of May 2023 is getting unacceptable
  • Amazonbot does not support crawl-delay, and aggressively crawls with no obvious upside to letting them do it
  • bgp.tools does not find SEO backlink bots useful.
  • For crawling, please see: https://bgp.tools/kb/api
  • facebookexternalhit appears to now do more than just linkunfurling
  • it appears to be crawling, I can only assume for AI reasons
  • Crawls so fast it may as well be a DoS
Back to top