brookpub.com
robots.txt

Robots Exclusion Standard data for brookpub.com

Resource Scan

Scan Details

Site Domain brookpub.com
Base Domain brookpub.com
Scan Status Ok
Last Scan2025-06-09T16:14:06+00:00
Next Scan 2025-07-09T16:14:06+00:00

Last Scan

Scanned2025-06-09T16:14:06+00:00
URL https://www.brookpub.com/robots.txt
Domain IPs 173.248.128.21
Response IP 173.248.128.21
Found Yes
Hash 18e9a9468a41131c1123f560d7a17433d60cb18f0491c5130bc2f8c47a882b43
SimHash 34111b10e2e2

Groups

adsbot-google
adsbot-google-mobile
adsbot-google-mobile-apps
adidxbot
applebot
applenewsbot
bingbot
bingpreview
bublupbot
ccbot
duckduckbot
duckduckgo-favicons-bot
facebookcatalog
facebookexternalhit
facebot
feedly
googlebot
googlebot-image
googlebot-mobile
googlebot-news
googlebot-video
lycos
mediapartners-google
microsoftpreview
mojeekbot
msnbot
msnbot-media
pinterest
slurp
twitterbot
whatsapp
yacybot
yepbot

Rule Path
Allow /

ahrefsbot
dotbot
majestic
mauibot
semrushbot
petalbot
yandexbot

Rule Path
Disallow /

*

Rule Path
Disallow /

Comments

  • ROBOTS.TXT
  • Updates and informantion can be found at:
  • https://www.ditig.com/publications/robots-txt-template
  • This document is licensed with a CC BY-NC-SA 4.0 license.
  • Last update: 2023-11-28
  • Specific list of good bots
  • google.com landing page quality checks
  • google.com app resource fetcher
  • bing ads bot
  • apple.com search engine
  • bing.com international search engine
  • bublup.com suggestion/search engine
  • commoncrawl.org open repository of web crawl data
  • duckduckgo.com international privacy search engine
  • facebook.com social network
  • feedly.com feed fetcher
  • google.com international search engine
  • lycos.com & hotbot.com international search engine
  • google.com adsense bot
  • Preview bot for Microsoft products
  • mojeek.com search engine
  • bing.com international search engine
  • pinterest.com social networtk
  • yahoo.com international search engine
  • twitter.com social media bot
  • whatsapp.com preview bot
  • yacy.net p2p search software
  • yep.com search engine
  • crawling rule(s) for above bots
  • Specific list of bad bots
  • disallow all other bots