weareexplorers.co
robots.txt

Robots Exclusion Standard data for weareexplorers.co

Resource Scan

Scan Details

Site Domain weareexplorers.co
Base Domain weareexplorers.co
Scan Status Ok
Last Scan2024-11-16T19:24:08+00:00
Next Scan 2024-11-23T19:24:08+00:00

Last Scan

Scanned2024-11-16T19:24:08+00:00
URL https://weareexplorers.co/robots.txt
Domain IPs 162.159.135.42
Response IP 162.159.135.42
Found Yes
Hash e0f0a84b77cc5f6e278a62f46bb8b6e684d7689e71517c1f43e7449ba49a97d6
SimHash 104a9b18aca2

Groups

*

Rule Path
Disallow /wp-admin/
Disallow *sort%3D
Disallow *?vo=
Disallow */s%3D
Disallow */wp-signup.php
Disallow /feed/*
Disallow */?*
Disallow */feed/
Disallow */events/2024*
Disallow */events/category/*

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

amazonadbot

Rule Path
Disallow /

adsbot

Rule Path
Disallow /

adsbot/3.1

Rule Path
Disallow /

criteobot

Rule Path
Disallow /

molokaibot

Rule Path
Disallow /

pinterestbot

Rule Path
Disallow /

dataforseobot/1.0

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

twengabot

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

storebot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

zumbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

molokaibot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

coccocbot-image

Rule Path
Disallow /

coccocbot-web

Rule Path
Disallow /
Disallow CVS
Disallow .svn
Disallow .idea
Disallow .sql
Disallow .tgz
Disallow *post_type%3Dtribe_events*
Disallow *hide_subsequent_recurrences%3D*
Disallow *tribe-bar-date%3D*
Disallow *tribe-venue%3D*
Disallow *eventDisplay%3D*
Disallow *eventDate%3D*
Disallow *paged%3D*
Disallow *pagename%3D*
Disallow *shortcode%3D*
Disallow *ical%3D*
Disallow *outlook-ical%3D*
Disallow *related_series%3D*
Disallow *tribe_geofence%3D*
Disallow *tribe_organizer%3D*

*

Rule Path
Disallow /wp-content/uploads/wp-import-export-lite/

Other Records

Field Value
sitemap https://weareexplorers.co/sitemap_index.xml

Comments

  • Block most common AI bots
  • Block most common non-AU search engines
  • Do not crawl development files and folders: CVS, svn directories and dump files
  • For Events Calendar
  • WP Import Export Rule

Warnings

  • 2 invalid lines.
  • `host` is not a known field.