pesi23.com
robots.txt

Robots Exclusion Standard data for pesi23.com

Resource Scan

Scan Details

Site Domain pesi23.com
Base Domain pesi23.com
Scan Status Ok
Last Scan2024-05-18T05:15:39+00:00
Next Scan 2024-06-17T05:15:39+00:00

Last Scan

Scanned2024-05-18T05:15:39+00:00
URL https://pesi23.com/robots.txt
Redirect https://www.pesi.com/robots.txt
Redirect Domain www.pesi.com
Redirect Base pesi.com
Domain IPs 3.13.31.214
Redirect IPs 216.56.243.154
Response IP 216.56.243.154
Found Yes
Hash 92548a74972c74060fd9ae7d6a141874eca8dfa514184b1ad4bd2fa03bdf7785
SimHash 3356383943a5

Groups

*

Rule Path
Allow /
Disallow /*?title=
Disallow /*?keyword=
Disallow /*?keywordSearchType=
Disallow /*?fromDuration=
Disallow /*?pagesize=
Disallow /*%26title%3D
Disallow /*%26hsLang%3D
Disallow /*%26keyword%3D
Disallow /*%26keywordSearchType%3D
Disallow /*%26fromDuration%3D
Disallow /*%26pagesize%3D
Disallow /*?_gl=
Disallow /*?_ga=
Disallow /*?utm_source=
Disallow /*?utm_medium=
Disallow /*?utm_campaign=
Disallow /*?utm_term=
Disallow /*?utm_content=
Disallow /*?__hstc=
Disallow /*?__hssc=
Disallow /*?__hsfp=
Disallow /*?hsLang=
Disallow /*?internalad=
Disallow /*%26_gl%3D
Disallow /*%26_ga%3D
Disallow /*%26utm_source%3D
Disallow /*%26utm_medium%3D
Disallow /*%26utm_campaign%3D
Disallow /*%26utm_term%3D
Disallow /*%26utm_content%3D
Disallow /*%26__hstc%3D
Disallow /*%26__hssc%3D
Disallow /*%26__hsfp%3D
Disallow /*%26hsLang%3D
Disallow /*%26internalad%3D
Disallow /*?response_type=
Disallow /*?scope=
Disallow /*?response_mode=
Disallow /*?nonce=
Disallow /*%26response_type%3D
Disallow /*%26response_mode%3D
Disallow /*%26scope%3D
Disallow /*%26nonce%3D
Disallow /bin/
Disallow /scripts/
Disallow /fonts/
Disallow /account/
Disallow /cetest/
Disallow /kajabi/
Disallow /orderhistory/

ahrefsbot

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

riddlerbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

whatcms

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

linkfluence

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

riddlerbot

Rule Path
Disallow /

mediatoolkitbot

Rule Path
Disallow /

aspiegelbot

Rule Path
Disallow /

domainstatsbot

Rule Path
Disallow /

cincraw

Rule Path
Disallow /

nimbostratus

Rule Path
Disallow /

httrack

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

megaindex

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

semanticbot

Rule Path
Disallow /

cocolyzebot

Rule Path
Disallow /

domcopbot

Rule Path
Disallow /

traackr

Rule Path
Disallow /

bomborabot

Rule Path
Disallow /

linguee

Rule Path
Disallow /

webtechbot

Rule Path
Disallow /

domainstatsbot

Rule Path
Disallow /

clickagy

Rule Path
Disallow /

sqlmap

Rule Path
Disallow /

seekport

Rule Path
Disallow /

internet-structure-research-project-bot

Rule Path
Disallow /

awariosmartbot

Rule Path
Disallow /

onalyticabot

Rule Path
Disallow /

sbl-bot

Rule Path
Disallow /

pubmatic crawler bot

Rule Path
Disallow /

bvbot

Rule Path
Disallow /

sogou

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

admantx

Rule Path
Disallow /

adbeat

Rule Path
Disallow /

embed.ly

Rule Path
Disallow /

semantic-visions

Rule Path
Disallow /

voluumdsp

Rule Path
Disallow /

wc-test-dev-bot

Rule Path
Disallow /

gulperbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.pesi.com/sitemap.xml

Comments

  • Hi Crawlers
  • Disallow parameters - filters and sorting
  • Disallow parameters - hubspot and google tracking links
  • Disallow parameters - single sign on
  • Disallow folders
  • Disallow agents