Log In Sign Up

photokit.com
robots.txt

Robots Exclusion Standard data for photokit.com

Resource Scan

Scan Details

Site Domain photokit.com
Base Domain photokit.com
Scan Status Ok
Last Scan2024-11-14T01:42:20+00:00
Next Scan 2024-11-21T01:42:20+00:00

Last Scan

Scanned2024-11-14T01:42:20+00:00
URL https://photokit.com/robots.txt
Domain IPs 240d:c010:81:2::13f, 43.154.43.206
Response IP 43.154.43.206
Found Yes
Hash 350623f1892c170fcf646df1af40ea9734f3defaee93d9830fca18554a37ca11
SimHash 494c9dc2eb30

Groups

*

Rule Path
Allow *
Disallow /paypal/
Disallow /landing/
Disallow /app/
Disallow /login/
Disallow /colors/app/
Disallow /editor/assets/

yandexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

pinterest

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

ahrefsbot

Rule Path
Disallow /

seokicks-robot

Rule Path
Disallow /

sistrix crawler

Rule Path
Disallow /

uptimerobot/2.0

Rule Path
Disallow /

ezooms robot

Rule Path
Disallow /

perl lwp

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mj12bot/v1.4.3

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

semrushbot-bm

Rule Path
Disallow /

splitsignalbot

Rule Path
Disallow /

grapeshot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://photokit.com/sitemap/sitemaps.xml
Back to top

Comments

  • Yandex tends to be rather aggressive, may be worth keeping them at arms lenght
  • Crawlers Setup
  • User-agent: *
  • Block Ahrefs
  • Block SEOkicks
  • Block SISTRIX
  • Block Uptime robot
  • Block Ezooms Robot
  • Block Perl LWP
  • Block BlexBot
Back to top

Warnings

  • 2 invalid lines.
Back to top