Log In Sign Up

photowall.nl
robots.txt

Robots Exclusion Standard data for photowall.nl

Resource Scan

Scan Details

Site Domain photowall.nl
Base Domain photowall.nl
Scan Status Ok
Last Scan2024-09-21T04:10:12+00:00
Next Scan 2024-10-21T04:10:12+00:00

Last Scan

Scanned2024-09-21T04:10:12+00:00
URL https://photowall.nl/robots.txt
Domain IPs 108.128.253.100, 52.31.107.109
Response IP 108.128.253.100
Found Yes
Hash bc5a024a147327ce33e33addd303ebba71a52b4d6d12ca5711924e0283835e06
SimHash f0586fb36f87

Groups

seznambot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

yandex

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

ltx71 - (http://ltx71.com/)

Rule Path
Disallow /
Disallow /

serpstatbot

Rule Path
Disallow /

vebot

Rule Path
Disallow /

barkrowler/0.9

Rule Path
Disallow /

zoominfobot

Rule Path
Disallow /

gowikibot

Rule Path
Disallow /

gowikibot

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

lumtelbot

Rule Path
Disallow /

coccocbot-web

Rule Path
Disallow /

uptimebot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

pinterestbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

chatgpt-user
ccbot
perplexitybot
anthropic-ai
claudebot
claude-web
bytespider
applebot
alexa
bingbot/2.0
imagesiftbot
omgili
omgilibot
friendlycrawler
awariorssbot
awariosmartbot
dataforseobot
diffbot
img2dataset
magpie-crawler
meltwater
peer39_crawler
piplbot
scoop.it
seekr
youbot
cohere-ai

Rule Path
Disallow /

*

Rule Path
Disallow /*?
Disallow /checkout/
Disallow /*/checkout/
Disallow /admin$
Disallow /admin/
Disallow /api$
Disallow /api/
Disallow /*/api$
Disallow /*/api/
Disallow /klarna-checkout
Disallow /*/klarna-checkout
Disallow /cart$
Disallow /cart/
Disallow /*/cart$
Disallow /*/cart/
Back to top

Comments

  • https://napoveda.seznam.cz/en/seznambot-crawler/
  • http://www.opensiteexplorer.org/dotbot
  • https://www.semrush.com/bot/
  • https://www.seokicks.de/robot.htm
  • https://yandex.com/support/webmaster/robot-workings/robot.html
  • http://webmeup-crawler.com/
  • https://mj12bot.com/
  • http://ltx71.com/
  • https://velen.io/
  • https://serpstatbot.com/
  • https://www.ve.com/vebot
  • https://babbar.tech/crawler
  • https://www.zoominfo.com/
  • http://www.gowikibot.com/
  • https://cliqz.com/en/cliqzbot
  • https://lumtel.com/
  • http://help.coccoc.com/searchengine
  • https://uptime.com/uptimebot
  • https://ahrefs.com/robot
  • Ai Bots
Back to top

Warnings

  • `useragent` is not a known field.
Back to top