Log In Sign Up

weesp.nl
robots.txt

Robots Exclusion Standard data for weesp.nl

Resource Scan

Scan Details

Site Domain weesp.nl
Base Domain weesp.nl
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-09-10T17:40:08+00:00
Next Scan 2024-11-09T17:40:08+00:00

Last Successful Scan

Scanned2024-06-20T09:01:30+00:00
URL https://weesp.nl/robots.txt
Domain IPs 46.17.24.186
Response IP 46.17.24.186
Found Yes
Hash 0af02da6c1875a2e3d4046c3c893b0f8275bfbdd9e7d8d9d481d959e522dde71
SimHash 275b7a0004d3

Groups

simplepie

Rule Path
Disallow /

curl

Rule Path
Disallow /

python urllib

Rule Path
Disallow /

osce

Rule Path
Disallow /

wget

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

genieo

Rule Path
Disallow /

jobdiggerspider

Rule Path
Disallow /

exabot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

abonti

Rule Path
Disallow /

linkchecker

Rule Path
Disallow /

jetslide

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

vagabondo

Rule Path
Disallow /

eknip

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

kingspider

Rule Path
Disallow /

openindexspider

Rule Path
Disallow /

googlebot

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

bingbot

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

gsa-crawler-internet-amsterdam

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

gsa-crawler-intranet-amsterdam

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

cloudtrawl

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

ia_archiver

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

crawly

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

addsearchbot

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

duckduckbot

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

cloudflare-alwaysonline/1.0

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

twitterbot

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

googlebot-news

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

facebot

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

sherlock

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

swiftbot

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

facebookexternalhit

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

*

Rule Path
Disallow /

simplepie

Rule Path
Disallow /

curl

Rule Path
Disallow /

python urllib

Rule Path
Disallow /

osce

Rule Path
Disallow /

wget

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

genieo

Rule Path
Disallow /

jobdiggerspider

Rule Path
Disallow /

exabot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

abonti

Rule Path
Disallow /

linkchecker

Rule Path
Disallow /

jetslide

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

vagabondo

Rule Path
Disallow /

eknip

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

kingspider

Rule Path
Disallow /

openindexspider

Rule Path
Disallow /

googlebot

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

bingbot

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

gsa-crawler-internet-amsterdam

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

gsa-crawler-intranet-amsterdam

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

cloudtrawl

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

ia_archiver

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

crawly

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

addsearchbot

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

duckduckbot

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

cloudflare-alwaysonline/1.0

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

twitterbot

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

googlebot-news

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

facebot

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

sherlock

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

swiftbot

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

facebookexternalhit

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

*

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.amsterdam.nl/weesp/sitemap.xml
sitemap https://www.amsterdam.nl/stadsdelen/weesp/sitemap.xml
Back to top

Comments

  • Dit is de bot van Google
  • Voor de BING zoekmachine. Want iedereen gebruikt die toch...?
  • Dit is de bot van onze internet GSA zoekmachine
  • Dit is de bot van onze INTRANET GSA zoekmachine
  • Zie https://www.cloudtrawl.com/. Wordt volgens mij niet gebruikt
  • Dit is de bot van Archiefweb
  • Dit is de bot van searchly.com
  • http://www.addsearch.com/
  • Dit is de bot van de Duck Duck Go zoekmachine
  • https://www.cloudflare.com/always-online/
  • Dit is de bot van twitter
  • Dit is de bot van Google News
  • Dit is de bot van Facebook
  • Dit is de bot van Findability
  • Dit is de bot van Swiftype
  • Dit is de preview fetch robot van facebook
  • Dit is de bot van Google
  • Voor de BING zoekmachine. Want iedereen gebruikt die toch...?
  • Dit is de bot van onze internet GSA zoekmachine
  • Dit is de bot van onze INTRANET GSA zoekmachine
  • Zie https://www.cloudtrawl.com/. Wordt volgens mij niet gebruikt
  • Dit is de bot van Archiefweb
  • Dit is de bot van searchly.com
  • http://www.addsearch.com/
  • Dit is de bot van de Duck Duck Go zoekmachine
  • https://www.cloudflare.com/always-online/
  • Dit is de bot van twitter
  • Dit is de bot van Google News
  • Dit is de bot van Facebook
  • Dit is de bot van Findability
  • Dit is de bot van Swiftype
  • Dit is de preview fetch robot van facebook
Back to top

Warnings

  • 4 invalid lines.
Back to top