Log In Sign Up

amsterdam.nl
robots.txt

Robots Exclusion Standard data for amsterdam.nl

Resource Scan

Scan Details

Site Domain amsterdam.nl
Base Domain amsterdam.nl
Scan Status Ok
Last Scan2024-05-20T20:24:24+00:00
Next Scan 2024-06-03T20:24:24+00:00

Last Scan

Scanned2024-05-20T20:24:24+00:00
URL https://amsterdam.nl/robots.txt
Domain IPs 2a07:3500:1020:f524::110, 46.17.24.110
Response IP 46.17.24.110
Found Yes
Hash c4a8e0a26c64be8ba429cee646b1558d27056da9aacbf3b93e48fd6ff19060c4
SimHash 275b7a0124c3

Groups

simplepie

Rule Path
Disallow /

curl

Rule Path
Disallow /

python urllib

Rule Path
Disallow /

osce

Rule Path
Disallow /

wget

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

yandeximages

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

genieo

Rule Path
Disallow /

jobdiggerspider

Rule Path
Disallow /

exabot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

abonti

Rule Path
Disallow /

linkchecker

Rule Path
Disallow /

jetslide

Rule Path
Disallow /

seokicks

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

wbsearchbot

Rule Path
Disallow /

vagabondo

Rule Path
Disallow /

eknip

Rule Path
Disallow /

mail.ru_bot

Rule Path
Disallow /

kingspider

Rule Path
Disallow /

openindexspider

Rule Path
Disallow /

googlebot

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

bingbot

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

gsa-crawler-internet-amsterdam

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

gsa-crawler-intranet-amsterdam

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

cloudtrawl

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

ia_archiver

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

crawly

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

addsearchbot

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

duckduckbot

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

cloudflare-alwaysonline/1.0

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

twitterbot

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

googlebot-news

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

facebot

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

sherlock

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

swiftbot

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

facebookexternalhit

Rule Path
Disallow /aspx/
Disallow /*?*appidt=*

Other Records

Field Value
crawl-delay 3

*

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.amsterdam.nl/sitemap.xml
Back to top

Comments

  • Dit is de bot van Google
  • Voor de BING zoekmachine. Want iedereen gebruikt die toch...?
  • Dit is de bot van onze internet GSA zoekmachine
  • Dit is de bot van onze INTRANET GSA zoekmachine
  • Zie https://www.cloudtrawl.com/. Wordt volgens mij niet gebruikt
  • Dit is de bot van Archiefweb
  • Dit is de bot van searchly.com
  • http://www.addsearch.com/
  • Dit is de bot van de Duck Duck Go zoekmachine
  • https://www.cloudflare.com/always-online/
  • Dit is de bot van twitter
  • Dit is de bot van Google News
  • Dit is de bot van Facebook
  • Dit is de bot van Findability
  • Dit is de bot van Swiftype
  • Dit is de preview fetch robot van facebook
Back to top

Warnings

  • 2 invalid lines.
Back to top