Log In Sign Up

huizebiermakelaars.nl
robots.txt

Robots Exclusion Standard data for huizebiermakelaars.nl

Resource Scan

Scan Details

Site Domain huizebiermakelaars.nl
Base Domain huizebiermakelaars.nl
Scan Status Ok
Last Scan2024-10-05T06:46:02+00:00
Next Scan 2024-10-19T06:46:02+00:00

Last Scan

Scanned2024-10-05T06:46:02+00:00
URL https://huizebiermakelaars.nl/robots.txt
Domain IPs 87.233.219.34
Response IP 87.233.219.34
Found Yes
Hash d822d95932c2958ebefd726d3a89fa53f9222d62b0cfa56756a0141a3bfe7681
SimHash 175f19d0eab1

Groups

*

Rule Path
Disallow */checkout/
Disallow */order/
Disallow */loggedin/
Disallow *?filter*
Disallow *?s=*
Disallow *?q=*
Disallow *?_gl=*
Disallow *?getFormHtmlOnly*
Disallow *?formId=*
Disallow *?country_id=
Disallow *?cat_id=
Disallow /search/
Disallow /upload/scripts/
Disallow /specifications-pdf/
Disallow /food-checkpostalcode
Disallow *utm_medium%3Dtradetracker

petalbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

newspaper/0.2.8

Rule Path
Disallow /

infotigerbot

Rule Path
Disallow /

blexbot/1.0

Rule Path
Disallow /

israbot

Rule Path
Disallow /

python/3.9 aiohttp/3.8.1

Rule Path
Disallow /

qwantify/2.4w

Rule Path
Disallow /

proximic

Rule Path
Disallow /

orthogaffe

Rule Path
Disallow /

scrapy/2.5.0

Rule Path
Disallow /

ubicrawler

Rule Path
Disallow /

doc

Rule Path
Disallow /

zao

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

fetch

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

webzip

Rule Path
Disallow /

linko

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

xenu

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

cazoodlebot

Rule Path
Disallow /

fasterfox

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

ingrid

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

jyxobot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

yahoo! slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

rogerbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

siteauditbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

bytedance

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

bytespider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

serpstatbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

mediapartners-google

Rule Path
Disallow *utm

adsbot-google

Rule Path
Disallow *utm

adsbot-google-mobile

Rule Path
Disallow *utm

screaming frog seo spider

Rule Path
Disallow /

googlebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 0

googlebot-image

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 0

googlebot-news

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 0

googlebot-video

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 0

*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

*

No rules defined. All paths allowed.

Back to top

Other Records

Field Value
sitemap https://huizebiermakelaars.nl/sitemap-index.xml
Back to top

Comments

  • Algemene regel voor alle andere bots (Crawl-Delay van 5 seconden)
Back to top