Log In Sign Up

marebos.nl
robots.txt

Robots Exclusion Standard data for marebos.nl

Resource Scan

Scan Details

Site Domain marebos.nl
Base Domain marebos.nl
Scan Status Ok
Last Scan2025-05-18T12:21:23+00:00
Next Scan 2025-05-25T12:21:23+00:00

Last Scan

Scanned2025-05-18T12:21:23+00:00
URL https://marebos.nl/robots.txt
Domain IPs 2a02:4780:84:227f:4f04:79b1:b595:c62d, 84.32.84.253
Response IP 91.108.100.49
Found Yes
Hash f988bde0e36b6db3315705cae66e4bbb27e8f2dbd5711da5df52c1e2cd7faa51
SimHash f316d24eecb7

Groups

googlebot

Rule Path
Allow /ads.txt

swiftbot

Rule Path
Disallow

*

Rule Path
Disallow /search/

*

Rule Path Comment
Disallow / -
Disallow /wp-login.php -
Disallow /moories/ -
Disallow /activate/ har har
Disallow /cgi-bin/ MT refugees
Disallow /mshots/v1/ -
Disallow /next/ -
Disallow /public.api/ -
Disallow /wp-admin/admin.php?page=ai1wm_import -
Disallow /wp-admin/admin-ajax.php -
Disallow /wp-admin/options.php -

irl bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3600
Back to top

Other Records

Field Value
sitemap https//marebos.nl/sitemap.xml
Back to top

Comments

  • If you are regularly crawling Wordpress.com sites, please use our firehose to receive real-time push updates instead.
  • Please see https://developer.wordpress.com/docs/firehose/ for more details.
  • Sitemap archive
  • This file was generated on 31 jan 2024
Back to top