wohnung-jetzt.de
robots.txt

Robots Exclusion Standard data for wohnung-jetzt.de

Resource Scan

Scan Details

Site Domain wohnung-jetzt.de
Base Domain wohnung-jetzt.de
Scan Status Ok
Last Scan2024-09-27T20:41:24+00:00
Next Scan 2024-10-04T20:41:24+00:00

Last Scan

Scanned2024-09-27T20:41:24+00:00
URL https://wohnung-jetzt.de/robots.txt
Redirect https://www.wohnung-jetzt.de/robots.txt
Redirect Domain www.wohnung-jetzt.de
Redirect Base wohnung-jetzt.de
Domain IPs 104.26.10.54, 104.26.11.54, 172.67.75.108, 2606:4700:20::681a:a36, 2606:4700:20::681a:b36, 2606:4700:20::ac43:4b6c
Redirect IPs 104.26.10.54, 104.26.11.54, 172.67.75.108, 2606:4700:20::681a:a36, 2606:4700:20::681a:b36, 2606:4700:20::ac43:4b6c
Response IP 104.26.10.54
Found Yes
Hash 4fec5fbcc0688163038ba92b8fc3aab5af5ae8e9ea82b346ef9c1dda8425539d
SimHash 4b347cc08eb8

Groups

*

Rule Path
Disallow /export/
Disallow /popup/
Disallow /fb/
Disallow /secure_login/
Disallow /test/
Disallow /cache/
Disallow /wp-admin/
Disallow /wp-login.php
Disallow /ad/
Disallow /ad/*
Disallow /wp-admin/*
Allow /wp-admin/admin-ajax.php

googlebot

Rule Path
Allow /exposee/
Allow /exposee/*

facebookexternalhit

Rule Path
Allow /exposee/
Allow /exposee/*

telegrambot

Rule Path
Allow /exposee/
Allow /exposee/*

bingbot

Rule Path
Allow /exposee/
Allow /exposee/*

twiceler

Rule Path
Disallow /

nerdbynature.bot

Rule Path
Disallow /

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 50

sistrix

Rule Path
Disallow /exposee/

mj12bot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

spbot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

mauibot

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

rogerbot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

adidxbot

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.wohnung-jetzt.de/sitemap_index.xml
sitemap https://www.wohnung-jetzt.de/sitemap_landingpages.xml
sitemap https://www.wohnung-jetzt.de/sitemap_search.xml
sitemap https://www.wohnung-jetzt.de/sitemap_mietspiegel.xml

Comments

  • Disallow: /exposee/
  • Disallow: /exposee/*
  • Spezifische Bots erlauben
  • Andere Bots spezifisch behandeln
  • Zusätzliche Spambots blockieren
  • Sitemaps