monimmeuble.com
robots.txt

Robots Exclusion Standard data for monimmeuble.com

Resource Scan

Scan Details

Site Domain monimmeuble.com
Base Domain monimmeuble.com
Scan Status Ok
Last Scan2024-05-27T11:07:49+00:00
Next Scan 2024-06-03T11:07:49+00:00

Last Scan

Scanned2024-05-27T11:07:49+00:00
URL https://monimmeuble.com/robots.txt
Domain IPs 51.210.109.210
Response IP 51.210.109.210
Found Yes
Hash bf4c0464bb543e7028007b0ba4db3d489847838ea232044d7a27b9dc816de5b6
SimHash 6737d8132370

Groups

googlebot

Rule Path
Allow /wp-admin/admin-ajax.php
Allow /post-sitemap.xml
Allow /page-sitemap.xml
Allow /wp-content/uploads/
Disallow /readme.html
Disallow /refer/
Disallow /wp-admin/
Disallow /wp-login.php
Disallow /wp-includes/
Disallow /wp-content/plugins
Disallow /wp-content/themes
Disallow /uncategorized/
Disallow /tag/
Disallow */trackback
Disallow */feed
Disallow */?s=%7Bsearch_term_string%7D*
Disallow /?paged=*
Disallow /account
Disallow */wp-admin/admin-ajax.php?action=*
Allow /*css?*
Allow /*js?*
Allow /*?utm*
Allow /css/?
Allow /*.css$
Allow /wp-includes/css/*
Allow */autoptimize/css/*

googlebot-image

Rule Path
Allow /*

slurp

Rule Path
Allow /wp-admin/admin-ajax.php
Disallow /readme.html
Disallow /refer/
Disallow /wp-admin/
Disallow /wp-includes
Disallow /wp-content/themes
Disallow /uncategorized/
Disallow /tag/
Disallow */feed
Disallow */?s=
Disallow */?s=%7Bsearch_term_string%7D*
Disallow /?paged=*
Disallow /account
Disallow */wp-admin/admin-ajax.php?action=*

Other Records

Field Value
crawl-delay 5

bingbot

Rule Path
Allow /wp-admin/admin-ajax.php
Disallow /readme.html
Disallow /refer/
Disallow /wp-admin/
Disallow /wp-includes
Disallow /wp-content/themes
Disallow /uncategorized/
Disallow /tag/
Disallow */?s=%7Bsearch_term_string%7D*
Disallow */?s=
Disallow /?paged=*
Disallow /account
Disallow */wp-admin/admin-ajax.php?action=*

Other Records

Field Value
crawl-delay 1

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

scrappy

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 20

scrapy/2.10.1

Rule Path
Disallow /

nutch

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 20

ccbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

textbulkerbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://monimmeuble.com/sitemap_index.xml

Comments

  • Googlebot
  • Allow Directives
  • Disallow Directives
  • Allow Directives CSS JS Analytics
  • Googlebot-Image
  • Slurp
  • Allow Directives
  • Disallow Directives
  • Bingbot
  • Allow Directives
  • Disallow Directives
  • Bad user-agent specifics rules - Maj 16/11/2023
  • IA user-agent specific rules - MAJ 16/11/2023
  • Sitemap