womanidol.com
robots.txt

Robots Exclusion Standard data for womanidol.com

Resource Scan

Scan Details

Site Domain womanidol.com
Base Domain womanidol.com
Scan Status Ok
Last Scan2024-11-12T23:05:24+00:00
Next Scan 2024-11-19T23:05:24+00:00

Last Scan

Scanned2024-11-12T23:05:24+00:00
URL https://womanidol.com/robots.txt
Domain IPs 46.4.27.228
Response IP 46.4.27.228
Found Yes
Hash 34d53163638009c2086606c498afe23f3e8c8624a0afed3c1cef3ab2b907d3c3
SimHash 4a5dcc426e98

Groups

*

Rule Path
Disallow /search*
Disallow /search/*
Disallow *preview%3Dtrue*
Disallow /?preview_id=*
Disallow /*?
Disallow /page/*/?
Disallow /wp-admin/*
Disallow /wp-login.php
Disallow /archive/
Disallow /site/
Disallow /cgi-bin/
Disallow /wp-content/plugins/
Disallow /wp-includes/
Disallow /wp-content/themes/
Disallow /wp-content/logs/
Disallow /wp-content/xhc-xmt/
Disallow /wp-content/upgrade/
Disallow /author/
Disallow /trackback/
Disallow /xmlrpc.php
Disallow /readme.html
Disallow /*?*
Disallow *?replytocom
Disallow /comments/feed/
Allow /wp-admin/admin-ajax.php
Allow /feed/
Allow /tag/feed/
Allow /wp-includes/js/*
Allow /wp-content/uploads/
Allow /wp-content/themes/theissue/
Allow /wp-content/themes/theissue-child/

googlebot-news
googlebot

Rule Path
Allow *.css
Allow *.js
Allow /?s=*

yandexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

sentibot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3

exabot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3

blexbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 3

Other Records

Field Value
sitemap https://www.womanidol.com/sitemap_index.xml

Warnings

  • 1 invalid line.
  • `host` is not a known field.