Log In Sign Up

genealogyadventures.wpcomstaging.com
robots.txt

Robots Exclusion Standard data for genealogyadventures.wpcomstaging.com

Resource Scan

Scan Details

Site Domain genealogyadventures.wpcomstaging.com
Base Domain wpcomstaging.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-07-26T20:07:12+00:00
Next Scan 2024-10-24T20:07:12+00:00

Last Successful Scan

Scanned2023-03-12T10:13:46+00:00
URL https://genealogyadventures.wpcomstaging.com/robots.txt
Domain IPs 192.0.78.20
Response IP 192.0.78.20
Found Yes
Hash 4e984e9d4ab18b553cf0c023654bea02f244bf52e84f72c4d645a5902ea59363
SimHash 01889cc04db2

Groups

*

Rule Path
Allow /
Allow /wp-admin/admin-ajax.php
Disallow /wp-admin/
Disallow /readme.html$

mediapartners-google

Rule Path
Disallow /

adsbot-google

Rule Path
Disallow /

proximic

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

serpstatbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

httrack

Rule Path
Disallow /
Back to top

Comments

  • BEGIN Magic robots.txt
  • ---------------------------
  • General
  • Debug mode: search engines -> allow
  • Ad networks
  • Debug mode: ad networks -> block
  • Link analyzers
  • Debug mode: link analyzers -> block
  • Downloaders
  • Debug mode: downloaders -> block
  • Debug mode: Disabled sitemap
  • ---------------------------
  • END Magic robots.txt
Back to top