Log In Sign Up

au.rollingstone.com
robots.txt

Robots Exclusion Standard data for au.rollingstone.com

Resource Scan

Scan Details

Site Domain au.rollingstone.com
Base Domain rollingstone.com
Scan Status Ok
Last Scan2024-04-24T06:33:35+00:00
Next Scan 2024-05-01T06:33:35+00:00

Last Scan

Scanned2024-04-24T06:33:35+00:00
URL https://au.rollingstone.com/robots.txt
Domain IPs 172.105.164.54
Response IP 172.105.164.54
Found Yes
Hash 3e0a29b2376b6fdc2bfaf9db8a6379dec0e2e1f285b8436aacc012e4fe871a48
SimHash 693539044fb0

Groups

*

Rule Path
Disallow /beta/

googlebot

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php

mediapartners-google

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php

adsbot-google

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php

bingbot

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php

msnbot

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php

msnbot-media

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php

alexabot

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php

semrushbot-sa

Rule Path
Disallow

twitterbot

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Back to top

Other Records

Field Value
sitemap https://au.rollingstone.com/sitemap_index.xml
Back to top

Comments

  • For all robots
  • Allow search crawlers to discover the sitemap
Back to top