archive.divernet.com
robots.txt

Robots Exclusion Standard data for archive.divernet.com

Resource Scan

Scan Details

Site Domain archive.divernet.com
Base Domain divernet.com
Scan Status Ok
Last Scan2025-09-17T10:34:19+00:00
Next Scan 2025-10-17T10:34:19+00:00

Last Scan

Scanned2025-09-17T10:34:19+00:00
URL https://archive.divernet.com/robots.txt
Redirect https://divernet.com/robots.txt
Redirect Domain divernet.com
Redirect Base divernet.com
Domain IPs 104.21.36.84, 172.67.190.173, 2606:4700:3034::6815:2454, 2606:4700:3036::ac43:bead
Redirect IPs 104.21.36.84, 172.67.190.173, 2606:4700:3034::6815:2454, 2606:4700:3036::ac43:bead
Response IP 172.67.190.173
Found Yes
Hash 94967d24065b0042ce5d8f6206c2776c82e4fb3df5465bc0cd953f9bc7eeecf0
SimHash 900dcf805cf5

Groups

googlebot

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

googlebot-video

Rule Path
Allow /

googlebot-news

Rule Path
Allow /

storebot-google

Rule Path
Allow /

google-inspectiontool

Rule Path
Allow /

googleother

Rule Path
Allow /

googleother-image

Rule Path
Allow /

googleother-video

Rule Path
Allow /

google-cloudvertexbot

Rule Path
Allow /

google-extended

Rule Path
Allow /

*

Rule Path
Disallow /wp-admin/
Disallow /login/*
Disallow /wp-login.php
Disallow /wp-json/*
Allow /wp-admin/admin-ajax.php

semrushbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://divernet.com/sitemap_index.xml

Comments

  • Google common crawlers full access
  • All other bots