apprentus.com
robots.txt

Robots Exclusion Standard data for apprentus.com

Resource Scan

Scan Details

Site Domain apprentus.com
Base Domain apprentus.com
Scan Status Ok
Last Scan2024-04-24T16:08:41+00:00
Next Scan 2024-05-24T16:08:41+00:00

Last Scan

Scanned2024-04-24T16:08:41+00:00
URL https://apprentus.com/robots.txt
Redirect https://www.apprentus.com/robots.txt
Redirect Domain www.apprentus.com
Redirect Base apprentus.com
Domain IPs 104.26.12.135, 104.26.13.135, 172.67.69.39, 2606:4700:20::681a:c87, 2606:4700:20::681a:d87, 2606:4700:20::ac43:4527
Redirect IPs 104.26.12.135, 104.26.13.135, 172.67.69.39, 2606:4700:20::681a:c87, 2606:4700:20::681a:d87, 2606:4700:20::ac43:4527
Response IP 104.26.13.135
Found Yes
Hash a36bb28b013377943fbd659b0d32b8efce665bfe9454ae3f56e64616feeba80d
SimHash 6d1d9815b0d3

Groups

*

Rule Path
Disallow /search.php
Disallow /login
Disallow /signup
Disallow /auth/
Disallow /_switch_language
Disallow /favourites

nlux_iaharvester

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

mj12bot

Rule Path
Disallow /

claudebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

Other Records

Field Value
sitemap https://www.apprentus.com/sitemap_index.xml