cruisersforum.com
robots.txt

Robots Exclusion Standard data for cruisersforum.com

Resource Scan

Scan Details

Site Domain cruisersforum.com
Base Domain cruisersforum.com
Scan Status Ok
Last Scan2024-09-22T18:16:57+00:00
Next Scan 2024-09-29T18:16:57+00:00

Last Scan

Scanned2024-09-22T18:16:57+00:00
URL https://cruisersforum.com/robots.txt
Redirect https://www.cruisersforum.com/robots.txt
Redirect Domain www.cruisersforum.com
Redirect Base cruisersforum.com
Domain IPs 104.26.0.68, 104.26.1.68, 172.67.75.74
Redirect IPs 104.26.0.68, 104.26.1.68, 172.67.75.74
Response IP 172.67.75.74
Found Yes
Hash 1fc0ef68d0bbd6579d8831c10c101b46ddb5e394582293cfc73c2da2a5ca3911
SimHash 4abd9064e711

Groups

*

Rule Path
Disallow /forums/clientscript/
Disallow /forums/includes/
Disallow /forums/install/
Disallow /forums/customavatars/
Disallow /forums/signatureuploads/

*

Rule Path
Disallow /forums/printthread.php
Disallow /forums/cron.php

googlebot

Rule Path
Disallow /forums/external-link/

*

Rule Path
Disallow /ad_tags/

boardtracker

Rule Path
Disallow /

boardreader

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

grapeshot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

ia_archiver

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

amazonbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

the knowledge ai

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 15

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 15

Other Records

Field Value
sitemap https://www.cruisersforum.com/forums/sitemap_index.xml.gz

Comments

  • Forum Folders
  • Forum Files
  • Outbound Link Exclude
  • Backup Ad Tags