Log In Sign Up

theforumsite.com
robots.txt

Robots Exclusion Standard data for theforumsite.com

Resource Scan

Scan Details

Site Domain theforumsite.com
Base Domain theforumsite.com
Scan Status Ok
Last Scan2025-10-20T13:37:41+00:00
Next Scan 2025-10-27T13:37:41+00:00

Last Scan

Scanned2025-10-20T13:37:41+00:00
URL https://theforumsite.com/robots.txt
Domain IPs 69.16.219.69
Response IP 69.16.219.69
Found Yes
Hash a385d3901b205920d57359632c46f5b496e278763edacbbb6045456b45f9669d
SimHash 265c8c9b7911

Groups

baiduspider

Rule Path
Disallow /

flamingo_searchengine+(+http://www.flamingosearch.com/bot)

Rule Path
Disallow /

whitevector crawler

Rule Path
Disallow /

whitevector

Rule Path
Disallow /

white vector

Rule Path
Disallow /

yandex

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 220

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 10

mj12bot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

ubicrawler

Rule Path
Disallow /

doc

Rule Path
Disallow /

zao

Rule Path
Disallow /

mediapartners-google

Rule Path
Disallow

*

Rule Path
Disallow /admin/
Disallow /comments.php
Disallow /editprofile.php
Disallow /forgotpass.php
Disallow /register.php
Disallow /activate.php
Disallow /pm.php
Disallow /invite.php
Disallow /usersearch.php
Disallow /reportuser.php
Disallow /contact.php
Disallow /login.php

Other Records

Field Value
crawl-delay 10
Back to top

Comments

  • Crawlers that are kind enough to obey, but which we'd rather not have
  • unless they're feeding search engines.
Back to top