Log In Sign Up

thathairforum.com
robots.txt

Robots Exclusion Standard data for thathairforum.com

Resource Scan

Scan Details

Site Domain thathairforum.com
Base Domain thathairforum.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-03-20T16:20:34+00:00
Next Scan 2024-06-18T16:20:34+00:00

Last Successful Scan

Scanned2022-11-27T08:19:20+00:00
URL http://www.thathairforum.com/robots.txt
Response IP 18.213.166.18, 107.21.35.214
Found Yes
Hash 3e654e7f1257c4f52e696be701fb1a2ba1097b2d2f32b21619f8e14708af6e85
SimHash 2842d8dcdcc0

Groups

bubing
alphaseobot
ltx71
companybook-crawler
bdcbot
spbot
semrushbot
ahrefsbot
mj12bot
dotbot
omgili
blexbot
magpie-crawler
extlinksbot
netseer
weborama-fetcher
linkfluence
sentibot
seokicks
barkrowler
ccbot
trendictionbot
amazonbot
serpstatbot
petalbot
dataforseobot
censysinspect

Rule Path
Disallow /

googlebot-image

Rule Path
Disallow /
Allow /site/images/support/

facebookexternalhit

Rule Path
Disallow /images/
Disallow /pm
Disallow /album
Disallow /mbactions
Disallow /register
Disallow /email
Disallow /search
Disallow /upload
Disallow /printthread
Disallow /post/show_single_post
Disallow /post/printadd
Disallow /post/upost
Disallow /post/hpt
Disallow /subscribe
Disallow /calendar
Disallow /calendar/newevent
Disallow /calendar/daydetail?*&nav=
Disallow /calendar/display*view%3Dweekly
Disallow /calendar/display*view%3Dmonthly
Disallow /calendar/showbirthday
Disallow /external
Disallow /tool/view/gb/private
Disallow /tool/view/gb/email
Disallow /tool/pm
Disallow /tool/members/
Disallow /tool/ticket/
Disallow /cgi/view/poll.cgi
Disallow /cgi/view/topsites.cgi
Disallow /cgi/view/member.cgi
Disallow /cgi/view/out.cgi
Disallow /?authtoken=
Disallow *?*&sort=
Disallow *?sort=
Allow /site/images/support/

Other Records

Field Value
crawl-delay 15

mediapartners-google

Rule Path
Disallow /images/
Disallow /pm
Disallow /album
Disallow /mbactions
Disallow /register
Disallow /email
Disallow /search
Disallow /profile
Disallow /file
Disallow /thumb
Disallow /upload
Disallow /printthread
Disallow /post/show_single_post
Disallow /post/printadd
Disallow /post/upost
Disallow /post/hpt
Disallow /subscribe
Disallow /calendar
Disallow /calendar/newevent
Disallow /calendar/daydetail?*&nav=
Disallow /calendar/display*view%3Dweekly
Disallow /calendar/display*view%3Dmonthly
Disallow /calendar/showbirthday
Disallow /external
Disallow /tool/view/gb/private
Disallow /tool/view/gb/email
Disallow /tool/pm
Disallow /tool/members/
Disallow /tool/ticket/
Disallow /cgi/view/poll.cgi
Disallow /cgi/view/topsites.cgi
Disallow /cgi/view/member.cgi
Disallow /cgi/view/out.cgi
Disallow /?authtoken=

Other Records

Field Value
crawl-delay 15

*

Rule Path
Disallow /images/
Disallow /pm
Disallow /album
Disallow /mbactions
Disallow /register
Disallow /email
Disallow /search
Disallow /profile
Disallow /file
Disallow /thumb
Disallow /upload
Disallow /printthread
Disallow /post/show_single_post
Disallow /post/printadd
Disallow /post/upost
Disallow /post/hpt
Disallow /subscribe
Disallow /calendar
Disallow /calendar/newevent
Disallow /calendar/daydetail?*&nav=
Disallow /calendar/display*view%3Dweekly
Disallow /calendar/display*view%3Dmonthly
Disallow /calendar/showbirthday
Disallow /external
Disallow /tags
Disallow /tool/view/gb/private
Disallow /tool/view/gb/email
Disallow /tool/pm
Disallow /tool/members/
Disallow /tool/ticket/
Disallow /cgi/view/poll.cgi
Disallow /cgi/view/topsites.cgi
Disallow /cgi/view/member.cgi
Disallow /cgi/view/out.cgi
Disallow /?authtoken=
Disallow /contact?*subject=
Disallow /post*?goto=
Disallow /post*%26goto%3D
Disallow /post*?id=
Disallow *?*&sort=
Disallow *?sort=
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.gif$
Disallow /*.png$
Disallow /tool/members/login?action=logout
Allow /site/images/support/
Allow /tool/members/signup
Allow /tool/members/login

Other Records

Field Value
crawl-delay 15
Back to top

Other Records

Field Value
sitemap http://www.thathairforum.com/sitemap.xml
Back to top

Comments

  • Most desirable images are hosted on S3. These images will just be icons and stuff, so block them.
  • allowing access to image scripts and topic pages for proper sharing
  • allowing access to topic pages for proper context ads
  • Disallow pages with very little content, duplicate content, or different links pointing to the same content
Back to top