Log In Sign Up

conquerorforum.com
robots.txt

Robots Exclusion Standard data for conquerorforum.com

Resource Scan

Scan Details

Site Domain conquerorforum.com
Base Domain conquerorforum.com
Scan Status Ok
Last Scan2024-10-10T14:51:15+00:00
Next Scan 2024-10-17T14:51:15+00:00

Last Scan

Scanned2024-10-10T14:51:15+00:00
URL https://conquerorforum.com/robots.txt
Redirect https://www.conquerorforum.com/robots.txt
Redirect Domain www.conquerorforum.com
Redirect Base conquerorforum.com
Domain IPs 104.21.41.207, 172.67.193.149, 2606:4700:3036::ac43:c195, 2606:4700:3037::6815:29cf
Redirect IPs 104.21.41.207, 172.67.193.149, 2606:4700:3036::ac43:c195, 2606:4700:3037::6815:29cf
Response IP 172.67.193.149
Found Yes
Hash f0417968b1143a1d1871c1b4710708d08b4db22996a93f49e3b78a9a465e2466
SimHash 2842d9dedcd1

Groups

bubing
alphaseobot
ltx71
companybook-crawler
bdcbot
spbot
semrushbot
ahrefsbot
mj12bot
dotbot
omgili
blexbot
magpie-crawler
extlinksbot
netseer
weborama-fetcher
linkfluence
sentibot
seokicks
barkrowler
ccbot
trendictionbot
amazonbot
serpstatbot
petalbot

Rule Path
Disallow /

googlebot-image

Rule Path
Disallow /
Allow /site/images/support/

facebookexternalhit

Rule Path
Disallow /images/
Disallow /pm
Disallow /album
Disallow /mbactions
Disallow /register
Disallow /email
Disallow /search
Disallow /upload
Disallow /printthread
Disallow /post/show_single_post
Disallow /post/printadd
Disallow /post/upost
Disallow /subscribe
Disallow /calendar/newevent
Disallow /calendar/daydetail?*&nav=
Disallow /calendar/display*view%3Dweekly
Disallow /calendar/display*view%3Dmonthly
Disallow /calendar/showbirthday
Disallow /tool/view/gb/private
Disallow /tool/view/gb/email
Disallow /tool/pm
Disallow /tool/members/
Disallow /tool/ticket/
Disallow /cgi/view/poll.cgi
Disallow /cgi/view/topsites.cgi
Disallow /cgi/view/member.cgi
Disallow /cgi/view/out.cgi
Disallow /?authtoken=
Disallow *?*&sort=
Disallow *?sort=
Allow /site/images/support/

Other Records

Field Value
crawl-delay 15

mediapartners-google

Rule Path
Disallow /images/
Disallow /pm
Disallow /album
Disallow /mbactions
Disallow /register
Disallow /email
Disallow /search
Disallow /profile
Disallow /file
Disallow /thumb
Disallow /upload
Disallow /printthread
Disallow /post/show_single_post
Disallow /post/printadd
Disallow /post/upost
Disallow /subscribe
Disallow /calendar/newevent
Disallow /calendar/daydetail?*&nav=
Disallow /calendar/display*view%3Dweekly
Disallow /calendar/display*view%3Dmonthly
Disallow /calendar/showbirthday
Disallow /tool/view/gb/private
Disallow /tool/view/gb/email
Disallow /tool/pm
Disallow /tool/members/
Disallow /tool/ticket/
Disallow /cgi/view/poll.cgi
Disallow /cgi/view/topsites.cgi
Disallow /cgi/view/member.cgi
Disallow /cgi/view/out.cgi
Disallow /?authtoken=

Other Records

Field Value
crawl-delay 15

*

Rule Path
Disallow /images/
Disallow /pm
Disallow /album
Disallow /mbactions
Disallow /register
Disallow /email
Disallow /search
Disallow /profile
Disallow /file
Disallow /thumb
Disallow /upload
Disallow /printthread
Disallow /post/show_single_post
Disallow /post/printadd
Disallow /post/upost
Disallow /subscribe
Disallow /calendar/newevent
Disallow /calendar/daydetail?*&nav=
Disallow /calendar/display*view%3Dweekly
Disallow /calendar/display*view%3Dmonthly
Disallow /calendar/showbirthday
Disallow /tool/view/gb/private
Disallow /tool/view/gb/email
Disallow /tool/pm
Disallow /tool/members/
Disallow /tool/ticket/
Disallow /cgi/view/poll.cgi
Disallow /cgi/view/topsites.cgi
Disallow /cgi/view/member.cgi
Disallow /cgi/view/out.cgi
Disallow /?authtoken=
Disallow /post*?goto=
Disallow /post*%26goto%3D
Disallow /post*?id=
Disallow *?*&sort=
Disallow *?sort=
Disallow /*.jpg$
Disallow /*.jpeg$
Disallow /*.gif$
Disallow /*.png$
Allow /site/images/support/
Allow /tool/members/signup
Allow /tool/members/login

Other Records

Field Value
crawl-delay 15
Back to top

Other Records

Field Value
sitemap https://www.conquerorforum.com/sitemap.xml
Back to top

Comments

  • Most desirable images are hosted on S3. These images will just be icons and stuff, so block them.
  • allowing access to image scripts and topic pages for proper sharing
  • allowing access to topic pages for proper context ads
  • Disallow pages with very little content, duplicate content, or different links pointing to the same content
Back to top