Log In Sign Up

qgis.org
robots.txt

Robots Exclusion Standard data for qgis.org

Resource Scan

Scan Details

Site Domain qgis.org
Base Domain qgis.org
Scan Status Ok
Last Scan2024-09-01T09:51:57+00:00
Next Scan 2024-10-01T09:51:57+00:00

Last Scan

Scanned2024-09-01T09:51:57+00:00
URL https://qgis.org/robots.txt
Domain IPs 95.217.26.231
Response IP 95.217.26.231
Found Yes
Hash e580106af4c239d17abbad11cf29e87ccb380d074f1b91bf2bae24b37abd04b4
SimHash 041e7b60a6b2

Groups

*

Rule Path
Disallow /cgi-bin/
Disallow /tmp/
Disallow /private/
Disallow /admin/
Disallow /login/
Disallow /register/
Disallow /cart/
Disallow /wp-admin/
Disallow /xmlrpc.php
Disallow /wp-login.php
Disallow /*.php$
Disallow /*.cgi$
Disallow /*.asp$
Disallow /*.aspx$
Disallow /*.jsp$
Allow /$
Allow /docs/
Allow /about/
Allow /download/
Allow /support/

ahrefsbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

sogou

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

openai-user

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

badbot

Rule Path
Disallow /

evilbot

Rule Path
Disallow /

nastybot

Rule Path
Disallow /

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://qgis.org/sitemap.xml
Back to top

Comments

  • robots.txt for qgis.org
  • Protecting against malicious bots, unwanted crawling, and AI bots
  • Disallow crawling of sensitive sections
  • Disallow specific file types
  • Allow crawling of the main sections
  • Specify the location of the sitemap
  • Block known bad bots and AI bots
  • Block AI bots and data scrapers
  • Block specific user agents (examples)
  • Allow specific user agents (examples)
Back to top