divx.com
robots.txt

Robots Exclusion Standard data for divx.com

Resource Scan

Scan Details

Site Domain divx.com
Base Domain divx.com
Scan Status Ok
Last Scan2025-08-31T14:38:37+00:00
Next Scan 2025-09-07T14:38:37+00:00

Last Scan

Scanned2025-08-31T14:38:37+00:00
URL https://divx.com/robots.txt
Domain IPs 162.159.134.42
Response IP 162.159.134.42
Found Yes
Hash d4fff89a5bd4b9c2f439ba305df99d622aa72bcddd336aa2bb95ffd6b8919ba2
SimHash 7000b9c0c117

Groups

*

Rule Path Comment
Disallow /wp-admin/ -
Allow /wp-admin/admin-ajax.php Allow AJAX for front-end functionality
Disallow /en/dwam/reset_password_page -
Disallow /downloads/ -
Disallow /wp-content/uploads/2020/09/IMG_0477_Kwanna.srt -
Disallow /wp-content/uploads/2020/09/IMG_0477_Kwanna_burned-in.mkv -

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

youbot

Rule Path
Disallow /

Comments

  • WP
  • DWAM
  • Paths (clean URLs)
  • AI Bots (disallow entire site for these specific bots)