Log In Sign Up

squabbled.net
robots.txt

Robots Exclusion Standard data for squabbled.net

Resource Scan

Scan Details

Site Domain squabbled.net
Base Domain squabbled.net
Scan Status Ok
Last Scan2025-10-07T10:36:53+00:00
Next Scan 2025-10-21T10:36:53+00:00

Last Scan

Scanned2025-10-07T10:36:53+00:00
URL https://squabbled.net/robots.txt
Redirect https://www.squabbled.net/robots.txt
Redirect Domain www.squabbled.net
Redirect Base squabbled.net
Domain IPs 104.21.21.168, 172.67.199.162, 2606:4700:3030::6815:15a8, 2606:4700:3036::ac43:c7a2
Redirect IPs 104.21.21.168, 172.67.199.162, 2606:4700:3030::6815:15a8, 2606:4700:3036::ac43:c7a2
Response IP 172.67.199.162
Found Yes
Hash ba69e9d485251df1965e97b7dad481c6d5f5a105307727874056f778c226a725
SimHash 70510981cfd4

Groups

omgili

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /
Disallow /

claudebot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

stablediffusionbot

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

mozilla/5.0 applewebkit/537.36 (khtml, like gecko); compatible; chatgpt-user/1.0; +https://openai.com/bot

Rule Path
Disallow /

sentibot

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

mozilla/5.0 (compatible; ahrefsbot/7.0; +http://ahrefs.com/robot/)

Rule Path
Disallow /

mozilla/5.0 (compatible; censysinspect/1.1; +https://about.censys.io/)

Rule Path
Disallow /

mozilla/5.0 (compatible; dotbot/1.2; +https://opensiteexplorer.org/dotbot; help@moz.com)

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili/0.5 +http://omgili.com

Rule Path
Disallow /

mozilla/5.0 (compatible; semrushbot/7~bl; +http://www.semrush.com/bot.html)

Rule Path
Disallow /

mozilla/5.0 zgrab/0.x

Rule Path
Disallow /

mozilla/5.0 (compatible;petalbot;+https://webmaster.petalsearch.com/site/petalbot)

Rule Path
Disallow /

grok

Rule Path
Disallow /

grokai

Rule Path
Disallow /

xai

Rule Path
Disallow /

twitterbot

Rule Path
Disallow /

xbot

Rule Path
Disallow /

mozilla/5.0 (compatible; blexbot/1.0; +http://webmeup-crawler.com/)

Rule Path
Disallow /

youbot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

screaming frog seo spider

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mediapartners-google*

Rule Path
Disallow /

k2spider

Rule Path
Disallow /

npbot

Rule Path
Disallow /

sitecheck.internetseer.com

Rule Path
Disallow /

zealbot

Rule Path
Disallow /

msiecrawler

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

fetch

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

webzip

Rule Path
Disallow /

linko

Rule Path
Disallow /

httrack

Rule Path
Disallow /

microsoft.url.control

Rule Path
Disallow /

xenu

Rule Path
Disallow /

larbin

Rule Path
Disallow /

libwww

Rule Path
Disallow /

zyborg

Rule Path
Disallow /

download ninja

Rule Path
Disallow /

webreaper

Rule Path
Disallow /

*

Rule Path
Disallow /fonts/
Disallow /abuseipdb-verification.html
Disallow /style.css
Disallow /blog/style.css
Disallow /assets/
Disallow /images/sharex/
Disallow /template
Allow /blog/
Allow /paste
Allow /projects
Allow /sharedarchive

mozilla/5.0 (compatible; archive.org_bot; wayback machine live record; +http://archive.org/details/archive.org_bot)

Rule Path
Allow /

ia_archiver

Rule Path
Allow /

googlebot

Rule Path
Disallow /fonts/
Disallow /abuseipdb-verification.html
Disallow /style.css
Disallow /blog/style.css
Disallow /assets/
Disallow /images/sharex/
Disallow /template
Allow /blog/
Allow /paste
Allow /projects
Allow /sharedarchive

google-extended

Rule Path
Disallow /

bingbot

Rule Path
Disallow /fonts/
Disallow /abuseipdb-verification.html
Disallow /style.css
Disallow /blog/style.css
Disallow /assets/
Disallow /images/sharex/
Disallow /template
Allow /blog/
Allow /paste
Allow /projects
Allow /sharedarchive

slurp

Rule Path
Disallow /fonts/
Disallow /abuseipdb-verification.html
Disallow /style.css
Disallow /blog/style.css
Disallow /assets/
Disallow /images/sharex/
Disallow /template
Allow /blog/
Allow /paste
Allow /projects
Allow /sharedarchive

duckduckbot-https/1.1; (+https://duckduckgo.com/duckduckbot)

Rule Path
Disallow /fonts/
Disallow /abuseipdb-verification.html
Disallow /style.css
Disallow /blog/style.css
Disallow /assets/
Disallow /images/sharex/
Disallow /template
Allow /blog/
Allow /paste
Allow /projects
Allow /sharedarchive

mozilla/5.0 (compatible; discordbot/2.0; +https://discordapp.com)

Rule Path
Allow /

telegrambot (like twitterbot)

Rule Path
Allow /

torubot

Rule Path
Allow /
Back to top

Comments

  • Do not use any content hosted on this website for artificial intelligence training, or otherwise.
  • Archival for personal use is allowed and encouraged. You may automate this with browser extensions or bots, however please don't hog my bandwidth. Please write to me if you plan on using it commercially.
  • SEO scraping services aren't allowed.
Back to top

Warnings

  • `‍user-agent` is not a known field.
Back to top