studio.com
robots.txt

Robots Exclusion Standard data for studio.com

Resource Scan

Scan Details

Site Domain studio.com
Base Domain studio.com
Scan Status Ok
Last Scan2024-06-16T00:57:45+00:00
Next Scan 2024-07-16T00:57:45+00:00

Last Scan

Scanned2024-06-16T00:57:45+00:00
URL https://studio.com/robots.txt
Domain IPs 76.76.21.21
Response IP 76.76.21.21
Found Yes
Hash beb6fcdbb0c8c2d49dba24ce072fc9d01eeb6c16458fb271ef4e022d7e16fcec
SimHash 8c212df53553

Groups

*

Rule Path
Disallow /logos/
Disallow /static/
Disallow /*/classroom
Disallow /dashboard
Disallow /v2/dashboard
Disallow /settings
Disallow /creator
Disallow /claim-gift/
Disallow /admin/
Disallow /homepage
Disallow /andrew$
Disallow /blanks$
Disallow /casey$
Disallow /charlie$
Disallow /daria$
Disallow /davidblaine$
Disallow /her$
Disallow /glitterbomb$
Disallow /kygo$
Disallow /kyle$
Disallow /louisbell$
Disallow /markarduino$
Disallow /markdesign$
Disallow /markrober$
Disallow /MarkRober1$
Disallow /MarkRober10$
Disallow /MarkRober2$
Disallow /MarkRober3$
Disallow /MarkRober4$
Disallow /MarkRober5$
Disallow /MarkRober6$
Disallow /MarkRober7$
Disallow /MarkRober8$
Disallow /MarkRober9$
Disallow /MarkRoberClass$
Disallow /MarkRoberFacebook$
Disallow /MarkRoberFB$
Disallow /MarkRoberMonthly$
Disallow /MarkRoberYouTube$
Disallow /MarkRoberYT$
Disallow /ninjasquirrel$
Disallow /pasekandpaul$
Disallow /pentatonix$
Disallow /ryan$
Disallow /ryan1c$
Disallow /ryan1fb$
Disallow /ryan1ig$
Disallow /ryan1tw$
Disallow /ryan1yt$
Disallow /ryan_facebook$
Disallow /ryan_facebook0521$
Disallow /ryanig$
Disallow /ryan_instagram$
Disallow /ryan_ln0521$
Disallow /ryan_rd$
Disallow /ryan_stories$
Disallow /ryan_tiktok0521$
Disallow /ryan_tt0521$
Disallow /ryantw$
Disallow /ryantwm$
Disallow /ryan_yt0521$
Disallow /shoesurgeon$
Disallow /simone$
Disallow /stevie$
Disallow /stevie-singing$
Disallow /tenhun$
Disallow /tenhun-patreon$
Disallow /tosi$
Disallow /tosicookies$
Disallow /tosicornflake$

Other Records

Field Value
sitemap https://studio.com/sitemap.xml

Comments

  • In general, we want to exclude the below from _all_ crawlers
  • Assets
  • Auth-walled
  • Marketing links used by instructors
  • TIP: regenerate with the following:
  • $ git grep 'shortUrl: "/' | cut -f2 -d'"' | sed 's/.*/Disallow: &$/' | sort