Log In Sign Up

worldstarhiphop.com
robots.txt

Robots Exclusion Standard data for worldstarhiphop.com

Resource Scan

Scan Details

Site Domain worldstarhiphop.com
Base Domain worldstarhiphop.com
Scan Status Ok
Last Scan2024-04-26T03:31:03+00:00
Next Scan 2024-05-03T03:31:03+00:00

Last Scan

Scanned2024-04-26T03:31:03+00:00
URL https://worldstarhiphop.com/robots.txt
Domain IPs 151.101.0.193, 151.101.128.193, 151.101.192.193, 151.101.64.193
Response IP 151.101.192.193
Found Yes
Hash 9d5ec6813aa415de8e3f3bc94a2704fc557ec936818dbccfead2e9cf32b319e8
SimHash 4501ef2bc2fb

Groups

*

Rule Path
Allow /
Disallow /embed/*
Disallow /u.php/*
Disallow /utils/*
Disallow /api/v1/*
Disallow /video.php
Disallow /videos/e/*
Disallow /recaptcha/*
Disallow /videos/js/*
Disallow /video-c.php
Disallow /video/*
Disallow /videos/assets/*
Disallow /videos/images/*
Disallow /videos/search.php
Allow /videos/search.php?s=fight%2Bcompilation$
Allow /videos/search.php?s=fight$
Allow /videos/search.php?s=fight%2Bcomp$
Allow /videos/search.php?s=detroitfights$
Allow /videos/search.php?s=fight%2Bcomp$
Allow /videos/search.php?s=Fight%2BCompilation$
Allow /videos/search.php?s=girl%2Bfights$
Allow /videos/search.php?s=woman%2Bfight$
Allow /videos/search.php?s=girls%2Bfighting$
Allow /videos/search.php?s=girls%2Bfighting$
Allow /videos/search.php?s=hood%2Bfight$
Allow /videos/search.php?s=shooting$
Allow /videos/search.php?s=street%2Bfight$
Allow /videos/search.php?filter=*&s=fight%2Bcompilation*&start=*$
Allow /videos/search.php?filter=*&s=fight*&start=*$
Allow /videos/search.php?filter=*&s=fight%2Bcomp*&start=*$
Allow /videos/search.php?filter=*&s=detroit%2Bfights*&start=*$
Allow /videos/search.php?filter=*&s=fight%2Bcomp*&start=*$
Allow /videos/search.php?filter=*&s=Fight%2BCompilation*&start=*$
Allow /videos/search.php?filter=*&s=girl%2Bfights*&start=*$
Allow /videos/search.php?filter=*&s=woman%2Bfight*&start=*$
Allow /videos/search.php?filter=*&s=girls%2Bfighting*&start=*$
Allow /videos/search.php?filter=*&s=hood%2Bfight*&start=*$
Allow /videos/search.php?filter=*&s=shooting*&start=*$
Allow /videos/search.php?filter=*&s=street%2Bfight*&start=*$
Allow /videos/search.php?s=fight%2Bcompilation*&start=*$
Allow /videos/search.php?s=fight*&start=*$
Allow /videos/search.php?s=fight%2Bcomp*&start=*$
Allow /videos/search.php?s=detroit%2Bfights*&start=*$
Allow /videos/search.php?s=fight%2Bcomp*&start=*$
Allow /videos/search.php?s=Fight%2BCompilation*&start=*$
Allow /videos/search.php?filter=*&s=girl%2Bfights*&start=*$
Allow /videos/search.php?s=woman%2Bfight*&start=*$
Allow /videos/search.php?s=girls%2Bfighting*&start=*$
Allow /videos/search.php?s=hood%2Bfight*&start=*$
Allow /videos/search.php?s=shooting*&start=*$
Allow /videos/search.php?s=street%2Bfight*&start=*$
Allow /videos/search.php?filter=*&s=fight%2Bcompilation$
Allow /videos/search.php?filter=*&s=fight$
Allow /videos/search.php?filter=*&s=fight%2Bcomp$
Allow /videos/search.php?filter=*&s=detroit%2Bfights$
Allow /videos/search.php?filter=*&s=fight%2Bcomp$
Allow /videos/search.php?filter=*&s=Fight%2BCompilation$
Allow /videos/search.php?filter=*&s=girl%2Bfights$
Allow /videos/search.php?filter=*&s=woman%2Bfight$
Allow /videos/search.php?filter=*&s=girls%2Bfighting$
Allow /videos/search.php?filter=*&s=hood%2Bfight$
Allow /videos/search.php?filter=*&s=shooting$
Allow /videos/search.php?filter=*&s=street%2Bfight$
Back to top

Other Records

Field Value
sitemap https://worldstarhiphop.com/sitemap.xml
Back to top

Comments

  • allow all paths
  • block all search pages
  • allow specific search pages
  • allow specific search pages with any trailing parameters
  • allow specific search pages with any trailing parameters
  • sitemap URLs
Back to top