hiphopwired.com
robots.txt

Robots Exclusion Standard data for hiphopwired.com

Resource Scan

Scan Details

Site Domain hiphopwired.com
Base Domain hiphopwired.com
Scan Status Ok
Last Scan2024-10-29T03:32:51+00:00
Next Scan 2024-11-05T03:32:51+00:00

Last Scan

Scanned2024-10-29T03:32:51+00:00
URL https://hiphopwired.com/robots.txt
Domain IPs 108.157.254.18, 108.157.254.42, 108.157.254.62, 108.157.254.8, 2600:9000:2753:2400:4:c844:2d40:93a1, 2600:9000:2753:3000:4:c844:2d40:93a1, 2600:9000:2753:3a00:4:c844:2d40:93a1, 2600:9000:2753:c00:4:c844:2d40:93a1, 2600:9000:2753:d200:4:c844:2d40:93a1, 2600:9000:2753:e00:4:c844:2d40:93a1, 2600:9000:2753:f200:4:c844:2d40:93a1, 2600:9000:2753:f400:4:c844:2d40:93a1
Response IP 108.157.254.18
Found Yes
Hash 76a9590668fd1bbcd76cf74c06a98c2f9c408e76f09145ae9e4a499d314ace97
SimHash 611cd941a112

Groups

*

Rule Path
Disallow
Disallow /?s=*

gptbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

youbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://hiphopwired.com/sitemap.xml
sitemap https://hiphopwired.com/news-sitemap.xml