Log In Sign Up

cybernews.com
robots.txt

Robots Exclusion Standard data for cybernews.com

Resource Scan

Scan Details

Site Domain cybernews.com
Base Domain cybernews.com
Scan Status Ok
Last Scan2024-11-09T01:56:54+00:00
Next Scan 2024-11-16T01:56:54+00:00

Last Scan

Scanned2024-11-09T01:56:54+00:00
URL https://cybernews.com/robots.txt
Domain IPs 172.66.40.59, 172.66.43.197, 2606:4700:3108::ac42:283b, 2606:4700:3108::ac42:2bc5
Response IP 172.66.43.197
Found Yes
Hash fa536b016151fc3e36337edfed587329541d7566cd4f3626aeb9ae68c580ee97
SimHash 49955868ec53

Groups

*

Rule Path
Disallow /collections/vendors*
Disallow /api/
Disallow /search/*
Disallow /?s=*

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

googlebot

Rule Path
Disallow /*?utm_source*
Disallow /*?ref*
Disallow /*?trk*
Disallow /*?utm_campaign*
Back to top

Other Records

Field Value
sitemap https://cybernews.com/sitemap_index.xml
sitemap https://cybernews.com/news-sitemap.xml
Back to top