dagesite.com
robots.txt

Robots Exclusion Standard data for dagesite.com

Resource Scan

Scan Details

Site Domain dagesite.com
Base Domain dagesite.com
Scan Status Ok
Last Scan2025-10-28T03:05:54+00:00
Next Scan 2025-11-27T03:05:54+00:00

Last Scan

Scanned2025-10-28T03:05:54+00:00
URL https://dagesite.com/robots.txt
Domain IPs 104.21.36.127, 172.67.194.93, 2606:4700:3033::6815:247f, 2606:4700:3034::ac43:c25d
Response IP 172.67.194.93
Found Yes
Hash c2cb0ff6a01d23cd98e181ef7f22ed165d21e9f12e68d78a7eddd3ca90988043
SimHash 8064a976b1a3

Groups

*

Rule Path
Allow /cn/
Allow /en/
Allow /jp/
Allow /ko/
Allow /ru/
Allow /es/
Allow /de/
Allow /fr/
Allow /cn/apps/
Allow /en/apps/
Allow /jp/apps/
Allow /ko/apps/
Allow /ru/apps/
Allow /es/apps/
Allow /de/apps/
Allow /fr/apps/
Allow /cn/favorites/
Allow /en/favorites/
Allow /jp/favorites/
Allow /ko/favorites/
Allow /ru/favorites/
Allow /es/favorites/
Allow /de/favorites/
Allow /fr/favorites/
Allow /cn/navigation/
Allow /en/navigation/
Allow /jp/navigation/
Allow /ko/navigation/
Allow /ru/navigation/
Allow /es/navigation/
Allow /de/navigation/
Allow /fr/navigation/
Allow /cn/sites/
Allow /en/sites/
Allow /jp/sites/
Allow /ko/sites/
Allow /ru/sites/
Allow /es/sites/
Allow /de/sites/
Allow /fr/sites/
Allow /cn/tags/
Allow /en/tags/
Allow /jp/tags/
Allow /ko/tags/
Allow /ru/tags/
Allow /es/tags/
Allow /de/tags/
Allow /fr/tags/
Allow /cn/video/
Allow /en/video/
Allow /jp/video/
Allow /ko/video/
Allow /ru/video/
Allow /es/video/
Allow /de/video/
Allow /fr/video/
Disallow /cn/search
Disallow /en/search
Disallow /jp/search
Disallow /ko/search
Disallow /ru/search
Disallow /es/search
Disallow /de/search
Disallow /fr/search
Disallow /cn/users
Disallow /en/users
Disallow /jp/users
Disallow /ko/users
Disallow /ru/users
Disallow /es/users
Disallow /de/users
Disallow /fr/users
Disallow /?*

Other Records

Field Value
sitemap https://dage.top/api/sitemap.xml
sitemap https://dage.top/api/sitemap_jp.xml
sitemap https://dage.top/api/sitemap_ru.xml
sitemap https://dage.top/api/sitemap_de.xml
sitemap https://dage.top/api/sitemap_es.xml
sitemap https://dage.top/api/sitemap_fr.xml
sitemap https://dage.top/api/sitemap_ko.xml
sitemap https://dage.top/api/sitemap_en.xml
sitemap https://dage.top/api/sitemap_video.xml

Comments

  • 允许抓取多语言主页目录
  • 允许抓取每个语言目录下的特定页面类型
  • 禁止抓取的页面