Log In Sign Up

toutiao.com
robots.txt

Robots Exclusion Standard data for toutiao.com

Resource Scan

Scan Details

Site Domain toutiao.com
Base Domain toutiao.com
Scan Status Ok
Last Scan2024-05-16T18:15:33+00:00
Next Scan 2024-05-30T18:15:33+00:00

Last Scan

Scanned2024-05-16T18:15:33+00:00
URL https://toutiao.com/robots.txt
Redirect https://www.toutiao.com/robots.txt
Redirect Domain www.toutiao.com
Redirect Base toutiao.com
Domain IPs 122.14.229.38, 122.14.229.39
Redirect IPs 125.56.219.35, 96.17.72.9
Response IP 23.210.250.67
Found Yes
Hash cac19574c23a7ee81b073b16adcd8b3397edb26b4498c4e4fcea8763e3a66be5
SimHash e145d9707f82

Groups

*

Rule Path
Disallow /search

bytespider

Rule Path
Disallow

toutiaospider

Rule Path
Disallow

baiduspider

Rule Path
Allow /article/*
Allow /w/*
Disallow /article/*?*
Disallow /w/*?*
Disallow /search

baiduspider-image

Rule Path
Allow /article/*
Allow /w/*
Disallow /article/*?*
Disallow /w/*?*
Disallow /search

googlebot

Rule Path
Allow /article/*
Allow /w/*
Disallow /search

bingbot

Rule Path
Allow /article/*
Allow /w/*
Disallow /article/*?*
Disallow /w/*?*
Disallow /search

sogou web spider

Rule Path
Allow /article/*
Allow /w/*
Disallow /article/*?*
Disallow /w/*?*
Disallow /search

sogou inst spider

Rule Path
Allow /article/*
Allow /w/*
Disallow /article/*?*
Disallow /w/*?*
Disallow /search

sogou spider2

Rule Path
Allow /article/*
Allow /w/*
Disallow /article/*?*
Disallow /w/*?*
Disallow /search

sogou blog

Rule Path
Allow /article/*
Allow /w/*
Disallow /article/*?*
Disallow /w/*?*
Disallow /search

sogou news spider

Rule Path
Allow /article/*
Allow /w/*
Disallow /article/*?*
Disallow /w/*?*
Disallow /search

sogou orion spider

Rule Path
Allow /article/*
Allow /w/*
Disallow /article/*?*
Disallow /w/*?*
Disallow /search

sosospider

Rule Path
Allow /article/*
Allow /w/*
Disallow /article/*?*
Disallow /w/*?*
Disallow /search

yisouspider

Rule Path
Allow /article/*
Allow /w/*
Disallow /article/*?*
Disallow /w/*?*
Disallow /search
Back to top

Warnings

  • 6 invalid lines.
Back to top