theweb.com
robots.txt

Robots Exclusion Standard data for theweb.com

Resource Scan

Scan Details

Site Domain theweb.com
Base Domain theweb.com
Scan Status Failed
Failure ReasonScan timed out.
Last Scan2024-09-06T15:50:54+00:00
Next Scan 2024-12-05T15:50:54+00:00

Last Successful Scan

Scanned2022-11-13T04:31:28+00:00
URL https://www.theweb.com/robots.txt
Response IP 13.224.250.14, 13.224.250.2, 13.224.250.57, 13.224.250.54
Found Yes
Hash c36038f68d31ed8131910e30c74bf61cafed0157138de412d1697facdbca96dd
SimHash 2400d0c05592

Groups

mediapartners-google

Rule Path
Disallow

*

Rule Path
Disallow /api
Disallow /click_ad
Disallow /clk_ad
Disallow /wr_clk

adsbot-google

Rule Path
Disallow

Comments

  • DWNDSO-2922: SEM Campaign Addition