adweek.com
robots.txt

Robots Exclusion Standard data for adweek.com

Resource Scan

Scan Details

Site Domain adweek.com
Base Domain adweek.com
Scan Status Ok
Last Scan2024-11-08T15:42:42+00:00
Next Scan 2024-11-15T15:42:42+00:00

Last Scan

Scanned2024-11-08T15:42:42+00:00
URL https://adweek.com/robots.txt
Redirect https://www.adweek.com/robots.txt
Redirect Domain www.adweek.com
Redirect Base adweek.com
Domain IPs 192.0.66.74
Redirect IPs 3.165.82.15, 3.165.82.39, 3.165.82.42, 3.165.82.50
Response IP 3.165.82.15
Found Yes
Hash ccdd2afabd0a66bab7ad1d6e3858b31768d77501eb0b9897768d5305dc1d332e
SimHash 610949418613

Groups

googlebot

Rule Path
Allow /wp-content/plugins/
Allow /wp-content/themes/
Allow /wp-content/uploads/
Disallow */feed/

adsbot-google

Rule Path
Allow /wp-content/plugins/
Allow /wp-content/themes/

googlebot-image

Rule Path
Allow /wp-content/plugins/
Allow /wp-content/themes/
Allow /wp-content/uploads/

twitterbot

Rule Path
Allow /sponsored/
Allow /partner-articles/

ccbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

linkcheck by siteimprove.com

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

youbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

meta-externalfetcher

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.adweek.com/sitemap.xml
sitemap https://www.adweek.com/sitemap-1.xml
sitemap https://www.adweek.com/tvnewser/sitemap.xml
sitemap https://www.adweek.com/tvspy/sitemap.xml
sitemap https://www.adweek.com/agencyspy/sitemap.xml
sitemap https://www.adweek.com/lostremote/sitemap.xml

Warnings

  • 3 invalid lines.