agendaweek.com
robots.txt
Robots Exclusion Standard data for agendaweek.com
Resource Scan
Scan Details
| Site Domain | agendaweek.com |
| Base Domain | agendaweek.com |
| Scan Status | Failed |
| Failure Stage | Fetching resource. |
| Failure Reason | Server returned a client error. |
| Last Scan | 2025-10-25T20:37:35+00:00 |
| Next Scan | 2026-01-23T20:37:35+00:00 |
Last Successful Scan
| Scanned | 2024-12-30T15:56:51+00:00 |
| URL | https://agendaweek.com/robots.txt |
| Redirect | https://www.agendaweek.com:443/robots.txt |
| Redirect Domain | www.agendaweek.com |
| Redirect Base | agendaweek.com |
| Domain IPs | 52.202.147.100, 54.196.164.200, 54.235.10.40 |
| Redirect IPs | 52.202.147.100, 54.196.164.200, 54.235.10.40 |
| Response IP | 54.196.164.200 |
| Found | Yes |
| Hash | 4620f019885bbdd4bed161550723f245004eba02cb28818aec5666d53cdd1e23 |
| SimHash | bade59a18cd0 |
Groups
gptbot
ccbot
my-tiny-bot
fidget-spinner-bot
test-bot
trendictionbot
dataforseobot
impendoom
| Rule | Path |
|---|---|
| Disallow | / |
*
| Rule | Path |
|---|---|
| Disallow | / |
| Allow | /ads.txt |
| Allow | /login |
| Allow | /about |
| Allow | /subscribers |
| Allow | /user/email |
| Allow | /advertising |
| Allow | /jobblast |
| Allow | /trial |
| Allow | /user/trial |
| Allow | /user/password |
| Allow | /privacy |
| Allow | /copyright |
| Allow | /legal |
| Allow | /cookie-policy |
| Allow | /reprints |
| Allow | /newsbrief |
| Allow | /sitemap.xml |
Other Records
| Field | Value |
|---|---|
| sitemap | https://www.agendaweek.com/sitemap.xml |