thegatewaypundit.com
robots.txt
Robots Exclusion Standard data for thegatewaypundit.com
Resource Scan
Scan Details
Site Domain | thegatewaypundit.com |
Base Domain | thegatewaypundit.com |
Scan Status | Ok |
Last Scan | 2024-11-08T18:01:49+00:00 |
Next Scan | 2024-11-15T18:01:49+00:00 |
Last Scan
Scanned | 2024-11-08T18:01:49+00:00 |
URL | https://thegatewaypundit.com/robots.txt |
Domain IPs | 104.22.4.85, 104.22.5.85, 172.67.41.88, 2606:4700:10::6816:455, 2606:4700:10::6816:555, 2606:4700:10::ac43:2958 |
Response IP | 104.22.4.85 |
Found | Yes |
Hash | 8de900925826761862da3a2064caed23f624588c5367d96feedbf4d47f5e0ccb |
SimHash | 78035802eca8 |
Groups
*
No rules defined. All paths allowed.
Other Records
Field | Value |
---|---|
crawl-delay | 30 |
parler
parlerstaging
bingbot
msnbot
msnbot-media
Rule | Path |
---|---|
Disallow | /?s= |
Disallow | /? |
Disallow | /?* |
Disallow | /search/ |
Disallow | /wp-admin/ |
Disallow | /wp-login.php |
Disallow | /members/ |
Disallow | /admin_page/ |
Disallow | /admin_page/* |
Disallow | /campaign/ |
Disallow | /twitter/ |
Disallow | /youtube/ |
Allow | /wp-admin/admin-ajax.php |
Other Records
Field | Value |
---|---|
crawl-delay | 120 |
*
Rule | Path |
---|---|
Disallow | /*?p=* |
Disallow | /*%26p%3D* |
Disallow | /*?s=* |
Disallow | /*%26s%3D* |
Disallow | /*?ical=1 |
Disallow | /*%26ical%3D1 |
Disallow | /*?tribe-bar-date=* |
Disallow | /*%26tribe-bar-date%3D* |
Disallow | /?author=* |
Disallow | /*wp-comments* |
Disallow | /*wp-trackback* |
Disallow | /*wp-feed* |
Disallow | /*replytocom%3D* |
Disallow | /*?preview=* |
Disallow | /*%26preview%3D* |
Disallow | /*add-to-cart%3D* |
Disallow | /*add_to_wishlist%3D* |
Disallow | /*cart/* |
Disallow | /*checkout/* |
Disallow | /*my-account/* |
Disallow | /*myaccount/* |
Disallow | /*?ajaxCalendar=1* |
Allow | /*/plugins/* |
Other Records
Field | Value |
---|---|
sitemap | https://www.thegatewaypundit.com/sitemap_index.xml |
sitemap | https://www.thegatewaypundit.com/news-sitemap.xml |
sitemap | https://www.thegatewaypundit.com/video-sitemap.xml |
Comments