Log In Sign Up

justjared.com
robots.txt

Robots Exclusion Standard data for justjared.com

Resource Scan

Scan Details

Site Domain justjared.com
Base Domain justjared.com
Scan Status Ok
Last Scan2024-05-19T03:29:38+00:00
Next Scan 2024-06-02T03:29:38+00:00

Last Scan

Scanned2024-05-19T03:29:38+00:00
URL https://www.justjared.com/robots.txt
Domain IPs 104.18.2.201, 104.18.3.201, 2606:4700::6812:2c9, 2606:4700::6812:3c9
Response IP 104.18.2.201
Found Yes
Hash 42693c5feba0e312a238bea459ae074edc477f71cb9dbe5d99bb690a72fd7688
SimHash 48b4db512ee5

Groups

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /?s=
Disallow /*/?s=
Disallow /search/
Disallow /search?
Disallow /*preview%3Dtrue
Disallow /*theme_preview%3Dtrue

Other Records

Field Value
crawl-delay 5

ccbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

chatgpt

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

bingbot

Rule Path
Disallow /*?s=*

googlebot

Rule Path
Disallow /*?s=*

grapeshot

Rule Path
Disallow

Other Records

Field Value Comment
crawl-delay 5 Apply the 10-second crawl delay

ia_archiver

Rule Path
Disallow /

archive.org_bot

Rule Path
Disallow /

cxensebot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

swiftbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5
Back to top

Other Records

Field Value
sitemap https://www.justjared.com/sitemapindex.xml
Back to top

Comments

  • Sitemap location
  • Global settings for all user agents
  • Block various specific bots completely
  • Specific settings for Bing's crawler (from previous configurations)
  • Specific settings for Google's crawler (from previous configurations)
  • Allow Grapeshot crawler full access and set crawl-delay
  • Block the Internet Archive's crawlers from archiving the site
  • cXensebot crawler settings
  • Swiftbot crawler settings
Back to top