comedy.com
robots.txt

Robots Exclusion Standard data for comedy.com

Resource Scan

Scan Details

Site Domain comedy.com
Base Domain comedy.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonCouldn't connect to server.
Last Scan2024-06-06T08:22:21+00:00
Next Scan 2024-09-04T08:22:21+00:00

Last Successful Scan

Scanned2024-02-08T08:21:05+00:00
URL https://comedy.com/robots.txt
Domain IPs 108.156.133.11, 108.156.133.123, 108.156.133.127, 108.156.133.91
Response IP 108.138.189.119
Found Yes
Hash ac86e09c31dae6a5840d571b66f315110e1a5059310748700d8aa55ff53a5aa6
SimHash 017eda508413

Groups

*

Rule Path
Disallow /uncategorized/
Disallow /search?term=
Disallow /search/
Disallow /en/

googlebot

Rule Path
Disallow /en/
Disallow /de/
Disallow /es/
Disallow /fr/
Disallow /ja/
Disallow /it/
Disallow /pt/

discobot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 4

yacybot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://comedy.com/sitemap/en