beritasatu.com
robots.txt

Robots Exclusion Standard data for beritasatu.com

Resource Scan

Scan Details

Site Domain beritasatu.com
Base Domain beritasatu.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-04-18T10:49:12+00:00
Next Scan 2024-05-18T10:49:12+00:00

Last Successful Scan

Scanned2024-03-19T19:01:49+00:00
URL https://beritasatu.com/robots.txt
Redirect https://www.beritasatu.com/robots.txt
Redirect Domain www.beritasatu.com
Redirect Base beritasatu.com
Domain IPs 108.156.133.129, 108.156.133.22, 108.156.133.78, 108.156.133.83
Redirect IPs 108.156.133.129, 108.156.133.22, 108.156.133.78, 108.156.133.83, 2600:9000:2755:2a00:0:9fe7:7e40:93a1, 2600:9000:2755:2e00:0:9fe7:7e40:93a1, 2600:9000:2755:5400:0:9fe7:7e40:93a1, 2600:9000:2755:8600:0:9fe7:7e40:93a1, 2600:9000:2755:8e00:0:9fe7:7e40:93a1, 2600:9000:2755:a00:0:9fe7:7e40:93a1, 2600:9000:2755:a600:0:9fe7:7e40:93a1, 2600:9000:2755:d200:0:9fe7:7e40:93a1
Response IP 108.156.133.78
Found Yes
Hash 06e02b6eefd80036b2cbfecf8639c592772258df2fa00c6be7052b833b461476
SimHash 4814d0708133

Groups

chatgpt-user

Rule Path
Disallow /

openai

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

moreover

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

paqlebot

Rule Path
Disallow /

*

Rule Path
Allow /

Other Records

Field Value
sitemap https://www.beritasatu.com/sitemap.xml