ergaster.org
robots.txt

Robots Exclusion Standard data for ergaster.org

Resource Scan

Scan Details

Site Domain ergaster.org
Base Domain ergaster.org
Scan Status Ok
Last Scan4/30/2025, 7:51:37 AM
Next Scan 5/30/2025, 7:51:37 AM

Last Scan

Scanned4/30/2025, 7:51:37 AM
URL https://ergaster.org/robots.txt
Domain IPs 104.21.0.142, 172.67.128.18, 2606:4700:3033::ac43:8012, 2606:4700:3035::6815:8e
Response IP 172.67.128.18
Found Yes
Hash 47ad9912d2f54bbfd2d1a80a85d64f4bdd1a495f868a78f9271fcbb93a956a5b
SimHash 4a0098608234

Groups

*

Rule Path
Allow /

ccbot

Rule Path
Disallow /

img2dataset

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

Other Records

Field Value
sitemap https://ergaster.org/sitemap-index.xml