Log In Sign Up

arbeits-abc.de
robots.txt

Robots Exclusion Standard data for arbeits-abc.de

Resource Scan

Scan Details

Site Domain arbeits-abc.de
Base Domain arbeits-abc.de
Scan Status Ok
Last Scan2024-11-13T00:39:39+00:00
Next Scan 2024-11-20T00:39:39+00:00

Last Scan

Scanned2024-11-13T00:39:39+00:00
URL https://arbeits-abc.de/robots.txt
Domain IPs 2001:8d8:100f:f000::2ff, 217.160.0.232
Response IP 217.160.0.232
Found Yes
Hash 07b72fb19c38016ac71e03f08c7f2d68b2a3b1051a022ccae965818b0f4b52a1
SimHash f01ec94a80a2

Groups

*

Rule Path
Disallow

gptbot

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

yeti

Rule Path
Disallow /

sentibot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

timpibot

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

lcc

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://arbeits-abc.de/news-sitemap.xml
Back to top