maroelamedia.co.za
robots.txt

Robots Exclusion Standard data for maroelamedia.co.za

Resource Scan

Scan Details

Site Domain maroelamedia.co.za
Base Domain maroelamedia.co.za
Scan Status Ok
Last Scan2024-10-03T04:36:06+00:00
Next Scan 2024-10-10T04:36:06+00:00

Last Scan

Scanned2024-10-03T04:36:06+00:00
URL https://maroelamedia.co.za/robots.txt
Domain IPs 104.22.18.125, 104.22.19.125, 172.67.24.153, 2606:4700:10::6816:127d, 2606:4700:10::6816:137d, 2606:4700:10::ac43:1899
Response IP 104.22.18.125
Found Yes
Hash 9fb37d9859126eb20cc1dffa5d6940af4c99768e5057705d1ff429f94348a166
SimHash 7a185955e8a0

Groups

*

Rule Path
Allow /
Disallow /readme.html
Disallow /wp-admin/
Disallow /wp-includes/
Disallow /wp-signup.php?*
Disallow /xmlrpc.php
Disallow /comments/feed/
Disallow /*?utm_source=
Disallow /*?gclid=
Disallow /*?sa=
Disallow /*?origin=

anthropic-ai

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

omgili

Rule Path
Disallow /

dataprovider-com

Rule Path
Disallow /

dcrawl

Rule Path
Disallow /

httrack

Rule Path
Disallow /

httrack-3-0

Rule Path
Disallow /

metainspector

Rule Path
Disallow /

newspaper

Rule Path
Disallow /

nutch

Rule Path
Disallow /

offline-explorer

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

criteobot/0.1

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

domainstatsbot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

hypestat

Rule Path
Disallow /

linkdexbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

screaming-frog-seo-spider

Rule Path
Disallow /

screaming

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

siteauditbot

Rule Path
Disallow /

semrushbot-ba

Rule Path
Disallow /

semrushbot-si

Rule Path
Disallow /

semrushbot-swa

Rule Path
Disallow /

semrushbot-ct

Rule Path
Disallow /

splitsignalbot

Rule Path
Disallow /

semrushbot-coub

Rule Path
Disallow /

yandexbot

Rule Path
Disallow /

zoombot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://maroelamedia.co.za/sitemap_index.xml

Warnings

  • 4 invalid lines.