mrgift.ro
robots.txt

Robots Exclusion Standard data for mrgift.ro

Archived Snapshots

Resource Scan

Scan Details

Site Domain	mrgift.ro
Base Domain	mrgift.ro
Scan Status	Failed
Failure Stage	Fetching resource.
Failure Reason	Server returned a client error.
Last Scan	2025-09-03T13:21:17+00:00
Next Scan	2025-12-02T13:21:17+00:00

Last Successful Scan

Scanned	2025-05-07T12:52:55+00:00
URL	https://mrgift.ro/robots.txt
Domain IPs	104.26.0.146, 104.26.1.146, 172.67.70.63, 2606:4700:20::681a:192, 2606:4700:20::681a:92, 2606:4700:20::ac43:463f
Response IP	104.26.0.146
Found	Yes
Hash	b36362ffbfaffd9f055c34bb3a55c51717d557124f6d7f882b5354c40a6a5cc9
SimHash	36574981c4c7

Groups

ai2bot

Rule	Path
Disallow	/

Rule

Path

Disallow

amazonbot

Rule	Path
Disallow	/

Rule

Path

Disallow

amazon-kendra

Rule	Path
Disallow	/

Rule

Path

Disallow

anthropic-ai

Rule	Path
Disallow	/

Rule

Path

Disallow

applebot

Rule	Path
Disallow	/

Rule

Path

Disallow

applebot-extended

Rule	Path
Disallow	/

Rule

Path

Disallow

awariorssbot

Rule	Path
Disallow	/

Rule

Path

Disallow

awariosmartbot

Rule	Path
Disallow	/

Rule

Path

Disallow

brightbot

Rule	Path
Disallow	/

Rule

Path

Disallow

bytespider

Rule	Path
Disallow	/

Rule

Path

Disallow

chatgpt-user

Rule	Path
Disallow	/

Rule

Path

Disallow

claudebot

Rule	Path
Disallow	/

Rule

Path

Disallow

diffbot

Rule	Path
Disallow	/

Rule

Path

Disallow

duckassistbot

Rule	Path
Disallow	/

Rule

Path

Disallow

facebookbot

Rule	Path
Disallow	/

Rule

Path

Disallow

friendlycrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

google-extended

Rule	Path
Disallow	/

Rule

Path

Disallow

gptbot

Rule	Path
Disallow	/

Rule

Path

Disallow

iaskspider/2.0

Rule	Path
Disallow	/

Rule

Path

Disallow

icc-crawler

Rule	Path
Disallow	/

Rule

Path

Disallow

img2dataset

Rule	Path
Disallow	/

Rule

Path

Disallow

kangaroo bot

Rule	Path
Disallow	/

Rule

Path

Disallow

linerbot

Rule	Path
Disallow	/

Rule

Path

Disallow

machinelearningforpeacebot

Rule	Path
Disallow	/

Rule

Path

Disallow

meltwater

Rule	Path
Disallow	/

Rule

Path

Disallow

meta-externalagent

Rule	Path
Disallow	/

Rule

Path

Disallow

meta-externalfetcher

Rule	Path
Disallow	/

Rule

Path

Disallow

nicecrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

oai-searchbot

Rule	Path
Disallow	/

Rule

Path

Disallow

omgili

Rule	Path
Disallow	/

Rule

Path

Disallow

omgilibot

Rule	Path
Disallow	/

Rule

Path

Disallow

pangubot

Rule	Path
Disallow	/

Rule

Path

Disallow

perplexitybot

Rule	Path
Disallow	/

Rule

Path

Disallow

perplexity-user

Rule	Path
Disallow	/

Rule

Path

Disallow

petalbot

Rule	Path
Disallow	/

Rule

Path

Disallow

piplbot

Rule	Path
Disallow	/

Rule

Path

Disallow

qualifiedbot

Rule	Path
Disallow	/

Rule

Path

Disallow

scoop.it

Rule	Path
Disallow	/

Rule

Path

Disallow

seekr

Rule	Path
Disallow	/

Rule

Path

Disallow

semrushbot-ocob

Rule	Path
Disallow	/

Rule

Path

Disallow

sidetrade indexer bot

Rule	Path
Disallow	/

Rule

Path

Disallow

timpibot

Rule	Path
Disallow	/

Rule

Path

Disallow

velenpublicwebcrawler

Rule	Path
Disallow	/

Rule

Path

Disallow

webzio-extended

Rule	Path
Disallow	/

Rule

Path

Disallow

youbot

Rule	Path
Disallow	/

Rule

Path

Disallow

Comments

Cloudflare Managed Robots.txt to block AI related bots.

mrgift.rorobots.txt

Resource Scan

Scan Details

Last Successful Scan

Groups

ai2bot

amazonbot

amazon-kendra

anthropic-ai

applebot

applebot-extended

awariorssbot

awariosmartbot

brightbot

bytespider

chatgpt-user

claudebot

diffbot

duckassistbot

facebookbot

friendlycrawler

google-extended

gptbot

iaskspider/2.0

icc-crawler

img2dataset

kangaroo bot

linerbot

machinelearningforpeacebot

meltwater

meta-externalagent

meta-externalfetcher

nicecrawler

oai-searchbot

omgili

omgilibot

pangubot

perplexitybot

perplexity-user

petalbot

piplbot

qualifiedbot

scoop.it

seekr

semrushbot-ocob

sidetrade indexer bot

timpibot

velenpublicwebcrawler

webzio-extended

youbot

Comments

mrgift.ro
robots.txt