transfermarkt.ro
robots.txt

Robots Exclusion Standard data for transfermarkt.ro

Resource Scan

Scan Details

Site Domain transfermarkt.ro
Base Domain transfermarkt.ro
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2024-05-14T21:35:30+00:00
Next Scan 2024-07-13T21:35:30+00:00

Last Successful Scan

Scanned2024-03-15T12:02:10+00:00
URL https://transfermarkt.ro/robots.txt
Redirect https://www.transfermarkt.ro/robots.txt
Redirect Domain www.transfermarkt.ro
Redirect Base transfermarkt.ro
Domain IPs 5.159.57.194, 5.159.57.195
Redirect IPs 18.172.226.111, 18.172.226.12, 18.172.226.61, 18.172.226.78, 2600:9000:20fe:2000:1d:ac59:edc0:93a1, 2600:9000:20fe:5a00:1d:ac59:edc0:93a1, 2600:9000:20fe:6600:1d:ac59:edc0:93a1, 2600:9000:20fe:6800:1d:ac59:edc0:93a1, 2600:9000:20fe:9a00:1d:ac59:edc0:93a1, 2600:9000:20fe:a000:1d:ac59:edc0:93a1, 2600:9000:20fe:a600:1d:ac59:edc0:93a1, 2600:9000:20fe:e00:1d:ac59:edc0:93a1
Response IP 18.165.171.115
Found Yes
Hash 9fd22887e9239d1a35054a5fb86763554940ebd951ce9e4ecc8d10a86eada5f9
SimHash 4542d6304715

Groups

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

mediapartners-google

Rule Path
Disallow

wget

Rule Path
Disallow /

*

Rule Path
Allow /