spa-networks.co.uk
robots.txt

Robots Exclusion Standard data for spa-networks.co.uk

Resource Scan

Scan Details

Site Domain spa-networks.co.uk
Base Domain spa-networks.co.uk
Scan Status Ok
Last Scan2024-10-06T16:30:09+00:00
Next Scan 2024-10-13T16:30:09+00:00

Last Scan

Scanned2024-10-06T16:30:09+00:00
URL https://spa-networks.co.uk/robots.txt
Domain IPs 31.22.15.18
Response IP 31.22.15.18
Found Yes
Hash ecde18d2fc40e7e9e839d7614be63fe2ecb6587ce2c944293198a36921617156
SimHash da78d4306aa1

Groups

ltx71

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

seekportbot

Rule Path
Disallow /

urlmetrics

Rule Path
Disallow /

netestate ne crawler (+http://www.website-datenbank.de/)

Rule Path
Disallow /

semrushbot

Rule Path
Disallow /

semrushbot-sa

Rule Path
Disallow /

the knowledge ai

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

megaindex.ru

Rule Path
Disallow /

megaindex.com

Rule Path
Disallow /

searchmetricsbot

Rule Path
Disallow /

dataforseobot

Rule Path
Disallow /

msnbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

bingbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

facebookbot

Rule Path
Disallow /

Other Records

Field Value
crawl-delay 300

facebookexternalhit

Rule Path
Disallow /

*

Rule Path
Disallow /SLM
Disallow /s_images
Disallow /phpforum
Disallow /s66programme
Disallow /lobby
Disallow /iphone
Disallow /mobile
Disallow /m
Disallow /mobi
Disallow /apple-touch-icon.png
Disallow /logs
Disallow /GrampianRoadForum
Disallow /server-status
Disallow /BackupForum/search.php
Disallow /BackupForum/register.php
Disallow /BackupForum/login.php
Disallow /BackupForum/userlist.php
Disallow /BackupForum/help.php
Disallow /BackupForum/misc.php
Disallow /BackupForum/post.php
Disallow /BackupForum/rss.php?tid=62
Disallow /BackupForum/rss.php?tid=40
Disallow /BackupForum/rss.php?tid=43
Disallow /BackupForum/rss.php?tid=39
Disallow /BackupForum/rss.php?tid=38
Disallow /BackupForum/rss.php?tid=41
Disallow /BackupForum/rss.php?tid=42
Disallow /punforum/search.php
Disallow /punforum/register.php
Disallow /punforum/login.php
Disallow /punforum/userlist.php
Disallow /punforum/help.php
Disallow /punforum/misc.php
Disallow /punforum/post.php
Disallow /*search.php
Disallow /*register.php
Disallow /*login.php
Disallow /*userlist.php
Disallow /*help.php
Disallow /*misc.php
Disallow /*post.php
Disallow /HannahMartinWedding.htm

Other Records

Field Value
sitemap https://backupforum.progforums.com/sitemap-bf.xml
sitemap https://www.progforums.com/sitemap.xml