planetgong.co.uk
robots.txt
Robots Exclusion Standard data for planetgong.co.uk
Resource Scan
Scan Details
| Site Domain | planetgong.co.uk |
| Base Domain | planetgong.co.uk |
| Scan Status | Ok |
| Last Scan | 2024-11-09T20:04:52+00:00 |
| Next Scan | 2024-11-16T20:04:52+00:00 |
Last Scan
| Scanned | 2024-11-09T20:04:52+00:00 |
| URL | https://planetgong.co.uk/robots.txt |
| Domain IPs | 35.214.119.107 |
| Response IP | 35.214.119.107 |
| Found | Yes |
| Hash | 95815de5aa1865ba499af4a5cbc39cf491363004e9c09d5536c61945f6075049 |
| SimHash | 147a5151ae22 |
Groups
archive.org_bot
heritrix
ia_archiver
ia_archiver-web.archive.org
mastodon
uptimebot.org
uptimerobot
| Rule | Path |
|---|---|
| Allow | / |
chatgpt-user
duckassistbot
meta-externalfetcher
ai2bot
anthropic-ai
bytespider
ccbot
claudebot
claude-web
cohere-ai
dataprovider.com
dcrawl
diffbot
facebookbot
google-extended
gptbot
httrack
httrack 3.0
meta-externalagent
metainspector
newspaper
nutch
offlineexplorer
omgili
scrapy
simplescraper
timpibot
webzio-extended
perplexitybot
youbot
linkedinbot
mail.ru_bot
pinterestbot
twitterbot
whatsapp
aihitbot
anderspinkbot
webzio
aisearchbot
bot-pge.chlooe.com
bot.araturka.com
emailcollector
emailsiphon
emailwolf
facebot
megaindex.ru
omgilibot
pinterest
pr-cy.ru
qqdownload
slackbot-linkexpanding
tencenttraveler
| Rule | Path |
|---|---|
| Disallow | / |
*
| Rule | Path |
|---|---|
| Disallow | /wp-login.php |
| Disallow | */menus/ |
| Disallow | */styles/ |
| Disallow | /zzz/ |
| Disallow | */zzz/ |
| Disallow | *.php |
| Disallow | *.re.shtml |
| Disallow | /archives/lyrics/songs/*.txt |
| Allow | /archives/lyrics/songs/a-z.shtml |
| Disallow | /archives/tabs/tunes/*.txt |
| Allow | /archives/tabs/tunes/a-z.shtml |
| Disallow | /av/ |
| Disallow | /bazaar/a-list.shtml |
| Disallow | /bazaar/brief.shtml |
| Disallow | /bazaar/badges/ |
| Disallow | /bazaar/books/*.html |
| Disallow | /bazaar/cd/*.html |
| Disallow | /bazaar/dvd/*.html |
| Disallow | /bazaar/postcards/ |
| Disallow | /bazaar/posters/ |
| Disallow | /bazaar/tape/*.html |
| Disallow | /bazaar/threads/ |
| Disallow | /bazaar/vinyl/*.html |
| Disallow | /bits/ |
| Disallow | /cgi-bin/ |
| Disallow | /digital/a-list.shtml |
| Disallow | /digital/brief.html |
| Disallow | /digital/linkloki/ |
| Disallow | /digital/logs/ |
| Disallow | /digital/music/*.html |
| Disallow | /digital/posters/*.html |
| Disallow | /digital/ringtones/*.html |
| Disallow | /digital/words/*.html |
| Disallow | /gigs/briefs/ |
| Disallow | /gigs/agenda.shtml |
| Disallow | /gigs/gignet.shtml |
| Disallow | /gigs/time-machine.shtml |
| Disallow | /graphics/ |
| Disallow | /headers/ |
| Disallow | /images/ |
| Disallow | /news/a-list.html |
| Disallow | /news/brief.shtml |
| Disallow | /newsletter/ |
| Disallow | /outland/forum/ |
| Disallow | /outland/accessibility.shtml |
| Disallow | /outland/cookies.html |
| Disallow | /outland/cookies.shtml |
| Disallow | /outland/privacy.shtml |
| Disallow | /tail.html |
Other Records
| Field | Value |
|---|---|
| sitemap | https://planetgong.co.uk/sitemap.xml |
Warnings
- 4 invalid lines.
- `iser-agent` is not a known field.
Comments