Log In Sign Up

pacmanfan.co.nz
robots.txt

Robots Exclusion Standard data for pacmanfan.co.nz

Resource Scan

Scan Details

Site Domain pacmanfan.co.nz
Base Domain pacmanfan.co.nz
Scan Status Ok
Last Scan2024-09-16T18:57:58+00:00
Next Scan 2024-10-16T18:57:58+00:00

Last Scan

Scanned2024-09-16T18:57:58+00:00
URL https://pacmanfan.co.nz/robots.txt
Domain IPs 13.227.254.13, 13.227.254.25, 13.227.254.74, 13.227.254.88, 2600:9000:200a:1c00:17:39fd:3f80:93a1, 2600:9000:200a:4800:17:39fd:3f80:93a1, 2600:9000:200a:5a00:17:39fd:3f80:93a1, 2600:9000:200a:600:17:39fd:3f80:93a1, 2600:9000:200a:8000:17:39fd:3f80:93a1, 2600:9000:200a:9400:17:39fd:3f80:93a1, 2600:9000:200a:e600:17:39fd:3f80:93a1, 2600:9000:200a:f800:17:39fd:3f80:93a1
Response IP 13.227.254.25
Found Yes
Hash 77960c53a2354670a115f88036837a4b7df1ca0b7efb0ecdee3c2ebb8487452b
SimHash 542c5b52a104

Groups

adsbot-google
amazonbot
anthropic-ai
applebot
awariorssbot
awariosmartbot
bytespider
ccbot
chatgpt-user
claudebot
claude-web
cohere-ai
dataforseobot
facebookbot
friendlycrawler
google-extended
googleother
gptbot
imagesiftbot
magpie-crawler
meltwater
omgili
omgilibot
peer39_crawler
peer39_crawler/1.0
perplexitybot
piplbot
seekr
youbot

Rule Path
Disallow /

wellknownbot

Rule Path
Disallow /

*

Rule Path
Disallow /api/
Disallow /auth/
Disallow /oauth/
Disallow /check_your_email
Disallow /wait_for_approval
Disallow /account_disabled
Disallow /signup
Disallow /.well-known/
Disallow /fileserver/
Disallow /users/
Disallow /emoji/
Disallow /admin
Disallow /user
Disallow /settings/
Disallow /about/suspended

Other Records

Field Value
crawl-delay 500
Back to top

Comments

  • GoToSocial robots.txt -- to edit, see internal/web/robots.go
  • More info @ https://developers.google.com/search/docs/crawling-indexing/robots/intro
  • AI scrapers and the like.
  • https://github.com/ai-robots-txt/ai.robots.txt/
  • Well-known.dev crawler. Indexes stuff under /.well-known.
  • https://well-known.dev/about/
  • Rules for everything else.
  • API endpoints.
  • Auth/Sign in endpoints.
  • Well-known endpoints.
  • Fileserver/media.
  • Fedi S2S API endpoints.
  • Settings panels.
  • Domain blocklist.
Back to top