neoseeker.com
robots.txt

Robots Exclusion Standard data for neoseeker.com

Resource Scan

Scan Details

Site Domain neoseeker.com
Base Domain neoseeker.com
Scan Status Ok
Last Scan2024-09-18T02:56:33+00:00
Next Scan 2024-09-25T02:56:33+00:00

Last Scan

Scanned2024-09-18T02:56:33+00:00
URL https://neoseeker.com/robots.txt
Redirect https://www.neoseeker.com/robots.txt
Redirect Domain www.neoseeker.com
Redirect Base neoseeker.com
Domain IPs 104.26.4.80, 104.26.5.80, 172.67.69.121, 2606:4700:20::681a:450, 2606:4700:20::681a:550, 2606:4700:20::ac43:4579
Redirect IPs 104.26.4.80, 104.26.5.80, 172.67.69.121, 2606:4700:20::681a:450, 2606:4700:20::681a:550, 2606:4700:20::ac43:4579
Response IP 104.26.5.80
Found Yes
Hash 84ce4f198230e3472dbc54c6dc941b44fedaaefb573304dcc7e7005a97d7aded
SimHash 751cc052c180

Groups

mediapartners-google

Rule Path
Disallow

*

Rule Path
Disallow /cgi-bin/
Disallow /templates/
Disallow /wusage/
Disallow /admin/
Disallow /api_proxy.php
Disallow /api_handler.php
Disallow /core/
Disallow /tools/
Disallow /Games/themes/
Disallow /forums/tags/
Disallow /f/*/?tag=
Disallow /forums/*/?tag=
Disallow /forums/index.php?fn=alert_mod*
Disallow /forums/index.php?fn=post_reply*
Disallow /forums/index.php?fn=post_thread*
Disallow /forums/index.php?fn=add_favforum*
Disallow /websitelink.php

gptbot

Rule Path
Disallow /

chatgpt-user

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

amazonbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

claude-web

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

omgili

Rule Path
Disallow /

omgilibot

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

applebot

Rule Path
Disallow /

youbot

Rule Path
Disallow /

rogerbot
dotbot

Rule Path
Disallow /

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 5

magpie-crawler

Rule Path
Disallow /

cliqzbot

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

trendictionbot

Rule Path
Disallow /

piplbot

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

naverbot
yeti

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

semrushbot
semrushbot-sa

Rule Path
Disallow /

grapeshot

Rule Path
Disallow

blexbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

quetextbot
bspider
megaindex
exabot
seekportbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.neoseeker.com/walkthroughs_sitemap_index.xml.php
sitemap https://www.neoseeker.com/tags_sitemap.xml
sitemap https://www.neoseeker.com/members_remove_http_sitemap.xml
sitemap https://www.neoseeker.com/members_remove_https_sitemap.xml
sitemap https://www.neoseeker.com/news_sitemap_index.xml.php
sitemap https://www.neoseeker.com/articles_sitemap_index.xml.php

Comments

  • Disallow: /seeker.php
  • Disallow: /tags/
  • Disallow: /api_proxy/
  • Disallow: /forums/lastreply.php
  • Disallow: /forums/directmessage.php
  • AI bots
  • Search Engines/AI
  • Moz
  • brandwatch
  • Majestic
  • trendiction.com
  • Naver (KR)
  • SoGou (CN)
  • Youdao (CN)
  • SEMRush
  • Other misc bots: