Log In Sign Up

gigazine.net
robots.txt

Robots Exclusion Standard data for gigazine.net

Resource Scan

Scan Details

Site Domain gigazine.net
Base Domain gigazine.net
Scan Status Ok
Last Scan2024-09-20T16:35:55+00:00
Next Scan 2024-09-27T16:35:55+00:00

Last Scan

Scanned2024-09-20T16:35:55+00:00
URL https://gigazine.net/robots.txt
Domain IPs 104.20.26.5, 104.20.27.5, 172.67.34.58, 2606:4700:10::6814:1a05, 2606:4700:10::6814:1b05, 2606:4700:10::ac43:223a
Response IP 104.20.26.5
Found Yes
Hash 69e8412193352cb3d97522dbc486c78a6f1c46bb07d3f135ae44b5d5ef959ad8
SimHash e53a52c17575

Groups

facebot

Rule Path
Disallow /admin.php

mediapartners-google

Rule Path
Disallow /admin.php
Disallow /index.php
Disallow /Backnumber/
Disallow /cgi-bin/
Disallow /Images/
Disallow /Library/
Disallow /Link/
Disallow /News/
Disallow /gsc/
Disallow /*/P
Disallow /*/C
Disallow /contact
Allow /gsc/comic_viewer/
Allow /gsc/movie_sitemap
Allow /gsc/rss_flipboard_znoj5lrwoe/

Other Records

Field Value
crawl-delay 100

googlebot

Rule Path
Disallow /index.php
Disallow /Backnumber/
Disallow /cgi-bin/
Disallow /Images/
Disallow /Library/
Disallow /Link/
Disallow /News/
Disallow /gsc/
Disallow /news/*/*/
Disallow /*/P
Disallow /*/C
Disallow /contact
Allow /gsc/comic_viewer/
Allow /gsc/movie_sitemap
Allow /gsc/rss_flipboard_znoj5lrwoe/

msnbot

Rule Path
Disallow /index.php
Disallow /Backnumber/
Disallow /cgi-bin/
Disallow /Images/
Disallow /Library/
Disallow /Link/
Disallow /News/
Disallow /gsc/
Disallow /amp/
Disallow /*/P
Disallow /*/C
Allow /gsc/comic_viewer/

yahoo! slurp

Rule Path
Disallow /index.php
Disallow /Backnumber/
Disallow /cgi-bin/
Disallow /Images/
Disallow /Library/
Disallow /Link/
Disallow /News/
Disallow /gsc/
Disallow /amp/
Disallow /*/P
Disallow /*/C
Allow /gsc/comic_viewer/

yahoo-newscrawler

Rule Path
Disallow /index.php
Disallow /Backnumber/
Disallow /cgi-bin/
Disallow /Images/
Disallow /Library/
Disallow /Link/
Disallow /News/
Disallow /gsc/
Disallow /amp/
Disallow /*/P
Disallow /*/C
Allow /gsc/comic_viewer/

becomebot

Rule Path
Disallow /index.php
Disallow /Backnumber/
Disallow /cgi-bin/
Disallow /Images/
Disallow /Library/
Disallow /Link/
Disallow /News/
Disallow /gsc/
Disallow /amp/
Disallow /*/P
Disallow /*/C
Allow /gsc/comic_viewer/

Other Records

Field Value
crawl-delay 100

fasterfox

Rule Path
Disallow /

*

Rule Path
Disallow /index.php
Disallow /Backnumber/
Disallow /cgi-bin/
Disallow /Images/
Disallow /Library/
Disallow /Link/
Disallow /News/
Disallow /gsc/
Disallow /amp/
Disallow /news/*/*/
Disallow /*/P
Disallow /*/C
Disallow /contact
Allow /gsc/comic_viewer/
Allow /gsc/movie_sitemap
Allow /gsc/rss_flipboard_znoj5lrwoe/

Other Records

Field Value
crawl-delay 100

megalodon

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

bubing

Rule Path
Disallow /
Back to top

Comments

  • /robots.txt file for Disallow: /
  • 2008/04/07 11:51
  • 2013/11/05 10:17 add ia_archiver by takaki
  • 2016/04/13 12:30 modify
  • 2018/12/26 09:42 refactored by log1d
  • 2020/04/16 10:26 modify by log1d
  • 2021/08/24 10:26 modify by log1d
  • 2022/10/21 10:26 modify by log1c
  • 2024/04/10 15:53 modify by log_mk
  • 2024/04/11 13:45 modify by log_mk
  • 2024/04/17 08:45 modify by log_mk
Back to top