Log In Sign Up

nintendoreporters.com
robots.txt

Robots Exclusion Standard data for nintendoreporters.com

Resource Scan

Scan Details

Site Domain nintendoreporters.com
Base Domain nintendoreporters.com
Scan Status Ok
Last Scan2024-05-17T13:52:32+00:00
Next Scan 2024-05-24T13:52:32+00:00

Last Scan

Scanned2024-05-17T13:52:32+00:00
URL https://nintendoreporters.com/robots.txt
Redirect https://www.nintendoreporters.com/robots.txt
Redirect Domain www.nintendoreporters.com
Redirect Base nintendoreporters.com
Domain IPs 104.17.15.101
Redirect IPs 104.17.15.101
Response IP 104.17.15.101
Found Yes
Hash 524fd7278c58116030573f2b5a0c81996e0c8277183be8346a314f6bca8ee47a
SimHash 7804d7c165b1

Groups

*

Rule Path
Allow /nieuws/

*

Rule Path
Allow /nieuws/nintendo-switch/

*

Rule Path
Allow /nieuws/algemeen/

*

Rule Path
Allow /releases/

*

Rule Path
Allow /en/news/

*

Rule Path
Allow /en/news/nintendo-switch/

*

Rule Path
Allow /en/news/general/

*

Rule Path
Allow /en/releases/

fasterfox

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

exabot

Rule Path
Disallow /

proximic

Rule Path
Disallow /

youdaobot

Rule Path
Disallow /

discobot

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

yacybot

Rule Path
Disallow /

googlebot-image

Rule Path
Allow /storage/files/

adsbot-google

Rule Path
Allow /

googlebot-mobile

Rule Path
Allow /

ahrefssiteaudit

Rule Path
Allow /

ahrefsbot

Rule Path
Allow /

*

Rule Path
Disallow /cgi-bin/
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php

Other Records

Field Value
crawl-delay 3
Back to top

Other Records

Field Value
sitemap https://www.nintendoreporters.com/sitemap_index.xml
sitemap https://www.nintendoreporters.com/newsarticle-sitemap.xml
sitemap https://www.nintendoreporters.com/video-sitemap.xml
Back to top

Comments

  • robots.txt - all ur data belongs to us ;)
  • Created in the distant future (2XXX) after a robotic uprising wiped out all humans.
  • Notice: Collection of any data on NintendoReporters.com through automated means is
  • prohibited unless you have express written permission from NintendoReporters staff
  • and it may only be conducted for the limited purpose contained in said permission.
  • Highlight content - NL
  • Highlight content - EN
  • Disallow certain user-agents
  • Specific allows
  • Crawl delays
  • General WordPress
  • Add sitemap index, news sitemap and video sitemap
Back to top

Warnings

  • 2 invalid lines.
Back to top