Log In Sign Up

networkworld.com
robots.txt

Robots Exclusion Standard data for networkworld.com

Resource Scan

Scan Details

Site Domain networkworld.com
Base Domain networkworld.com
Scan Status Ok
Last Scan2025-09-23T15:53:08+00:00
Next Scan 2025-09-30T15:53:08+00:00

Last Scan

Scanned2025-09-23T15:53:08+00:00
URL https://networkworld.com/robots.txt
Redirect https://www.networkworld.com/robots.txt
Redirect Domain www.networkworld.com
Redirect Base networkworld.com
Domain IPs 192.0.66.49
Redirect IPs 192.0.66.49, 2a04:fa87:fffd::c000:4231
Response IP 192.0.66.49
Found Yes
Hash 363971088875a87d958df1003fac6043336bc6a4998a751311542687fd40004a
SimHash 5af9d952c9b3

Groups

*

Rule Path
Disallow /search*
Disallow */filter/*
Disallow /*?utm_*
Disallow /*//*
Disallow /wp-admin/
Disallow /wp-login.php
Disallow /wp-register.php
Disallow /wp-includes/
Disallow /xmlrpc.php
Disallow /wp-json/
Disallow /feed/
Disallow /*/feed/
Disallow /*LICENSE.txt$
Disallow /*README.md$
Disallow /*README.txt$
Disallow /*license.txt$
Disallow /*readme.html$
Disallow /*readme.md$
Disallow /*readme.txt$
Disallow /*package.json$
Disallow /*composer.json$
Disallow /*composer.lock$
Disallow /*yarn.lock$

ai2bot-dolma

Rule Path
Disallow /

ai2bot

Rule Path
Disallow /

anthropic-ai

Rule Path
Disallow /

applebot-extended

Rule Path
Disallow /

arquivo-web-crawler

Rule Path
Disallow /

awariorssbot

Rule Path
Disallow /

awariosmartbot

Rule Path
Disallow /

barkrowler

Rule Path
Disallow /

bytespider

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

cohere-ai

Rule Path
Disallow /

cohere-training-data-crawler

Rule Path
Disallow /

crawler4j

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

facebookbot

Rule Path
Disallow /

friendlycrawler

Rule Path
Disallow /

google-cloudvertexbot

Rule Path
Disallow /

google-extended

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

https://hada.news

Rule Path
Disallow /

icc-crawler

Rule Path
Disallow /

imagesiftbot

Rule Path
Disallow /

img2dataset

Rule Path
Disallow /

isscyberriskcrawler

Rule Path
Disallow /

jenkersbot

Rule Path
Disallow /

kangaroo bot

Rule Path
Disallow /

meltwater

Rule Path
Disallow /

meta-externalagent

Rule Path
Disallow /

meta-externalfetcher

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

pangubot

Rule Path
Disallow /

perplexitybot

Rule Path
Disallow /

scrapy

Rule Path
Disallow /

seekr

Rule Path
Disallow /

timpibot

Rule Path
Disallow /

velenpublicwebcrawler

Rule Path
Disallow /

webzio-extended

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://www.networkworld.com/sitemap-index.xml
sitemap https://www.networkworld.com/sitemap-news-en.xml
sitemap https://www.networkworld.com/sitemap-articles-en.xml
sitemap https://www.networkworld.com/sitemap-podcast-episodes-en.xml
sitemap https://www.networkworld.com/sitemap-profiles.xml
sitemap https://www.networkworld.com/sitemap-video-episodes-en.xml
sitemap https://www.networkworld.com/sitemap-pages.xml
sitemap https://www.networkworld.com/sitemap-video-series.xml
sitemap https://www.networkworld.com/sitemap-podcast-series.xml
sitemap https://www.networkworld.com/sitemap-blog-series.xml
sitemap https://www.networkworld.com/sitemap-brandposts.xml
Back to top

Comments

  • ------------------------------------------------------
  • Global Rules
  • ------------------------------------------------------
  • ------------------------------------------------------
  • Unwated Bots
  • ------------------------------------------------------
  • ------------------------------------------------------
  • Sitemaps
  • ------------------------------------------------------
Back to top