Log In Sign Up

net.fr
robots.txt

Robots Exclusion Standard data for net.fr

Resource Scan

Scan Details

Site Domain net.fr
Base Domain net.fr
Scan Status Ok
Last Scan2025-03-18T06:55:53+00:00
Next Scan 2025-04-17T06:55:53+00:00

Last Scan

Scanned2025-03-18T06:55:53+00:00
URL https://www.net.fr/robots.txt
Domain IPs 104.21.44.143, 172.67.200.197, 2606:4700:3032::ac43:c8c5, 2606:4700:3034::6815:2c8f
Response IP 104.21.44.143
Found Yes
Hash 9aa59dfc5240561a5d34e7754b2a187a92074676a9ad9254d667185af1dab2cb
SimHash a35d76f2d481

Groups

*

Rule Path
Disallow /private/
Disallow /tmp/
Disallow /junk/

ahrefsbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

badbot

Rule Path
Disallow /

blackwidow

Rule Path
Disallow /

bot\+http

Rule Path
Disallow /

casper

Rule Path
Disallow /

cazoodlebot

Rule Path
Disallow /

cherrypicker

Rule Path
Disallow /

copyrightcheck

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

download\ demon

Rule Path
Disallow /

exabot

Rule Path
Disallow /

ezooms

Rule Path
Disallow /

feedfinder

Rule Path
Disallow /

fetch\ api\ request

Rule Path
Disallow /

gigabot

Rule Path
Disallow /

grapeshotcrawler

Rule Path
Disallow /

httrack

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

joomlawget

Rule Path
Disallow /

magpie-crawler

Rule Path
Disallow /

mj12bot

Rule Path
Disallow /

mnogosearch

Rule Path
Disallow /

nutch

Rule Path
Disallow /

offline\ explorer

Rule Path
Disallow /

offline\ navigator

Rule Path
Disallow /

pagegrabber

Rule Path
Disallow /

qwantify

Rule Path
Disallow /

sitesnagger

Rule Path
Disallow /

slurp

Rule Path
Disallow /

smartdownload

Rule Path
Disallow /

sogou

Rule Path
Disallow /

spbot

Rule Path
Disallow /

teleport

Rule Path
Disallow /

teleportpro

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

ubicrawler

Rule Path
Disallow /

unisterbot

Rule Path
Disallow /

web\ image\ collector

Rule Path
Disallow /

web\ sucker

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

webfetch

Rule Path
Disallow /

webleacher

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webwhacker

Rule Path
Disallow /

webzip

Rule Path
Disallow /

wget

Rule Path
Disallow /

wisenutbot

Rule Path
Disallow /

yandex

Rule Path
Disallow /

zeus

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap /sitemap.xml
sitemap /sitemap_index.xml
Back to top

Comments

  • Exclure les bots néfastes
  • Sitemaps
Back to top

Warnings

  • 2 invalid lines.
Back to top