Log In Sign Up

sigenit.pro
robots.txt

Robots Exclusion Standard data for sigenit.pro

Resource Scan

Scan Details

Site Domain sigenit.pro
Base Domain sigenit.pro
Scan Status Ok
Last Scan2024-11-03T15:55:57+00:00
Next Scan 2024-12-03T15:55:57+00:00

Last Scan

Scanned2024-11-03T15:55:57+00:00
URL https://sigenit.pro/robots.txt
Domain IPs 104.21.11.206, 172.67.192.161, 2606:4700:3030::6815:bce, 2606:4700:3032::ac43:c0a1
Response IP 172.67.192.161
Found Yes
Hash 0b6c2a513a6eb2022217a66682d19fb680502ad1eb14d37384fd0eb04bae9eb3
SimHash 449f0e02acd4

Groups

*

Rule Path
Disallow /wp-admin/
Allow /wp-admin/admin-ajax.php
Disallow /xmlrpc.php

Other Records

Field Value
crawl-delay 100

dotbot

Rule Path
Disallow /

giftghostbot

Rule Path
Disallow /

seznam

Rule Path
Disallow /

paperlibot

Rule Path
Disallow /

genieo

Rule Path
Disallow /

dataprovider6.101

Rule Path
Disallow /

dataprovidersiteexplorer

Rule Path
Disallow /

dazoobot1.0

Rule Path
Disallow /

diffbot

Rule Path
Disallow /

domainstatsbot1.0

Rule Path
Disallow /

dotbot1.1

Rule Path
Disallow /

dubaiindex

Rule Path
Disallow /

ecommercebot

Rule Path
Disallow /

expertsearchspider

Rule Path
Disallow /

feedbin

Rule Path
Disallow /

fetch2.0a

Rule Path
Disallow /

ffbot1.0

Rule Path
Disallow /

focusbot1.1

Rule Path
Disallow /

huaweisymantecspider

Rule Path
Disallow /

huaweisymantecspider1.0

Rule Path
Disallow /

jobdiggerspider

Rule Path
Disallow /

lemurwebcrawler

Rule Path
Disallow /

lipperheylinkexplorer

Rule Path
Disallow /

lssrocketcrawler1.0

Rule Path
Disallow /

lyt.srv1.5

Rule Path
Disallow /

miadev0.0.1

Rule Path
Disallow /

najdi.si3.1

Rule Path
Disallow /

bountiibot

Rule Path
Disallow /

experibot_v1

Rule Path
Disallow /

bixocrawler

Rule Path
Disallow /

bixocrawler testcrawler

Rule Path
Disallow /

crawler4j

Rule Path
Disallow /

crowsnest0.5

Rule Path
Disallow /

cukbot

Rule Path
Disallow /

dataprovider6.92

Rule Path
Disallow /

dblbot1.0

Rule Path
Disallow /

diffbot0.1

Rule Path
Disallow /

digg deeperv1

Rule Path
Disallow /

discobot1.0

Rule Path
Disallow /

discobot1.1

Rule Path
Disallow /

discobot2.0

Rule Path
Disallow /

discoverybot2.0

Rule Path
Disallow /

dlvr.it1.0

Rule Path
Disallow /

drupact0.7

Rule Path
Disallow /

ezooms1.0

Rule Path
Disallow /

fastbot crawler beta 2.0

Rule Path
Disallow /

fastbot crawler beta 4.0

Rule Path
Disallow /

feedly social

Rule Path
Disallow /

feedly1.0

Rule Path
Disallow /

feedlybot1.0

Rule Path
Disallow /

feedspot

Rule Path
Disallow /

feedspotbot1.0

Rule Path
Disallow /

clickagy intelligence bot v2

Rule Path
Disallow /

classbot

Rule Path
Disallow /

cispa vulnerability notification

Rule Path
Disallow /

cirrusexplorer1.1

Rule Path
Disallow /

checksemnutch-1.10

Rule Path
Disallow /

catchbot5.0

Rule Path
Disallow /

catchbot3.0

Rule Path
Disallow /

catchbot2.0

Rule Path
Disallow /

catchbot1.0

Rule Path
Disallow /

camontspider1.0

Rule Path
Disallow /

buzzbot1.0

Rule Path
Disallow /

buzzbot

Rule Path
Disallow /

businessseek.biz_spider

Rule Path
Disallow /

bubing

Rule Path
Disallow /

nail

Rule Path
Disallow /

fyberspider1.3

Rule Path
Disallow /

findlinks1.1.6-beta5

Rule Path
Disallow /

g2reader-bot1.0

Rule Path
Disallow /

findlinks1.1.6-beta6

Rule Path
Disallow /

findlinks2.0

Rule Path
Disallow /

findlinks2.0.1

Rule Path
Disallow /

findlinks2.0.2

Rule Path
Disallow /

findlinks2.0.4

Rule Path
Disallow /

findlinks2.0.5

Rule Path
Disallow /

findlinks2.0.9

Rule Path
Disallow /

findlinks2.1

Rule Path
Disallow /

findlinks2.1.5

Rule Path
Disallow /

findlinks2.1.3

Rule Path
Disallow /

findlinks2.2

Rule Path
Disallow /

findlinks2.5

Rule Path
Disallow /

findlinks2.6

Rule Path
Disallow /

findlinks1.0

Rule Path
Disallow /

findlinks1.1.3-beta8

Rule Path
Disallow /

findlinks1.1.3-beta9

Rule Path
Disallow /

findlinks1.1.4-beta7

Rule Path
Disallow /

findlinks1.1.6-beta1

Rule Path
Disallow /

findlinks1.1.6-beta1 yacy

Rule Path
Disallow /

findlinks1.1.6-beta2

Rule Path
Disallow /

findlinks1.1.6-beta3

Rule Path
Disallow /

findlinks1.1.6-beta4

Rule Path
Disallow /

bixo

Rule Path
Disallow /

bixolabs1.0

Rule Path
Disallow /

crawlera1.10.2

Rule Path
Disallow /

bingbot

Rule Path
Allow /

yandex

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

adsbot-google-mobile

Rule Path
Allow /

googlebot

Rule Path
Allow /
Back to top

Other Records

Field Value
sitemap https://sigenit.pro/sitemap.xml
sitemap https://sigenit.pro/sitemap.rss
Back to top

Warnings

  • 6 invalid lines.
  • `clean-param` is not a known field.
Back to top