wigsdo.com
robots.txt

Robots Exclusion Standard data for wigsdo.com

Resource Scan

Scan Details

Site Domain wigsdo.com
Base Domain wigsdo.com
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-07-17T12:43:58+00:00
Next Scan 2025-10-15T12:43:58+00:00

Last Successful Scan

Scanned2025-02-25T03:59:58+00:00
URL https://wigsdo.com/robots.txt
Redirect https://www.wigsdo.com/robots.txt
Redirect Domain www.wigsdo.com
Redirect Base wigsdo.com
Domain IPs 104.21.48.44, 172.67.177.106, 2606:4700:3032::ac43:b16a, 2606:4700:3033::6815:302c
Redirect IPs 104.21.48.44, 172.67.177.106, 2606:4700:3032::ac43:b16a, 2606:4700:3033::6815:302c
Response IP 172.67.177.106
Found Yes
Hash a83b6aa6b37f6084e37ad6d49d5afcc1214f8424b1c760bd6057089eb7a0203c
SimHash f369d7eec611

Groups

*

Rule Path
Allow /
Disallow /fpc/catalog_product/view/id/
Disallow /*%2C*%2C*.html
Disallow /*%21*.html
Disallow /s/
Disallow /catalog/
Disallow /catalogsearch/
Disallow /app/
Disallow /cgi-bin/
Disallow /downloader/
Disallow /errors/
Disallow /includes/
Disallow /account/
Disallow /lib/
Disallow /magento/
Disallow /pkginfo/
Disallow /report/
Disallow /scripts/
Disallow /shell/
Disallow /stats/
Disallow /var/
Disallow /index.php/
Disallow /checkout/
Disallow /control/
Disallow /contacts/
Disallow /customer/
Disallow /customize/
Disallow /newsletter/
Disallow /contact-us/
Disallow /review/
Disallow /sendfriend/
Disallow /wishlist/
Disallow /directory/
Disallow /help/
Disallow /webhelper/
Disallow /wp-admin/

mj12bot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

ahrefsbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

barkrowler

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

dotbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 20

img2dataset

Rule Path
Disallow /

petalbot

Rule Path
Disallow /

claudebot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

almaden

Rule Path
Disallow /

aspseek

Rule Path
Disallow /

axmo

Rule Path
Disallow /

baiduspider

Rule Path
Disallow /

booch

Rule Path
Disallow /

dts agent

Rule Path
Disallow /

downloader

Rule Path
Disallow /

emailcollector

Rule Path
Disallow /

emailsiphon

Rule Path
Disallow /

emailwolf

Rule Path
Disallow /

expired domain sleuth

Rule Path
Disallow /

franklin locator

Rule Path
Disallow /

gaisbot

Rule Path
Disallow /

grub

Rule Path
Disallow /

hughcrawler

Rule Path
Disallow /

iaea.org

Rule Path
Disallow /

lcabotaccept

Rule Path
Disallow /

iconsurf

Rule Path
Disallow /

iltrovatore-setaccio

Rule Path
Disallow /

indy library

Rule Path
Disallow /

iupui

Rule Path
Disallow /

kittiecentral

Rule Path
Disallow /

iaea.org

Rule Path
Disallow /

larbin

Rule Path
Disallow /

lwp-trivial

Rule Path
Disallow /

metatagrobot

Rule Path
Disallow /

missigua locator

Rule Path
Disallow /

netresearchserver

Rule Path
Disallow /

nextgensearch

Rule Path
Disallow /

npbot

Rule Path
Disallow /

nutch

Rule Path
Disallow /

objectssearch

Rule Path
Disallow /

oracle ultra search

Rule Path
Disallow /

peerbot

Rule Path
Disallow /

pictureofinternet

Rule Path
Disallow /

plantynet

Rule Path
Disallow /

quepasacreep

Rule Path
Disallow /

scspider

Rule Path
Disallow /

soft411

Rule Path
Disallow /

spider.acont.de

Rule Path
Disallow /

sqworm

Rule Path
Disallow /

ssm agent

Rule Path
Disallow /

tamu

Rule Path
Disallow /

theusefulbot

Rule Path
Disallow /

turnitinbot

Rule Path
Disallow /

tutorial crawler

Rule Path
Disallow /

tutorgig

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

webzip

Rule Path
Disallow /

zipppbot

Rule Path
Disallow /

xenu

Rule Path
Disallow /

wotbox

Rule Path
Disallow /

wget

Rule Path
Disallow /

naverbot

Rule Path
Disallow /

mozdex

Rule Path
Disallow /

sosospider

Rule Path
Disallow /

sogou web spider

Rule Path
Disallow /

sogou spider

Rule Path
Disallow /

Other Records

Field Value
sitemap https://www.wigsdo.com/sitemap.xml