netvesti.com
robots.txt

Robots Exclusion Standard data for netvesti.com

Resource Scan

Scan Details

Site Domain netvesti.com
Base Domain netvesti.com
Scan Status Ok
Last Scan2024-09-27T09:59:33+00:00
Next Scan 2024-10-04T09:59:33+00:00

Last Scan

Scanned2024-09-27T09:59:33+00:00
URL https://netvesti.com/robots.txt
Domain IPs 104.21.77.82, 172.67.205.188, 2606:4700:3033::6815:4d52, 2606:4700:3037::ac43:cdbc
Response IP 172.67.205.188
Found Yes
Hash bc049330924356cd676677f3ef3c12a0079fdd6fe7e716782fe2dfdd1a7f16bd
SimHash 6d5ddf35d653

Groups

googlebot

Rule Path
Allow

googlebot-image

Rule Path
Allow

googlebot-mobile

Rule Path
Allow

msnbot

Rule Path
Allow

slurp

Rule Path
Allow

yahoo-mmcrawler

Rule Path
Allow

*

Rule Path
Disallow
Disallow /cgi-bin/

facebookexternalhit

Rule Path
Allow /imgres
Allow /search
Disallow /groups
Disallow /hosted/images/
Disallow /m/

facebookexternalhit

Rule Path
Allow /

facebookexternalhit

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 0

Other Records

Field Value
sitemap https://netvesti.com/sitemap_index.xml