jgvdata.com
robots.txt
Robots Exclusion Standard data for jgvdata.com
Resource Scan
Scan Details
| Site Domain | jgvdata.com |
| Base Domain | jgvdata.com |
| Scan Status | Ok |
| Last Scan | 2025-12-08T18:21:53+00:00 |
| Next Scan | 2026-01-07T18:21:53+00:00 |
Last Scan
| Scanned | 2025-12-08T18:21:53+00:00 |
| URL | https://jgvdata.com/robots.txt |
| Domain IPs | 104.21.95.10, 172.67.142.125, 2606:4700:3031::ac43:8e7d, 2606:4700:3037::6815:5f0a |
| Response IP | 172.67.142.125 |
| Found | Yes |
| Hash | cc25faf2f05fd40f140efe80ed3afaf61210339bff82e71de857421e0aa06213 |
| SimHash | f11b514be68b |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /wp-admin/ |
| Allow | / |
*
| Rule | Path |
|---|---|
| Disallow | /api/ |
| Disallow | /_next/ |
| Disallow | /admin/ |
| Disallow | /server-status |
| Disallow | /search/ |
| Disallow | /contactus/ |
| Disallow | /privacy/ |
| Disallow | /term/ |
| Allow | / |
google-extended
gptbot
ccbot
beautifulsoup
scrapy
ia_archiver
archive.org_bot
ia_archiver-web.archive.org
pinterestbot
petalbot
haosouspider
mj12bot
mediapartners-google*
israbot
orthogaffe
ubicrawler
doc
zao
sitecheck.internetseer.com
zealbot
msiecrawler
sitesnagger
webstripper
webcopier
fetch
offline explorer
teleport
teleportpro
webzip
linko
httrack
microsoft.url.control
xenu
larbin
libwww
zyborg
download ninja
fast
wget
grub-client
k2spider
npbot
webreaper
proximic
wget
teleport
webcopy
sitesucker
goo
y!j*
y!j-srd/1.0
y!j-mbs/1.0
y!j-brw/1.0 crawler
y!j-brj/yats crawler
y!j-brl/yatss crawler
y!j-brm/yatsd crawler
y!j-brn/yatsa crawler
y!j-bry/yatsh crawler
y!j-brz/yatsha crawler
y!j-bri/0.0.1 crawler
cyotekwebcopy
cyotekwebcopy/1.0 cyotekwebcrawler/1.0
cyotekwebcopy/1.0 cyotekhttp/2.0
cyotekwebcopy/1.6 cyotekhttp/2.0
cyotekwebcopy/1.8 cyotekhttp/2.0
lightspeedsystemscrawler
| Rule | Path |
|---|---|
| Disallow | / |
Other Records
| Field | Value |
|---|---|
| sitemap | https://jgvdata.com/sitemap_index.xml |
Comments