dgitraining.com
robots.txt
Robots Exclusion Standard data for dgitraining.com
Resource Scan
Scan Details
| Site Domain | dgitraining.com |
| Base Domain | dgitraining.com |
| Scan Status | Ok |
| Last Scan | 2025-11-24T02:27:06+00:00 |
| Next Scan | 2025-12-24T02:27:06+00:00 |
Last Scan
| Scanned | 2025-11-24T02:27:06+00:00 |
| URL | https://dgitraining.com/robots.txt |
| Domain IPs | 192.200.160.13 |
| Response IP | 192.200.160.13 |
| Found | Yes |
| Hash | f447e3885cc35c7d6cce517412b0110d2240479800266903fc6502da40837eae |
| SimHash | 45ee6311c3c4 |
Groups
*
| Rule | Path |
|---|---|
| Disallow | /account.php |
| Disallow | /cart.php |
| Disallow | /checkout.php |
| Disallow | /finishorder.php |
| Disallow | /login.php |
| Disallow | /orderstatus.php |
| Disallow | /postreview.php |
| Disallow | /productimage.php |
| Disallow | /productupdates.php |
| Disallow | /remote.php |
| Disallow | /search.php |
| Disallow | /viewfile.php |
| Disallow | /wishlist.php |
| Disallow | /admin/ |
| Disallow | /*sort%3D |
| Disallow | /__socialshop/ |
| Disallow | /dgi-employee-portal-94063/ |
| Disallow | /corp-accounts/ |
| Disallow | /aerovironment/ |
| Disallow | /pages/Cargill.html |
| Disallow | /pages/dhl.html |
| Disallow | /dow/ |
| Disallow | /pages/emo-trans.html |
| Disallow | /flint/ |
| Disallow | /content/pdf/employeehandbook.pdf |
| Disallow | /pages/general-motors.html |
| Disallow | /givaudan/ |
| Disallow | /content/pdf/contractorguidelines.pdf |
| Disallow | /webinar-info/ |
| Disallow | /content/pdf/SATforportal.pdf |
| Disallow | /content/pdf/staffdirectory.pdf |
| Disallow | /content/pdf/Calendar.pdf |
| Disallow | /haeco/ |
| Disallow | /jas-forwarding.html |
| Disallow | /jll/ |
| Disallow | /pages/kintetsu.html |
| Disallow | /landmark/ |
| Disallow | /landstar/ |
| Disallow | /menzies-aviation/ |
| Disallow | /mid-america-overseas/ |
| Disallow | /netjets-aviation/ |
| Disallow | /novitex-enterprise-solutions/ |
| Disallow | /rr-donnelley/ |
| Disallow | /qualcomm/ |
| Disallow | /saic/ |
| Disallow | /pages/uti.html |
| Disallow | /tyco/ |
| Disallow | /estee/ |
| Disallow | /pages/cargill.html |
| Disallow | /pages/dow.html |
| Disallow | /pages/emo-trans.html |
| Disallow | /pages/flint.html |
| Disallow | /pages/general-motors.html |
| Disallow | /givaudan/ |
| Disallow | /helix-logistics/ |
| Disallow | /landstar/ |
| Disallow | /menzies-aviation |
| Disallow | /mid-america-overseas/ |
| Disallow | /saic/ |
| Disallow | /pages/aerovirnment.html |
| Disallow | /pages/dhl.html |
| Disallow | /haeco/ |
| Disallow | /pages/IACAC.html |
| Disallow | /pages/kintetsu.html |
| Disallow | /netjets-aviation/ |
| Disallow | /rr-donnelley/ |
| Disallow | /pages/jas-forwarding.html |
| Disallow | /xpo-logistics/ |
| Disallow | /ch-robinson/ |
| Disallow | /dsv/ |
| Disallow | /kuehne-nagel-special-rates/ |
| Disallow | /pages/ceva.html |
| Disallow | /dyncorp/ |
| Disallow | /flash-global-logistics/ |
| Disallow | /genentech/ |
| Disallow | /loccitane/ |
| Disallow | /pages/lufthansa.html |
| Disallow | /pages/nnr.html |
| Disallow | /pages/nippon.html |
| Disallow | /dgiportal94063/ |
| Disallow | /content/pdf/employeehandbook.pdf |
| Disallow | /content/pdf/Calendar2019.pdf |
| Disallow | /expense-report/ |
| Disallow | /content/pdf/codeofethics.pdfg.com/content/pdf/staffdirectory.pdf |
| Disallow | /content/pdf/SATforportal.pdf |
| Disallow | /content/pdf/publicseminarchecklist2010.pdf |
| Disallow | /content/pdf/codeofethics.pdf |
| Disallow | /content/pdf/staffdirectory.pdf |
| Disallow | /content/pdf/contractorguidelines.pdf |
| Disallow | /dhl |
*
| Rule | Path |
|---|---|
| Disallow | /*?_bc_fsnf=1* |
| Disallow | /*%26_bc_fsnf%3D1* |
ai2bot
ai2bot-dolma
amazonbot
applebot
applebot-extended
bytespider
ccbot
chatgpt-user
claude-web
claudebot
diffbot
facebookbot
friendlycrawler
gptbot
google-extended
googleother
googleother-image
googleother-video
icc-crawler
isscyberriskcrawler
imagesiftbot
kangaroo bot
meta-externalagent
meta-externalfetcher
oai-searchbot
perplexitybot
petalbot
scrapy
sidetrade indexer bot
timpibot
velenpublicwebcrawler
webzio-extended
youbot
anthropic-ai
cohere-ai
facebookexternalhit
iaskspider/2.0
img2dataset
omgili
omgilibot
No rules defined. All paths allowed.
Other Records
| Field | Value |
|---|---|
| crawl-delay | 10 |
Warnings
- 1 invalid line.