Log In Sign Up

atinfotech.com
robots.txt

Robots Exclusion Standard data for atinfotech.com

Resource Scan

Scan Details

Site Domain atinfotech.com
Base Domain atinfotech.com
Scan Status Ok
Last Scan2025-05-21T12:21:05+00:00
Next Scan 2025-06-20T12:21:05+00:00

Last Scan

Scanned2025-05-21T12:21:05+00:00
URL https://atinfotech.com/robots.txt
Domain IPs 103.133.215.2
Response IP 103.133.215.2
Found Yes
Hash 7a62f7a3b17bd0af3acb1001ba85e4f9052188f0c92cab932724aab3188cc943
SimHash 655059460dbe

Groups

*

Rule Path Comment
Allow /*.png$ -
Allow /*.jpg$ -
Allow /*.jpeg$ -
Allow /*.gif$ -
Allow /*.svg$ -
Allow /*.webp$ -
Disallow /ATinfotech-admin/ -
Disallow /search -
Disallow /temporary/ Example for disallowing unnecessary dynamically generated content
Disallow /private/ Block access to private content if applicable

httrack

Rule Path
Disallow /

netcaptor

Rule Path
Disallow /

offline explorer

Rule Path
Disallow /

spiderku/0.9

Rule Path
Disallow /

steeler

Rule Path
Disallow /

webcopier

Rule Path
Disallow /

webcrawler

Rule Path
Disallow /

web downloader

Rule Path
Disallow /

webgather

Rule Path
Disallow /

webstripper

Rule Path
Disallow /

webzip

Rule Path
Disallow /

wget

Rule Path
Disallow /

zao

Rule Path
Disallow /

zeus 2.6

Rule Path Comment
Disallow / -
Disallow /tag/ -
Disallow /category/ -
Disallow /archive/ -
Disallow /*?* Block URLs with query parameters if not necessary for indexing
Back to top

Other Records

Field Value
sitemap https://atinfotech.com/sitemap.xml
Back to top

Comments

  • General instructions for all user agents
  • Sitemap reference
  • Block specific web scraping and downloading tools
  • Additional Crawling Efficiency Suggestions:
  • Consider disallowing common pages that may generate duplicate content or low-value pages:
Back to top