Log In Sign Up

gantech.com.ng
robots.txt

Robots Exclusion Standard data for gantech.com.ng

Resource Scan

Scan Details

Site Domain gantech.com.ng
Base Domain gantech.com.ng
Scan Status Failed
Failure StageFetching resource.
Failure ReasonServer returned a client error.
Last Scan2025-08-28T21:49:08+00:00
Next Scan 2025-09-11T21:49:08+00:00

Last Successful Scan

Scanned2025-08-19T21:11:56+00:00
URL https://gantech.com.ng/robots.txt
Domain IPs 65.181.111.166
Response IP 65.181.111.166
Found Yes
Hash 4befe97033023437c468aee251c235326b82f4753ce89251f3c8fa0213e56211
SimHash 24284b20e6a8

Groups

*

Rule Path Comment
Disallow /private/ Disallow crawling of private directories
Disallow /tmp/ Disallow crawling of temporary files or directories
Disallow /404.html Disallow access to custom 404 error page
Disallow /?* Disallow crawling of URL parameters
Disallow /search Disallow crawling of internal search results
Disallow /admin/ Disallow crawling of admin section
Disallow /login/ Disallow crawling of login pages
Disallow /register/ Disallow crawling of registration pages
Disallow /feed/ Disallow crawling of feeds
Disallow /tag/ Disallow crawling of tag pages
Disallow /author/ Disallow crawling of author pages
Disallow /404/ Disallow crawling of 404 pages
Disallow /duplicate-content/ Disallow crawling of directories that can create duplicate content
Disallow /cgi-bin/ Disallow crawling of the cgi-bin directory
Allow /posts/ Allow crawling of posts
Allow /categories/ Allow crawling of categories
Allow /pages/ Allow crawling of pages
Allow /images/ Allow crawling of images

googlebot

Rule Path
Allow /

bingbot

Rule Path
Allow /

badbot

Rule Path
Disallow /
Back to top

Other Records

Field Value
sitemap https://gantech.com.ng/sitemap_index.xml
Back to top

Comments

  • Allow all user agents to crawl posts, categories, pages, and images
  • Allow crawling of important directories
  • Specify sitemap location
  • Allow specific user agents (if necessary)
  • Block specific user agents (if necessary)
Back to top