clickworker.com
robots.txt

Robots Exclusion Standard data for clickworker.com

Resource Scan

Scan Details

Site Domain clickworker.com
Base Domain clickworker.com
Scan Status Ok
Last Scan2024-06-03T05:03:59+00:00
Next Scan 2024-06-17T05:03:59+00:00

Last Scan

Scanned2024-06-03T05:03:59+00:00
URL https://clickworker.com/robots.txt
Redirect https://www.clickworker.com/robots.txt
Redirect Domain www.clickworker.com
Redirect Base clickworker.com
Domain IPs 104.22.16.85, 104.22.17.85, 172.67.24.215, 2606:4700:10::6816:1055, 2606:4700:10::6816:1155, 2606:4700:10::ac43:18d7
Redirect IPs 104.22.16.85, 104.22.17.85, 172.67.24.215, 2606:4700:10::6816:1055, 2606:4700:10::6816:1155, 2606:4700:10::ac43:18d7
Response IP 104.22.16.85
Found Yes
Hash e969ca94fabf26d3946b11f60ff3e11bff26bfffc0590797dd7176cee63e8c21
SimHash 691558120933

Groups

*

Rule Path
Disallow /wp-admin/
Disallow /wp-admin/admin-ajax.php
Disallow /apple-app-site-association
Disallow /.well-known
Disallow /product-wizard
Disallow /tagesdateien01
Disallow /community_advent_calendar/

surveybot

Rule Path
Disallow /

shopwiki

Rule Path
Disallow /

jakarta commons-httpclient

Rule Path
Disallow /

aboutusbot

Rule Path
Disallow /

semager

Rule Path
Disallow /

linguee

Rule Path
Disallow /

updownerbot

Rule Path
Disallow /

gonzo

Rule Path
Disallow /

zeusbot

Rule Path
Disallow /

*

Rule Path
Disallow

Other Records

Field Value
sitemap https://www.clickworker.com/sitemap_index.xml/