technology.org
robots.txt

Robots Exclusion Standard data for technology.org

Resource Scan

Scan Details

Site Domain technology.org
Base Domain technology.org
Scan Status Ok
Last Scan2024-06-05T04:53:12+00:00
Next Scan 2024-06-12T04:53:12+00:00

Last Scan

Scanned2024-06-05T04:53:12+00:00
URL https://technology.org/robots.txt
Redirect https://www.technology.org/robots.txt
Redirect Domain www.technology.org
Redirect Base technology.org
Domain IPs 94.130.140.146
Redirect IPs 172.66.40.152, 172.66.43.104, 2606:4700:3108::ac42:2898, 2606:4700:3108::ac42:2b68
Response IP 172.66.40.152
Found Yes
Hash 5d4fb9c9662dc645ad4fe8531eaa0c5b5f21a0d748f91ed1c02ac617feb86772
SimHash 6f5d5895c281

Groups

googlebot

Rule Path
Disallow

Other Records

Field Value
crawl-delay 10

mediapartners-google

Rule Path
Disallow

Other Records

Field Value
crawl-delay 5

adsbot-google

Rule Path
Disallow

Other Records

Field Value
crawl-delay 5

adsbot-google-mobile

Rule Path
Disallow

Other Records

Field Value
crawl-delay 5

googlebot-news

Rule Path
Disallow

Other Records

Field Value
crawl-delay 10

googlebot-image

Rule Path
Disallow /

facebookexternalhit

Rule Path
Disallow

Other Records

Field Value
crawl-delay 10

bingbot

Rule Path
Disallow /2012/
Disallow /2013/
Disallow /2014/
Disallow /2015/
Disallow /2016/
Disallow /2017/
Disallow /2018/
Disallow /2019/
Disallow /2020/
Disallow /2021/
Disallow /2022/
Disallow /2023/

Other Records

Field Value
crawl-delay 30

*

Rule Path
Disallow /
Disallow /thank-you/

Other Records

Field Value
sitemap https://www.technology.org/sitemap_index.xml
sitemap https://www.technology.org/sitemap_news.xml