thediggings.com
robots.txt

Robots Exclusion Standard data for thediggings.com

Resource Scan

Scan Details

Site Domain thediggings.com
Base Domain thediggings.com
Scan Status Ok
Last Scan2024-09-27T03:29:31+00:00
Next Scan 2024-10-04T03:29:31+00:00

Last Scan

Scanned2024-09-27T03:29:31+00:00
URL https://thediggings.com/robots.txt
Domain IPs 172.66.40.193, 172.66.43.63, 2606:4700:3108::ac42:28c1, 2606:4700:3108::ac42:2b3f
Response IP 172.66.43.63
Found Yes
Hash 44dffd9e0bc66ec7acd3fe190143c87adff1cc2cb9e1cb929e93e88b1e4e94d7
SimHash 4d1042f257f5

Groups

googlebot

Rule Path
Allow /

googlebot-mobile

Rule Path
Allow /

msnbot

Rule Path
Allow /

bingbot

Rule Path
Allow /

mediapartners-google

Rule Path
Allow /

twitterbot

Rule Path
Allow /

facebot

Rule Path
Allow /

adsbot-google

Rule Path
Allow /

googlebot-image

Rule Path
Allow /

googlebot-news

Rule Path
Allow /

googlebot-video

Rule Path
Allow /

msnbot-media

Rule Path
Allow /

adidxbot

Rule Path
Allow /

duckduckbot

Rule Path
Allow /

bingpreview

Rule Path
Allow /

panscient.com

Rule Path
Disallow /

ia_archiver-web.archive.org

Rule Path
Disallow /

ia_archiver

Rule Path
Disallow /

yandex

Rule Path
Disallow /

dotbot

Rule Path
Disallow /

ahrefsbot

Rule Path
Disallow /

*

Rule Path
Disallow /