pond5.com
robots.txt

Robots Exclusion Standard data for pond5.com

Resource Scan

Scan Details

Site Domain pond5.com
Base Domain pond5.com
Scan Status Ok
Last Scan2024-05-24T02:04:35+00:00
Next Scan 2024-06-23T02:04:35+00:00

Last Scan

Scanned2024-05-24T02:04:35+00:00
URL https://pond5.com/robots.txt
Redirect https://www.pond5.com/robots.txt
Redirect Domain www.pond5.com
Redirect Base pond5.com
Domain IPs 13.33.88.27, 13.33.88.28, 13.33.88.6, 13.33.88.76
Redirect IPs 13.33.88.27, 13.33.88.28, 13.33.88.6, 13.33.88.76
Response IP 13.33.88.6
Found Yes
Hash 0278ce8c7b913d8c6f36d615e16638fc72716aa2936be802c567a168ac164b76
SimHash 9845f4318277

Groups

*

Rule Path
Disallow *page%3Dartists
Disallow /*/mightlike
Disallow /artist-directory
Disallow *membership%3D1
Disallow *page%3Dthundercloud
Disallow *page%3Dlightbox_license
Disallow *page%3Dlightbox_terms
Disallow /ajax/recent/items/add
Disallow /ajax/user/*
Disallow /ajax/user-account/*
Disallow /ajax/collections-bar
Disallow /ajax/dashboard/*
Disallow /ajax/cart/*
Disallow /ajax/bin/*
Disallow /ajax/item/*
Disallow /ajax/search-filters/*
Disallow /ajax/translation*
Disallow /ajax/search/suggestion*
Disallow /*/ajax/recent/items/add
Disallow /*/ajax/user/*
Disallow /*/ajax/user-account/*
Disallow /*/ajax/collections-bar
Disallow /*/ajax/dashboard/*
Disallow /*/ajax/cart/*
Disallow /*/ajax/bin/*
Disallow /*/ajax/item/*
Disallow /*/ajax/search-filters/*
Disallow /*/ajax/translation*

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

rogerbot

Rule Path
Disallow /

sitebot

Rule Path
Disallow /

coccoc

Rule Path
Disallow /

facebookexternalhit/*

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

facebookexternalhit/1.1

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

facebookexternalhit*

Rule Path
Allow *

baiduspider

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

jikespider

Rule Path
Disallow /

twengabot

Rule Path
Disallow /

blekkobot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2

linguee

Rule Path
Disallow /

sistrix

Rule Path
Disallow /

yioopbot

Rule Path
Disallow /

blexbot

Rule Path
Disallow /

ccbot

Rule Path
Disallow /

gptbot

Rule Path
Disallow /

Other Records

Field Value
sitemap https://sitemap.pond5.com/en-sitemaps-index-static-pages-1.xml.gz
sitemap https://sitemap.pond5.com/en-sitemaps-index-artists-1.xml.gz
sitemap https://sitemap.pond5.com/en-sitemaps-index-collections-1.xml.gz
sitemap https://sitemap.pond5.com/en-sitemaps-index-tags-1.xml.gz
sitemap https://sitemap.pond5.com/en-sitemaps-index-searches-1.xml.gz
sitemap https://sitemap.pond5.com/en-sitemaps-index-pd-1.xml.gz
sitemap https://sitemap.pond5.com/en-sitemaps-index-detail-pages-1.xml.gz