Log In Sign Up

picfair.com
robots.txt

Robots Exclusion Standard data for picfair.com

Resource Scan

Scan Details

Site Domain picfair.com
Base Domain picfair.com
Scan Status Ok
Last Scan2025-11-27T00:09:12+00:00
Next Scan 2025-12-27T00:09:12+00:00

Last Scan

Scanned2025-11-27T00:09:12+00:00
URL https://picfair.com/robots.txt
Redirect https://www.picfair.com/robots.txt
Redirect Domain www.picfair.com
Redirect Base picfair.com
Domain IPs 104.26.2.235, 104.26.3.235, 172.67.68.232, 2606:4700:20::681a:2eb, 2606:4700:20::681a:3eb, 2606:4700:20::ac43:44e8
Redirect IPs 104.26.2.235, 104.26.3.235, 172.67.68.232, 2606:4700:20::681a:2eb, 2606:4700:20::681a:3eb, 2606:4700:20::ac43:44e8
Response IP 172.67.68.232
Found Yes
Hash 8a6b3cb6cfd8279783f26bab140b8cd5dd66c07079729fbe435798e8a9fd2ff2
SimHash 8b944bacdd65

Groups

slurp

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 1

*

Rule Path
Disallow /styleguide
Disallow /admin
Disallow /administration
Disallow /basket
Disallow /users/
Disallow /pics/
Disallow /tags
Disallow /search
Disallow /categories
Disallow /curated-collections
Disallow /help
Disallow /basket
Disallow /download
Disallow /users/new?*
Disallow /session/new?*
Disallow /search?*
Disallow /*mode%3Dquery*
Disallow /*reset%3Dtrue*

petalbot

Rule Path
Disallow /tags/

Other Records

Field Value
crawl-delay 1

semrushbot

No rules defined. All paths allowed.

Other Records

Field Value
crawl-delay 2
Back to top

Other Records

Field Value
sitemap https://www.picfair.com/sitemap.xml
Back to top

Comments

  • See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
  • Robots shouldn't try to share content,
  • or index the style guide
  • Market
  • 2020SEAP21
Back to top