artnet.com
robots.txt

Robots Exclusion Standard data for artnet.com

Resource Scan

Scan Details

Site Domain artnet.com
Base Domain artnet.com
Scan Status Ok
Last Scan2024-06-27T20:13:50+00:00
Next Scan 2024-07-04T20:13:50+00:00

Last Scan

Scanned2024-06-27T20:13:50+00:00
URL https://artnet.com/robots.txt
Redirect https://www.artnet.com/robots.txt
Redirect Domain www.artnet.com
Redirect Base artnet.com
Domain IPs 45.223.18.250, 45.223.28.250
Redirect IPs 45.223.22.250
Response IP 45.223.22.250
Found Yes
Hash 97e9f6cdc2d1309d77343fa8dbae3482f40f20657038212b3788632e5870283a
SimHash f33074b5abc7

Groups

*

Rule Path
Disallow /bannerinclude_DE
Disallow /bannerinclude_US
Disallow /bannerinclude
Disallow /bannerinclude_fr
Disallow /FinancialServices
Disallow /Intranet
Disallow /Intranet-PDB
Disallow /includes/*.asp$
Disallow /fiut
Disallow /MarketAlert/AppsSecure
Disallow /MarketAlert/EmailNotifications
Disallow /MarketAlert/bin
Disallow /net/AppsSecure
Disallow /net/EmailNotifications
Disallow /net/bin
Disallow /net/Templates
Disallow /usernet/agadmin
Disallow /usernet/amt
Disallow /usernet/include
Disallow /usernet/security
Disallow /usernet/tracking
Disallow /net/galleries/search_artworks_results.aspx
Disallow /net/galleries/address.aspx
Disallow /net/galleries/search_galleries_results.aspx
Disallow /ag/fulltextsearch_binfo.asp
Disallow /ag/fulltextsearch.asp
Disallow /admin
Disallow /content
Disallow /customerror
Disallow /ArtnetIntegrationService
Disallow /auctions/BrowseAuctions.asmx
Disallow /auctions/WebService.asmx
Disallow /auctions/SecuredWebService.asmx
Disallow /auctions/Pages/Lots/image.ashx
Disallow /auctions/Pages/Lots/Private
Disallow /auctions/ordercomplete.aspx
Disallow /auctions/assets/files/
Disallow /auctions/fts/
Disallow /auctions/search/
Disallow /auctions/Pages/Lots/Private/
Disallow /*fairs.artnet.*
Disallow /*web.artnet.*
Disallow /*admin.artnet.*
Disallow /*dev.artnet.*
Disallow /decart1
Disallow /iptest
Disallow /iptest.aspx
Disallow /mobileservices
Disallow /artnetmktg
Disallow /auctions/Pages/Lots/Private/54622.aspx?lotId=54622
Disallow /auctions/Pages/Lots/Private/54622.aspx?q=%2Fauctions%2Fmodern-art%2F5
Disallow /services/WebServices/VATService.asmx/ValidateVATID
Disallow /ArtnetLookupService
Disallow /dojo
Disallow /Solr
Disallow /Membersites
Disallow /Autosuggest
Disallow /search/
Disallow /Dart_banner
Disallow /home/SignupNewsletter.asmx
Disallow /products/getStatesByIdHandler.asp
Disallow /4408/art.net.com

Other Records

Field Value
crawl-delay 1

wget
webzip
webmirror
webcopy
czxt2s
trendictionbot
blp_bbot
ccbot
trendictionbot
sogou web spider
cliqzbot

Rule Path
Disallow /

Other Records

Field Value
sitemap http://www.artnet.com/SiteMap/com/sitemapindex.xml
sitemap https://www.artnet.com/auctions/sitemapindex.xml