The word is French, as you might have guessed, and means a small shop. Its origin is the Greek apotheke, meaning a store-house, cellar or, literally, 'place where things are put away'. The word was first used in English in the 18th century, usually in the context of exotic foreign places. The idea of the boutique as a shop selling trendy clothes to young people is American and began in the 1950s. Nowadays, you are as likely to find it used for a small business offering a highly specialised, almost personal service to a select clientele.
'Boutique hotels', offering a chi-chi alternative to the large chains, are common, but there are also 'boutique' brokers, investment banks and advertising agencies, all dedicated to the idea that small is beautiful.