The last time I checked, a Google search on confidence produced over 250 million results. Of those that I surveyed, not all were related to self-confidence, but most did focus on some aspect of self-help, such as boosting self-esteem, self-belief or self-love levels - with the underlying assumption that doing so will make us happier, healthier, wealthier, and more attractive.
In line with this, publication of self-help books has more than doubled in the past three decades, and the wider self-help industry (which also includes DVDs, podcasts, and seminars) is now worth an estimated £7bn.
And yet, there is actually very little evidence that one can deliberately boost one's self-confidence, let alone that it is beneficial to do so.
In fact, a great deal of psychological research suggests that deliberate attempts to increase confidence or self-esteem will probably backfire: either by distancing you from reality - this is what happens when you successfully persuade yourself that you are better than you really are; or by making you depressed - this is what happens when you don't succeed at it.
Consider the well-documented fact that levels of narcissism (a delusional, unstable and selfish form of self-admiration that disrupts all sorts of interpersonal relations) have been rising steadily in the past few decades. Unsurprisingly, this trend is most notable in the US, which is arguably the biggest global exporter of narcissism.
Far from making you happier or more successful, a culture of narcissism simply makes you feel more entitled, more self-centred, and unrealistic. For all the talk and discussion about so-called millennials or gen Y, who will represent 75% of the workforce by 2030, the only clear thing about them is the huge mismatch between their aspirations and their work ethic. They want to be Steve Jobs but with Paris Hilton's lifestyle - and, in fact, work a lot less than she does.
As for the world of management, the news is hardly any better. First, it is a well-documented fact that most people are not as good as they think they are at, well, virtually anything. If you ask people how well they drive, sing, or manage others, over 80% will tell you that they are better at these things than the average person, which is of course statistically impossible.
The world, then, is full of people who are more confident than competent, and there are clear disadvantages to displaying a 'confidence surplus' (overestimating your abilities): most people will find you arrogant and even intolerable.
However, people are famously prone to misinterpret displays of confidence as competence. The 'fake-it-till-you-make-it' rule does indeed work but it has short-term benefits at best. In the long run, it is more a matter of 'fake-it-till-you-make-it-till-you-break-it' - and everybody realises you were always no good.
Second, the systemic toxicity of some corporate cultures allows manipulative narcissists and bullish megalomaniacs to move up the power ranks of the organisational ladder, while stopping talented but modest employees from being promoted. This is why most CEOs and entrepreneurs fail, and why we have far fewer women in power than men.
In a recent study, researchers at UCL asked employees to describe their current bosses using just one word. The chart above is an uncensored wordcloud of their responses - I am sure you can relate to it.
Although few countries are as similar to each other as England and the US (think language, imperialism, and capitalism), there is a significant difference between these two cultures - the fact that Americans worship confidence and bragging, whereas the English indulge in displays of insecurity and self-deprecation.
(A personal side-note: after growing up in one of the most overconfident countries on earth - Argentina, a nation of charming underachievers - I moved to Britain almost 15 years ago and, even before I became a Brit, I learned to perfect the art of over-the-top modesty. Of course, I also learned to read between the lines and identify covert displays of hubris underlying fake modesty, something no psychology textbook could teach you.)
However, there are many reasons to be optimistic about British pessimism, and to prefer this underwhelmed, negative, and grumpy attitude to life over the too optimistic and overconfident American alternative (and, yes, I know I'm preaching to the converted).
Consider the following: there is virtually no difference between the leadership development and the self-help industries; both foment unrealistic confidence and a can-do attitude that is completely out of touch with reality.
The message is always the same: 'No matter how bad things are, don't worry, you are great and who cares about what other people think.' My feeling is that a commonsense alternative to this uncomfortably cheerful message should be: 'No matter how well things are going, keep worrying, you are not as great as you think or even as other people may think.' This is arguably the simplest recipe for a confidence antidote.
If you think this is just metaphysical speculation, how do you explain the current shift in power between the self-indulgent, self-promotional, and narcissistic west and the modest, perfectionist and self-knowledge-oriented cultures of the east? Perhaps Britain should start exporting pessimism to its fellow western nations, especially the US: it may open the doors for a bit of realism.
In a more or less singlehanded attempt to offer an alternative to the nonsensical message of the self-help industry, I decided to write a book about the true science of confidence and competence. This book is not just an effort to debunk the common myths about self-confidence, it also hopes to offer a solution - or range of solutions - for those who hope to gain competence (eg, in their career, personal relationships or health-related issues).
My theory is rather simple and can be illustrated with this matrix, which depicts the relationship between confidence (how good you think you are) and competence (how good you really are).
Although there are four possible types emerging from the interplay between high and low confidence and high and low competence, these four types are not equally distributed in the population.
Rather, most people are in the high confidence but low competence quadrant, which means they have incompetent confidence - and the cure for this problem is to get genuine feedback from honest, critical others. This solution is not as simple as it sounds because humans, in their attempt to be civil and display the required political etiquette, seem generally to prefer polite lies to honest criticism.
What's more, those who have incompetent or unwarranted confidence seem somehow prewired to neglect any negative feedback and attend to positive feedback instead, or even interpret neutral feedback as positive.
Why is incompetent confidence a problem? Because distorted positive self-views are the chief enemy of selfimprovement. The better you think you are at something, the more unlikely it is that you are interested in becoming better at that something - and instead of focusing your energies on self-improvement, you will be mostly focused on maintaining your high regard for yourself no matter what, distancing yourself from reality.
The second most common group is the low confidence-low competence quadrant and, contrary to what the self-help industry proclaims, people in this group enjoy an advantage over the incompetent confidence group, namely being self-aware or knowing what their limitations are. Indeed, this condition, which I call realistic self-doubt, is the first step forward for any individual with incompetent confidence who wants to improve.
It is what happens to cocky managers when they realise that they are not as smart as they thought they were... or, at least, this is the best thing that could happen to them: although initially the truth may hurt, the long-term implications are clearly more positive than negative.
Anyone with realistic self-doubt faces a clear self-improvement option, which is to become more competent, and an equally obvious cul-de-sac, which is to attempt to be more confident, and so downgrade to the incompetent confidence condition.
In other words, if you are aware of your limitations and you don't like what you see, then what better incentive could you have to actually get better?
The common mistake is to seek fake positive feedback in order to feel better - this is why the self-help book market is booming - when the actual goal is not to feel better but to get better. Instead of acting like the mythical ostrich that buries its head in the sand to avoid danger, you should try to confront your insecurities rather than ignore them, because they are there precisely to alert you to some of your problems.
There are plenty of drugs to treat your symptoms, but if your goal is to be healthy then you need to tackle the root cause.
People in the top two quadrants of the diagram are statistically much rarer. Though they are competent, they still face challenges. People with realistic confidence - who are not just competent but also aware of their competence - need to avoid complacency and stagnation.
This is a common problem in business, as most successful organisations focus so much on maintaining the status quo (precisely because they are successful) that they stop being innovative and competitive. The paradox, then, is that although innovation leads to growth, growth tends to inhibit innovation.
And the same happens at the individual level - that is, with people rather than organisations: when you are good at something and you know it, you have as little incentive to improve as when you are not as good at something as you think you are.
This is why political leaders tend to decay after too many years in power; why top athletes have their reigns threatened by hungrier contenders (in tennis, for example, first there was Federer, then Nadal, then Djokovic, and now Murray); and why the BlackBerry was overshadowed by the iPhone, which in turn has now lost its edge against Samsung and other Android devices.
A favourable reputation, just like a strong brand, is a double-edged sword: it leads to accomplishments for historical or nostalgic reasons, rather than true merit. There is no doubt a time lag between actual and perceived competence and when this gap widens you are almost destined to fall from the realistic confidence to the incompetent confidence quadrant.
As for the final quadrant (top left), it arguably illustrates the most fascinating condition, namely perfectionist self-criticism - these are the people who are their own worst critics and are overly harsh on themselves.
On the one hand, they should be more confident than they are - because they are more competent than they think. On the other hand, by being their own worst critic they avoid being complacent and remain more competent than their competitors.
Top performers and exceptional achievers in any domain of competence tend to fall into this quadrant. Furthermore, achieving cultures are much more likely to fall into this quadrant than any of the other three quadrants in the diagram.
Contrary to popular opinion, self-belief has never been responsible for economic prosperity - it is merely a symptom of a nation's realisation of its accomplishments. The real source of progress is ambition, and ambition is the ability to remain hungry and dissatisfied, no matter how much you have achieved.
When countries stagnate, or display signs of complacency and entitlement, it is almost certain that they have killed their ambition - perhaps because their accomplishments have sated their hunger, or because they feel as competent as they wanted to be.
It seems that western civilisation is finally waking up to the idea that perfectionist self-criticism is one of the driving forces underlying the Asian boom and, by the same token, one can only hope that we slowly start to acknowledge the fact that much of the decay of the west can be accounted for by overconfidence.
The lesson could not be clearer - if our primary goal is to feel good about ourselves, there's no point in getting better. But if our goal is to get better, we should try to avoid feeling good about ourselves, and at any cost.
Tomas Chamorro-Premuzic is a professor of business psychology at UCL and VP of Hogan Assessments. His latest book, Confidence - the suprising truth about how much you need - and how to get it, will be published in the UK by Profile Books on 7 November at £12.99.