How many times has an interviewer — either for an actual employer or for a recruiting agency — asked you: “On a scale of 1 to 10, how would you rate yourself at $insert_skill_here?”
If you ask candidates to rate their own skill levels — on any scale; it doesn’t matter if it’s 1-5, 1-10, or “beginner, intermediate, advanced” — you’re just asking to have the Dunning-Kruger Effect mess up your results.
Let’s say you have two candidates for your Java position: One is basically just one step below James Gosling — let’s call her Jane[1]. The other is the self-described “brillant[sic]” Ms. Paula Bean.
You ask Jane Gosling, “How would you rate yourself in Java, on a scale of 1 to 10?” And Jane thinks, “Well, I’ve got years of experience, I know my way around J2EE like the back of my hand. I still remember Hibernate pretty well, and my grasp of Design Patterns is mostly pretty solid. But lots of people have that; it’s totally standard! And my Struts and EJB are both rusty. I could stand to refresh on a few patterns… hell, I haven’t done a Memento in at least a year; I might even have to look it up to get it right. And I know there’s some new stuff coming out with closures, whatever those are. Well, I guess I’m better than average, so a little above a five.”
“Six,” says Jane. “Ummm… maybe a seven?”
Then you get Paula in, and of course, Paula has no idea how much stuff there is out there that she hasn’t even heard of. She knows she’s brilliant! “Ten!” she answers, without batting an eye.
If you believe either of them, you’re going to be wrong. We defined Jane such that if she isn’t a 10, nobody is. She’s just hyper-aware of her own imperfections — like most of the experts in any field — and she doesn’t think she’s all that special. And we all know Paula is a 1.
If you’re an interviewer, why even ask this question at all? It is nearly guaranteed to never get you accurate results.
And if you’re being interviewed, and someone asks you a question like this? Tell them how awful it is, and why. And if they insist on you giving a self-rating of your own skills? Then my advice is to break off negotiations, and tell them you’ll be looking elsewhere for employment.
Consider: Not only are they unwilling to learn when you point out that they’re doing something fundamentally broken, but also, they used this process to hire everyone else you’ll be working with, too. How many Paula Beans are in their office?
You can find someplace better.
[1] There’s a shortage of famous Java programmers. If this were a Ruby example, I could just use Dave Thomas, Yehuda Katz, or DHH; for Perl, there are Randal Schwartz, Damian Conway, Mark Jason Dominus, and Tom Christiansen. (Curiously, Python seems kind of anti-rock star; I can’t find any big names there easily.)
But the “brillant” Paula Bean makes such a great counter-example, I wanted to make this a Java story. Nothing against Java itself; this is all about Paula as a paragon of the Dunning-Kruger Effect.
Obviously, this is excluding the language authors themselves: They have to spend a lot of time working in C to write their languages’ compilers or interpreters, and so they can’t be spending as much time actually writing in their own languages. ↑↑