It's another variation on "Looking for the watch where the light is better."
This really doesn't fit. If there are dozens of watches scattered around, of course you first look where the light is good.
These situations aren't at all symmetric. A false positive always costs you (the company). A false negative is only a cost to you if you fail to hire well for that position.
Also the fact that a candidate you passed on does well elsewhere doesn't mean you were wrong to pass on them (it could, but it's not clear).
I was not equating this to a scenario where you are only looking for one watch. The expression (and anecdote behind it) is also used to describe managing according to metrics that are easy to collect, often at the expense of outcomes.
That applies in the case of caring more about false positives than false negatives, since nobody (or close enough) even knows how many false negatives they have.
Your statement that someone being successful elsewhere doesn't prove we have a false negative underscores the uncertainty involved. It doesn't invalidate what I'm saying in the least.
But if we want to be rigorous, we could do some direct testing. Let's say we give a telephone screening and four interviews to each candidate. Simpkifying somewhat, let's say that our policy is that a fail on any one eliminates the candidate.
If we hire lots of people, we can collect false positive and false negative stats by randomly passing someone who fails exactly one test.
If a lot of people who fail a particular test end up succeeding, we can infer that the test provides false negatives.
On the other hand, if everyone who gets the random "free pass" on one particular test ends up failing, we can presume it has low false negatives, and maybe abandon the random passes for that test.
I understood the analogy you were trying to draw, I just disagree with it.
The original joke/anectdote works because it leads to clearly suboptimal search - the guy knows his watch isn't near the streetlight, but he's looking there because he can see.
This is a very different situation. To abuse the analogy further, the guys best bet to find a watch seems to be just looking near the streetlight. After all, not only can you see better there, but most of the watches are near there too.
What your opponent is saying is that the cost of false negatives is unknown because it's hard to calculate. So people only calculate the cost of of false positive because it's easy to calculate then compare it to the unknown value and conclude that it's greater, not dissimilar to the gentleman who had been looking for the watch under a streetlight.
Your saying "false negative is only a cost to you if you fail to hire well for that position" could be easily disproved with examples of people, who were rejected and started their own competitive business (e.g. [1]). IMHO, one of the goals of having 10Ks of engineers on staff is to keep them from the competition. But since nobody is tracking this and the extreme cases like WhatsApp can be hand-waved away it's easy to pretend that the false negative's cost is miniscule.
If I’d seen any evidence that this cost wasn’t minuscule int the typical case, I’d be much more interested in expending the effort to try and measure it better and act on that - but that has an opportunity cost also. So you are suggesting incur a real cost just in case? I don’t buy it.
The guy with the watch analogy only works if there is good reason to believe he’d be better off searching elsewhere. But we don’t appear to have that here....
I am not sure you checked out the link I've given. It's quite a substantial evidence that not hiring Koum and Acton might have costed FB somewhere around $19B if one were to believe WhatsApp was acquired to neutralize competition and not for its amazing revenue.
Let's be generous and say the WhatsApp was a 10x unicorn so FB overpayed only 9B for it. Which makes 4.5B loss per false negative on each founder. If a false positive was, on average, 1M loss (which is an extremely generous in an organization such as FB in my recollection), it means you could have had 4000 false positives for each of these negatives and still come ahead.
And, it brings us back to the Type I and II errors - it's always cheaper to make one, which has a limited downside than the unknown (potentially very high one). Typical example given is that while you are on the way to an expensive cruise you realize that you might have forgotten to turn off the stove. If you go back you might miss your 10K cruise. If you cary on, you might lose you 1M house. On average, people who miss cruise in such circumstances fare much better than ones, who risk their houses.
I know that story - but think it is enough of an outlier (regardless of how it went down) to basically be irrelevant to almost all hires. The idea that people are routinely filtering out great hires in their thousands in ways that come back to haunt them ... doesn’t seem plausible.
I get what you guys are saying, there just doesn’t seem to be real evidence that is an issue nearly all cases. Whatsapp notwithstanding
No, the idea is that you don't know what you are filtering out. As I said - it's extremely easy to just hand-wave. And if you want to believe that you know the unknown downside I am not the one to persuade you otherwise. But acknowledging that it's just a personal belief not only unsubstantiated but contradicted by data could be a good first step :)
I think we are talking past each other. I have never said I know the unknown, that would be silly. But funds,entsllu can’t avoid false negatives without degrading other performance , assuming some reasonable things. Even without those assumption, there is a very real cost to what you are suggesting. You seem to be waving off this cost.
So in order for it to make sense to spend that money, you want some evidence there will be a positive return. Far from contradicting my position, what limited data I have seen does not support that effort, at least in the general case. Certainly whatsapp and the like don’t speak to it - they are very much outliers.
By all means try it. If your false positive rate explodes it will be an expensive lesson, but that’s always a risk.
One last point, the idea that nobody thinks about this and try’s thinks is laughable , which is worth noting because some comments in this thread almost suggest that it’s true,.
With my example of sacrificing a 10K cruise for a chance to not lose a 1M home, I figure?
> Far from contradicting my position, what limited data I have seen does not support that effort, at least in the general case.
Indeed, the data does not contradict a statement "on average, false positives are 4.5B each", which nobody has argued. However, it does contradict the statement "false negatives have any cost only when you are unable to fill the position as the result", which you have made in this thread.
>One last point, the idea that nobody thinks about this and try’s thinks is laughable , which is worth noting because some comments in this thread almost suggest that it’s true,.
Again, it would be silly to say that nobody thinks about it. Evidently, a number of people think the same way you do.
If there are dozens of watches scattered around, of course you first look where the light is good.
At first, yes. But when one has had enough experience looking for scarce commodities (of whatever kind: genuinely attractive apartments, interesting music / restaurants, romantic partners, etc) one learns that the "where the light is" search technique only gets you so far.
These situations aren't at all symmetric. A false positive always costs you (the company). A false negative is only a cost to you if you fail to hire well for that position.
Also the fact that a candidate you passed on does well elsewhere doesn't mean you were wrong to pass on them (it could, but it's not clear).