Subject: Log-likelihood ratio test as a probability


On Fri, Jun 21, 2013 at 8:25 AM, Dan Filimon <[EMAIL PROTECTED]>wrote:

You can't reliably combine p-values this way (repeated comparisons and all
that).

Also, in practice if you take the top 50-100 indicators of this sort the
p-values will be so astronomically small that frequentist tests of
significance are ludicrous.

That said, the assumptions underlying the tests are really a much bigger
problem.  The interesting problems of the world are often highly
non-stationary which can lead to all kinds of problems in interpreting
these results.  What does it mean if something shows a 10^-20 p value one
day and a 0.2 value the next? Are you going to multiply them?  Or just say
that something isn't quite the same?  But how do you avoid comparing
p-values in this case which is a famously bad practice.

To my mind, the real problem here is that we are simply asking the wrong
question.  We shouldn't be asking about individual features.  We should be
asking about overall model performance.  You *can* measure real-world
performance and you *can* put error bars around that performance and you
*can* see changes and degradation in that performance.  All of those
comparisons are well-founded and work great.  Whether the model has
selected too many or too few variables really is a diagnostic matter that
has little to do with answering the question of whether the model is
working well.