Hi all,

Let me preface by saying that I've never had any formal statistics training, am trying to teach myself, and I'm not sure if my question is even feasible.

I'm working on a project where I'm creating surveys for users to take. I'm trying to figure out which questions are the best, so I'm going to develop a "Question Quality Score" (QQS).


There are two factors that go into the QQS - the question's answer rate, and the standard deviation of the chosen answers (i.e, a question with stddev=0 is most valuable because it splits up the user population evenly. On the other hand, if I had a yes/no question where 100% people answered yes, that wouldn't help me split up the population at all).

My first thought was to just normalize the answer rates and stddev, and then average the normalized values together to get the QQS. However, I suspect that having a low stddev is more valuable than having a high answer rate. How could I quantify my suspicion? My end goal would be to prove that, say, stddev is twice as important as answer rate, then my QQS formula could factor that in.

SD is directly related to how many answers you get, not independent.

Here is the key to us helping you with your problem. What do you mean by "which questions are the best?" Do you mean the questions that are the most difficult? Do you mean the questions that are the most discriminative (i.e., the questions that best identify differences in the test takers' true skill levels)?

Depending on your answer, you might look into item response theory.

The questions that are the most discriminative is what I'm trying to accomplish.

Just Wiki'd IRT. Seems like pretty interesting stuff. Going to try to familiarize myself with it now and see if I can find it applicable.