Originally posted by Junix
View Post
It seems odd to me that a random sampling would have that much variation where some people get it "regularly", and others almost never. It seems like over a span of years, a random sampling would tend to average out so that any particular person would have roughly the same average frequency. If I roll a fair die once, a 1 is as likely as a 4 (or any other number), but if I roll the same die 60 times, the average is very likely to be close to 3.5.
The same principle should apply to the survey sampling. In any given month, random chance might just not give you the survey, but over the course of 60 months, it should tend to average out. Sure, there will still be occasional outliers who got exceptionally many or few surveys, but the vast majority of people should be relatively clustered around an average number of surveys that is based on what percentage of the users they sample each month.
Bad sampling might be equally effecting all operating systems, or it might be biasing the results, but if we can't trust the sampling, the data is basically garbage, at least with regard to OS popularity.
Personally, I'd like to see them include a few basic things like OS and processor in an automatic full-population survey that's automatically on by default, with an option to opt-out if you want. I'd also like them to include an account setting for "preferred OS", to capture information about how many people are using an OS other than their preferred one, along with which specific games they play in their non-preferred OS. It seems like this could be potentially useful data to game developers. It might be interesting to measure what percentage of people's usage time is in their preferred vs. non-preferred OS.
It seems to me that the current information they're collecting is of limited use, possibly inaccurate, and potentially more misleading than illuminating
Comment