Sunday, January 18, 2015

Research that Only Confirms What You Already Know, or Want to Believe, is a Problem

Surveys are only as good as the assumptions that underlie them. If the samples are not truly representative of the population one claims to study (women, men, Millennials, retirees, electric vehicle enthusiasts, Facebook users), then one cannot extrapolate from a survey. 

In addition, the survey instruments, methods and questions must be constructed in ways that do not overtly bias the results. 

So without implying any shortcomings in methodology, or any overt attempt at influence the results on the part of any firm that hires a research firm to conduct a study, the results of one recent study of potential Internet of Things demand are so out of line with the likely state of current awareness one has to assume the survey sample was chosen from an atypical group of subjects.

In all likelihood--and without alleging any effort to skew results--a survey claiming extraordinarily high confidence in the value of IoT by business users produces those results because the survey sample is of people whose job responsibilities involve IoT in some way.

That doesn't mean the findings are in any way deceptive or inaccurate, or the result of survey methodologies that were faulty.

It likely does mean that the survey sample was not typical of all business or technology executives and managers of businesses in the United States as a whole. 

Some might argue the study only confirms that the survey population primarily was of information technology executives who believe IoT will be important. 





No comments:

Many Winners and Losers from Generative AI

Perhaps there is no contradiction between low historical total factor annual productivity gains and high expected generative artificial inte...