Abstract
In this issue's Statistical Sidebar, I would like to take a step way back from looking at individual statistical tests and speak about the entire enterprise of statistics. Hopefully, this broad view will demonstrate how professionals in the field of visual impairment can, and should, be a part of research endeavors. Many people think of statistics just as a mathematical treatment of data; the application of statistical tests. To conduct statistical tests with confidence, however, a researcher needs to make sure that the data being used is of high quality. And, after the tests are run, the researcher needs to be able to interpret and convey the meaning of those tests. So “statistics” really incorporates elements of the entire research process. A person can think of statistics as being about collecting, analyzing, interpreting, and presenting data so that an answer to a question being asked might be arrived at with some level of confidence.
Collecting Information
How a researcher collects information sets the foundation for everything that comes afterward. Careful preparation for collecting information in the correct way is what will allow meaningful use of statistical testing later. There are a host of elements a researcher should consider when preparing to collect information, and some of these involve knowledge of principles that are fundamental to statistics. For example, research studies typically use samples of participants.
Knowing how different approaches to sampling might affect the generalizability of a study's results is useful. A researcher needs to be able to calculate how large a sample is necessary for the kinds of statistics that are planned. Related to this kind of calculation is an understanding of the normal distribution.
While many things that a researcher might collect information on are normally distributed (e.g., most of the scores will be closer to the average of all the scores with progressively fewer scores being much higher or lower than the average), not all data will be normally distributed. Thus, a researcher needs to know how to test for normality, what to do if data are not normally distributed, and what that might mean for a study's conclusions.
Finally, a researcher needs to pay attention to how information is collected. Research is generally an attempt to isolate just a few variables so that the effect of these variables on a few other variables can be investigated without other things intruding on the process. In other words, research studies try very hard to control the effect of outside influences.
There are many ways to mitigate outside influences in a research study, but using control groups is one of the best ways. In a control group design, the researcher has two groups of participants who are as similar to each other as possible on what the researcher thinks are variables that might affect the study's outcome. One group experiences some sort of treatment or intervention and the other group does not. It should be noted that researcher bias is always lurking in the background. Perhaps the research has discounted some variables that should have been included—one of the many aspects of research that make it difficult to draw solid conclusions from only one study. The other thing to pay attention to in preparing to collect information is in defining what is being measured and how it is being measured. A researcher needs to be confident that they are measuring what they intend to measure, they are doing it with precision, and they are not accidentally measuring other information tied up with what they intend to measure.
Analyzing Information
Information analysis is what most people tend to think of when they think of statistics. But, if the collection of data is not done carefully, often no analysis is possible that would allow confidence in the “answers” pulled from the data. This scenario is what is behind the saying “garbage in, garbage out.” When using statistical software, a researcher can input numbers and churn out some sort of numerical result. Without confidence in the numbers being input, however, the result is meaningless. Careful description of a study's methodology is also important, so that the reader of published research is able to ascertain whether the kind of data gathered makes sense for answering the questions being asked in the study.
As a dataset gets more complicated, with more variables able to be considered or compared to each other, there are more ways that the data could be looked at. Decisions on which variables should be used to answer a given question might not be as clear cut as a researcher thought when embarking on the study. There may appear to be, in the data, some effects of other variables that were not expected. Or, the effect of one variable on another might be impacted by a third or fourth variable and this complex relationship might not be apparent with a simple analysis of the data. It can take some time and a researcher may need to create a series of tables and graphs to visualize the data or organize the data in different ways in order to fully understand all the variable relationships within the dataset.
Interpreting Information
If a researcher has confidence in their measurement procedures and resulting data, then making sense of the results is the next potential stumbling block. Analysis and interpretation go somewhat hand in hand. A thorough analysis of the data should expose all the relationships in the dataset, the interpretation is both a way to give an overall impression not only to a reader of what the data are saying but also to illustrate the nuances within the data. The more complex a dataset, the more complicated the explanation of what is going on in the data will be. Harkening back to the first section on collecting information, clearly defining the goal of the research, identifying any hypotheses the researcher may have, doing what is possible to control for the influence of factors outside of the research, and precisely and systematically collecting all information relevant to the research questions will allow the researcher to be able to interpret the findings. Poor sampling, lack of control, poorly defined measures, and sloppy procedures will lead to results that are essentially meaningless.
Presenting Information
Once a researcher has run a well-controlled and defined study, analyzed the data to get at all of the nuances within the dataset, and made sense of the findings, an effective way to present the data is needed. This step is one on which many researchers fall short. An article with a results section ladened with statistical results and graphs pumped full of as much information as possible tends not to be the most effective way to communicate findings.
Although it is important to offer all the relevant details of data analysis, it needs to be presented in the results section alongside a thread of explanation. It helps readers to have a brief interpretation of a statistical test as soon as it is presented in an article. Then the conclusions section can offer a clear description of the broad strokes findings and some of the more impactful results of the research. Although not often done, a simplified graphic of the main take-home points can make the conclusions section more easily understandable.
In summary, statistics is simply a process by which researchers answer questions that are being asked. In the vision fields, there remain a huge number of questions that need answering. What is most pressing often comes from professionals in the field. If you have a question that has been nagging at you, I encourage you to team up with someone who can lead you through the sort of process I have laid out in this sidebar. The brief description of the research process that I have offered does not illustrate all of the nuances a researcher needs to be aware of or the pitfalls that might await a novice researcher, but that is why it is useful to form a team. I have found that the most impactful research I have been involved in over the years are projects where some team members are tied closely to service provision and other team members can offer guidance on the research process. And, of course, once your research is all done and ready to share, the pages of this journal are here to facilitate that sharing.