I alternate wearing two smartwatches. I wear a Garmin on days I go running, and a smaller FitBit on days I do not. I had a hunch that my FitBit made it easier to get to 10,000 steps. So one day I decided to wear both watches on the same wrist for the whole day. (This is the kind of nerdy thing people who write research methods books will do.) Turns out I was right: My FitBit counted about 20% more steps than my Garmin. One of my watches was wrong.
Smartwatches count more than just steps–they estimate calorie expenditure, heart rate, and more. How accurate are they, and how might we tell?
A study conducted by Dr. Jason Kostrna at Florida International University decided to find out. His team tested four commercial smartwatches. Their report is available open-access in PLOS, and was summarized by media outlets including NPR.
Here’s an excerpt from the NPR interview.
A new study finds the wearable devices that so many people use to track their daily exercise just aren’t that accurate. The research shows the smartwatches and fitness trackers overestimate how many calories are being burned. […]
FADEL: So, Jason, how inaccurate are the readings people are getting?
KOSTRNA: So in our study, we found that there’s a typical error of about 15% to 25%. And that varies in our study based on individuals’ body fat percentage.
…one of the big findings of our study, is that the body fat percentage of an individual was correlated with an increased error in how many calories the watch said you were burning. So individuals with higher body fat percentage were getting estimates that were less accurate.
FADEL: As in less accurate that they were burning less or more?
KOSTRNA: So the watch was telling them that they burned more calories. So in our study, if you were in a high body fat percentage of 35% or 40%, you might look down and see 100 calories on your watch. And in reality, you might’ve only burned 50 or 60.
In esssence, this was a criterion validity study. They were testing whether each of four “tests” (four watches) correlated with the actual behavior of burning calories. Here’s an important part of the story: How do they know how many calories people actually are burning? What is the “ground truth?”
To answer that question, here’s more detail from a Washington Post story:
The volunteers also strapped on high-tech face masks and other scientific paraphernalia to precisely measure metabolic changes.
Then they rode stationary bicycles, alternating two minutes of fast pedaling with five minutes of rest, until they’d completed 10 minutes of strenuous pedaling, plus the rest periods. The various devices tracked everyone’s energy expenditure throughout.
But the final measurements differed widely. Almost across the board, the smartwatches had overestimated people’s caloric burns by about 15 to 25 percent, compared to the rigorous, scientific readings.
Questions
a)Visit the PLOS journal article and find out the specific name of the “scientific paraphernalia” they used to “precisely measure metabolic changes.”
b) There were about 50 people in this study. Sketch a scatterplot of the correlation between people’s actual calorie burn (as measured in part a) and their Garmin watch estimate. Imagine that calorie burn ranges from 50 to 200 on each device. Your scatterplot should demonstrate the 15 to 25% overestimate of the Garmin device. (You can ignore body fat for this part.)
c) Draw a comparison between this criterion validity test and the criterion validity tests described in Chapter 5 for the Beck Depression Inventory. What do they have in common? How are they different? Your answer might use terms such as “self-report,” “behavior,” “correlation,” and “physiological measure.”
d) Which of the four big validities is this study about?