A Harvard biostatistician dropped the idea to use an Apple Watch in a research study due to the inconsistency of the smartwatch’s data. It was found that Apple Watch shows data after it’s analyzed and filtered through an algorithm, and the algorithm changes, frequently.
As reported by The Verge, researchers call algorithms of commercial devices ‘black boxes’ which are a consistent challenge. As companies update their software often to offer improvements but those changes are pushed without a warning which unfortunately changes the data collected from the same time period.
Unless researchers gain access to raw data on Apple Watch, the smartwatch is a reliable device to be used in studies
Aware of the ‘black boxes” impact, JP Onnela, associate professor of biostatistics at the Harvard T.H. Chan School of Public Health and developer of the open-source data platform Beiwe tested Apple Watch’s data reliability before starting the study.
Onnela checked the heart rate data of a collaborator exported for his Apple Watch twice: September 5th, 2020, and April 15th, 2021. Theoretically, the data should have been identical, but it was found:
Rather than showing the raw data collected by a device, the products usually only let researchers export information after it has been analyzed and filtered through an algorithm of some kind.
Companies change their algorithms regularly and without warning, so the September 2020 export may have included data analyzed using a different algorithm than the April 2021 export. “What was surprising was how different they were. This is probably the cleanest example that I have seen of this phenomenon.”
Furthermore, Olivia Walch, a sleep researcher who works with wearable and app data at the University of Michigan said:
Constantly changing algorithms makes it almost prohibitively difficult to use commercial wearables for sleep research. Sleep studies are already expensive. “Are you going to be able to strap four FitBits on someone, each running a different version of the software, and then compare them? Probably not.”
Companies have incentives to change their algorithms to make their products better. “They’re not super incentivized to tell us how they’re changing things.”
Looking at the Apple Watch heart rate data, Onnela concludes that “it was enough to make his team back away from plans to use the devices in studies. He thinks commercial wearables should only be used if raw data is available, or, at minimum, if researchers are able to get a heads-up when an algorithm is going to change.”
Read More:
