The Core of Big Data Is Predictive Power: Lessons from the Fitbit Litigation for Clinical Researchers

Fitbit has recently hit a major snag. The legal battle with rival Jawbone over trade secrets and patents has finally been settled, but the company now faces a class-action lawsuit regarding the accuracy of its heart rate data. Researchers from California State Polytechnic University, Pomona, have demonstrated that the margin of error can reach up to 20 beats per minute. This news has cast a chill on the previously optimistic outlook for biometric telemetry trials.

People have long been concerned about the usability of data provided by research data collection tools such as wearable devices, sensors, and monitors. Here, VCBeat (WeChat: vcbeat) has compiled for you insights fromCo-Founder of Litmus HealthDaphne Kis andSome Views of Dr. Samuel Volchenboum and the Current State of the Industry,I hope you have some correct insights into the above issues.

Speaking ofFitbit，Litmus Health isWe conducted a medical study involving 100 participants, which was approved by the Institutional Review Board (IRB). During the study, we utilized devices and companion mobile applications provided by Fitbit, thereby qualifying us to comment on this issue. The principal investigator of this study was Dr. David Rubin, a world-renowned gastroenterologist.

David’s objective is to understand whether and how sleep and activity influence the incidence of inflammatory bowel disease (IBD, including Crohn’s disease and ulcerative colitis). Our study is also sponsored by Takeda Pharmaceuticals U.S.A., Inc. The specific details of the study are as follows.

To date, many studies have focused on the intrinsic value and efficacy of medical devices themselves, but few researchers have actually integrated these devices into academic investigations of specific diseases. After reviewing extensive peer-reviewed literature on the use of sleep and activity monitoring devices, we selected Fitbit due to its ease of use and user-friendly application interface.

This lawsuit highlights the challenges we face. We must properly address errors in telemetry data, an unavoidable issue for any research data generated in hospitals, clinics, or home settings.

Many researchers and their sponsors believe that information for developing new treatments for challenging clinical diseases is ubiquitous.

“This summer, Dr. Sam Blackman from Juno Therapeutics stated at the breakfast session of the American Society of Clinical Oncology in Chicago: ‘We must leverage vast amounts of patient data. Quantifying health issues in non-clinical patients is crucial for developing faster and more effective treatments.’”

Although we strongly agree with this perspective, very few hardware devices have received FDA approval. Despite their popularity among consumers, scientific validation often presents challenges such as high difficulty, high costs, and operational complexities.

Data admissibility is also a concern, with data source issues being the most critical. For instance, currently available relatively simple and consistent timestamp and time-series protocols fail to meet the audit requirements stipulated in Title 21, Part 11 of the U.S. Code of Federal Regulations (CFR), and the claimed regulatory compliance falls far short of FDA standards. Moreover, these are not the only issues.

At a recent SXSW conference, the third-largest consumer device manufacturer expressed strong interest in collaborative research platforms. However, we were surprised to find that their fundamental understanding of data collection requirements in clinical trials was quite vague.

We should not blame this company or any other. If we want better equipment, we must more clearly communicate to manufacturers the specific dimensions and products we require. It is crucial that standards developed, promoted, and adopted by the research community enable data collection and storage, and facilitate the generation of reports that are straightforward and easy to use for all stakeholders.

Apple’s ResearchKit faces similar issues. Although Apple boasts a top-tier brand and public relations prowess, its Apple-centric technological worldview, closed-off approach, and compatibility issues between its research products and non-Apple platforms may severely hinder a substantial amount of research work.

At this point, the Institute of Electrical and Electronics Engineers (IEEE) has also put forward the same view. In the long run, the formulation of relevant standards is essential to improve the usability of these new data sources. Importantly, the FDA has issued a mandate requiring all electronic data to comply with standards established by organizations such as the Clinical Data Interchange Standards Consortium (CDISC). This constitutes official guidance from government authorities. As the deadline approaches, consumers have witnessed a series of encouraging initiatives.

Initially, we expected pharmaceutical companies to allocate most of their funding to late-stage data conversion rather than applying these standards across the entire data collection and storage process. The latter approach would greatly benefit efficient data collection firms such as Capsenta.

However, over time, the trickle-down effect will turn standards into a way of life, applied to every process and scenario. It will become increasingly easy to leverage the latest data for reliable innovation.

CDISC is a global non-profit organization dedicated to establishing and supporting clinical research data standards. While macro-level electronic data reporting standards have matured and gained widespread adoption, the domain of mobile devices remains underdeveloped. In fact, we have convened a series of working groups to address mobile data standards, and CDISC is highly interested in obtaining user-generated data from this summer and fall. Our critical task is to ensure that data are correctly exported; otherwise, innovation will be hindered, forcing continued reliance on outdated methods for data collection, storage, and transmission.

As we have come to appreciate, the world would be a better place if patients were not forced to frantically recall information from weeks prior and then haphazardly fill it out on clipboards in outpatient lobbies. Although new standards and data sources are not yet fully mature, they will undoubtedly lay the foundation for the future vision of clinical research.

Google’s upcoming clinical wearable device bodes well. It is likely that more specialized, lower-error, and highly modernized medical devices will soon hit the market.

Google X’s baseline research initiatives, along with Calico—the longevity program under Alphabet led by Art Levinson—have provided sufficient impetus for the internet giant to develop superior remote monitoring devices. The Google team also surveyed existing products on the market and identified numerous flaws. Consequently, Google’s decision to adopt a do-it-yourself (DIY) approach was entirely predictable.

But strictly speaking, Google’s clinical wearable devices are merely a means to achieve a greater goal. In the absence of data collection methods, patient data spanning weeks or months is largely based onTraditional clinical physician observation. ThisThis leads to limitations in the research findings of any major disease. For Google and all of us, this source of information has become outdated.

As we have long believed, quality of life is the greater goal and indeed the ultimate end of all things.

New therapies for various diseases are gradually shifting their focus from merely ensuring survival to improving patients’ quality of life. People may be more inclined to accept treatments that extend life by three months while enhancing quality of life, rather than those that prolong survival by six months but require patients to endure pain and suffering. For patients with chronic diseases, the ultimate goal is often to improve quality of life; however, for healthy individuals, quality of life also becomes increasingly important with age.

We believe that organizations or companies adept at measuring, interpreting, and ultimately establishing quality-of-life metrics will reap substantial economic benefits, driven by the growing emphasis on quality of life, increasing demand for personalized therapy, and the need for lean clinical trials. Undoubtedly, this motivation has guided Google’s business decisions in this field.

Reports suggest that Jawbone is developing a concept similar to Google’s, and its acquisition of Spectros confirms this. We believe that new competitors dedicated to clinical research will continue to enter the field.

Currently, a significant technological and experiential gap exists between consumer-centric modern medical devices and sensors, and traditional medical equipment that has long been battle-tested in healthcare delivery, with user experience being, at best, an afterthought. This delta will gradually close. Soon, we will no longer need to choose between usability and accuracy.

Until the devices themselves and the relevant standards are fully developed, we still face an imminent problem: the accuracy and precision of these devices remain relatively low, insufficient to support clinical decision-making.

Transferring data from proprietary devices to analytical environments remains a significant challenge. This issue is gradually being addressed on a device-by-device basis, while companies like Validic are betting on connectivity through parsing endpoints. As for device data, most manufacturers have their own methods for extracting information.

However, the most pressing need today lies in error modeling, which we affectionately refer to as MOE. No device manufacturer can address this challenge independently. Even data aggregators such as Validic are unable to develop such models, first because it is not necessary for them to do so, and second because they lack sufficient access to untransformed raw data.

In fact, MOE is the area most suited for data sharing and related open-source software. The specialized expertise required to transfer reliable device data into Electronic Data Capture (EDC) systems has, thus far, failed to deliver any tangible benefits to individual health IT startups or large corporate groups. Consequently, MOE has become a quintessential example of a sector experiencing broad-based growth.

Let us revisit wearable devices and other tracking technologies. Although volume, velocity, and variety are somewhat conventional terms, researchers have applied them to the data they generate. “Big data” has become a central concept in the research process. Researchers have collected vast amounts of information to facilitate the application of machine learning and other modeling approaches, thereby establishing sophisticated models for each type of data.

In the Google baseline study, by leveraging information on normal subjects collected by Google, it is possible to model errors for each data type, thereby enabling further analysis of the collected data. This constitutes the essence of MOE, and its public availability to clinicians and researchers is critical for the use of wearable devices and other equipment.

Even a straightforward comparison of Fitbit data with accelerometer and GPS data from iPhone and Android systems can help researchers establish ground truth and quantify margins of error. Consequently, heterogeneous data sources significantly outperform homogeneous ones, which is why we do not recommend adopting specific hardware solutions.

We should not allow the pursuit of perfect devices to hinder the progress of research. In fact, any expectation of high accuracy and high precision is unrealistic. Fitbit’s marketers may have been overly optimistic about their copywriting. The idea of using wrist-worn sensors equipped with only a few commercial-grade sensors to identify real-world conditions in completely unpredictable environments is absurd.

Researchers should not aim for perfection. Instead, there is an urgent need to clearly understand the margin of error associated with these devices.

We do not need to eliminate errors; rather, we need to predict them. Effective device error modeling will allow researchers to normalize data from different devices and experimental subjects.

It is evident that we are facing a shift in the way data is collected and used, which will have implications for our health and well-being. We are living in an era where the entire world has become a large-scale clinical trial, with patients contributing their data in a manner that respects privacy and provides highly granular control over access permissions and consent.

The Fitbit lawsuit does not indicate that these technologies and the data they generate are immature. On the contrary, as researchers, we recognize the need to scrutinize these technologies with the same rigor applied to other research tools.

As time goes on, these devices will continue to improve. At best, we are currently in an evolutionary stage akin to Apple’s early development. Nevertheless, the establishment of standards and error modeling helps address immediate challenges. In any case, remember that the way data is processed is far more important than the data itself. Patients reveal their health status through their daily behaviors.