Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals Measurement

  • Xin Liu ,
  • Josh Fromm ,
  • Shwetak Patel ,
  • Daniel McDuff

Neural Information Processing Systems (NeurIPS) |

Download BibTex

Telehealth and remote health monitoring have become increasingly important during the SARS-CoV-2 pandemic and it is widely expected that this will have a lasting impact on healthcare practices. These tools can help reduce the risk of exposing patients and medical staff to infection, make healthcare services more accessible, and allow providers to see more patients. However, objective measurement of vital signs is challenging without direct contact with a patient. We present a video-based and on-device optical cardiopulmonary vital sign measurement approach. It leverages a novel multi-task temporal shift convolutional attention network (MTTS-CAN) and enables real-time cardiovascular and respiratory measurements on mobile platforms. We evaluate our system on an ARM CPU and achieve state-of-the-art accuracy while running at over 150 frames per second which enables real-time applications. Systematic experimentation on large benchmark datasets reveals that our approach leads to substantial (20%-50%) reductions in error and generalizes well across datasets.

Publication Downloads

MTTS-CAN

December 2, 2020

Accompanies the paper Multi-Task Temporal Shift Attention Networks for On-Device Contactless Vitals Measurement (NeurIPS 2020)

Download Data

Camera-based non-contact health sensing webinar

The SARS-CoV-2 (COVID-19) pandemic is transforming the face of healthcare around the world. One example of this transformation can be seen in the number of medical appointments held via teleconference, which has increased by more than an order of magnitude because of stay-at-home orders and greater burdens on healthcare systems. Experts suggest that particular attention should be given to cardiovascular and pulmonary protection during treatment for COVID-19. However, in most telehealth scenarios physicians lack access to objective measurements of a patient’s condition because of the inability to capture vital signs.

In this webinar, Microsoft Principal Researcher Daniel McDuff and University of Washington PhD student Xin Liu will present an overview of computer vision methods that leverage ordinary webcams to measure physiological signals (for example, peripheral blood flow, heart rate, respiration, and blood oxygenation) without contact with the body. Learn about some examples of state-of-the-art neural models that enable on-device sensing even in resource-constrained settings and understand some of the challenges and exciting research opportunities in this space. This webinar will frame the application of these methods in the context of telehealth; however, non-contact physiological measurement also holds promise in broader health, well-being, and affective computing settings.

Together, you’ll explore:

  • The optical and physiological basis for camera-based sensing.
  • State-of-the-art, on-device algorithms for fast, scalable, and privacy-preserving measurement.
  • Future challenges, open research questions, and opportunities in this space.

Resource list:

*This on-demand webinar features a previously recorded Q&A session and open captioning.