Health screening is a common sense approach applied to many medical conditions.  It should be easy:  create a sensible tool and then validate it.  In the medical literature, this usually involves calculating performance measures such as the sensitivity, specificity, and predictive value of the tool.  Calculating each of these is simple -- the risk resides in the interpretation of these numbers [1]. 

     There are two rules of thumb that will protect anyone from mis-interpreting screening tool statistics.  The first, I call the “Rule of 100”:  If the sensitivity and specificity add up to 100, the tool is performing at chance level.  The higher the sum (and closer to 200 maximum), the more accurate the tool is.  If the sum adds to less than 100, then the performance is paradoxical (a positive result means the disease is less likely).  In statistics parlance, when the sum of sensitivity and specificity is 100, then the likelihood ratio =1, which means the pre-test probability equals the post-test probability, which means the test result added no information.
   
     The second rule is that you cannot interpret a screening test about disease probability without a key ingredient:  the pre-test probability.  This completes the Bayesian triad of ingredients, along with sensitivity and specificity, needed to interpret any screening test.  The information gained by the screen can be viewed as the difference between the pre-test probability and the post-test probability (also called the “predictive value”).   The predictive value can appear high because the pre-test probability was high, or because the test is very accurate.  Even a random coin toss, where heads is positive for the disease, will have a “good positive predictive value” if applied to a population with a high probability of the disease.
   
​     Unfortunately, many publications miss one or both of these points.  We could choose many examples, but today, a new publication will serve to demonstrate the problem [2].  The authors report one version of their screening tool for sleep apnea as having 24% sensitivity and 68% specificity.  This violates the “rule of 100”, and in fact shows that the test is paradoxical.  The supposedly improved versions of their screen showed sums of 100-103.  The Rule of 100 tells us that the screen tool added virtually nothing – because it is performing near chance. 

     Bayes’ Theorem is all too often missing in medical testing.  There are so many examples - one cannot publish letters to the editor for each (though I did recently for a particularly problematic example regarding the contentious topic of home sleep testing [3]).  Yet there is hope:  armed with just the Rule of 100, anyone can quickly glance at test performance and surmise the accuracy.  And for those that pass this Rule, then the second rule will guide interpretation of the predictive value. 
 
References:
[1] Bianchi MT, Alexander BM (2006) Evidence based diagnosis: does the language reflect the theory?
BMJ. 333(7565):442-5.   (full text at:  https://www.ncbi.nlm.nih.gov/pubmed/16931846)
 
[2] Laratta et al (2016) Validity of administrative data for identification of obstructive sleep apnea.
J Sleep Res. doi: 10.1111/jsr.12465.
 
[3] Bianchi (2015) Evidence that home apnea testing does not follow AASM practice guidelines--or Bayes' theorem. J Clin Sleep Med. 15;11(2):189.  (full text at: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4298779/)
 
Contributor:  Matt Bianchi MD PhD

  We applaud the authors of the SAVE trial for orchestrating such a large prospective trial in this important population(1).  Such high profile negative findings represent an opportunity to recognize and address key points of uncertainty in the field of sleep apnea from diagnosis to management.   Several factors likely contributed to “noise” in this trial that reduced its power to detect physiological benefits from CPAP therapy.  From a diagnosis standpoint, the device was a 2-channel “Level-4” at-home kit, using oximetry to quantify OSA and the flow channel to exclude those with >50% periodic breathing – neither of which are standard methods, even with “Level 3” kits which at least include effort belts. At-home kits do not provide information about periodic limb movements, which have been linked to vascular morbidity(2).   At-home kits are not validated to detect central apnea, for which this population may be at increased risk (and no data on opiate use, also a risk for central apnea, was provided).  Further, the use of at home kits for diagnosis was not in line with American Academy of Sleep Medicine standards, which require a pre-test probability of 80% for AHI>15(3,4), which is not met even in this high risk population.  There is no data regarding insomnia, a common comorbidity with sleep apnea, or the use of hypnotics which carry morbidity risk.  Even though these diagnostic points of uncertainty may have distributed evenly in randomization, they nevertheless contribute unaccounted-for variance that could reduce power. 

   Uncertainties in the treatment phase are likely to have played an even more important role.  Using auto-PAP to choose a fixed pressure is clinically commonplace, under the dual assumptions that a) CPAP is always effective, and b) machine algorithm pressure choice is equivalent to polysomnographic titration.  The first assumption ignores the role of interacting obstructive and chemoreflex phenotypes in sleep apnea pathogenesis.  Although the second assumption is supported in carefully selected populations, we do not have independent confirmation in this vascular population.  Like at home diagnostic kits, at-home auto-titration is assumed but not validated clinically to detect central apnea.  In fact, detecting pauses of any kind may not be as accurate as often assumed, as we recently showed manual scoring of machine waveforms revealed significantly higher indices than automated scoring(5).  Arguably the most important treatment-related finding was that CPAP compliance averaged only 3.3 hrs/night, and only ~42% averaged more than 4 hrs/night.  Was this enough to expect risk mitigation at all, not to mention the high risk differential that the trial was powered upon?  Although subjective endpoints of mood and the Epworth Scale improved, one of the most common expected physiological endpoints, blood pressure reduction, was not observed.  Performing careful subset analysis using propensity score matching for those with >4 hrs/night usage was also negative for vascular benefit, but this subset is expected to be under-powered.  Finally, although not measured in any clinical trials to date, the “apnea burden” also contributes variance, as off-PAP sleep time may contain significant disease(6,7).  When sleep apnea recurs during off-PAP sleep time, and even a 4-hour per night participant might only be treated for 50% of sleep time, this too may contribute to incomplete treatment in the CPAP group.     

​   In summary, we caution against an oversimplified conclusion of a study based on Level 4 diagnosis, absence of PSG at diagnosis or therapeutic phases, poor average compliance, and other contributors to noise in the data.   We hope that these issues and ongoing discussions can provide context to such “negative” outcomes, and mitigate the risk of de-prioritizing sleep apnea therapy clinically or even from a coverage perspective. The field faces real challenges in determining what is the most effective therapy for simple obstructive disease as well as phenotypically complex sleep-disordered breathing, what is the minimum effective threshold of PAP use, and how precisely do we need to understand therapy effectiveness to maximize power to ask the outcomes questions.  We also face a challenge of how research findings, especially when negative or unexpected, are discussed and disseminated.  Can one reasonably ask why the response to this trial was not an indictment of at-home methods for managing sleep apnea?  Of course, the trial was not designed to answer the in-lab versus at-home question, but when strong concerns exist for the questions it was designed to answer, we are beholden to think critically about how we can do better for our patients, and how we as a field choose to discuss research findings with the very audience who depend on us to curate biomedical knowledge for their benefit. 

Contributors:  Matt Bianchi MD PhD (MGH), and Robert Thomas MD (BIDMC)

References:
1.            McEvoy RD, Antic NA, Heeley E, et al. CPAP for Prevention of Cardiovascular Events in Obstructive Sleep Apnea. The New England journal of medicine. Sep 8 2016;375(10):919-931.
2.            Walters AS, Rye DB. Review of the relationship of restless legs syndrome and periodic limb movements in sleep to hypertension, heart disease, and stroke. Sleep. May 2009;32(5):589-597.
3.            Collop NA, Tracy SL, Kapur V, et al. Obstructive sleep apnea devices for out-of-center (OOC) testing: technology evaluation. Journal of clinical sleep medicine : JCSM : official publication of the American Academy of Sleep Medicine. Oct 15 2011;7(5):531-548.
4.            Collop NA, Anderson WM, Boehlecke B, et al. Clinical guidelines for the use of unattended portable monitors in the diagnosis of obstructive sleep apnea in adult patients. Portable Monitoring Task Force of the American Academy of Sleep Medicine. Journal of clinical sleep medicine : JCSM : official publication of the American Academy of Sleep Medicine. Dec 15 2007;3(7):737-747.
5.            Reiter J, Zleik B, Bazalakova M, Mehta P, Thomas RJ. Residual Events during Use of CPAP: Prevalence, Predictors, and Detection Accuracy. Journal of clinical sleep medicine : JCSM : official publication of the American Academy of Sleep Medicine. 2016;12(8):1153-1158.
6.            Boyd SB, Walters AS. Effectiveness of Treatment Apnea-Hypopnea Index: A Mathematical Estimate of the True Apnea-Hypopnea Index in the Home Setting. J Oral Maxillofac Surg. Jul 5 2012.
7.            Bianchi MT, Alameddine Y, Mojica J. Apnea burden: efficacy versus effectiveness in patients using positive airway pressure. Sleep medicine. Dec 2014;15(12):1579-1581.