The accuracy of using a wound care specialty clinicdatabase to study diabetic neuropathic foot ulcers

JONATHAN KANTOR, MAa; DAVID J. MARGOLIS, MD, MSCEa,b

Few epidemiologic studies have examined the effect of clinical risk factors on the probability that a patient with achronic wound will heal or develop another wound. Curative Health Services maintains one of the few databasesthat contain detailed patient record information on patients with chronic wounds. The purpose of this study was toevaluate the reliability and validity of using this database to study individuals with diabetic neuropathic foot ulcers.154 patient medical records were randomly selected from the database and abstracted using a standardized question-naire and protocol. We assessed three key variables: diagnosis of diabetic neuropathic foot ulcer, whether the patienthealed, and if the patient received an autologous product called platelet releasate. These variables in the databasevery accurately agreed with the information in the patient medical records, with positive predictive values of 98%(95% confidence interval [0.89, 0.99]), 93% (95% confidence interval [0.68, 0.99]), and 100%, respectively. We haveshown that, with respect to these three variables, the database is very accurate when compared to the medicalrecord. It therefore represents a valuable tool with which to study patients with diabetic insensate foot ulcers. (WOUNDREP REG 2000;8:169–173)

Epidemiologic study designs are often the only way to CHS Curative Health Servicesunderstand the burden of a disease on a population andto determine the effect that an exposure or risk factormay have on a disease. Chronic wounds, such as venousleg ulcers, diabetic foot ulcers, and pressure ulcers, resultin diminished quality of life, increased morbidity and to study large populations of individuals with chronic

wounds.mortality, and frequent hospitalizations and visits tohealth care providers.1–7 Unfortunately, very few epidemi- Curative Health Services (CHS) has been directly

involved in the care of individuals with chronic woundsologic studies have investigated the effect of clinical riskfactors on the probability that a patient with a chronic since 1988.8–10 Their involvement has been centered on

marketing, health care provider education, treatment al-wound will heal or develop another wound. These studydesigns require large samples of patients with chronic gorithms, and dispensing an autologous product that they

believe improves the probability that a chronic woundwounds, and very few databases exist that can be usedwill heal. CHS is associated with more than 130 woundcare centers in the United States. As part of this associa-

From the Departments of Biostatistics and Epidemiologya

tion, they have maintained an administrative databaseand Dermatology,b University of Pennsylvaniathat contains information on every patient seen in aSchool of Medicine, Philadelphia, Pennsylvania.wound care center since 1988. This database is newlyReprint requests: David J. Margolis, MD MSCE, Depart-

ments of Dermatology and Biostatistics, University available for investigative use. Administrative databasesof Pennsylvania School of Medicine, Room 815 primarily contain information necessary for billing. AtBlockley Hall; 423 Guardian Drive, Philadelphia, times, these databases also contain useful patient recordPA 19104. Fax: (215) 573-5315; Email: dmargoli@c-

information. The CHS database does contain some pa-ceb.med.upenn.edu.tient record information such as patient age, wound size,Copyright q 2000 by The Wound Healing Society.

ISSN: 1067-1927 $15.00 ` 0 duration, depth, wound-related hospitalizations, wound

169

WOUND REPAIR AND REGENERATIONMAY–JUNE 2000170 KANTOR AND MARGOLIS

treatments, wound diagnostic information, wound diag- tar surface of the foot or heel of an individual with diabe-tes. This individual must also have had adequate arterialnosis, and wound-associated medical illnesses.

In order to use the CHS database for high quality blood flow as defined by a transcutaneous oxygen ten-sion of greater than 30 mm Hg, an ankle brachial indexepidemiological studies, an assessment of the accuracy

of data in the database is essential. The purpose of this of > 0.90, or the presence of dorsal pedal pulse.6,11 Inthe database, patients were defined as having a diabeticstudy was to evaluate the reliability and validity of using

this database to study individuals with diabetic neuro- insensate foot ulcer if a diagnostic code was used thatwas consistent with a diabetic foot, and the patient hadpathic foot ulcers. Our current investigation includes an

assessment of three key variables: diagnosis of diabetic no codes consistent with a wound due to another etiologysuch as poor arterial blood flow to the lower extremity,neuropathic foot ulcer, whether the patient healed, and if

the patient received an autologous product called platelet venous leg ulcer, or a pressure ulcer.A wound was considered healed if the medical re-releasate.

cord noted that the wound was healed and that a bandagewas no longer used. If a bandage was used only to protect

MATERIALS AND METHODS a ‘‘healed’’ wound, the wound was only considered healedif the medical record noted the absence of discharge orAdministrative and treatment related information hasdrainage on the bandage.12 In the database, a wound wasbeen collected on all patients seen in a wound care centerconsidered healed if it was coded as 100% epithelializedassociated with CHS since 1988. This database is usedrequiring a dressing only for protection or as 100% epithe-by CHS to evaluate administrative compliance betweenlialized and no longer requiring a dressing.the local wound center and CHS. It is also used for wound

In the medical record, use of autologous plateletcare related quality assessment. We evaluated informa-releasate was defined by documentation that blood wastion that was entered in the database between Septemberobtained from the patient for this purpose, that the autol-of 1988 and July 1997. The purpose of this study was toogous product was manufactured, and that at least onedetermine the accuracy of the database diagnosis whendose was used. In the database, an appropriate treatmentcompared to the patient’s medical record.code and at least one entry that the product was used,Medical records were abstracted at wound care cen-confirmed the use of autologous product.ters, which were selected based on the proximity of the

wound care center to the investigators. Patient medicalStatisticsrecords were randomly selected using a computer basedAccuracy was determined by comparing information inpseudo-random number generator. The population ofthe database to the clinical ‘‘gold standard.’’ The clinicalrandomized patients in the database who were from all‘‘gold standard’’ was based on information obtained bypatients at the close-proximity wound care centers andthe medical record review. Accuracy was estimated byhad criteria developed by the investigators that could beevaluations of validity and reliability. Validity is reflectedused to identify that a patient had or did not have aby the sensitivity, specificity, and positive predictivediabetic insensate foot ulcer. Medical records were ab-value.13 For example, in our study the sensitivity, or truestracted using a standardized questionnaire and protocol.positive rate, is the probability that a healed wound,Each medical record was abstracted by one of two inves-according to the medical record, was rated as healed bytigators. Unique patient identifiers were not recorded onthe database; the specificity, or true negative rate, wasthe standardized questionnaire.the probability that an unhealed wound, according to theThe Institutional Review Board of the University ofmedical record, was rated by the database as unhealed;Pennsylvania approved this study.and the positive predictive value was the probability thata wound healed, according to the database, was healedVariables assessed and clinical definitions

The primary objective of this study was to determine according to the medical record review. Reliability wasassessed using a Kappa statistic to compare variables ofthe accuracy of three variables: patient diagnosis (i.e.,

diabetic insensate foot ulcer), whether they healed interest for patients based on the database and chartreview.14within 20 weeks of care, and whether they received treat-

ment with an autologous product called platelet relea- The full data set was obtained electronically fromCHS and warehoused in a relational database (FoxPro,sate. In all cases the patient’s first course of treatment

in a wound care center was used in our evaluation. Microsoft, Redmond, WA). Analyses were preformed ei-ther using Stata 6.0 (Stata Corporation, College Station,In the patient medical record, we defined a diabetic

neuropathic foot ulcer as an ulcer occurring on the plan- TX) or with the aid of a hand calculator.

WOUND REPAIR AND REGENERATIONVOL. 8, NO. 3 KANTOR AND MARGOLIS 171

Table 1. Estimates of agreement between the database and wounds belongs to CHS, which is newly available formedical record on whether a patient has a diabetic neuropathic

investigative use. Before any database can be used forulcerhigh quality epidemiologic studies, the accuracy of the

Estimate (95% confidenceinformation in the database should be evaluated. In thisAccuracy estimate interval*)

study, we evaluated variables in the CHS database thatPercent correct 93.4 (88.3, 96.8)

would be essential for epidemiologic studies on diabeticKappa 0.86 (0.79, 0.91)neuropathic foot ulcers. The accuracy of this databaseSensitivity 0.84 (0.72, 0.93)as compared to the medical record with respect to ascer-Specificity 0.99 (0.94, 0.99)taining who has a diabetic insensate foot ulcer, who re-Positive predictive value 0.98 (0.89, 0.99)

Negative predictive value 0.91 (0.84, 0.96) ceived autologous platelet releasate, and who healedwithin 20 weeks of care is very high. Therefore, we have*All 95% confidence intervals were calculated using exact methods.

shown that it is possible to use these variables from theCHS database for high quality epidemiologic studies.

Table 2. Estimates of agreement between the database and Traditionally, diagnostic tests (or, in this study, themedical record on whether a patient healed after 20 weeks of

use of a database to ascertain information about an indi-carevidual) are evaluated by reports of sensitivity and speci-

Estimate (95% confidenceficity. This is useful if the objective is to assess theAccuracy estimate interval*)

proportion of patients with and without disease who arePercent correct 0.90 (0.78, 0.97)correctly classified by a test. However, in the context ofKappa 0.78 (0.64, 0.88)this study, the appropriate approach to estimate accuracySensitivity 0.78 (0.53, 0.94)is predictive value. This is true since we are interested inSpecificity 0.97 (0.84, 0.99)

establishing the accuracy of a disease state (i.e., diabeticPositive predictive value 0.93 (0.68, 0.99)

Negative predictive value 0.89 (0.73, 0.97) insensate foot ulcer and healed) or exposure (i.e., autolo-gous releasate) given the presence of a particular data-*All 95% confidence intervals were calculated using exact methods.

base code and the probability of the absence of diseasegiven the absence of that database code. This approach

RESULTS has been used by other investigators;15,16 Since only sub-jects with database codes of interest can be analyzedFour of six centers agreed to participate in this studyin a validation study, information bias might appear tofrom which we requested 175 patient charts. 154 (88%)hamper these estimates. However, the target populationmedical records were successfully reviewed. Of the 21for an epidemiologic study is those individuals that canmedical records that we were unable to obtain for review,be identified, and therefore, this does not represent a10 were obtained for the correct patients but did notproblem. Secondly, verification bias, a type of selectioninclude the desired date, and the remaining 11 were un-bias, can be a significant problem when estimating theavailable for review. Prevalence of diabetic insensatesensitivity and specificity of a diagnostic test if the sub-foot ulcers in the sample was 32.5%.ject’s selection is dependent on test positivity (e.g., diag-The database definition of a diabetic neuropathicnostic code recognition of disease).17–21 When using afoot ulcer very nicely agreed with the medical recordpredictive value approach, this is not a source of error(Table 1) and the reliability was very good across all fouras long as the subjects selected for the validation studycenters (Kappa ranging from 0.79 to 1.00). As would beare randomly selected from the main database, as wasexpected, since this is an administrative database, alldone in this study.17,21(100%) of the 50 patients with diabetic insensate foot

More simply stated, an important aspect of using aulcers who, by database criteria, received autologouslarge database to study a disease is confidently knowingplatelet releasate received it by medical record criteria.that an individual that is being studied actually has theFinally, 93% of patients that were healed as defined bydisease of interest. Positive predictive value is the bestdatabase criteria were in fact healed according to theway to assess the accuracy of correctly ascertaining sub-medical record (Table 2).jects to study from a database. For this study, with re-spect to patient diagnosis, the positive predictive value is

DISCUSSION the probability that a person truly has a diabetic insensatefoot ulcer when the database indicates that they have aVery few databases exist that contain useful clinical infor-

mation on patients with chronic wounds. One of the diabetic insensate foot ulcer. We have shown that if thedatabase ascertained that a person has a diabetic insen-largest that exclusively contains information on chronic

WOUND REPAIR AND REGENERATIONMAY–JUNE 2000172 KANTOR AND MARGOLIS

sate foot ulcer, then the medical record agreed 98% of have shown that, with respect to these three variables,the database is very accurate when compared to thethe time. Similarly, the database performed exceptionally

when ascertaining whether an individual received plate- medical record. Using the CHS database, which is newlyavailable for investigative use, in lieu of the primary medi-let releasate, and if they healed.

Few studies have evaluated the validity of databases cal record should therefore result in minimal error, andthis database represents a very useful tool to study pa-related to skin conditions. Our study’s results appear to

be as good as, or superior to other reports of the valida- tients with diabetic insensate foot ulcers.tion of large databases. For example, one study of ery-thema multiforme (EM) utilizing Medicaid data revealed

ACKNOWLEDGMENTSthat 94.5% of patients coded as having EM actually hadsome sort of skin disease, and that 85.2% actually had This study was supported in part by an unconditionalEM.22 Similarly, a study of cardiovascular health using grant from Curative Health Services to the Trustees ofMedicare eligibility lists revealed that 76.6% of men and the University of Pennsylvania and grant AG 00715–0165.1% of women that reported a history of a myocardial from the National Institutes of Health. Neither grantinginfarction actually had a myocardial infarction docu- agency was involved with the design or analysis of thismented in their medical record.23 A study examining the study. We thank Drs Brian L. Strom, MD MPH and Jessevalidity of self-reports for fractures in postmenopausal A. Berlin, ScD for their assistance in the design andwomen found that only 84% of reported fractures repre- analysis of this study. Finally, we thank Ms. Sandy Masiaksented true fractures, yet concluded that the accuracy for her secretarial and editorial assistance in preparingof patient reporting is a reasonable method of obtaining this manuscript for submission and George Davis frominformation about prior fractures.24 An excellent review Curative Health Services for his technical guidance onof the use of Medicaid data in epidemiologic studies has issues related to the functioning of the database. Thealso highlighted some of the limitations of using Medicaid authors of this study have no financial interest in Curativedata due to problems with validity.25 Our results appear Health Services. Curative Health Services, as perto be more convincing than those of previous validation agreement with the Trustees of the University of Pennsyl-studies. This is likely related to the fact that while the vania, had no editorial rights with respect to the publica-CHS database is primarily administrative, it is also used tion of this manuscript.for quality assurance. Therefore, greater effort is likelyexpended by both CHS and the individual wound carecenters to maintain accurate records, which in turn in- REFERENCEScreases the validity of their data and its usefulness for 1. Phillips TJ, Dover JS. Leg ulcers. J Am Acad Dermatol 1991;25:

965–87.clinical research.2. Phillips T, Stanton B, Provan A, Lew R. A study of the impact ofThere are several limitations to this study. Database

leg ulcers on quality of life: Financial, social, and psychologicvalidation studies are manpower intensive. While the re- implications. J Am Acad Dermatol 1994;31:49–53.sults of this type of study are critical to understanding 3. Ruckley CV. Socioeconomic impact of chronic venous insuffi-

ciency and leg ulcers. Angiology 1997;48:67–9.the limitations of a database, these studies are seldom4. Bergstrom N, Allman RM, Carlson CE, Eaglstein W, Frantz RA,the primary reason for undertaking an epidemiologic

Garber SL, Gosnell D, Jackson BS. Pressure ulcers in adults:study. In order to minimize costs, charts were randomly prediction and prevention (1st edition). Rockville, MD: Agency

for Health Care Policy and Research, 1992: 1–63.selected from four wound care centers within a 70-mile5. Lindholm C, Bjellerup M, Christensen OB, Zederfeldt B. Qualityradius of the investigators. It is therefore, possible that

of life in chronic leg ulcer patients. An assessment according toour results only represent the accuracy of database infor- the Nottingham Health Profile. Acta Dermato-Venereol 1993;73:

440–3.mation from these four centers. While this is possible, it is6. American Diabetes Association. Consensus development confer-not very likely. CHS uses this database for administrative

ence on diabetic wound care. Diabetes Care 1999;22:1354–60.purposes, and they have a fiscal interest in insuring that 7. Centers for Disease Control and Prevention. Diabetes—A se-the database is properly maintained by all of their cen- rious public health problem. 1–5. Atlanta: U.S. Department of

Health and Human Services, 1998. www.cdc.gov/diabetes/pubs/ters.glance.htmIn conclusion, critical information in a database

8. Glover JL, Weingarten MS, Buchbinder DS, Poucher RL, Deitrickshould be evaluated before it is used in an epidemiologic GA, Fylling CP. A 4-year outcome-based retrospective study of

wound healing and limb salvage in patients with chronic wounds.study. Databases are not frequently validated due to ex-Adv Wound Care 1997;10:33–8.pense and time. In this study, we describe the accuracy

9. Doucette MM, Fylling C, Knighton DR. Amputation preventionof three variables found in the CHS database that would in a high-risk population through comprehensive wound-healing

protocol. Arch Phys Med Rehab 1989;70:780–5.be critical to studying diabetic insensate foot ulcers. We

WOUND REPAIR AND REGENERATIONVOL. 8, NO. 3 KANTOR AND MARGOLIS 173

10. Knighton DR, Fiegel VD, Doucette MM, Fylling CP, Cerra F. The use 19. Ransohoff DF, Feinstein AR. Problems of spectrum and bias inevaluating the efficacy of diagnostic tests. N Engl J Med 1978;of topically applied platelet growth factors in chronic nonhealing

wounds: a review. Wounds 1989;1:71–8. 299:926–30.11. Steed DL. Foundations of good ulcer care. Am J Surg 1998;176 20. Begg CB, Greenes RA. Assessment of diagnostic tests when dis-

(Suppl. 2A):20S–5S. ease verification is subject to selection bias. Biometrics 1983;39:12. Lazarus GS, Cooper DM, Knighton DR, Margolis DJ, Percoraro 207–15.

RE, Rodeheaver G, Robson MC. Definitions and guideline for 21. Choi BCK. Sensitivity and specificity of a single diagnostic testassessment of wounds and evaluation of healing. Arch Dermatol in the presence of work-up bias. J Clin Epidemiol 1992;45:581–6.1994;130:489–93. 22. Strom BL, Carson JL, Halpern AC, Schinnar R, Snyder ES, Shaw

13. Hennekens CH, Buring JE. Epidemiology in medicine (1st edition). M, Tilson HH, Joseph M, Dai WS, Chen D, Stern RS, Bergman U,Boston: Little, Brown, 1987: 1–383. Stolley PD. A population-based study of Stevens–Johnson syn-

14. Fleiss JL. Statistical methods for rates and proportions (2nd edi- drome. Incidence and antecedent drug exposures. Arch Dermatoltion). New York: John Wiley and Sons, 1981: 211–36. 1991;127:831–8.

15. Strom BL, Carson JL, Halperin AC, Schinnar R, Snyder ES, Stolley 23. Psaty BM, Kuller LH, Bild D, Burke GL, Kittner SJ, MittelmarkPD. Using a claims database to investigate drug-induced M, Price TR, Rautaharju PM, Robbins J. Methods of assessingStevens–Johnson syndrome. Stat Med 1991;10:565–76. prevalent cardiovascular disease in the Cardiovascular Health

16. Raiford DS, Gutthann SP, Rodriguez LAG. Positive predictive value Study. Ann Epidemiol 1995;5:270–7.of ICD-9 codes in the identifcation of cases of complicated peptic 24. Honkanen K, Honkanen R, Heikkinen L, Kroger H, Saarikoski S.ulcer disease in Saskatchewan hospital automated database. Epi- Validity of self-reports of fractures in perimenopausal women. Amdemiology 1996;7:101–4. J Epidemiol 1999;150:511–6.

17. Zhou XH. Verification bias on positive and negative predictive 25. Bright RA, Avorn J, Everitt DE. Medicaid data as a resource forvalues. Stat Med 1994;13:1737–45. epidemiologic studies. Strengths and limitations. J Clin Epidemiol

18. Bates AS, Margolis PA, Evans AT. Verification bias in pediatric 1989;42:937–45.studies evaluating diagnostic tests. J Pediatr 1993;122:585–90.