The inter-sonographer reliability of carotid duplex ultrasound
-
Upload
holly-thomson -
Category
Documents
-
view
219 -
download
3
Transcript of The inter-sonographer reliability of carotid duplex ultrasound
INTRODUCTIONStroke is the third most common cause of death in Australia.1
The risk of stroke associated with carotid artery disease is high
and relates to the degree of narrowing of the internal carotid
artery (ICA).2 Current techniques for the assessment of carotid
artery disease include digital subtraction angiography (DSA),
computed tomography angiography (CTA), magnetic reson-
ance angiography (MRA) and carotid duplex ultrasound
(CDUS).3 CDUS is considered to be safe, relatively inexpensive
and repeatable. It is also reported to be an operator dependant
tool of investigation with varying levels of accuracy between
different sites.4–6
Carotid duplex ultrasound is based on the combination of
high resolution B mode imaging and spectral Doppler.
Variations in blood flow velocity within an artery can be
used to estimate the degree of stenosis caused by the pre-
sence of plaque within the vessel.7 These estimates require
the measurement of one or a combination of the following
parameters: peak systolic velocity (PSV), end diastolic velocity
(EDV) at the point of maximum stenosis or the peak systolic
velocity ratio (PSVR, peak velocity at the point of maximum
stenosis in the ICA divided by PSV in the unobstructed common
carotid artery (CCA) 1 cm or more below bifurcation).7,8
In 1991, the results of the North American Symptomatic
Carotid Endarterectomy Trial (NASCET)2 were published.
These results demonstrated a significant reduction in the num-
ber of strokes after carotid endarterectomy (CEA) compared
with the best medical treatment in symptomatic patients with
high grade (>70%) stenosis. Since then, numerous studies
have compared CDUS measurements of ICA stenosis to DSA
DiagnosticRadiology
The inter-sonographer reliability of carotid duplexultrasoundHolly Thomson,1 Anthony E Woods,2 John lannos3 and Michael Sage1
1Radiology Department, Repatriation General Hospital, Daw Park, 2School of Pharmacy and Medical Science, University of South
Australia, 3Division of Surgery, Repatriation General Hospital, Daw Park, South Australia, Australia
SUMMARY
Carotid duplex ultrasound (CDUS) is a non-invasive technique used to assess the severity of carotid artery stenosis. Ithas been shown to have good correlation with digital subtraction angiography (DSA) but has been criticised for itsvariability. One source of this is the variation in results between studies responsible for re-validating velocity criteria tomatch the established treatment thresholds of internal carotid artery (ICA) stenosis. The aim of this study was todevelop velocity criteria and determine the presence of inter-sonographer variation of CDUS when grading ICAstenosis in our department. Five sonographers measured the degree of ICA stenosis with CDUS in 33 patients who alsounderwent DSA. Receiver operator characteristic curve analysis was used to develop optimal velocity criteria for the50%, 70% and 90% ICA stenosis thresholds as a group and for each individual sonographer. A peak systolic velocityratio of ≥ 3.25 was shown to have the highest accuracy (91.5%) for predicting a 70% stenosis. A moderate value of κ (0.53 ± 0.027) was calculated if the optimum velocity criterion was employed for each sonographer. There was nosignificant variation between the ability of sonographers to grade ICA stenosis (P > 0.05) and an excellent ICC of 0.911was calculated. This study provides evidence to suggest that CDUS in our department is not an operator-dependanttest for the investigation of ICA stenosis.
Key words: carotid duplex ultrasound; digital subtraction angiography; internal carotid artery; inter-sonographervariation, stenosis.
H Thomson BAppSc (Hons); A Woods PhD; J Iannos BSc; M Sage FRANZCR
Correspondence: H Thomson, Repatriation General Hospital, Radiology Department, Daws Road, Daw Park, South Australia 5041, Australia.
Submitted 18 November 1999; resubmitted 19 April 2000; accepted 4 September 2000.
Australasian Radiology (2001) 45, 19–24
measurements,9 using NASCET DSA protocol, and developed
CDUS criteria to match the treatment threshold (Table 1).
Variation between studies may indicate potential inter-
sonographer and inter-equipment variation. Similarly in a study
by Moneta et al.,9 distribution plots correlating PSV and EDV
measures in the ICA to DSA displayed a wide distribution of
points, attributed to inter-sonographer variation as well as
patient and equipment variation. Additionally Meyer et al.10
showed that the flow velocity varied along the length of the
CCA, suggesting that unless the position of CCA measurement
is controlled, inconsistent PSVR measurements could result.
In contrast to these results, Fillinger et al.,11 while assessing
inter-equipment variation, noted that the sonographers contri-
buted little to observed variation. Two scanners of the same make
and model were used, but with different operators. By observing
linear regression lines correlating each scanner to DSA results, it
appeared that there was no significant difference between the
two scanners and hence sonographers. Finally Ranke et al.,12
who tested inter-equipment and inter-sonographer variation for
CDUS, demonstrated a good correlation between two sono-
graphers for both PSV (r = 0.92) and mean velocity ratios
(r = 0.94). However, no assessment of velocity criteria and
variation between grading categories was made.
The aim of this study was therefore to determine the
presence of inter-sonographer variation of CDUS when grading
ICA stenosis in our department, based on a velocity criterion
developed to grade ICA stenosis preoperatively.
METHODSEach subject underwent a total of five CDUS examinations
(each performed by a different sonographer) and a carotid
angiogram.
A total of 33 subjects, 30 males and three females, ranging
in age from 58 to 85 years (mean age 71.8), referred for a
carotid angiogram for the diagnosis and assessment of carotid
artery disease, participated in the study. Both left and right
internal carotid arteries were evaluated.
Carotid duplex ultrasoundThe ultrasound examinations were performed on an Acuson
Model 128XP/10 (Acuson Corporation, California, USA) using a
5 MHz linear transducer. Real time B mode imaging in both the
longitudinal and transverse directions was performed to gain
vessel information, such as location, size and course. Doppler
flow sampling was taken in the CCA and ICA, with any areas of
stenosis being thoroughly evaluated and the highest peak
velocity obtained. The Doppler angle (angle of incidence) was
restricted to ≤ 60° at all times.13 Each of the five sonographers
measured the PSV in the internal and common carotid arteries
and the PSVR was calculated. Each Sonographer was blind to
the results of any previous CDUS done.
Digital subtraction angiographyTo accommodate clinical demands, three DSA systems were
used: (i) a Fischer Imaging Xerox 1000 (Fischer, Colorado, USA),
(ii) a Toshiba DFP–50 A (Toshiba, Tochigi-ken, Japan); and
(iii) a Siemens Polytron TOP (Siemens, München, Germany).
Each of the right and left carotid arteries was selectively
catheterized, and a series of digital subtraction images were
obtained as 5–10 mL of Ultravist 370 (Sehering, Berlin,
Germany) was hand-injected. The DSA projections normally
included an anterio-posterior (AP), lateral and either a right
anterior oblique (RAO) or left anterior oblique (LAO) projection.
These projections were photographed on to hard copy. From
these images, one of two radiologists measured (in mm) the
degree of ICA stenosis according to the NASCET derived
protocol.2 The view showing the greatest degree of ICA
narrowing was chosen for analysis. For the first 20 subjects,
both radiologists interpreted the DSA images to obtain a
measure of inter-observer reliability.
Data analysisExpert statistical advice was sought and received from Dr
Brenton Dansie (School of Mathematics, University of South
Australia) for the study design and statistical interpretation.
All data were entered into a personal computer and the
receiver operator characteristic (ROC) curve data were
calculated and plots generated as described by Sumner.14 In
combination, this data allowed the selection of the velocity
criterion based on the optimum levels of sensitivity, specificity,
postitive predictive value (PPV), negative predictive value
(NPV) and accuracy predicted by the velocity thresholds. ROC
curve analysis was carried out for all five sonographers as a
group, as well as for each individual.
The agreement between CDUS and DSA was measured
with the kappa (k) statistic, a calculation of the difference
20 H THOMSON ET AL.
Table 1. Summary of findings for velocity criteria ≥ 70% North American Symptomatic Carotid Endarterectomy Trial stenosis
Study Parameter Sensitivity(%) Specificity (%) Positive predictive Negative predictive Accuracy(%)
value (%) value (%)
Carpenter et al.5 PSV > 210 cm/s 94 77 68 96 83
Moneta et al.6 PSVR > 4 91 87 76 96 88
Faught et al.4 PSV > 130 cm/s 81 98 89 96 95
EDV > 100 cm/s
between the observed agreement and chance. This calculation
was based upon the selected optimum velocity criteria.
Calculation of the inter-observer variability of the CDUS
operators was based on the optimum velocity criterion that was
selected through ROC curve analysis. An ANOVA was used to
determine if the observed differences among the sonographers
were significant. An intra-class correlation coefficient (ICC) was
calculated as a measure of the reliability of the sonographers.
An ICC of 1.0 is considered good reliability and an ICC of 0.0 is
considered poor reliability.
RESULTSOf the 33 subjects who participated in the study, one vessel was
excluded from the analysis due to recanalization, which was
only detected with further investigation using spiral CT. Ten
subjects had incomplete CDUS evaluations (less than the
required 5), and so 23 subjects (46 arteries) were used for the
analysis of inter-sonographer variation. All data (complete and
incomplete) was used to calculate the optimum velocity criteria
and the agreement between DSA and CDUS. A total of 303
measurements of the 65 vessels were made.
Optimum velocity criteria were determined for the 50%, 70%
and 90% thresholds of angiographic stenosis, defining the
0–49%, 50–69%, 70–89%, 90–99% stenosis bands. According
to DSA measurements these grades were present in 44.6%,
16.9%, 27.7%, 7.7% of cases, respectively. Selection of
velocity criteria was based on the PSVR that provided the best
combination of sensitivity and specificity (that is the highest
overall accuracy) (Table 2).
The distribution of the data (correlating DSA to CDUS) when
the selected optimum velocity criteria are employed to grade
the stenosis is given in Table 3 (this distribution displays the
spread of true and false positives and negatives). Thirty percent
of the stenoses were overestimated and 10% were under-
estimated by CDUS. There were two occlusions, both being
diagnosed accurately by each sonographer (one subject had
incomplete measurements).
Receiver operator characteristic curve analysis was also
performed for each individual sonographer in order to assess if
there was any variability of PSVR between sonographers. Table
4 displays the predicted levels of sensitivity, specificity, NPV,
PPV and accuracy for 50%, 70% and 90% ICA stenosis for
individual optimum thresholds of each stenographer. The pre-
dicted values of sensitivity, specificity, NPV, PPV and accuracy
for each individual sonographer when the group velocity criteria
are employed are given (Table 5).
If the optimum velocity criteria are employed by each
sonographer, then the 95% confidence limits of k are 0.53 ±
0.027. Perfect agreement occurred in 67% of cases. Statistical
analysis revealed there was no significant variation between
the ability of each sonographer to grade the degree of stenosis
(P > 0.05) (Tables 6 and 7). To follow on from this, a calculation
of the intra-class correlation coefficients was made to obtain a
measure of reliability between sonographers. An ICC (2,1) of
0.911 was computed.
It is important to acknowledge that the ICC was calculated
according to the categories the stenosis was graded into, and
not on the velocity measurements.
21INTER-SONOGRAPHER RELIABILITY
Table 2. Optimum velocity criteria
Degree of Optimum peak Sensitivity (%) Specificty (%) Positive predictive Negative predictive Accuracy (%)
internal carotid systolic velocity value (%) value (%)
artery stenosis (%) ratio
50 < 2.25 84.24 90.57 91.44 82.78 87.12
70 ≥ 3.25 93.04 90.96 86.30 95.53 91.47
90 ≥ 5 93.75 80.22 38.46 99.09 83.33
Table 3. Optimum carotid duplex ultrasound (CDUS) velocity criteria (peak systolic velocity ratio) versus carotid digital subtraction angiography
(DSA)
Angiographic percent stenosis
0–49% 50–69% 70–89% 90–99% 100% Total
< 2.25 125* 22 4 0 0 151
≥ 2.25 to < 3.25 8 16* 4 0 0 28
≥ 3.25 to < 5 5 5 34* 2 0 46
≥ 5 0 7 41 21* 0 69
No flow 0 0 0 0 9* 9
Total 138 50 83 23 9 303
*Denotes perfect agreement between DSA and CDUS.
DISCUSSIONIn this study, the optimum velocity criteria were selected on the
basis of the highest overall accuracy. This study aimed to
develop velocity criteria that were capable of preoperative
assessment of ICA stenosis, which meant that a false positive
was considered of equal consequence to a false negative.
Other studies that have also developed velocity criteria based
on the highest overall accuracy have achieved both different
optimum velocity criteria and different values of sensitivity,
specificity, PPV, NPV and accuracy (Table 1).4–6 While these
differences could be attributed to inter-sonographer variation,
they may also reflect the use of different equipment in studies,
as it has been shown that not only is there is a difference in
optimal velocity criteria between units,11 but also that the
relationship between Doppler velocity measurements and the
angiographic measurement of stenosis differs between
ultrasound equipment as reported by Howard et al.15 In the
same study, a comparison of 63 devices found that only 21%
were capable of achieving good sensitivity (> 80%) with a PPV
> 90%. This suggests that regardless of the selected optimal
velocity criteria for a particular system, it is possible for one to
be achieving results superior to another.
The main reason for inter-sonographer differences as a
cause for CDUS variation can be attributed to the scanning
technique and the many potential factors associated with the
velocity measurements. Doppler velocity measurements are
affected by the position of the Doppler cursor, angle of
incidence, and selection for the site along the length of stenosis
to be interrogated. These factors are difficult to control between
sonographers and it is probable that they are the major reason
for any variation between operators. Several studies have
utilized variation in velocity criteria between ultrasound units as
a basis for determining differences between these.11,15 Using a
similar approach, the present study assessed inter-sono-
grapher variation by determining if the velocity criteria varied
between sonographers. For example, the individual criterion for
sonographers differentiating 70% ICA stenosis ranged from a
PSVR ≥ 3.0 to ≥3.5 (as shown in Table 4). Although the
individual criteria did vary, the predicted levels of sensitivity,
specificity, PPV, NPV and accuracy did not differ significantly
between the individual group criterion for each sonographer.
For instance, the individual optimum criterion for Sonographer
22 H THOMSON ET AL.
Table 4. Sensitivity, Specificity, positive predictive value (PPV), negative predictive value (NPV) and accuracy for the individual optimal thresholds
of each sonographer
Sonographer Optimal threshold Sensitivity (%) Specificity (%) PPV (%) NPV(%) Accuracy (%)
50%
1 2.00 88.20 89.6 90.9 86.6 88.8
2 2.75 77.7 96.1 88.0 78.1 85.9
3 2.25 87.8 96.5 93.5 86.6 90.2
4 2.25 83.3 86.1 88.2 80.6 84.6
5 2.25 83.8 92.3 92.8 82.7 87.7
70%
1 3.25 95.8 89.5 85.2 97.2 88.0
2 3.50 90.5 94.4 90.4 94.4 92.8
3 3.00 95.6 92.1 88.0 97.2 92.8
4 3.25 96.0 92.5 88.8 97.3 93.8
5 3.50 94.5 97.1 95.5 97.1 96.5
90%
1 5 100 78.9 33.3 100 80.9
2 5 100 84.3 42.8 100 80.9
3 5 85.7 83.3 40.0 97.8 83.6
4 7 100 97.9 36.3 91.1 82.1
5 5 100 86.2 46.1 100 87.7
Table 5. Sensitivity, specificity, positive predictive value (PPV),
negative predictive value (NPV) and accuracy of individual
sonographers with group criteria
Sonographer Sensitivity Specificity PPV NPV Accuracy
(%) (%) (%) (%) (%)
50%
1 85.3 93.1 93.5 84.4 88.0
2 80.6 88.5 82.1 79.3 84.2
3 87.8 96.5 93.5 86.6 80.2
4 83.3 86.1 88.2 80.6 84.6
5 83.8 92.3 92.8 82.7 87.7
70%
1 95.8 89.5 85.2 97.2 88.0
2 90.5 86.1 79.2 93.3 87.8
3 94.7 92.1 90.9 92.3 91.8
4 96.0 92.5 88.8 97.3 93.8
5 95.5 91.4 87.5 96.9 92.8
90%
1 100 79.3 33.3 97.9 80.0
3 was a PSVR ≥ 3.0; this provided an accuracy of 92.8%
(sensitivity 95.6%, specificity 92.1%, PPV 88.0%, NPV 97.2%).
If sonographer 3 employed the group criterion, a PSVR ≥3.25,
an accuracy of 91.8% (sensitivity 94.7%, specificity 92.1%,
PPV 90.9% and NPV 92.3%) was predicted (as shown in Table
5). In the clinical situation it is practical that only a group
criterion is employed, from which no sonographer in the current
study achieved unacceptable results. Analyzing each sono-
grapher individually allows the identification of their boundaries
to which they can perform.
The ROC curve analysis of each sonographer indicates that
there are differences between their optimum threshold, but the
current study showed that there was no significant difference
between sonographers in their grading of stenosis (P > 0.05). It
is essential to recognize that this analysis was not carried out
on the measurement of PSVR by the sonographer, but on the
categories the stenosis was graded into (based on the selected
optimum velocity criterion 0–49%, 50–69%, 70–89%, 90–99%
and occluded). This is unlike Ranke et al.,12 who showed good
correlation between two sonographers in measurements of
both the mean systolic ratio (r = 0.94) and intra-stenotic PSV
(r = 0.92). The category into which the stenosis is graded will
determine the treatment and consequently any variation in the
grading of stenosis by sonographers is particularly important.
The current study calculated an ICC (2,1) of 0.911, indicating
strong reliability between sonographers.
The sonographers who participated in the study ranged in
experience (two were experienced, two were moderately
experienced and one was a junior sonographer). However, each
of them was trained in the same setting. It would be expected,
therefore, that their scanning techniques were relatively similar.
Whether results similar to this study would be achieved with
sonographers from different centres who have trained in a
variety of ways remains to be determined.
In the current study the 95% confidence limits of k were
0.53 ± 0.027 when all five grading categories were being
evaluated. Perfect agreement between CDUS and DSA
occurred in 67% of cases. Faught et al.4 reported a k value of
0.68 ± 0.02 and perfect correlation in 82% of cases. However,
Faught et al.4 did not assess 90% angiographic stenosis, which
was the main cause for a low k value in our study. Recalculating
the present values and ommitting 90% stenosis gives a k
value of 0.71 ± 0.12, closer to Faught et al. In conclusion, this
study has shown that there is no significant variation between
the ability of sonographers to grade the degree of ICA stenosis.
Only moderate agreement between CDUS and DSA was
achieved. However, the predicted levels of sensitivity,
specificity, NPV, PPV and accuracy of the developed criteria
were acceptable, but did differ from validated velocity criteria
from other centres. These results therefore suggest that while
CDUS may not be an operator dependant measurement, each
centre should be responsible for validating its own velocity
criteria and maintain continuous evaluation of it.
ACKNOWLEDGEMENTSWe would like to acknowledge and thank the following people
for their help and guidance on this research project: R Morcom,
C Lott, J Bestt, C Kapsis, S Jenke, M Gourlay, all staff at the
Repatriation General Hospital and B Dansie.
REFERENCES1. National Stroke Foundation (Stroke Australia task force). National
Stroke Strategy. National Stroke Foundation, Melbourne,
Australia, 1997.
2. NASCET collaborators. Beneficial effect of carotid endarterectomy
in symptomatic patients with high grade stenosis. N Engl J Med
1991; 325: 445–53.
3. Polak JF, Bajakian RL, O’Leary DH, Anderson MR, Donaldson
MC, Jolesz FA. Detection of internal carotid artery stenosis:
23INTER-SONOGRAPHER RELIABILITY
Table 6. Single factor ANOVA
Source of SS df MS F ratio P value F critical
variance
Between subjects 349.39 45 7.746 52.52 1.63 1.43
Within Subjects 27.20 184 0.147
Total 376.59 229
Intra-class correlation coefficient (1,1) = 0.911.
Table 7. Two factor ANOVA without replication
Source of SS df MS F ratio P value F critical
variance
Between subjects 349.39 45 7.760 52.60 2.96 1.44
Between Sonographers 0.6340 4 0.158 1.075 0.37 2.42
Error 26.560 180 0.147
Total 376.59
Intra-class correlation coefficient (2,1) = 0.911; (3,1) = 0.912.
comparison of MR angiography, color Doppler sonography, and
arteriography. Radiology 1992; 182: 35–40.
4. Faught WE, Mattos MA, Bernmele PS et al. Color-flow duplex
scanning of carotid arteries: New velocity criteria based on
receiver operator characteristic curve analysis for threshold
stenoses used in the symptomatic and asymptomatic carotid trials.
J. Vasc Surg 1994; 19: 818–28.
5. Carpenter JP, Lexa FJ, Davis JT. Determination of duplex Doppler
ultrasound criteria appropriate to the North American symptomatic
endarterectomy trial. Stroke 1996; 27: 695–9.
6. Moneta GL, Edwards JM, Chitwood RW et al. Correlation of North
American symptomatic carotid endarterectomy trial; angiographic
definition of 70–99% internal carotid artery stenosis with duplex
scanning. J Vasc Surg 1993; 17: 152–9.
7. Faccenda F, Usui Y, Spencer M. Doppler Measurement of the
pressure drop caused by arterial stenosis: An experimental study:
A case report. Angiology 1985; 36: 899–905.
8. Bluth EI, Merritt CR. Carotid arteries and vertebral arteries. In:
Merritt CR (ed.) Doppler color imaging. Churchill Livingstone, New
York, 1992; 61–96.
9. Moneta GL, Edwards JM, Papanicolaou G et al. Screening for
asymptornatic internal carotid artery stenosis: Duplex criteria for
discriminating 60–99% stenosis. J Vasc Surg 1995; 21: 989–94.
10. Meyer JI, Khalil RM, Obuchowski NA, Baus LK. Common carotid
artery: Variability of Doppler ultrasound measurements. Radiology
1997; 204: 339–41.
11. Fillinger MF, Baker RJ, Zwolak RM et al. Carotid duplex criteria for
60% or greater angiographic stenosis: Variation according to
equipment. J Vasc Surg 1996; 24: 856–64.
12. Ranke C, Trappe HJ. Blood flow velocity measurements for carotid
stenosis estimation: Interobserver variations and interequipment
variability. VASA 1997; 26: 210–14.
13. Beasly MG, Balu JN, Gosling RG. Changes in internal carotid
artery flow velocities with cerebral vasodilation and constriction.
Stroke 1979; 19: 331–5.
14. Sumner DS. Evaluation of noninvasive testing procedures data
analysis interpretation. In: Bernstein (ed.) Noninvasive Diagnostic
Techniques in Vascular Disease. CV Mosby, St Louis, 1985;
861–89.
15. Howard G, Baker YM, Chambless LE, Howard VJ, Jones AM,
Toole JF. An approach for the use of Doppler ultrasound as a
screening tool for hemodynamically significant stenosis (despite
heterogenity of Doppler performance). A multi center experience.
Asymptomatic carotid atherosclerosis study investigators. Stroke
1996; 27: 1951–7.
24 H THOMSON ET AL.