INTRODUCTIONStroke is the third most common cause of death in Australia.1

The risk of stroke associated with carotid artery disease is high

and relates to the degree of narrowing of the internal carotid

artery (ICA).2 Current techniques for the assessment of carotid

artery disease include digital subtraction angiography (DSA),

computed tomography angiography (CTA), magnetic reson-

ance angiography (MRA) and carotid duplex ultrasound

(CDUS).3 CDUS is considered to be safe, relatively inexpensive

and repeatable. It is also reported to be an operator dependant

tool of investigation with varying levels of accuracy between

different sites.4–6

Carotid duplex ultrasound is based on the combination of

high resolution B mode imaging and spectral Doppler.

Variations in blood flow velocity within an artery can be

used to estimate the degree of stenosis caused by the pre-

sence of plaque within the vessel.7 These estimates require

the measurement of one or a combination of the following

parameters: peak systolic velocity (PSV), end diastolic velocity

(EDV) at the point of maximum stenosis or the peak systolic

velocity ratio (PSVR, peak velocity at the point of maximum

stenosis in the ICA divided by PSV in the unobstructed common

carotid artery (CCA) 1 cm or more below bifurcation).7,8

In 1991, the results of the North American Symptomatic

Carotid Endarterectomy Trial (NASCET)2 were published.

These results demonstrated a significant reduction in the num-

ber of strokes after carotid endarterectomy (CEA) compared

with the best medical treatment in symptomatic patients with

high grade (>70%) stenosis. Since then, numerous studies

have compared CDUS measurements of ICA stenosis to DSA

DiagnosticRadiology

The inter-sonographer reliability of carotid duplexultrasoundHolly Thomson,1 Anthony E Woods,2 John lannos3 and Michael Sage1

1Radiology Department, Repatriation General Hospital, Daw Park, 2School of Pharmacy and Medical Science, University of South

Australia, 3Division of Surgery, Repatriation General Hospital, Daw Park, South Australia, Australia

SUMMARY

Carotid duplex ultrasound (CDUS) is a non-invasive technique used to assess the severity of carotid artery stenosis. Ithas been shown to have good correlation with digital subtraction angiography (DSA) but has been criticised for itsvariability. One source of this is the variation in results between studies responsible for re-validating velocity criteria tomatch the established treatment thresholds of internal carotid artery (ICA) stenosis. The aim of this study was todevelop velocity criteria and determine the presence of inter-sonographer variation of CDUS when grading ICAstenosis in our department. Five sonographers measured the degree of ICA stenosis with CDUS in 33 patients who alsounderwent DSA. Receiver operator characteristic curve analysis was used to develop optimal velocity criteria for the50%, 70% and 90% ICA stenosis thresholds as a group and for each individual sonographer. A peak systolic velocityratio of ≥ 3.25 was shown to have the highest accuracy (91.5%) for predicting a 70% stenosis. A moderate value of κ (0.53 ± 0.027) was calculated if the optimum velocity criterion was employed for each sonographer. There was nosignificant variation between the ability of sonographers to grade ICA stenosis (P > 0.05) and an excellent ICC of 0.911was calculated. This study provides evidence to suggest that CDUS in our department is not an operator-dependanttest for the investigation of ICA stenosis.

Key words: carotid duplex ultrasound; digital subtraction angiography; internal carotid artery; inter-sonographervariation, stenosis.

H Thomson BAppSc (Hons); A Woods PhD; J Iannos BSc; M Sage FRANZCR

Correspondence: H Thomson, Repatriation General Hospital, Radiology Department, Daws Road, Daw Park, South Australia 5041, Australia.

Submitted 18 November 1999; resubmitted 19 April 2000; accepted 4 September 2000.

Australasian Radiology (2001) 45, 19–24

measurements,9 using NASCET DSA protocol, and developed

CDUS criteria to match the treatment threshold (Table 1).

Variation between studies may indicate potential inter-

sonographer and inter-equipment variation. Similarly in a study

by Moneta et al.,9 distribution plots correlating PSV and EDV

measures in the ICA to DSA displayed a wide distribution of

points, attributed to inter-sonographer variation as well as

patient and equipment variation. Additionally Meyer et al.10

showed that the flow velocity varied along the length of the

CCA, suggesting that unless the position of CCA measurement

is controlled, inconsistent PSVR measurements could result.

In contrast to these results, Fillinger et al.,11 while assessing

inter-equipment variation, noted that the sonographers contri-

buted little to observed variation. Two scanners of the same make

and model were used, but with different operators. By observing

linear regression lines correlating each scanner to DSA results, it

appeared that there was no significant difference between the

two scanners and hence sonographers. Finally Ranke et al.,12

who tested inter-equipment and inter-sonographer variation for

CDUS, demonstrated a good correlation between two sono-

graphers for both PSV (r = 0.92) and mean velocity ratios

(r = 0.94). However, no assessment of velocity criteria and

variation between grading categories was made.

The aim of this study was therefore to determine the

presence of inter-sonographer variation of CDUS when grading

ICA stenosis in our department, based on a velocity criterion

developed to grade ICA stenosis preoperatively.

METHODSEach subject underwent a total of five CDUS examinations

(each performed by a different sonographer) and a carotid

angiogram.

A total of 33 subjects, 30 males and three females, ranging

in age from 58 to 85 years (mean age 71.8), referred for a

carotid angiogram for the diagnosis and assessment of carotid

artery disease, participated in the study. Both left and right

internal carotid arteries were evaluated.

Carotid duplex ultrasoundThe ultrasound examinations were performed on an Acuson

Model 128XP/10 (Acuson Corporation, California, USA) using a

5 MHz linear transducer. Real time B mode imaging in both the

longitudinal and transverse directions was performed to gain

vessel information, such as location, size and course. Doppler

flow sampling was taken in the CCA and ICA, with any areas of

stenosis being thoroughly evaluated and the highest peak

velocity obtained. The Doppler angle (angle of incidence) was

restricted to ≤ 60° at all times.13 Each of the five sonographers

measured the PSV in the internal and common carotid arteries

and the PSVR was calculated. Each Sonographer was blind to

the results of any previous CDUS done.

Digital subtraction angiographyTo accommodate clinical demands, three DSA systems were

used: (i) a Fischer Imaging Xerox 1000 (Fischer, Colorado, USA),

(ii) a Toshiba DFP–50 A (Toshiba, Tochigi-ken, Japan); and

(iii) a Siemens Polytron TOP (Siemens, München, Germany).

Each of the right and left carotid arteries was selectively

catheterized, and a series of digital subtraction images were

obtained as 5–10 mL of Ultravist 370 (Sehering, Berlin,

Germany) was hand-injected. The DSA projections normally

included an anterio-posterior (AP), lateral and either a right

anterior oblique (RAO) or left anterior oblique (LAO) projection.

These projections were photographed on to hard copy. From

these images, one of two radiologists measured (in mm) the

degree of ICA stenosis according to the NASCET derived

protocol.2 The view showing the greatest degree of ICA

narrowing was chosen for analysis. For the first 20 subjects,

both radiologists interpreted the DSA images to obtain a

measure of inter-observer reliability.

Data analysisExpert statistical advice was sought and received from Dr

Brenton Dansie (School of Mathematics, University of South

Australia) for the study design and statistical interpretation.

All data were entered into a personal computer and the

receiver operator characteristic (ROC) curve data were

calculated and plots generated as described by Sumner.14 In

combination, this data allowed the selection of the velocity

criterion based on the optimum levels of sensitivity, specificity,

postitive predictive value (PPV), negative predictive value

(NPV) and accuracy predicted by the velocity thresholds. ROC

curve analysis was carried out for all five sonographers as a

group, as well as for each individual.

The agreement between CDUS and DSA was measured

with the kappa (k) statistic, a calculation of the difference

20 H THOMSON ET AL.

Table 1. Summary of findings for velocity criteria ≥ 70% North American Symptomatic Carotid Endarterectomy Trial stenosis

Study Parameter Sensitivity(%) Specificity (%) Positive predictive Negative predictive Accuracy(%)

value (%) value (%)

Carpenter et al.5 PSV > 210 cm/s 94 77 68 96 83

Moneta et al.6 PSVR > 4 91 87 76 96 88

Faught et al.4 PSV > 130 cm/s 81 98 89 96 95

EDV > 100 cm/s

between the observed agreement and chance. This calculation

was based upon the selected optimum velocity criteria.

Calculation of the inter-observer variability of the CDUS

operators was based on the optimum velocity criterion that was

selected through ROC curve analysis. An ANOVA was used to

determine if the observed differences among the sonographers

were significant. An intra-class correlation coefficient (ICC) was

calculated as a measure of the reliability of the sonographers.

An ICC of 1.0 is considered good reliability and an ICC of 0.0 is

considered poor reliability.

RESULTSOf the 33 subjects who participated in the study, one vessel was

excluded from the analysis due to recanalization, which was

only detected with further investigation using spiral CT. Ten

subjects had incomplete CDUS evaluations (less than the

required 5), and so 23 subjects (46 arteries) were used for the

analysis of inter-sonographer variation. All data (complete and

incomplete) was used to calculate the optimum velocity criteria

and the agreement between DSA and CDUS. A total of 303

measurements of the 65 vessels were made.

Optimum velocity criteria were determined for the 50%, 70%

and 90% thresholds of angiographic stenosis, defining the

0–49%, 50–69%, 70–89%, 90–99% stenosis bands. According

to DSA measurements these grades were present in 44.6%,

16.9%, 27.7%, 7.7% of cases, respectively. Selection of

velocity criteria was based on the PSVR that provided the best

combination of sensitivity and specificity (that is the highest

overall accuracy) (Table 2).

The distribution of the data (correlating DSA to CDUS) when

the selected optimum velocity criteria are employed to grade

the stenosis is given in Table 3 (this distribution displays the

spread of true and false positives and negatives). Thirty percent

of the stenoses were overestimated and 10% were under-

estimated by CDUS. There were two occlusions, both being

diagnosed accurately by each sonographer (one subject had

incomplete measurements).

Receiver operator characteristic curve analysis was also

performed for each individual sonographer in order to assess if

there was any variability of PSVR between sonographers. Table

4 displays the predicted levels of sensitivity, specificity, NPV,

PPV and accuracy for 50%, 70% and 90% ICA stenosis for

individual optimum thresholds of each stenographer. The pre-

dicted values of sensitivity, specificity, NPV, PPV and accuracy

for each individual sonographer when the group velocity criteria

are employed are given (Table 5).

If the optimum velocity criteria are employed by each

sonographer, then the 95% confidence limits of k are 0.53 ±

0.027. Perfect agreement occurred in 67% of cases. Statistical

analysis revealed there was no significant variation between

the ability of each sonographer to grade the degree of stenosis

(P > 0.05) (Tables 6 and 7). To follow on from this, a calculation

of the intra-class correlation coefficients was made to obtain a

measure of reliability between sonographers. An ICC (2,1) of

0.911 was computed.

It is important to acknowledge that the ICC was calculated

according to the categories the stenosis was graded into, and

not on the velocity measurements.

21INTER-SONOGRAPHER RELIABILITY

Table 2. Optimum velocity criteria

Degree of Optimum peak Sensitivity (%) Specificty (%) Positive predictive Negative predictive Accuracy (%)

internal carotid systolic velocity value (%) value (%)

artery stenosis (%) ratio

50 < 2.25 84.24 90.57 91.44 82.78 87.12

70 ≥ 3.25 93.04 90.96 86.30 95.53 91.47

90 ≥ 5 93.75 80.22 38.46 99.09 83.33

Table 3. Optimum carotid duplex ultrasound (CDUS) velocity criteria (peak systolic velocity ratio) versus carotid digital subtraction angiography

(DSA)

Angiographic percent stenosis

0–49% 50–69% 70–89% 90–99% 100% Total

< 2.25 125* 22 4 0 0 151

≥ 2.25 to < 3.25 8 16* 4 0 0 28

≥ 3.25 to < 5 5 5 34* 2 0 46

≥ 5 0 7 41 21* 0 69

No flow 0 0 0 0 9* 9

Total 138 50 83 23 9 303

*Denotes perfect agreement between DSA and CDUS.

DISCUSSIONIn this study, the optimum velocity criteria were selected on the

basis of the highest overall accuracy. This study aimed to

develop velocity criteria that were capable of preoperative

assessment of ICA stenosis, which meant that a false positive

was considered of equal consequence to a false negative.

Other studies that have also developed velocity criteria based

on the highest overall accuracy have achieved both different

optimum velocity criteria and different values of sensitivity,

specificity, PPV, NPV and accuracy (Table 1).4–6 While these

differences could be attributed to inter-sonographer variation,

they may also reflect the use of different equipment in studies,

as it has been shown that not only is there is a difference in

optimal velocity criteria between units,11 but also that the

relationship between Doppler velocity measurements and the

angiographic measurement of stenosis differs between

ultrasound equipment as reported by Howard et al.15 In the

same study, a comparison of 63 devices found that only 21%

were capable of achieving good sensitivity (> 80%) with a PPV

> 90%. This suggests that regardless of the selected optimal

velocity criteria for a particular system, it is possible for one to

be achieving results superior to another.

The main reason for inter-sonographer differences as a

cause for CDUS variation can be attributed to the scanning

technique and the many potential factors associated with the

velocity measurements. Doppler velocity measurements are

affected by the position of the Doppler cursor, angle of

incidence, and selection for the site along the length of stenosis

to be interrogated. These factors are difficult to control between

sonographers and it is probable that they are the major reason

for any variation between operators. Several studies have

utilized variation in velocity criteria between ultrasound units as

a basis for determining differences between these.11,15 Using a

similar approach, the present study assessed inter-sono-

grapher variation by determining if the velocity criteria varied

between sonographers. For example, the individual criterion for

sonographers differentiating 70% ICA stenosis ranged from a

PSVR ≥ 3.0 to ≥3.5 (as shown in Table 4). Although the

individual criteria did vary, the predicted levels of sensitivity,

specificity, PPV, NPV and accuracy did not differ significantly

between the individual group criterion for each sonographer.

For instance, the individual optimum criterion for Sonographer

22 H THOMSON ET AL.

Table 4. Sensitivity, Specificity, positive predictive value (PPV), negative predictive value (NPV) and accuracy for the individual optimal thresholds

of each sonographer

Sonographer Optimal threshold Sensitivity (%) Specificity (%) PPV (%) NPV(%) Accuracy (%)

50%

1 2.00 88.20 89.6 90.9 86.6 88.8

2 2.75 77.7 96.1 88.0 78.1 85.9

3 2.25 87.8 96.5 93.5 86.6 90.2

4 2.25 83.3 86.1 88.2 80.6 84.6

5 2.25 83.8 92.3 92.8 82.7 87.7

70%

1 3.25 95.8 89.5 85.2 97.2 88.0

2 3.50 90.5 94.4 90.4 94.4 92.8

3 3.00 95.6 92.1 88.0 97.2 92.8

4 3.25 96.0 92.5 88.8 97.3 93.8

5 3.50 94.5 97.1 95.5 97.1 96.5

90%

1 5 100 78.9 33.3 100 80.9

2 5 100 84.3 42.8 100 80.9

3 5 85.7 83.3 40.0 97.8 83.6

4 7 100 97.9 36.3 91.1 82.1

5 5 100 86.2 46.1 100 87.7

Table 5. Sensitivity, specificity, positive predictive value (PPV),

negative predictive value (NPV) and accuracy of individual

sonographers with group criteria

Sonographer Sensitivity Specificity PPV NPV Accuracy

(%) (%) (%) (%) (%)

50%

1 85.3 93.1 93.5 84.4 88.0

2 80.6 88.5 82.1 79.3 84.2

3 87.8 96.5 93.5 86.6 80.2

4 83.3 86.1 88.2 80.6 84.6

5 83.8 92.3 92.8 82.7 87.7

70%

1 95.8 89.5 85.2 97.2 88.0

2 90.5 86.1 79.2 93.3 87.8

3 94.7 92.1 90.9 92.3 91.8

4 96.0 92.5 88.8 97.3 93.8

5 95.5 91.4 87.5 96.9 92.8

90%

1 100 79.3 33.3 97.9 80.0

3 was a PSVR ≥ 3.0; this provided an accuracy of 92.8%

(sensitivity 95.6%, specificity 92.1%, PPV 88.0%, NPV 97.2%).

If sonographer 3 employed the group criterion, a PSVR ≥3.25,

an accuracy of 91.8% (sensitivity 94.7%, specificity 92.1%,

PPV 90.9% and NPV 92.3%) was predicted (as shown in Table

5). In the clinical situation it is practical that only a group

criterion is employed, from which no sonographer in the current

study achieved unacceptable results. Analyzing each sono-

grapher individually allows the identification of their boundaries

to which they can perform.

The ROC curve analysis of each sonographer indicates that

there are differences between their optimum threshold, but the

current study showed that there was no significant difference

between sonographers in their grading of stenosis (P > 0.05). It

is essential to recognize that this analysis was not carried out

on the measurement of PSVR by the sonographer, but on the

categories the stenosis was graded into (based on the selected

optimum velocity criterion 0–49%, 50–69%, 70–89%, 90–99%

and occluded). This is unlike Ranke et al.,12 who showed good

correlation between two sonographers in measurements of

both the mean systolic ratio (r = 0.94) and intra-stenotic PSV

(r = 0.92). The category into which the stenosis is graded will

determine the treatment and consequently any variation in the

grading of stenosis by sonographers is particularly important.

The current study calculated an ICC (2,1) of 0.911, indicating

strong reliability between sonographers.

The sonographers who participated in the study ranged in

experience (two were experienced, two were moderately

experienced and one was a junior sonographer). However, each

of them was trained in the same setting. It would be expected,

therefore, that their scanning techniques were relatively similar.

Whether results similar to this study would be achieved with

sonographers from different centres who have trained in a

variety of ways remains to be determined.

In the current study the 95% confidence limits of k were

0.53 ± 0.027 when all five grading categories were being

evaluated. Perfect agreement between CDUS and DSA

occurred in 67% of cases. Faught et al.4 reported a k value of

0.68 ± 0.02 and perfect correlation in 82% of cases. However,

Faught et al.4 did not assess 90% angiographic stenosis, which

was the main cause for a low k value in our study. Recalculating

the present values and ommitting 90% stenosis gives a k

value of 0.71 ± 0.12, closer to Faught et al. In conclusion, this

study has shown that there is no significant variation between

the ability of sonographers to grade the degree of ICA stenosis.

Only moderate agreement between CDUS and DSA was

achieved. However, the predicted levels of sensitivity,

specificity, NPV, PPV and accuracy of the developed criteria

were acceptable, but did differ from validated velocity criteria

from other centres. These results therefore suggest that while

CDUS may not be an operator dependant measurement, each

centre should be responsible for validating its own velocity

criteria and maintain continuous evaluation of it.

ACKNOWLEDGEMENTSWe would like to acknowledge and thank the following people

for their help and guidance on this research project: R Morcom,

C Lott, J Bestt, C Kapsis, S Jenke, M Gourlay, all staff at the

Repatriation General Hospital and B Dansie.

REFERENCES1. National Stroke Foundation (Stroke Australia task force). National

Stroke Strategy. National Stroke Foundation, Melbourne,

Australia, 1997.

2. NASCET collaborators. Beneficial effect of carotid endarterectomy

in symptomatic patients with high grade stenosis. N Engl J Med

1991; 325: 445–53.

3. Polak JF, Bajakian RL, O’Leary DH, Anderson MR, Donaldson

MC, Jolesz FA. Detection of internal carotid artery stenosis:

23INTER-SONOGRAPHER RELIABILITY

Table 6. Single factor ANOVA

Source of SS df MS F ratio P value F critical

variance

Between subjects 349.39 45 7.746 52.52 1.63 1.43

Within Subjects 27.20 184 0.147

Total 376.59 229

Intra-class correlation coefficient (1,1) = 0.911.

Table 7. Two factor ANOVA without replication

Source of SS df MS F ratio P value F critical

variance

Between subjects 349.39 45 7.760 52.60 2.96 1.44

Between Sonographers 0.6340 4 0.158 1.075 0.37 2.42

Error 26.560 180 0.147

Total 376.59

Intra-class correlation coefficient (2,1) = 0.911; (3,1) = 0.912.

comparison of MR angiography, color Doppler sonography, and

arteriography. Radiology 1992; 182: 35–40.

4. Faught WE, Mattos MA, Bernmele PS et al. Color-flow duplex

scanning of carotid arteries: New velocity criteria based on

receiver operator characteristic curve analysis for threshold

stenoses used in the symptomatic and asymptomatic carotid trials.

J. Vasc Surg 1994; 19: 818–28.

5. Carpenter JP, Lexa FJ, Davis JT. Determination of duplex Doppler

ultrasound criteria appropriate to the North American symptomatic

endarterectomy trial. Stroke 1996; 27: 695–9.

6. Moneta GL, Edwards JM, Chitwood RW et al. Correlation of North

American symptomatic carotid endarterectomy trial; angiographic

definition of 70–99% internal carotid artery stenosis with duplex

scanning. J Vasc Surg 1993; 17: 152–9.

7. Faccenda F, Usui Y, Spencer M. Doppler Measurement of the

pressure drop caused by arterial stenosis: An experimental study:

A case report. Angiology 1985; 36: 899–905.

8. Bluth EI, Merritt CR. Carotid arteries and vertebral arteries. In:

Merritt CR (ed.) Doppler color imaging. Churchill Livingstone, New

York, 1992; 61–96.

9. Moneta GL, Edwards JM, Papanicolaou G et al. Screening for

asymptornatic internal carotid artery stenosis: Duplex criteria for

discriminating 60–99% stenosis. J Vasc Surg 1995; 21: 989–94.

10. Meyer JI, Khalil RM, Obuchowski NA, Baus LK. Common carotid

artery: Variability of Doppler ultrasound measurements. Radiology

1997; 204: 339–41.

11. Fillinger MF, Baker RJ, Zwolak RM et al. Carotid duplex criteria for

60% or greater angiographic stenosis: Variation according to

equipment. J Vasc Surg 1996; 24: 856–64.

12. Ranke C, Trappe HJ. Blood flow velocity measurements for carotid

stenosis estimation: Interobserver variations and interequipment

variability. VASA 1997; 26: 210–14.

13. Beasly MG, Balu JN, Gosling RG. Changes in internal carotid

artery flow velocities with cerebral vasodilation and constriction.

Stroke 1979; 19: 331–5.

14. Sumner DS. Evaluation of noninvasive testing procedures data

analysis interpretation. In: Bernstein (ed.) Noninvasive Diagnostic

Techniques in Vascular Disease. CV Mosby, St Louis, 1985;

861–89.

15. Howard G, Baker YM, Chambless LE, Howard VJ, Jones AM,

Toole JF. An approach for the use of Doppler ultrasound as a

screening tool for hemodynamically significant stenosis (despite

heterogenity of Doppler performance). A multi center experience.

Asymptomatic carotid atherosclerosis study investigators. Stroke

1996; 27: 1951–7.

24 H THOMSON ET AL.