Observer Performance Studies

Observer Detection Performance in Radiology Using a Retransmission-Free Network Communication Protocol

The goal of this study [4] was to measure the effect of image data loss on diagnostic accuracy to test the possibility of using a retransmission-free network communication protocol for transferring radiologic images.

Methods

Images

  • Images

-25 lesion-free
-25 with single subtle microcalcification cluster
-Number of microcalfications per cluster determined by experienced mammographer
-Images were digitized at 50 microns with 12-bit contrast resolution

  • Simulating Transmission Loss

-Burst-oriented packet loss (0%, 15%, 25%) was incurred for each image using a scattered-encoding mechanism with a packet group size that was normally distributed with a mean of 8 & SD of 1 packet
-A 4-neighbor linear interpolation recovery scheme was then run on each image
-Images were printed back to film yielding total set of 150 images - 50 basic with 0%, 15%, 25% loss

Display

Standard viewbox

Procedure

-10 observers
-Viewed images from the various loss conditions in random orders

Task

1) Report cluster present or absent
2) Report confidence in decision using 5-level scale
3) Report number of microcalcifications in each cluster detected

Results

ROC Performance Analysis

Figure 1. Mean ROC Az values for 0%, 15% and 25% loss conditions. The difference between 0% & 15% loss was not statistically significant, but 25% loss performance was significantly lower than 0% & 15% (F (2,18) = 2.12, p <.001).

Figure 1. Mean ROC Az values for 0%, 15% and 25% loss conditions. The difference between 0% & 15% loss was not statistically significant, but 25% loss performance was significantly lower than 0% & 15% (F (2,18) = 2.12, p <.001).

Table 1. Mean percentages of True-Positive (TP) & False-Positive (FP) Responses for the 3 Loss Conditions

0% Loss 15% Loss 25% Loss
TP FP TP FP TP FP
79 25 83 22 76 31

Number of Reported Microcalcifications

In all 3 conditions the correlation between number of reported microcalcifications and the actual number of microcalcifications ranged from r = 0.73 to r = 0.97.  There were no differences between the 3 loss conditions.

Conclusions

  • Up to 15% packet loss can be tolerated without affecting diagnostic performance
  • Even 25% loss of information does not dramatically impact of diagnostic accuracy - the TP rate stays fairly high but the FP
    rate increases somewhat
  • In some radiologic applications retransmission-free network communication protocols may be useful (e.g., battlefield
    conditions)

Return to Observer Performance Studies page