Interrater and intermethod reliability of default mode network selection

Near perfect agreement was found between human raters selecting the default mode network (DMN) component from resting-state fMRI data, but agreement between human raters and an automated template-matching method was only moderate. The automated method's performance improved significantly when a weighted combination of anterior and posterior cingulate nodes was used. The study involved 42 healthy controls and compared interrater and intermethod reliability for identifying DMN activation, highlighting the need for careful template design in automated component selection.

Study at a glance

Abstract

There has been a growing interest in the neuroimaging community regarding resting state data (i.e., passive mental activity) and the subsequent activation of the so-called default mode network (DMN). Although this network was originally characterized by a pattern of deactivation during active cognitive states, more recent applications of data-driven techniques such as independent component analysis (ICA) have permitted the analysis of brain activation during extended periods of truly passive mental activity. However, ICA requires the resultant components to be evaluated for "goodness of fit" via either human raters or more automated techniques. To our knowledge, an investigation on the reliability of either technique in determining the component that best corresponds to default-mode activity has not been performed. Moreover, it is not clear how automated techniques, which are necessarily dependent upon a template mask, are affected by the structures used to compose the mask. The current study investigated both interrater (human-human) reliability and intermethod (human-machine) reliability for determining DMN activation in 42 healthy controls. Results indicated that near perfect interrater reliability was achieved, whereas intermethod reliability was only within the moderate range. The latter was significantly improved via a weighted combination of the anterior and posterior cingulate nodes of the DMN. Implications for fully automating the component selection process are discussed.