Evaluation of the Consonant Vowel Consonant (CVC) Test as an Assessment and Screening Instrument for Mild Cognitive Impairment

Objective: Traditional word learning tasks have been criticised for being affected by ceiling effects. The Consonant Vowel Consonant (CVC) test is a non-word verbal learning task designed to be more difficult and therefore have a lower risk of ceiling effects. Method: The current study examines the psychometric properties of the CVC in 404 middle-aged persons and evaluates it as a screening instrument for mild cognitive impairment by comparing it to the Montreal Cognitive Assessment (MoCA). Differences between currently depressed and non-depressed participants were also examined. Results: CVC characteristics are similar to traditional verbal memory tasks but with reduced likelihood of a ceiling effect. Using the standard cut-off on the MoCA as an indication of mild cognitive impairment, the CVC performed only moderately well in predicting this. Depressed participants scored significantly lower on the CVC compared with non-depressed individuals. Conclusions: The CVC may be similar in psychometric properties to the traditional word learning tests but with a higher ceiling. Scores are lower in depression.


Introduction
Verbal memory functions are frequently tested with list learning tests such as the Rey Auditory Verbal Learning Task (RAVLT; Rey, 1964) and the California Verbal Learning Test (CVLT; Delis, Kramer, Kaplan, & Ober, 2000).Both these tasks involve repeated presentations of word lists for immediate and delayed recall and the assessment of verbal learning performance.In addition, a distractor list is included to test for susceptibility to interference and a recognition task to assess discriminability.The RAVLT has been shown to be useful in predicting psychosocial functioning after a traumatic brain injury (Ross, Millis, & Rosenthal, 1997).In addition, different degenerative diseases lead to different outcome patterns on the RAVLT.For example, patients with early Alzheimer dementia recall only a few words in trial one and slightly increase word recall over the other acquisition trials (Bigler, Rosa, Schultz, Hall, & Harris, 1989).They also show poor performance after a delay that includes distraction (Ferman et al., 2006).In contrast, patients with multiple sclerosis show low overall low recall (Bravin, Kinsella, Ong, & Vowels, 2000).Performance on the total number of recalled items and the number of items recalled after the long delay in the CVLT was shown to be useful in distinguishing patients with mild cognitive impairment from control participants (Rabin et al., 2009).Although widely used in clinical and research settings (Elwood, 1995;Hawkins, Dean, & Pearlson, 2004), both tests have been criticized for being affected by ceiling effects (Elwood, 1995;Uttl, 2005).Ceiling effects cause significant problems because of diminished variability of scores in the top range, affecting means and standard deviations.Sensitivity can also be reduced as differences between groups may not be detected.Ceiling effects are more likely to occur in non-clinical groups, thus making comparisons between clinical and non-clinical participant particularly vulnerable.
To avoid such ceiling effects, in a study of mild to moderately severe depression, Bourke et al. (2012) developed a Consonant Vowel Consonant non-word verbal learning task (CVC) which uses non-words instead of words, but otherwise follows the same structure as the RAVLT and the CVLT.Their goal was to create a task that would be more sensitive and able to show subtle differences in verbal learning and declarative memory between depressed and healthy subjects.Both the RVALT and the CVC were used to compare participants with major depression, social anxiety disorder and a healthy control group (Bourke et al., 2012).The study showed verbal memory deficits in participants with major depression compared with controls and participants with social anxiety disorder.The effect size for immediate recall performance was larger in the CVC test than in the RVALT and a difference in learning was observed only on the CVC, suggesting greater sensitivity of the CVC in detecting subtle deficits in verbal learning in this setting.With regard to delayed memory recall, the RVALT revealed a significant difference between the depressed and control groups, but the CVC test did not.This mixed pattern suggests the possibility of different psychometric properties in the CVC task, therefore we aimed to examine these in a large sample.
The current paper reports CVC performance of 404 randomly selected middle-aged people.In addition, since the CVC may be a valuable screening tool for mild cognitive impairment we examined the relationship between scores on the CVC and the Montreal Cognitive Assessment (MoCA; Nasreddine et al., 2005), a screener to evaluate general cognitive functions in this sample.If the CVC follows the same patterns that were found for other word lists learning tests, we would expect a relationship between low MoCA scores and low scores on CVC total and delayed recall.

Participants
Participants were recruited as part of a large prospective cohort study-the Canterbury Health, Ageing and Lifecourse (CHALICE) study.Participants with an age between 49-51 years were randomly selected from the electoral rolls, registration on which is compulsory in New Zealand.A total sample of 404 participants was recruited and the participation rate was 62.0%.After informed written consent was obtained, assessment was conducted at the University of Otago, Christchurch.Assessment took about 5-6 hours and included laboratory measures, such as conducting an echocardiogram and measuring blood pressure, questions pertaining to demographics, physical and mental health history, diet and exercise, cognitive functioning, and attitudes and beliefs.Standard of living was assessed with the Economic Living Standard Index Short Form (ELSI SF , Jensen, Spittal, & Krishnan, 2005) which is an instrument developed in New Zealand.The ELSI SF asks about a person's consumption and personal possessions and categorises participant into mild, moderate, and high standard of living.The study received ethical approval from the Upper South A Regional Ethics Committee and is described in detail in Schluter et al. (2013).

CVC
The non-words for the CVC were selected from the ACR Non-word Database (Rastle, Harrington, & Coltheart, 2002).The chosen words were each 3 letters long, began and ended with a consonant, with a vowel in the middle.Care was taken to avoid non-words that allowed any associations to meaning including commonly used acronyms.Several lists containing 15 non-words were generated of which only one was used in this study.To standardise presentation, each non-word was pre-recorded and played to participants while the non-word was also displayed on the computer screen controlled by E-Prime (Schneider, Eschman, & Zuccolotto, 2002).The presentation rate was about 1 word per second.The list was presented five times.After each presentation the participant was asked to recall as many non-words as possible.Delayed recall performance of this list was tested after a period of 15 minutes.Immediately afterwards, the recognition task was conducted.For this task 30 non-words were presented as before, by audio track and on screen, in random order.Fifteen of these items were new non-words, from another list, whereas the other 15 non-words were from the original list.Participants were asked to discriminate whether they had seen the word before pressing either the "z" and the "/" key on the keyboard.The z key was labeled with a "Y" and served to indicate a non-word from the original list and the "/" key was labeled with an "N" and served to indicate a new non-word.Participants had five seconds to respond to each word.Accuracy and response time was recorded.

MoCA
The MoCA has been designed to assess mild cognitive impairment and tests several cognitive functions, such as short-term memory, visuospatial abilities, executive functions, attention and working memory, language and orientation.Participants with a score of 25 or less (out of 30) are considered impaired.The MoCA has been shown to have excellent sensitivity and good specificity.Test-retest reliability, internal consistency, and content validity are also excellent (Dalrymple-Alford et al., 2010;Nasreddine et al., 2005).

Mini-International Neuropsychiatric Interview
The Mini-International Neuropsychiatric Interview (M.I.N.I.; Sheehan et al., 1998) is a brief structured diagnostic interview based on the Diagnostic and Statistical Manual of Mental Disorders (American Psychiatric Association, 2000) and generates psychiatric diagnoses as defined by DSM-IV.

Procedure
The session began with the five repeated trials of the CVC.During the 15 minute delay period the MoCA and the Edinburgh Handedness Inventory (Oldfield, 1971) were conducted.The session concluded with the delayed CVC recall and the recognition component of the CVC.

Statistical Analysis
The characteristics of the CVC task were analysed using repeated measurement analysis of variance.The Huynh-Feldt correction was used where the assumption of sphericity was violated.
Relationship of the CVC to the standard cut-off on the MoCA was analysed using receiver operating characteristic (ROC) curves and sensitivity/specificity data derived from these.

Results
The data of 19 participants were excluded: for six participants, data was missing due to technical problems and 12 more participants were excluded because of pre-existing problems, such as blindness, partial deafness, severe memory problems as a result of head injury, an inability to read and write English or other health problems.One person refused to complete the test.This left 385 participants for the analysis of the five acquisition trials of the CVC.Two further participants had missing data due to technical difficulties for the delayed recall only, reducing the participant number further to 383 participants.Data for the CVC recognition was excluded for six more participants (two participants had a tremor and could not properly operate the keys, one did not follow instructions and for three other participants data was missing due to technical problems) leading to a total sample of 379 participants for the recognition data.

Demographic Data
The demographic data of the remaining 385 participants who completed the five acquisition trials is displayed in Table 1.One hundred and seventy-nine males and 206 females participated in the study.The average age of the participants was 50.85 (SD=0.69)years.7.8% of the participants were judged as currently depressed as determined by the M.I.N.I. (Sheehan et al., 1998).

CVC Recognition
The data for the recognition part of the CVC is displayed in Table 2.The frequency distributions for hits and false alarms are displayed in Figures 3. As can be seen in the graphs most participants found it easy to recognise the non-words that have been presented previously, but many participants also identified new non-words as being from the presented lists, leading to a high number of false alarms.
Figures 3. Frequency distribution for the number of hits (left) and false alarms (right) in the CVC recognition.There were 15 words in the list We calculated d-Prime to estimate discriminability of new versus old non-words.The value of -2.43 indicates perfect performance.Values range quite dramatically from -2.43 to 8.13, which indicates a wide range of discriminability abilities in the sample.

Other Factors Impacting on CVC Performance
Scores in word list learning tests are known to be influenced by demographic variables such as gender and education (Longenecker et al., 2010).In addition, meta-analytic reviews have shown an effect of depression on memory performance (Kindermann & Brown, 1997;Veiel, 1997).We therefore analysed the influence of education, gender, work situation and depression to identify whether these variables also impact on the CVC outcome measures by using these variables as fixed factors in the repeated measures analysis.
There was a main effect of education [F(3, 379)=10.03,p<0.001].In general, the higher the education achieved, the more non-words were recalled.There was also a significant trial by education interaction [F(12.159,1536.082)=1.959,p=0.024].Participants with secondary school education correctly recalled more non-words in the 5 th trial of the acquisition phase than the other groups.
In addition, we found a main effect of gender on the total number of non-words recalled in the CVC [F(1,383)=9.981,p=0.002], with females recalling significantly more non-words overall than males (31.4 vs. 27.8).
For the analysis of work situation we excluded the three participants in the "other" category.The current work situation had no effect on any part of the CVC.
Comparisons between medicated and un-medicated participants (the total sample) and the medicated and un-medicated currently depressed participants revealed no differences in the CVC total score, the delayed recall, d-Prime and total MoCA scores.

Comparison of MoCA and CVC Scores
The MoCA total score was moderately, but significantly, positively correlated with the CVC total score

CVC Sensitivity and Specificity
To determine sensitivity and specificity values of the CVC in relationship to the MoCA results we divided MoCA total results in cases (scores<26) and non-cases (≥26).This categorization resulted in 102 cases and 283 non-cases.We created ROC curves for each CVC trial and the delayed recall condition separately and then for cumulative CVC trials.Because of the significantly lower scores in the depressed patients, we repeated this analysis without the data from the currently depressed participants and for depressed participants only.The areas of the curves and the standard errors are displayed in Table 3.After an evaluation of the ROC curves for cumulative trials, it appeared that sensitivity and specificity is already close to the values for the total score after only three acquisition trials.We therefore present sensitivity and specificity values for data using the cumulative score of CVC trial 1+2+3 and the CVC total score only, as these seem to be the most predictive measures (see Table 4).

Discussion
In this study we examined the psychometric properties of the CVC in a large group of middle-aged participants and evaluated the CVC as a screening tool for mild cognitive impairment.
Our findings show that the characteristics of the CVC are comparable to those described in traditional verbal memory tasks.The total number of words increased gradually across the five acquisition trials.This learning curve is very similar to the patterns observed in other verbal learning tasks such as the RAVLT and the CVLT.Further, the primacy and recency pattern found in our study is also similar to patterns described in word recall studies with the same age group (Mitrushina, Satz, Chervinsky, & Delia, 1991).Words mentioned early in the sequence are remembered best followed by words presented last; with middle items recalled to a lesser extent (Graf & Uttl, 1995).Whereas learning curves in these tasks usually increase from around seven words in the first trial to 13 words in the fifth trial (Bleecker, Bollawilson, Agnew, & Meyers, 1988;Kramer, Delis, & Daniel, 1988), non-word recall in our study increased from around three to eight items over the five acquisition trails.This finding clearly confirms that the CVC is unlikely to display ceiling effects.
Effects of gender, education, and depression observed in our study mirror those found in memory tasks using words.With regard to gender, women generally recall more words, but have a similar learning curve to men (Kramer et al., 1988;Reite, Cullum, Stocker, Teale, & Kozora, 1993).The difference in recall is on average one word more per trial (Kramer et al., 1988).In our study the difference for non-words was 0.71 on average, which matches the much lower number of non-words recalled overall.We observed a positive effect of education on memory performance.In general, the higher the education the more words were recalled.Similar results have been reported for word recall tasks (Donders, 2008;Messinis, Tsakona, Malefaki, & Papathanasopoulos, 2007;Norman, Evans, Miller, & Heaton, 2000;Teruya, Ortiz, & Minett, 2009).
Subjects who currently fulfilled criteria for a diagnosis of current major depressive disorder had significantly lower scores on the CVC.This is in keeping with the results of Bourke et al. (2012) examining the CVC in patients with depression and confirms, in a community study, that the CVC appears sensitive to the effects of depression.However, only 30 patients were depressed and of these half were on medication.The numbers are too small to determine clearly whether medication has a significant effect on the results.
In order to evaluate the possible usefulness of the CVC to detect mild cognitive impairment we compared CVC results to the results of the MoCA.The correlation between these two measures was within the expected range with a correlation of r=0.39.Schweizer et al. (2012) reported a correlation of equal size between the total of the MoCA and the CVLT in patients after aneurysmal subarachnoid haemorrhage.Similarly, Mitrushina and Satz (1991) found associations between 0.31 and 0.39 for the different components of the RAVLT and the MMSE, a screening tool similar to the MoCA.As expected the CVC outcome measures that correlated most with the MoCA were total recall and delayed recall, measures that have been associated with mild cognitive impairment (Rabin et al., 2009).Assuming the MoCA is the current "gold standard" psychometric test for evaluating mild cognitive impairment, the CVC performs only moderately well in predicting impairment as defined by the MoCA.The ROC curves for cumulative trials suggest that after three acquisition trials of the CVC the maximum possible values for specificity and sensitivity are reached.This increases the value of the CVC as it limits the time needed for assessment and thus the demand on the patient.Since it focusses on memory, it is possible that the CVC task may be more sensitive in detecting early Alzheimer's disease.There are several studies that support this notion.Chang et al. (Chang et al., 2010) evaluated an abbreviated version of the CVLT as a screener for Alzheimer's disease and confirmed good predictive validity for the test.In addition, word list recall has been shown to differentiate successfully between people with late onset depression and Alzheimer's disease.Participants affected by Alzheimer's recalled significantly fewer words than depressed participants (Kuenig et al., 2006).Furthermore an earlier meta-analysis showed that free recall of words after a short delay were effective in separating depressed patients from demented patients (Lachner & Engel, 1994).
The current study has several limitations.All participants were within a very narrow age bracket.Therefore we cannot extrapolate across the age range or examine effects of age in this sample.Furthermore, as yet, we do not know which of the subjects will go on to develop cognitive impairment.Cut-off scores on the MoCA are only a best estimate of who has cognitive impairment which is likely to progress.Clearly longitudinal follow up will add significantly to the data.Lastly, the data in the depressed subjects is hampered by the relatively small number (30) of depressed participants and a potential medication effect in the 14 who were also using antidepressants.
In summary, we have demonstrated that the CVC, in this population operates in the middle range of word recall and therefore is likely to avoid ceiling effects.This may make it more sensitive compared with other verbal learning tests while offering the same range of outcome measures, especially when comparing clinical and non-clinical groups.There is some risk that the CVC is more susceptible to floor effects, especially in impaired participants.Similarly, the test is relatively unpopular with participants, due to its difficulty.This may be a limit for certain participant groups, although in our sample only one person refused to do the CVC.
While the CVC performed only moderately well in predicting cognitive impairment as defined by the MoCA, the true value of the CVC as a screening device remains to be seen since the development of significant cognitive impairment has not been determined and requires further follow up.A follow-up for the study population is planned at age 60.
The CVC has already been shown to be sensitive to impairment in depressed patients, where it showed a greater sensitivity in demonstrating deficits in verbal learning than the RAVLT (Bourke et al., 2013).In this study we have confirmed significant differences in patients with depression.
In conclusion, the CVC may be a promising tool to assess verbal learning, when a ceiling effect may occur and as a screener to identify mild cognitive impairment.
3.2 CVCAcross the five acquisition trials the number of recalled items for each subsequent trial increased significantly[F(3.279,1259.122)=512.904,p<0.001].The average number of items recalled per trial is displayed in Figure1.

Figure 1 .
Figure 1.The mean and standard deviation of the number of words recalled across the five acquisitions trials and the delayed recall

Figure 2 .
Figure 2. Recall of words over the five acquisitions trials and the delayed recall as a function of position in the list of the CVC test [r=.388, p<0.001].In addition, when dividing MoCA scores into those above and below the cut-off value of 26, a main effect of this factor emerged [F(1, 381)=40.914,p≤0.001].Furthermore, the number of words recalled across all five acquisition trials and the delayed recall differed significantly between subjects with scores above or below the cut-off [F(4.054,1544.418)=5.246,p<.001].This interaction is displayed in Figure 4.

Figure 4 .
Figure 4. Recall of words over the five acquisitions trials and the delayed recall as a function of scores above and below cut-off in the MoCA

Table 1 .
Demographic data for the 385 participants with percentages displayed in brackets

Table 3 .
Areas and standard error (in brackets) under ROC curves for participants scoring below or above cut-off on the MoCA

Table 4 .
Cut points, specificity, false negative rate, and false positive rate for different sensitivity values for CVC total score and CVC Trial 1+2+3.