Overcoming Limitations of the Independent Clusters Model for CFA by Means of Bayes-Estimation and Buffered Simple Structure

The model of buffered simple structure is discussed as a method for modeling cross-loadings in confirmatory factor analysis. This method introduces assumptions from item sampling theory into confirmatory factor analysis. The independent clusters model, buffered simple structure, and Bayes estimation were compared by means of a simulation study based on three different population types. Population type A had zero cross-loadings, population type B had symmetrically distributed nonzero cross-loadings, and population type C had asymmetrically distributed nonzero cross-loadings. It turned out for population A that, although the independent clusters model yields the best loading estimate, it did not outperform Bayes estimation and buffered simple structure with respect to the factor inter-correlation estimate and model fit. One reason for this unexpected result could be that the specification of zero-cross loadings is suboptimal even when only sampling error introduces some cross-loadings. For populations B and C Bayes estimation and buffered simple structure clearly outperformed the independent clusters model. Overall, the results indicate that depending on the structure of cross-loadings in the population and depending on the focus on loading estimates or factor inter-correlation estimates, different modeling approaches might be appropriate.


Introduction
Factor analysis is regularly performed, especially in order to find a measurement model for a set of variables and, more specifically, in the context of scale development. Simple structure has been proposed in order to facilitate the interpretation of the factor loadings (Thurstone, 1947). Simple structure implies that each factor is represented by some marker variables with large loadings and that the other variables have rather low loadings on the respective factor. A strong version of simple structure is the independent clusters model (ICM), which implies that there are some large, salient loadings and that the non-salient loadings are exactly zero. The ICM is investigated in the context of confirmatory factor analysis (CFA), where the non-salient loadings are fixed to zero while the loadings of the marker variables are freely estimated. The fit of the ICM to personality models has been questioned (McCrae, Zonderman, Costa, Bond, & Paunonen, 1996;Vassend & Skrondal, 1997) and consequences of possible misfit of the ICM for the evaluation of model fit (Raykov, 1998;Beauducel & Wittmann, 2005) as well as the usefulness of the ICM have been discussed (Marsh et al., 2009).
Exploratory structural equation modeling (ESEM; ) may overcome some limitations of ICM. In ESEM exploratory factor analysis is combined with confirmatory path modeling. Therefore, ESEM does not propose a new or relaxed form of confirmatory factor modeling and suffers from rotational indeterminacy unless Target-rotation is performed or unless the exploratory factor model is used as a basis for further CFA as proposed by Marsh, Morin, Parker, and Kaur (2014). Although Target-rotation allows for the specification of a priori hypotheses for loadings, the model fit is not altered by factor rotation. Therefore, differences in model fit cannot be used in order to compare different loading patterns or measurement models. However, Bayesian structural equation modeling (BSEM; Muthé n & Asparouhov, 2012) may allow to overcome the unnecessarily strict ICM, replacing parameter specifications of exact zeros with more realistic, approximate zeros based on informative, small-variance priors. Thus, in BSEM one might assume that a loading is very small, but it is not necessary to assume that it is exactly zero. The possibility to overcome the limitations of the ICM by means of ESEM and BSEM has also been investigated by Xiao, Liu, and Hau series of model modifications may capitalize on chance (e.g., MacCallum, Roznowski, & Necowitz, 1992). Overall, population loadings can more precisely be estimated by means of ESEM than by means of ICM, but ESEM depends on the method of factor rotation and even on the specification of parameters of the rotation method (Xiao et al., 2019;). In consequence, ESEM might be used as an alternative to ICM in several research contexts, but it is not a general alternative for those, who want to investigate specific hypotheses on loading patterns or intended measurement models. As the present study is devoted to the consequences of hypotheses on loading patterns for model fit, ESEM will not be considered further here.
Even when researchers should not use small-variance priors mechanically (MacCallum, Edwards, & Cai, 2012) and even when a sensitivity analysis for priors has been recommended (Depaoli & van de Schoot, 2017), it is notable that the use of correct zero priors with small variance instead of exact zero loadings in BSEM resulted in loading estimates that were close to the population loadings (Xiao et al., 2019). This is in line with previous results indicating that BSEM might be an interesting alternative to ICM (Muthé n & Asparouhov, 2012;Reis, 2017) when hypotheses on loading patterns are investigated. It is certainly helpful to replace the unrealistic exact-zero hypothesis for cross-loadings by means of the softer small prior variance hypothesis. Since only CFA and not complete structural equation models (SEM) will be considered in the following, the CFA based on Bayes-specification/estimation will be denoted as BAYES. Although BAYES is promising from a conceptual perspective as well as according to simulation studies, it might also be interesting to try to overcome the limitations of the ICM within the frequentist approach to CFA. It has rarely been considered whether besides the small prior variance hypothesis of the BAYES approach and besides specification searches following ICM, a more realistic global hypothesis for the cross-loadings might also allow for an appropriate model specification. The investigation of this possibility could improve the range of measurement models for cross-loadings and could thereby help to find the appropriate model in a given theoretical and empirical setting. This instrumental perspective of modeling may help to resolve issues that are due to diverging conceptual interpretations of cross-loadings (Asparouhov, Muthé n & Morin, 2015;Stromeyer, Miller, Sriramachandramurthy, & DeMartino, 2015). Beauducel (2017) proposed to replace the hypothesis of an exact zero of each cross-loading by a hypothesis on a zero-mean of the cross-loadings across the variables of a factor. This hypothesis is less strict than the exact-zero hypothesis for each cross-loading, does not leave the specific framework of maximum-likelihood estimation, and goes back to the 'buffered scales' that have been proposed by Cattell and Tsujioka (1964) as a way to represent factors by means of scales. There are different ways for the specification of a zero-mean hypothesis for cross-loadings. One could simply specify a zero mean for all cross-loadings of a factor, without taking into account whether the variables are markers of different factors. This would typically lead to very few constraints on the loading matrix and could result in non-identified models. More complex patterns of zero-constraints for cross-loadings that are weighted by the respective salient loadings have therefore been proposed (Beauducel, 2017). Since the weights introduce additional assumptions that might not be compelling in several research settings, we propose a simplified set of constraints that is based solely on the zero mean for the cross-loadings of each block of marker variables of each factor. These constraints can be illustrated by the following example of a three-factor loading pattern A composed by 3  1 submatrices L ij of salient loadings and 3  1 submatrices NL ij of non-salient loadings, with The buffered-cross-loadings constraints on A are   6  9  3  9  3  6   1  1  2  2  3  3  4  7  1  7  1  4   0 , 0 , 0 , 0 , 0 , 0 .
Since the constraints are based on the blocks of cross-loadings that are based on variables marking the same factor, the number of constraints is q(q -1). A corresponding set of zero-constraints could be a more realistic alternative for the ICM and will be termed buffered-simple-structure (BSS) in the following. It should be noted that although BAYES represents an interesting alternative to the ICM, it needs a specification of the prior variance of a cross-loading distribution (Muthé n & Asparouhov, 2012). As different values might be used for the cross-loading variance in different studies, it might be difficult to compare the results from different studies. It might therefore be interesting to consider the BSS as an alternative to the ICM that will lead to the same model specification across studies.
Moreover, the BSS could also be of interest because it introduces assumptions from item-sampling theory (Lord & Novick, 1968) into CFA model specification. Whereas in the ICM the expectation of the cross-loadings for the population of individuals is zero, in the BSS the expectation of cross-loadings for the population of individuals and for the population of measurements of the same factor is zero, i.e., E(NL ij ) = 0. Thus, the cross-loadings are considered as resulting from item-sampling error. Although items are often not sampled at random, the problem of CFA modeling is that items are typically not constructed in a systematic way which might allow to predict the size of cross-loadings. As long as item construction is rather unsystematic, no hypothesis on the cross-loadings of the items can be stated. Under these circumstances it could be reasonable to consider items with substantial loadings on the same factor as members of the same population of items. In any case, it is possible to conceive an item population from which the items loading on the same factor have randomly been drawn (Lord & Novick, 1968). Instead of assuming that all cross-loadings are zero, it might therefore reasonable to assume that the cross-loadings of the items loading on the same factor spread randomly around zero. Thus, the BSS should be of interest, whenever the assumptions of item-sampling hold. If there are strong reasons to expect that the assumptions of item-sampling theory do not hold, other models like BAYES or ICM might be preferred.
Although previous simulation studies already provided important results on the ICM, ESEM, and BAYES (Xiao et al., 2019), the BSS has previously not been systematically compared to BAYES. Moreover, previous simulation studies on the BSS were based on a more complicated procedure using salient loading weights and on only three common factors (Beauducel, 2017). The aim of the present simulation study was therefore to compare the quality of the three model specifications (ICM, BAYES, and BSS) for three different population types. As a further extension to the simulation of Xiao et al. (2019) we provided a population model based on uncorrelated factors. Even when uncorrelated factor models are probably more rarely computed than correlated factor models, the differences between CFAs based on the ICM, BSS, and BAYES should also be investigated for this type of population models. Accordingly, we compared models specified according to the ICM, BSS, and BAYES for predefined population models with uncorrelated factors and with correlated factors corresponding to the ICM as well as for population models with a clear simple structure but with substantial cross-loadings. It is thereby investigated how well a predefined model comprising a set of factors can be identified by means of model specifications according to the ICM, BSS, and BAYES, when substantial cross-loadings occur in the population.

Method
The specified models were evaluated in terms of fit to the data, i.e., whether the reproduced observed correlation matrices are similar to the population observed correlation matrices. This aspect was represented by the standardized root mean squared residual (SRMR). The quality of the models was also evaluated in terms of the root mean squared difference between the loadings estimated for the specified models and the predefined loadings (RMSD(L)) as well as the root mean squared difference between the estimated factor inter-correlations and the predefined factor inter-correlations (RMSD( nd )). The simulations were carried out at both population and sample level. At the sample level, the RMSD(L) and the RMSD( nd ) were calculated for each sample in comparison with the predefined population values. Three types of population models were used in each case (see Table 1), where model type A (Pop A) denotes a predefined ICM, model type B (Pop B) corresponds to a predefined BSS, and model type C (Pop C) has a disturbed simple structure in which the secondary loadings per block and factor do not add up to zero.
The conception of the Bayes model pursued here, in which no secondary loading is fixed, is therefore compatible with all three model types, the buffered model is compatible with type A and type B, and the ICM only corresponds to model type A. It was therefore expected that the Bayesian method would have the best overall properties in terms of data fit (reproduced observed correlation matrices) and estimation of model parameters (loadings and factor inter-correlations). The ICM, on the other hand, should have the worst overall characteristics, since it corresponds to only one of the three model types. The three population types (see Table 1) were realized by means of combinations of two salient loading sizes (l 1 , l 2 ) and one non-salient loading size (nl). The population types were analyzed for orthogonal population models with all non-diagonal elements of the factor inter-correlation matrix being zero ( nd = .0) and for oblique population models with  nd = .4. For each model type, 32 populations were investigated, resulting for model type A from the combination of q ( 3; 5)  p/q  ( 6; 12)  {l 1 , l 2 , nl=0} ( .8,.8,0; .8,.4,0; .6,.6,0; .6,.3,0)   nd ( 0; .4), for model type B by q ( 3; 5)  p/q  ( 6; 12)  {l 1 , l 2 , nl} ( .8,.8,.2; .8,.4,.2; .6,.6,.15; .6,.3,.15)   nd ( 0; .4), and for model type C by q ( 3; 5)  p/q  ( 6; 12)  {l 1 , l 2 , nl} ( .8,.8,.35; .8,.4,.35; .6,.6,.25; .6,.3,.25)   nd ( 0; .4), where q denotes the number of factors, p/q the number of variables with salient loadings per factor, l 1 and l 2 the salient loadings and nl the non-salient loadings, and  nd the uniform correlation of the factors. Note that nl denotes the absolute value of the non-salient loading and is used in models B and C with alternating signs (see Table 1). For the conditions with p/q  12 variables the loading blocks of the smaller models with p/q  6 were doubled. This resulted in 96 conditions at population level.
For the simulation at sample level, 1,000 samples of size n = 200 and n = 600 were drawn from these populations for each condition, resulting in 192 test conditions. Uniform pseudo-random numbers were generated with the Mersenne Twister integrated in SPSS 23 (IBM Corp., 2015), which were converted into multivariate normal distributed variables according to Box and Muller (1958). All factor analyses were performed in Mplus 7.4 (Muthé n & Muthé n, 1998-2015) using maximum likelihood estimation for ICM and BSS and using Bayes estimation for BAYES, the default values for the number of iterations and convergence criteria. For BAYES, the mean of the priors of the cross-loadings was zero and as in Xiao et al. (2019) the variance of the normal distributed cross-loading priors was small, i.e., N(0,0.01). The factor variance were fixed to one and the factor inter-correlations were always freely estimated, i.e. also in the populations with  nd = 0. The Supplementary Material, Appendix A contains exemplary scripts for the conditions shown in Table 1. An example sample data file for the condition q = 3, p/q=6, l 1 =.6, l 2 =.3, nl=.15,  nd = 0, running with the scripts is given in the Supplementary Material (sample.csv).
While the ML estimator used for the ICM and the buffered model almost always converged, the Bayesian algorithm experienced convergence problems, especially with five-factor models. If no convergence has been achieved at population level, the (preliminary) result output by Mplus is reported. The detailed tables in the Supplementary Material, Appendix B show these provisional solutions in brackets. Similarly, the results are bracketed at sample level (c.f. Supplementary Material, Appendix C), when the convergence criterion was not met in more than five percent of the samples. The summary statistics in Appendix B and Appendix C are shown separately for all conditions and for only those conditions in which the Bayesian method has (sufficiently often) converged; this means that BAYES has converged in at least 95 % of the samples. As no significant differences in the comparisons between the models resulted from these separate analyses, all conditions (including the preliminary Bayesian data) are always listed in the following results report.

Results
Univariate analyses of variance were calculated in order to assess the effect sizes of the five independent variables (Population type, Loading size, p, p/q,  nd ) on the dependent variables (SRMR, RMSD(L), and RMSD( nd )) at the population level. Of course, the cells of these analyses contain only one value at population level, so that the error variances are zero and only the effect size measures are of interest. Appendix B (Table B1) lists all the measures of explained variance (² ) as well as the total sums of squares. These analyses show that the variability of the criterion measures was mainly affected by the type of population and the amount of the loadings. Furthermore, the total sums of squares indicate that the variability of the results of ICM was highest overall and that of BSS lowest. The results of the population simulation averaged across q, p/q as well as  nd and broken down by type of population and loading size, are shown in Table 2. The detailed results for q, p/q, and  nd can be found in the Appendix (Table B2 for Pop A, Table B3 for Pop B, Table B4 for Pop C).   Table 1,  nd  factor inter-correlation, ICM  independent clusters model, BAYES = Bayes estimator, BSS  buffered simple structure, SRMR  standardized root mean square residual, RMSD(L)  root mean square deviation between the predefined and estimated loading matrix, RMSD( nd )  RMSD between the predefined and estimated factor inter-correlation matrix; underlined are the best results per criterion; the cells give mean values of 8 test conditions each.
As expected, ICM always had a perfect fit only for the corresponding population type A, but among the two other population types B and C, it showed the worst fit measures compared to BAYES and BSS. Also in line with expectations, BSS consistently showed perfect fit in both population type A and population type B. With regard to the model fit of BAYES, however, two findings were remarkable: Firstly, BAYES did not show a perfect fit for any population type, and secondly, BAYES also showed an overall worse fit than BSS for population type C. It is also noticeable that BAYES was not suitable for determining the predefined factor inter-correlation in population type B. Overall, in the population simulation BSS was one of the best model specification in order to recover the observed correlations (SRMR), factor loadings and factor inter-correlations across population types.
In the sample simulation, univariate variance analyses were also initially calculated to assess the effects of the manipulated variables (see Table C1). Across the various criteria and estimation methods, the variables population type, loading size, and sample size were essential for the variability of the data. Therefore, Figures 1 to 3 show all simulation results as a function of these independent variables.
For population type A the ICM had the best properties for estimating the underlying loading matrices (Figure 2). The factor inter-correlations were similarly well estimated under population type A by all three methods (Figure 3). And BSS showed the best reproduction of the observed correlation matrices (Figure 1). Under population type B, BSS consistently had the best fit values for all criteria. What is noteworthy here is the advantage that BSS has over BAYES. Especially with regard to the estimation of the factor inter-correlation (Figure 3), BSS proved to be superior to BAYES. But even under population type C, which corresponds most closely to the model specifications of BAYES, BSS is at least on a par with BAYES with regard to all criteria. Under type C, BSS achieved the best fit in terms of SRMR ( Figure 1) and RMSD( nd ) ( Figure 3) and a similarly good fit in terms of RMSD(L) as BAYES (Figure 2).    Table 1,  nd  factor inter-correlation, ICM  independent clusters model, BAYES = Bayes estimator, BSS  buffered simple structure, SRMR  standardized root mean square residual, RMSD(L)  root mean square deviation between the predefined and estimated loading matrix, RMSD( nd )  RMSD between the predefined and estimated factor inter-correlation matrix; underlined are the best results per criterion; the cells give mean values of 64 test conditions each.

Discussion
In order to overcome the limitations of the ICM, different methods of CFA model specification/estimation were compared in a simulation study. BAYES estimation/specification was investigated because it has been shown that this method is especially promising in order to overcome the limitations of the ICM (Xiao et al., 2019). Since BAYES needs an a priori specification of the variance of the cross-loadings, it may introduce some variability in prior specification that might hamper the comparison of models. Buffered simple structure (BSS) was therefore considered as a possibility to provide a single testable hypothesis for each factor that might compete with the ICM hypothesis. The ICM, BAYES, and BSS were compared in a population model corresponding to the ICM (Pop A), in a population model corresponding to a buffered simple structure (Pop B), and in a population with a disturbed simple structure with a few substantial cross-loadings per factor that do not add up to zero (Pop C). It was expected that the ICM model specification yields the best model fit and parameter estimation for Pop A, that BSS model specification yields the best model fit and parameter It turned out at the sample level that, although the ICM yields the best loading estimate for Pop A, it did not outperform BAYES and BSS with respect to the factor inter-correlation estimate and model fit. One reason for this unexpected result could be that the specification of zero-cross loadings is suboptimal even when only sampling error introduces some cross-loadings. As expected, BSS yields the best loading estimate, factor inter-correlation estimate and model fit for Pop B. For Pop C BAYES and BSS clearly outperformed the ICM, which was also expected. However, it was not expected that BAYES and BSS yielded similar model fit and had similar overall quality of the model parameter estimates for Pop C. Whereas BAYES yielded slightly superior loading estimates, BSS yielded slightly superior factor inter-correlation estimates for Pop C. Thus, when there is a small number of cross-loadings that do not cancel out and when the focus is on the estimation of the loadings rather than on the estimation of the factor inter-correlations, BAYES should be preferred. Overall, the results indicate that depending on the structure of cross-loadings in the population, different modeling approaches might be appropriate. Therefore, we give no general recommendation against or in favor of the ICM, BAYES and BSS, but the recommendation to explicitly relate the hypotheses on the cross-loadings to model specification. If cross-loadings are solely due to sampling error, the ICM would be appropriate, if they are due to some unknown systematic effect, BAYES would be appropriate, and if the cross-loadings can be regarded as being the result of an item-universe with symmetrically distributed positive and negative cross-loadings, the BSS would be appropriate. In other words, the more a set of items might be regarded as a random sample of items drawn from a population of items measuring the same factor, the more the BSS should be an appropriate model specification.
Especially, when method effects are controlled by means of method factors within the multitrait-multimethod approach (Campbell & Fiske, 1959;Marsh & Bailey, 1991), the residual variances of the items might be considered as being drawn at random from a population of item variances representing the respective factor, so that the expectation of the cross-loadings of the residual items in such models could be zero. This idea, as well as the general relationship of BSS to item-sampling theory (Lord & Novick, 1968) might be further explored in simulation studies.
As in all simulation studies, the limitations of the search space should be acknowledged. First of all, only three and five factor models were investigated although models with a larger number of factors would be of interest. The present study shares this limitation with Xiao et al. (2019) where also a maximum of five factors was investigated. As the computation time for BAYES for five factor models was already substantial, a simulation that goes beyond five factors would need an extremely long computation time. Another limitation is that only a few fixed population loading sizes were investigated. A larger variability of population loadings might reveal further differences between the ICM, BSS, and BAYES. Finally, it should also be acknowledged that only a single, small prior variance was investigated for the cross-loadings. Perhaps, the choice of a very small prior, followed by a sequence of increasing priors (Asparouhov et al., 2015) might have improved further the results for BAYES and might have reduced the number of convergence problems.

Appendix B
.  Table 1,  nd  factor inter-correlation, q  number of factors, p/q  number of variables with salient loadings per factor, l 1 , l 2  salient loadings, nl  non-salient loadings, ICM  independent clusters model, BAYES = Bayes estimator, BSS  buffered simple structure, SRMR  standardized root mean square residual, RMSD(L)  root mean square deviation between the predefined and estimated loading matrix, RMSD()  RMSD between the predefined and estimated factor inter-correlation matrix; in brackets are results for which the convergence criterion was not fulfilled, underlined are the best results per criterion.  Table 1,  nd  factor inter-correlation, q  number of factors, p/q  number of variables with salient loadings per factor, l 1 , l 2  salient loadings, nl  non-salient loadings, ICM  independent clusters model, BAYES = Bayes estimator, BSS  buffered simple structure, SRMR  standardized root mean square residual, RMSD(L)  root mean square deviation between the predefined and estimated loading matrix, RMSD( nd )  RMSD between the predefined and estimated factor inter-correlation matrix; in brackets are results for which the convergence criterion was not fulfilled, underlined are the best results per criterion. Table B4. Results of population simulation for population type C ICM BAYES BSS  q p/q l 1 l 2 nl SRMR RMSD(L) RMSD( nd ) SRMR RMSD(L) RMSD( nd ) SRMR RMSD(L) RMSD ( nd Table 1,  nd  factor inter-correlation, q  number of factors, p/q  number of variables with salient loadings per factor, l 1 , l 2  salient loadings, nl  non-salient loadings, ICM  independent clusters model, BAYES = Bayes estimator, BSS  buffered simple structure, SRMR  standardized root mean square residual, RMSD(L)  root mean square deviation between the predefined and estimated loading matrix, RMSD( nd )  RMSD between the predefined and estimated factor inter-correlation matrix; in brackets are results for which the convergence criterion was not fulfilled, underlined are the best results per criterion.

Copyrights
Copyright for this article is retained by the author(s), with first publication rights granted to the journal.
This is an open-access article distributed under the terms and conditions of the Creative Commons Attribution license (http://creativecommons.org/licenses/by/4.0/).