A Bibliometrics Analysis of Canadian Electrical and Computer Engineering Institutions ( 1996-2006 ) Based on IEEE Journal Publications

This paper presents a bibliometric assessment of Canadian institutions in the discipline of Electrical and Computer Engineering (ECE) from 1996 to 2006. The paper is a first step in identifying the top institutions over a 10-year period in Canada. The rankings are calculated using three metrics: (1) simple count of papers, (2) journal impact factors, and (3) journal impact factor of each institution normalized by its faculty size. The venues/journals considered are 71 flagship IEEE Transaction journals in different areas of ECE which are perceived as the most prestigious venues in the discipline. Using the three metrics, the top-ranked institutions are identified as: the University of Waterloo (by two metrics), and Queen’s University’s (by one metric). Our study also reveals other interesting results, such as: (1) Researchers from the universities of Waterloo and Toronto, combined, authored about a third of all the Canadian papers published in the IEEE Transaction journals during the time period under study (691 of 2,540 papers). (2) Canadian provinces have different levels of ECE research productivity and efficiency, and (3) ECE ranking of the Canadian institutions has similarities and differences versus a recently-published software engineering ranking of the same institutions. While this study is in the context of Canadian ECE institutions, our approach can be easily adapted to rank the institutions of any other nation and/or in any other discipline.


Introduction
Bibliometric assessment of academic and industrial research institutions of a nation, together with their scholars, can help identify the leading organizations and researchers of a country in a given discipline (Bellis, 2009).The benefits of such an assessment include the followings:  It would reveal outstanding institutions and scholars, allowing graduate students and researchers to better choose where they want to study or work.


It can allow employers to recruit the most qualified potential graduate students/researchers, and also industrial firms to choose the most promising academic research teams for collaboration. Such an assessment would assist internal administrators in making influential decisions, e.g., promotions and internal funding.(Higher Education Funding Council for England, 2011).
It is interesting and somewhat surprising that there are only a handful number of bibliometric studies in the context of IEEE community and Electrical and Computer Engineering (ECE) disciplines, e.g., (Dimitrios Katsaros, Vassilios Matsoukas, & Yannis Manolopoulos, 2008;Tsay et al., 2000;Zhang, 2010;Ziegler, 2009).Many universities world-wide have a single department or separate departments in one or both of the Electrical and Computer Engineering (ECE) disciplines.Many universities in Canada have a single combined department named Electrical and Computer Engineering (ECE), e.g., Universities of Waterloo, Toronto, British Columbia, and Alberta.
To the best knowledge of the authors, there exists no bibliometric study which assesses the Canadian ECE institutions.This study is a first step to assess and rank the ECE research productivity in Canada based on publications in a flagship selected list of journals and using three research performance metrics: (1) simple count of papers, (2) impact factor (Garfield, 2005), and (3) impact factor of each department normalized by its faculty size.The flagship journals we have selected for bibliometric analysis are those published by the Institute of Electrical and Electronics Engineers (IEEE), believed by many to be the most prestigious world-wide association in the ECE discipline.
Although this study has been conducted in the context of Canadian researchers and institutions, the framework and approach that we propose can be used to conduct world-wide or other regional studies in the ECE or any other discipline.
In summary, structured based on the Goal, Question, Metric (GQM) methodology (Basili, 1992), the goal of this work is to conduct a bibliometric study in the context of Canadian ECE institutions for the purpose of identifying the top academic and industrial research institutions in the period of [1996][1997][1998][1999][2000][2001][2002][2003][2004][2005][2006], analyze the trends over different years, and also to rank Canadian provinces and their research efficiency (output versus input).Based on the above goal, we raise the following research questions (RQ's) which this study attempts to investigate:  RQ 1-What are the top Canadian ECE institutions? RQ 2-How does institutions' performance change over years? RQ 3-What are the top institutions by ECE sub-areas (e.g., power engineering, wireless communications)? RQ 4-How do the different Canadian provinces rank in terms of their ECE research productivity? RQ 5-What is efficiency of each province in ECE research? RQ 6-How does the ranking of Canadian ECE institutions compare to other related Canadian institutions rankings?
It is important to note, at the outset, that this study focuses on the discipline of ECE, and not, for example, on its related disciplines, e.g., computer science, software engineering, information technology (IT), or information systems.We note that the field of Computer Engineering has slightly different meanings in different parts of the world.We use the IEEE/ACM definition of the field of Computer Engineering (which has also been adapted in many but not all universities): "Computer engineering, also called computer systems engineering, is a discipline that integrates several fields of electrical engineering and computer science required to develop computer systems" (IEEE Computer Society & ACM, 2004).In this definition, computer engineering does not include software engineering, for example.
From another perspective, as per the authors' experience in discussions with colleagues and other researchers, the issue of bibliometric studies can often be the subject of debate and controversy (Narin, Olivastro, & Stevens, 1994), i.e., whether there is any bias involved in a given bibliometric study or metric, and/or the paper pool used for a study.To prevent future debates on the results of our study, and to maximize transparency and replicability of our methods and results, we discuss the details of our methodology first before reporting the bibliometric results.In reviewing the large literature of the existing bibliometric studies (Campbell et al., 2010;Canadian Institutes of Health Research, 2011;Gaughan & Bozeman, 2002;Glass, 1995), it is somewhat surprising that most studies do not include details to assess the transparency and reproducibility of those studies.To address this issue, we have made all the effort to make our study 100% reproducible by discussing our methodology in detail and also by providing the entire data set we have used and generated for our analysis in an online webpage (Garousi, 2011).
The remainder of this article is structured as follows.A brief survey of the related work is presented in Section 0. Our bibliometrics methodology and the bibliometric software tool we have used are discussed in Section 3. Section 4 presents the results (structured based on the six above questions).Finally, Section 5 concludes the article and discusses future work directions.

Related Works
We present a brief survey of the related works in this section based on the following classifications:  Bibliometrics assessment of universities and institutions  Bibliometrics in the ECE context  Bibliometrics in the Canadian context  Bibliometrics in other engineering disciplines

Bibliometrics Assessment of Universities and Institutions
There are various university ranking systems which are either nation-wide or international, such as the US News' ranking of American colleges and universities (US News, 2010a), and the Maclean's magazine's ranking of Canadian colleges and universities (Maclean's Magazine, 2010).Although these studies conducts comprehensive rankings and provide useful and interesting data for public, they are mostly focused on undergraduate level and do not analyze research and publications productivity in depth.
There are a few ranking systems for graduate schools such as the annual "Best Graduate Schools" ranking by the US News (US News, 2010b).By reviewing the criteria on its online website, one can see that this particular ranking considers factors such as research spending per faculty member, number of faculty members, tuition, and PhD student/faculty ratio.However, it is not clear whether this ranking considers the research/publications productivity in rankings, a factor which should be considered for ranking graduate schools.
In more technically-focused areas, there are studies such as (Dimitrios et al., 2008;Moed & Visser, 2007;Tsay, 2011) which assess universities and institutions for a particular technical subject.The work in (Dimitrios et al., 2008), for example, evaluates Greek departments of Electrical Engineering, Computer Science/Engineering using bibliometric indices.

Bibliometrics in the ECE Discipline
By a search in the literature, we found several bibliometric studies in the ECE discipline: (Dimitrios et al., 2008;Lufrano & Staiti, 2009;Tsay et al., 2000;Zhang, 2010;Ziegler, 2009).As discussed above, the study reported in (Dimitrios et al., 2008) evaluated Greek departments of Electrical Engineering, Computer Science and Engineering and also faculty members using bibliometric indices.Among the metric used were: Hirsch's h-index, Egghe's g-index, Sidiropoulos's contemporary h-index, and age-weighted citation rate (Bellis, 2009).
Research by (Ziegler, 2009) is a Master's thesis in the Massachusetts Institute of Technology (MIT), which proposed new methods and software implementation for bibliometric analysis and, as a case study, applied the methods to the area of renewable energy.The work used and developed sophisticated mathematical concepts such as eigenvector centrality to conduct bibliometric analysis such as hit-count extraction, growth-rate analysis and latent semantic analysis.The thesis conducts the bibliometric analysis in the sub-areas of the renewable energy such as: energy technology, bio-fuels, and global warming, and reports their growth rate in the literature.
A bibliometric study of semiconductor literature in term period of 1978-1997 is reported in (Tsay et al., 2000).
The study investigated the growth of semiconductor literature based on the database of INSPEC.Bibliometric techniques, such as Bradford-Zipf's plot and Lotka's law were applied to explore the characteristics of the semiconductor literature.Quantitative results on the literature growth, research treatment, publishing country and language, author productivity and affiliates were reported.Moreover, from the Bradford-Zipf's plot, 25 core journals in semiconductor were identified and analyzed.
Work by (Zhang, 2010) reported a bibliometric analysis in the area of virtual reality (a sub-area in the discipline of computer science/engineering) in the world-wide scale.The study found that the volume of papers on virtual reality in the world was in steady growth during period 1997-2009, and the papers contributed by Chinese researchers had the highest growth rate in recent years.The study also found that the computer science and medicine are the main subject areas of virtual reality, and that the citation peak of a paper was on average 3-4 years after the paper was published.
The article by (Lufrano & Staiti, 2009) is a bibliometric analysis of the international literature in the area of super capacitors (an ECE area).The following metrics were used: number of papers, citations, Hirsh index, and citations per paper.The analysis of aggregate records shows that the number of publications has increased remarkably since 2004.The study attributes the root cause of this trend as follows: in the last few years, there has been a rapid growth in publications coming from China and other Asian countries (Korea, Taiwan and India).
China was found to be the leading country with 459 published papers and five authors on the list of the 30 most prolific scientists per number of publications.However, the Western countries also such as the United States, Canada and Europe, along with France, Poland, Italy and Germany have shown a high level of productivity on the basis of other metric indicators.
In our literature survey, we have also found two bibliometric Canadian ECE-related studies (Dalpé & Longpré, 1995;Dawson, Dalpé, Longpré, & Caron, 1996), however as of this writing we have been unable to have access to full text of these articles.

Bibliometrics in the Canadian Context
Bibliometrics in the context of Canadian research community is an active area.There are many Canada-related bibliometric studies, e.g., (Davignon, Gingras, & Godin, 1998;Godin, 2011; The Committee of the state of science and technology in Canada -Council of Canadian Academies, 2006).There are also Canadian companies specializing in providing service in this area (Science Metrix Inc., 2011).For the example, work by (Godin, 2011) is a bibliometric study of the social sciences in Canada.The article by (Davignon et al., 1998)  In summary, the report confirmed Canada's research strength is.When the bibliometric data are viewed in their entirety, Canada's broad strength in published research was apparent.The report noted that:  For 38 percent of the 125 areas analyzed, both publication quality (ARIF) and intensity (SI) were above the world average.


In only 10 percent of the 125 disciplines were quality and intensity both below the world average.


Almost 70 percent of the 125 disciplines had publication quality ratings above the world average.


In only 11 of the 125 disciplines was publication quality rated at less than 90 percent of the world average From the list of 16 broad S&T areas, areas of "engineering" and "physical, math & computer sciences", in order, ranked 5th and 6th in terms of average strength, correspondingly.
Methodology guidelines for bibliometric studies have also been published in the Canadian context.For example, the work of (Gauthier, 1998), commissioned by Statistics Canada, provides an overview usage of bibliometric methods and techniques, including an extensive bibliography.It also provides technical specifications on the database of Canadian authors that has been developed, with support from Statistics Canada.
To our knowledge, there exists no bibliometric study which assesses the Canadian ECE institutions in depth as it is done by this article.

Bibliometrics in Other Engineering Disciplines
Based on the literature review, except in software engineering, bibliometric studies are not that common in other engineering disciplines (e.g., mechanical civil or chemical engineering).It seems that bibliometrics has not been adapted in engineering disciplines as much as it has been in social science and medicine disciplines.
As discussed in Section 1, the Software Engineering (SE) discipline is an exception.There are numerous bibliometric studies in SE, e.g., (Garousi & Varma, 2010;Glass, 1995;Glass & Chen, 2001;2002;2003;Ren & Taylor, 2007b;Tse, Chen, & Glass, 2006) are just a selection.The SE discipline and community is so mature in this regard that even ranking of scholars on an annual basis is quite usual and there is virtually no controversy in the community around this type of sensitive rankings (i.e., of scholars).The series of annual papers by Glass et al., e.g., (Glass, 1995;Glass & Chen, 2001;2002;Tse et al., 2006), have assessed the systems and software engineering scholars and institutions in a worldwide context since 1996.
We conducted a recent study (Garousi & Varma, 2010) which ranks the Canadian SE scholars and institutions from 1996-2006.This study found Carleton University as the top institutions in the area, and identified Lionel Briand (formerly with Carleton University) and Gail Murphy from UBC the as top-ranked scholars based on either of the two metrics (impact factor and h-index).The current article is actually the continuation of our work from that earlier paper.
Study by (Ren & Taylor, 2007b) and the Java tool they have developed (Ren & Taylor, 2007a) incorporates the impact factors of publication venues and have used those to more precisely calculate the scores for SE and computer science institutions and scholars.We actually use their Java tool (Ren & Taylor, 2007a) to automate the process of rankings in the current article.Details will be discussed in Section 3.4.
Works in (Eckel, 2009) and (Rong-ying et al., 2009) are among the few bibliometric studies in other engineering disciplines.The work in (Eckel, 2009) is a citation analysis of theses and dissertations at Western Michigan University and intends to find the "emerging engineering scholar(s)".The work in (Rong et al., 2009) is a bibliometric study on research competitiveness of engineering universities in China.

Methodology
To prevent future debate on the results of our study, and to maximize transparency and replicability of our methods and results, we discuss the details of our bibliometric methodology in this section, before reporting the results in Section 4. The following discussions are presented next:  An overview (Section 0)  Data source (Section 0)  Article search criteria and choosing the time-period under analysis (Section 0)  Bibliometric software tool (Section 0)  Score criteria and ranking parameters (Section 0)

An Overview
We adapt a typical bibliometric approach.To conduct our publication-based ranking, we first extracted the entire pool of papers written by Canadian ECE researchers.To score publications and venues, we then selected a group of publication venues that are considered prestigious, representative, and influential for the ECE discipline.
Based on the impact factor of each venue, a score is assigned to each paper an institution or an author has published, and finally institutions are ranked using sums of the scores.
An overview of our bibliometric methodology is shown as an activity diagram in Figure 1.Details of each step are discussed in the next sections.
Note that, in addition to the ranking of institutions as output, the bibliometric software tool we used (the Ranking tool which will be discussed in Section 3.4) also generates the ranking of authors (researchers).However, according to informal discussions with several collogues in the Canadian ECE community, it seems that since bibliometric studies and rankings are not common in the ECE discipline, ranking of ECE researchers is usually considered a sensitive issue.Thus, to prevent potential controversies, although we had generated the Canadian ECE authors ranking as well, we are not reporting its results in this article.
Figure 1.An overview of our bibliometric assessment methodology
For more than 40 years, the Institute for Scientific Information (ISI, now part of Thomson Reuters) produced the only available bibliographic databases from which bibliometric analysts could compile large-scale bibliometric indicators.ISI's citation indexes, now regrouped under the Web of Science (WoS), were the major sources of bibliometric data until 2004, when Scopus was launched by the publisher Reed Elsevier.For those who perform bibliometric analyses and comparisons of countries, scholar or institutions, the existence of these two major databases raises the important question of the comparability and stability of statistics obtained from different data sources.
There are studies such as (Archambault et al., 2009;Gorraiz & Schloegl, 2008), which compare the two databases WoS and Scopus in terms of the comparability and stability of the data extracted from the two of them.
The study in (Archambault et al., 2009) used macro-level bibliometric indicators to compare results obtained from the WoS and Scopus.It showed that the correlations between the measures obtained with both databases for the number of papers and the number of citations received by countries, as well as for their ranks, are extremely high (R 2 ≈0.99).There is also a very high correlation when papers are broken down by field.The paper provided evidence that indicators of scientific production and citations are stable and largely independent of the database.
The study concluded that "the two databases offer robust tools for measuring science" (Archambault et al., 2009).
The study by (Gorraiz & Schloegl, 2008) compared the two databases on a focus scientific area (i.e., pharmacology and pharmacy journals) and revealed the same conclusion as (Archambault et al., 2009).
Since Scopus provided easier user interface to query the data we were looking for in this study, we chose Scopus as our data source for the publications.Scopus is owned by Elsevier and, as of late 2011, covers over 19,000 titles (journals and conference proceedings) from more than 5,000 international publishers in all areas of science.
It has been launched in November 2004 and, according to (Elsevier, 2011), it is now the largest abstract and citation database containing both peer-reviewed research literature and quality web sources.It has 45.5 million records, including 4.6 million conference papers.

Article Search Criteria and Time-Period Under Analysis
For our bibliometric study, the first step was to extract the pool of papers published by the Canadian ECE researchers from the huge 45.5 million-record database of Scopus.Our search (query) method for this purpose is visualized in Figure 2.
To extract from the pool of papers published by the Canadian ECE researchers, in the Scopus search interface, we entered the word "Canada" in the field "Affiliation Country" and the phrase "Electrical and Computer Engineering" in the field "Affiliation".To ensure accuracy of the paper pool extracted from Scopus, we took several steps.As the first step, we verified the venues of the papers.In the Scopus search result page, venues are referred to as "Source Titles" (the top ranked ones are shown in Figure 3).We randomly verified to see if the pool contains each and every venue from the set all well-known ECE venues (IEEE journals and conference proceedings).The answer was yes.

Figure 3. Source titles in the Scopus results page
To further validate the accuracy of the papers in our pool, we considered two well-known accuracy metrics from the information retrieval and data-mining literature: precision and recall (Witten, Frank, & Hall, 2011).Precision is the fraction of retrieved instances that are relevant, while recall is the fraction of relevant instances that are retrieved.
We did not have to worry about precision since, according to the search query that we used, all the papers in our pool were published by the Canadian ECE researchers.
To assess recall, our measure was to count the number of relevant papers actually published by the Canadian ECE researchers that are present in our data pool.Obviously, it would have been tedious to do this task manually for all the papers.To keep our workload manageable, we randomly selected a manageable number of papers (around 200) from the Canadian ECE researchers' websites, or their online CV's.We then verified whether the generated pool of papers included those 200 papers.All the 200 papers, except two of them, were in the pool.Those two papers were published in venues not indexed by the ISI or Scopus.Thus, it was not really a miss in terms of the recall metric.This yielded the recall rate of our data set to be 100%.
With the recall rate of 100%, we gained confidence in the accuracy of our data set and paper pool and we thus continued the study using the pool of 26,581 papers.For readers interested in replicating the ranking of this study, we would like to remind that all the data we have used and generated in this study are available as text files in an online webpage (Garousi, 2011).
In terms of the publications time-period under analysis, although the study was conducted in 2009 and 2010, the analysis period was set to 1996-2006 due to the reasons discussed next.In many online sources such as Scopus, it usually takes a few months or even years until all the publications have been entered into the database.As per our manual investigation of a few known publications, 2006 showed to be a suitable year-end setting and buffer time for the papers to be included in our study.
The publications time-period is a feature supported by the bibliometric software tool we used (next section).Thus, we did not have to supply it in the Scopus search.

Bibliometric Software Tool
As it was discussed in Section 2, bibliometric studies are quite popular in the software engineering discipline and not in the ECE disciplines.Similar to several related studies such as in (Garousi & Varma, 2010;Ren & Taylor, 2007b), to perform the rankings based on the above large set of almost 26,500 papers, we used the Java software application developed by Ren and Taylor at the University of California Irvine which is available online as open-source (Ren & Taylor, 2007a).
The tool does not have a particular name and we refer to it as just the Ranking tool in this article.It is a free tool and has a simple interface.The guidelines on how to use it are provided online in (Ren & Taylor, 2007a).As a brief overview of its features used in this article, its menus, and how to create a new ranking project in this tool are shown in Figure 4.
The steps that should be taken (and we took) using the Ranking tool to generate the rankings in this article are explained as a simple step-by-step algorithm in the following: 1 Create a new ranking using the "New Ranking…" menu in Figure 4-a and setting the parameters in the window that comes up (Figure 4-d) as follows: 1.1 Specify ranking type: Institutions or Persons (Scholars): We chose "Institutions".
1.2 Specify Score Distribution: The five possible options are shown in Figure 4-d.The score may optionally be given to only the first institution/author, each participating institution/author (so each institution or author will receive a score for each paper it published), evenly distributed among each institution/author, distributed using the scheme proposed by Journal of Systems and Software, or distributed according to some distribution.In the last case, an input string like "0.5, 0.4, 0.3" in the input box give the first author 50% of the score, the second author 40%, and the third author 30%.If there are more than three authors for one reference, each following author will get the same percentage as the last listed percentage.The total of the percentages does not have to be 100%.We chose the option 2 since it is a popular choice and has been used in other studies such as (Garousi & Varma, 2010;Ren & Taylor, 2007b).

2
Set the options for the ranking using the menus in Figure 4-b.Details for the options are discussed in Section 3.5.

3
Load the papers pool using the "Add References" menu in Figure 4-a.We loaded the text file containing the pool of 26,581 papers into the tool.

4
Do the actual ranking and provide the results using the "Rank" menu in Figure 4-a and setting the parameters in the window that comes up, as Figure 4-c

Score Criteria and Ranking Parameters
To assess research output and productivity, there are a variety of metrics in the bibliometric community: (1) the simplest one is the just the number of papers published in any venue by an author, (2) impact factor (citation count) (Garfield, 2005), and (3) h-index (Hirsch, 2005).
While many bibliometric papers, e.g., (Glass & Chen, 2003, 2005), have used the above first metric (i.e., count of papers) due to its simplicity, many researchers such as (Parnas, 2007) and decision makers (e.g., funding agencies) question it.For example, David Parnas (Parnas, 2007) who is a well-known Canadian researcher who is one of the early pioneers of software engineering, published a prolific article called "Stop the Numbers Game" (Parnas, 2007), where he advocated for stopping counting the numbers of papers published by researchers.He believes that "counting papers slows the rate of scientific progress".Among other drawbacks in such an approach, he also believes that: "It [counting papers] encourages superficial research", "It encourages repetition", "It encourages small, insignificant studies", and "It rewards publication of half-baked ideas".
Instead of just counting the number of papers, many studies such as the one by (Ren & Taylor, 2007b) assign a score to each published paper, e.g., based on the venue's weight (journal impact factor values) published by (Thomson, 2011a).The work of (Ren & Taylor, 2007b) incorporated the impact factors of publication venues and used those to calculate the scores for software engineering institutions and scholars in a world-wide scale.We also followed the same scoring criteria in our recent paper (Garousi & Varma, 2010).
Impact factor (Garfield, 2005) and h-index (Hirsch, 2005) can be measured for different types of entities: e.g., for a journal, scholar, institution or even country.The impact factor of a journal is usually calculated as follows.In a given year, the impact factor of a journal is the average number of citations to those papers that were published during the two preceding years.For example, the 2003 impact factor of a journal would be calculated as follows: The h-index metric is defined as follows.The h-index of a scholar is based on the set of a scientist's most cited papers and the number of citations that they have received in other people's publications.A scholar with an index of h has published h papers each of which has been cited by others at least h times.Thus, the h-index reflects both the number of publications and the number of citations per publication.The h-index is designed to improve upon simpler measures such as the total number of citations or publications.
The Ranking tool supports assigning impact factor values to the journals in its analysis.This is supported in the tool using a feature called "Score Criteria", shown in the menu of Figure 4-b.To do the ranking we had to select a list of journals and then feed their list plus their impact factor values to the Ranking tool.
The other decision we had to make in this study was the list of journals we wanted to consider.The Institute of Electrical and Electronics Engineers (IEEE) is generally considered the most prestigious flagship association for practitioners and researchers in the ECE discipline.The IEEE website reports (IEEE, 2011) that the IEEE produces 30 percent of the world's literature in the electrical and computer engineering fields, publishing well over 100 peer-reviewed journals and transactions.Many people in the ECE community agree that the IEEE journals are the most prestigious venues to publish ECE-related research.
From the list of those 100+ IEEE journals, we selected, as per our experience, 71 journals which had a core focus in various areas of ECE.For example, we excluded the IEEE Transactions on Software Engineering, IEEE Transactions on Professional Communications and IEEE Transactions on Knowledge and Data Engineering since they were more focused on software and systems areas.We have provided in our online page (Garousi, 2011) the complete list of the 71 ECE-related "IEEE Transactions" journals that we selected including their 2006 impact factor values.The impact factors were retrieved from the online Journal Citation Reports (JCR) tool provided by (Thomson Reuters, 2011a), which is a widely accepted source for impact factors.
We have listed in Table 1 the top 5 and the bottom 2 of the 71 "IEEE Transactions" journals as ranked by their year 2006 impact factor values.The list of all the 71 IEEE Transactions and their impact factor values were stored in a text file (available in our online page) which was then fed into the Ranking tool.The other tuning necessary when using the Ranking tool is the institution aliases.The data from Scopus are not that "clean" as more than one name (exact-text) may be used by different papers for an institution, e.g., we noticed both "Calgary Uni." and "University of Calgary" in the data we exported from Scopus.Once fed into the Ranking tool, this would result in wrong calculations and the tool counts the records as two different institutions!To accurately rank the institutions, all aliases of an institution should be explicitly fed into the tool so each paper/institution can be correctly accredited.
Using a proper list of institution aliases, the Ranking tool will merge known institution aliases under a single, proper name.We created a text file of all possible institution aliases and short names and used the Ranking tool menu (Figure 4-b) to feed into it.The file is also part of our online data file for this project.By several try-and-investigate iterations, we were able to finalize the file so that calculated rankings were all valid and every institution was actually counted as only one institution.

Results
Results of the bibliometric assessment are presented in this section, based on the list of RQ's raised in Section 1.

RQ 1-Top Institutions
To rank the top institutions, we use three approaches in the Ranking tool: (1) By calculating absolute score value for each institution, which is the summation of scores (journal impact factors) of all papers published by researchers working in that institution (in Section 0) (2) Normalizing the total score calculated in approach #1 by considering the number of faculty members in each institution (in Section 0) (3) Using the simplest bibliometric which is just counting the number of papers (in Section 0) Note that an "institution" in this context is a Canadian ECE department, a company or a governmental agency working in the ECE discipline.Also, we present an exclusive discussion on top Canadian corporations and governmental agencies contributing to the ECE research in Section 0.  in the 27 th place, and an industrial firm (Neil Squire Foundation based in Burnaby, BC), in the 28 th place.We will have a separate discussion in Section 0 about top Canadian ECE-related corporations' ranking in our assessment.As the second approach to rank institutions, it is a logical idea to normalize the total score calculated in approach #1 above by considering the number of faculty members in each institution.It is obvious that larger Institutions (those with more faculty members) will have the chance to publish more in the IEEE Transactions.Our intent in considering the number of faculty members in the ranking is to assess the balanced normalized share of institutions scores for each faculty member.
Our approach to extract the faculty size of each Canadian ECE department was as follows.The most precise data for this purpose would have been to have access to historical size of each ECE department during years 1996-2006.Authors attempted to query those statistics from two Canadian ECE departments.However, unfortunately they did not receive positive replies to those requests.The statistics could have been manually extracted by following the CVs of each faculty member in each ECE department to analyze their start dates, etc.However, on the other hand, it would be quite impossible to know the number of academics who have left a given ECE department in that period, without explicit external knowledge.If this entire data gathering was to be done without the cooperation of ECE department heads, this would turned out to be a very complex and tedious manual process.Thus, the authors had no choice but to come up with a work-around approach for this issue.The best alternative (but of course, less precise) approach they could think of was to review each Canadian ECE department's website as of this writing (November 2011) and extract the number of faculty members in that department.The simplifying assumption behind this less-precise approach is to suppose that all the Canadian ECE departments have had similar growth rates for their faculty members' number between 1996-2006, and thus the 2011 faculty size data could be a representative-enough indicator.
For the top 10 Canadian ECE departments based on the total publication score, the X-Y plot in Figure 7 visualizes the faculty size of each department versus its cumulative score.The solid line shows the trend (regression) on the 10 points.The dotted line connects the base point to the point corresponding the Queen's University's ECE department which has the highest slope among all points.It will be discussed in the following.The two metrics are somewhat correlated (the correlation R 2 = 0.91).We ranked the institutions by their cumulative score and extracted the top 10.We then calculated the above metrics for each institution.The ranking of the top 10 nations according to the above metric is shown in Figure 8.For easier comparison with ranking in Section 4.1.1,the ranks of that section are shown inside parentheses in Figure 8.
In terms of the faculty-size-normalized score, Queen's University's ECE department ranks first.With only 25 faculty members, this department has the cumulative paper score of 228.3 (and normalized score of 228.3/25=9.132 in Figure 8).By cumulative paper score, the department ranks 5 th , but in the normalized score, it ranks 1 st .University of Toronto's ECE department ranks the lowest in the top 10 in the normalized score.We also wanted to use the simplest bibliometric which is just counting the number of papers.Using the Ranking tool, this was quite easy to do.We set the impact factor of all IEEE Transactions to 1 instead of their actual values as discussed in Section 0. Results are show in Figure 10.The university names have been shorted for brevity, e.g., Waterloo instead of University of Waterloo.
There is no major change in the ranking, however, some institutions rank lower or higher in this simplest metric.For example, University of Alberta and University of British Columbia have switched (between 3 rd and 4 th ranks).
One root cause might be that almost all institutions publish in, more or less, all the IEE Transactions and thus the overall rank does not change that much with or without considering journal impact factors.
It is interesting to see that researchers from the universities of Waterloo and Toronto, combined, authored about a third of all the Canadian papers published in the IEEE Transaction journals during the time period under study (691 of 2,540 papers).Similarly, when combined, researchers from the top 5 universities (shown in Fgiure 10), authored more than half of all the papers in the pool (1,300 of 2,540 papers).

Top Canadian Corporations and Governmental Agencies Contributing to the ECE Research
Note that the affiliation of an author in a paper could be (1) an academic institution, (2) a governmental agency, or (3) a corporation.Based on the author information of the papers published in IEEE Transactions and entered in our paper pool file, the Ranking tool ranks the affiliations accordingly.As we observed in Figure 6, the top-30 list included a governmental research agency (the Communications Research Centre based in Ottawa) in the 27 th place, and an industrial firm (Neil Squire Foundation based in Burnaby, BC), in the 28 th place.
It is somewhat expect to see that almost all of the top ranks are taken by academic institutions and further that, governmental agencies and corporations are appearing towards the bottom of the list.We traversed the ranking to extract the list of top 10 Canadian governmental agencies and corporations who are contributing to the ECE research through publications in IEEE Transactions.Figure 9 shows the data, where we both present the rank of each institution among governmental agencies and corporations only, and among all institutions, e.g., General Electric Canada ranks #4 and #34, respectively.For brevity, we are not showing the academic institutions falling in between these ranks.

RQ 2-Variations in Top Institutions' Contribution over Years
It is expected that even top ranking institutions' research performance/contribution level vary over years.Variations of cumulative scores of top 5 institutions (as extracted in Figure 6) over years 1996-2006 are visualized in Figure 11.Note that the data are readily available from running the Ranking tool (see the screen shot in Figure 5).The corresponding box-plot of the data is shown in Figure 12.The most notable observations from these variations are discussed next. The general trend in cumulative impact factors values is an up-ward trend, meaning that all the top 5 institutions have been producing higher quantity and quality of publications (recall the impact factor of the journals under study) over the 10-year period.
 The yearly ranks of some institutions have stayed almost stable over the years, but the ranks of some others have changed.For example, University of Waterloo is the rank #1 in 6 of the 11 years under study.University of Toronto or UBC have ranked higher than UW in 5 of the 11 years.
 McMaster was the 5 th in the early years of this period, while it has ranked slightly higher in some of the years (e.g., 3 rd in 2004).The root-cause analysis of these types of rank changes would need careful analysis of various potential influential factors, e.g., (1) changes (additions) of the research personnel, and (2) major changes in research funding.The same trend is visible for the UofA.The latter might be also due to the aggressive rise of Alberta's economy after the new millennium, which has led to substantial increase in provincial research funding and also faculty hiring in Alberta.
 The highest single-year research output is by the Universities of Waterloo and Toronto in the year 2004.It is somewhat hard to explain why the research output has slightly decreased since 2004.

RQ 3-Top Institutions Ranking by ECE Sub-Areas
Thanks to the flexible and powerful design of the Ranking tool, it is easy to get a ranking of institutions for each sub-area of the ECE discipline (e.g., power engineering, information theory, biomedical engineering, and communications).More precisely, this can be configured through the input list of the journal names (IEEE Transactions in our case) and the "Score Criteria" feature of the tool (shown in Figure 4-b).
As two examples, we report in this article how the institutions ranking can be conducted for two example ECE sub-areas: power engineering, and information theory.The ranking can be easily done for any other ECE sub-area.
To rank the Canadian institutions by their publications in the area of power engineering, we identified the IEEE Transactions which relate to the power engineering field and extracted their impact factor values from the Journal Citation Reports (JCR) tool.Similar to Section 3.5, the following information was saved in a text file available in our online page (Garousi, 2011)  We then executed the Ranking tool again and the ranking was generated (see the snapshot in Figure 13).For the area of information theory, we identified the IEEE Transactions on Information Theory (with impact factor of 2.728) as the only IEEE Transaction journal relating to this area, and configured the Ranking tool similarly.
Top 15 institutions based on cumulative score in the areas of power engineering and information theory are shown in Figure 14 and Figure 15, respectively.

RQ 4-Ranking by Provinces
Based on the geographic location of each institution, we can also rank the research productivity of Canadian provinces and territories.For brevity, we report the score aggregations for provinces based on the approach/scores presented in Section 4.1.1(absolute impact-factor score values).The aggregation approach can easily be replicated using any other biblio-metric as well.Results are shown in Figure 16 as a bubble chart, where the size of the bubble denotes the number of universities from each province in the top 30 list as extracted in Section 4.1.1.The province of Ontario (ON) has 14 universities in the top 30 list and is the clear leader in terms of cumulative score.The province of British Columbia (BC) has 4 universities in the top 30 and is the 2 nd .Quebec (QC) and Alberta (AB) are the 3 rd and 4 th .The remaining provinces: Manitoba (MB), Nova Scotia (NS), Saskatchewan (SK), and New Brunswick (NB) rank lower.Newfoundland (NF) has a few universities, but none among the top 30 (Section 4.1.1).The Canadian territories have no public universities and thus not in the list.
In terms of the cumulative impact-factor score, the province of Ontario (ON) with 14 universities has scored roughly only twice of the British Columbia (BC) with only 4 universities (score of 2,062 versus 936).This might denote that the ECE research in ON is perhaps more spread compared to BC (in which it is perhaps more centralized), i.e., 14 versus 4 universities.Quebec (QC)'s bubble is larger than BC's.This might mean that with more universities in the top 30 (6 of them), QC has scored lower than the BC (with only 4 universities).

RQ 5-Analysis of Provincial Research Efficiency: Considering NSERC Grants as Input
One would expect that the research productivity of a region might relate to the amount of research grants of the researchers in that region, and their also industrial connections.If the amount of research grants in the area of ECE can be estimated for those cases, the relationship between the research grants (as input) and cumulative impact factors (as output) can be analyzed.
One such open data source for the amount of research grants in the Canadian context is available online through the (Natural Sciences and Engineering Research Council of Canada (NSERC), 2010).To search for the ECE-related grants in that database, we searched for the list of NSERC grants from [1996][1997][1998][1999][2000][2001][2002][2003][2004][2005][2006]  The scatter plot of provinces' research productivity (output) versus NSERC grants and scholarships (input) for the period 1996-2006 is shown in Figure 17.Note that the acronyms for four province names with the lowest amount of grants were discussed above in Section 4.4.In our data collection phase, there was absolutely no ECE paper (research output) from any of the four Canadian territories.For this reason, the four territories are not shown in this scatter plot. Ontario (the most populous Canadian province) has received the major portion of the NSERC funding in the ECE area with an amount which is greater than the 2 nd and 3 rd rank amounts added together (Quebec and British Columbia).
 The total research support for the ECE researchers in the bottom five provinces (Manitoba, NF, NB, NS and SK) together is almost equal to the support for British Columbian researchers alone.It is interesting that the summation of research productivity in those five provinces (score=155) is also almost equal to the British Columbian researchers (score=156).This denotes an almost equal efficiency in research publications in those cases.
 A regression trend line has been super-imposed on the scatter plot.Three provinces (Ontario, British Columbia and Alberta) are above this trend line, denoting their research efficiency is more than the national average (i.e., the trend line).On the other hand, researchers from Quebec and four other provinces seem to have a lower efficiency than the national average.The root-cause analysis of this particular phenomenon would need careful analysis of various influential factors, e.g., perhaps, it is that researchers from Quebec are publishing more in other venues than the IEEE journals, e.g., those in French language.Quebec is receiving more than twice from NSERC grants compared to British Columbia, but Quebecois researchers are publishing fewer papers in the IEEE Transactions (output).

Comparison to the Canadian Software Engineering Ranking
We had recently published a similar study (Garousi & Varma, 2010) where we reported a bibliometric assessment of Canadian software engineering scholars and institutions (1996)(1997)(1998)(1999)(2000)(2001)(2002)(2003)(2004)(2005)(2006).As a background terminology definition of the discipline of Software Engineering (SE), we should note that it is the application of a systematic, disciplined, quantifiable approach to the development, operation, and maintenance of software, and the study of these approaches; that is, the application of engineering to software.SE is usually considered a multi-disciplinary field among ECE, computer science and few other fields (e.g., economics and industrial engineering).
Figure 18 shows a comparison of the ECE rankings (from Section 4.1.1)with the SE rankings (adapted from (Garousi & Varma, 2010)).For better understanding, cross-rankings and scores have been explicitly mentioned and also visually shown (using arrows).There is a mix of trends.Some universities have similar rankings in both, but some have very different rankings.For example, University of Waterloo has ranked 1 st in the ECE, but 3 rd in the SE.Carleton University has ranked 1 st in SE, but 23 rd in the ECE.Both the guide and the rankings issue feature articles discussing Canadian universities and ranking them by order of quality.The rankings focus on taking a measure of the "undergraduate experience", comparing universities in three peer groupings: (1) primarily undergraduate, (2) comprehensive, and (3) medical doctoral.The guide defines each of the categories as follows:  Primarily undergraduate universities are largely focused on undergraduate education with relatively fewer graduate programs and graduate students.


Comprehensive universities have a significant degree of research activity and a wide range of programs at the undergraduate and graduate levels, including professional degrees.


Medical doctoral universities offer a broad range of Ph.D. programs and have medical schools.
Thus, if we want to compare the ECE rankings generated in our study, the most natural category will be the "comprehensive universities".Figure 19 shows the comparison of the ECE rankings (from Section 4.1.1)with the Maclean's University Ranking Guide, adapted from (Maclean's Magazine, 2010).For better understanding, cross-rankings have again been explicitly mentioned.Since the online Magazine's ranking only provides the top 10 list for free on its website, we did not have access to the list of universities below rank #10.Thus, we could not extract the exact Maclean's ranks for 8 of the top 10 ECE rankings and have labeled them as "10+".
In this outset, we should add that according to (CBC News, 2006), for the November 2006 University Rankings issue, 11 Canadian universities refused to provide information directly to Maclean's.To rank those universities, the magazine relied on data it collected itself, as well as data drawn from third party sources such as Statistics Canada.Among the universities that refused to provide information directly to Maclean's in the fall of 2006 were some of the top schools in our ECE ranking: University of British Columbia, University of Toronto, Dalhousie University, McMaster University, University of Alberta, and University of Calgary.Thus, the rankings might not be as precise as they could be.
Compared to the SE ranking (Figure 18), the comparison of ECE with Magazine's ranking (Figure 19) is not that correlated as only two of the top 10 universities appear in both list, i.e., University of Waterloo and University of Victoria.

Limitations and Threats to Validity
By referring to the design of empirical studies (Mitchell & Jolley, 2001), the following limitations and potential threats to validity are applicable for this study: internal, construct, and external validity.We addressed internal validity and reduced (selection) bias in two ways.Firstly, as discussed in Section 3.2, we chose the Scopus instead of other data sources such as IEEEXplore, ACM Digital Library, and Google Scholar since accordingly to our evaluation, Scopus has a more complete pool of ECE papers compared to the other sources.Secondly, to further validate the accuracy of the papers in our pool, we used the recall metric (Witten et al., 2011), a well-known accuracy metric from the information retrieval and data-mining literature (Section 3.3).
Construct validity denotes the extent to which what was to be measured was actually measured.In our context, our goal was to conduct a bibliometric assessment of Canadian ECE department.In addition to using the simplest bibliometric metrics, i.e., number of papers, for this approach (in Section 4.1.3),we used a more widely-accepted metric (i.e., impact factors) for ranking.
In terms of external validity, we believe the above steps that we have taken will help also increase the external validity of our results.Since reliable publicly-available data from Scopus were used, since our entire data set is provided online (Garousi, 2011), and also since our paper pool had high level of precision, we believe that it is possible to replicate and also generalize our results.

Conclusions and Future Works
This paper presented a bibliometric assessment of publications by the Canadian researchers in the field of Electrical and Computer Engineering (ECE) from 1996 to 2006, based on publications in IEEE Transactions journals.The survey identified the top institutions over a 10-year period.
The rankings were calculated based on the impact factor of about 27,000 papers published in 71 flag-ship IEEE journals in different areas of electrical and computer engineering.Using three metrics and approaches (Section 4.1.1-4.1.3),the top-ranked institutions were the University of Waterloo (by two metrics), and Queen's University's (by one metric).
Assessments such as the one reported in this paper can assist both internal and external administrators and funding agencies in making influential decisions, e.g., promotion.Although this study has been conducted in the context of Canadian researchers and institutions, the framework and approach that we propose can be used to conduct world-wide or other regional studies.We are planning to repeat the rankings in the upcoming years.

Figure 2 .
Figure 2. Our search method in Scopus: search keywords (top) and search results (bottom) Menus of the Ranking tool(Ren & Taylor, 2007a) and creating a new ranking project in this tool the number of times articles published in 2001 and 2002 were cited by indexed journals during 2003  B = the total number of "citable items" published in 2001 and 2002.("Citable items" are usually articles, reviews, proceedings, or notes; not editorials or Letters-to-the-Editor.) 2003 impact factor = A/B 4.1.1Top Institutions (by Absolute Score Values)To rank the Canadian ECE institutions by absolute paper score values, we executed the Ranking tool as discussed in Section 3.4-3.5.A file containing the pool of 26,581 papers, IEEE Transaction journal's score criteria and institutions alias files were fed into the tool and we then asked the tool to provide the ranking.A snapshot of the tool's output is shown in Figure5.The tool shows the institutions in the rank order together with cumulative scores values of their ECE papers.Once each institution is selected, its scores for each year under study and also the detailed list of its IEEE publications are shown.

Figure 5 .
Figure 5.A snapshot of the Ranking tool's (Ren & Taylor, 2007a) output ranking institutions based on cumulative score values

Figure 6 .
Figure 6.Top 30 institutions based on cumulative impact factors

Figure 7
Figure 7. X-Y plot of the Canadian top-10 ECE departments' faculty size versus their cumulative scores of publications

Figure 8 .
Figure 8. Top 10 institutions based on normalization of cumulative scores by faculty sizes

Figure 10 .
Figure 10.Top 30 institutions based on by simply the "number" of IEEE Transactions papers

Figure 11 .Figure 12
Figure 11.Variations of cumulative impact factors of top 5 institutions

Figure 13 .Figure 14 .Figure 15 .
Figure 13.A snapshot of the Ranking tool's (Ren & Taylor, 2007a) output ranking institutions based on cumulative score for papers in the area of power engineering

Figure 16 .
Figure 16.Ranking of provinces based on cumulative scores (as a bubble chart)

Figure 17 .
Figure 17.Scatter plot of provinces' research productivity (output) versus NSERC grants and scholarships (input): 1996-2006 studies the flows of knowledge across different disciplines in Canada as measured by bibliometrics.

Table 1 .
Top five and the last two of the 71 "IEEE Transactions" journals, ranked by their year 2006 impact factor values and was fed into the Ranking tool: Although the Canadian ECE researchers receive other research funds and grants than only from NSERC grants, we can assume that the ratios of all research funds to NSERC grants across all institutions are almost equal.Therefore, we conduct a comparative analysis of provinces' research productivity versus NSERC grants in the following.
Canadian weekly news magazine, reporting on Canadian issues such as politics, pop culture, and current events.The Maclean's Guide to Canadian Universities (Maclean's Magazine, 2010) is published annually in March.It is also known as Maclean's University Guide.The Guide includes information from the Maclean's University Rankings, an issue of the magazine proper that is published annually in November, primarily for students in their last year of high school and entering their first year in Canadian universities.The 21 st series of the ranking was published in 2011.