Quick Links

Evaluating Knowledge Production Systems in Security Studies and Health Sciences: Citation Network Analysis

McGill Sociological Review, Volume 4 (February 2014): 19-43

Alla G. Khadka, 

Johanna E. Steenrod, 

Jeffrey L. Carasiti, 

Abigail R. Stark, 

Megan Thomas, 

Jonas Gamso,

University of Pittsburgh

Abstract: Production of knowledge is a complex and political process. At the same time, there is no universally accepted tool that allows us to evaluate research systems across different disciplines. To remedy this gap, we develop a generalizable normative index based on Campbell’s epistemologically relevant internalist sociology of science and Gibbons et al. (1994) model of knowledge production and use it to evaluate research production systems in health sciences and security studies. Each system is elicited through a citation network analysis method used to capture all the publications on a specific issue that dominates a given field as an evolving network where ties among publications represent connections between authors, organizations and journals. The generated citation networks are analyzed on the subject of transdisciplinarity, heterogeneity and stakeholder representativeness. Analyzing knowledge production systems as networks allows us to measure how properties of the network affect the quality of the scholarship in different research systems. This, in turn enables us to better understand strengths and weaknesses of the system and inform the research agenda moving forward.

Key Words: knowledge production ; sociology of knowledge ; network analysis ; bibliometrics ; primary care; rural health ; Afghanistan ; counterinsurgency

 

Résumé: La production de la connaissance est un processus complexe et politique. En même temps, il n’y a pas d’outil universellement accepté qui nous permette d’évaluer les systèmes de connaissance de façon transversale et interdisciplinaire. Afin de combler cette lacune, nous développons un index normatif et généralisable basé sur la pertinence épistémologique de la conception internaliste de la sociologie de la science de Campbell et sur le modèle de production de la connaissance de Gibbons et al. (1994). Nous utilisons cet index pour évaluer les systèmes de production de connaissance dans les sciences de la santé et de la sécurité. Chaque système est abordé à l’aide d’une méthode d’analyse de réseaux de citations. Cette méthode vise à saisir l’ensemble des publications portant sur un enjeu spécifique dominant un champ donné comme un réseau en constante évolution, dont les liens entre les publications représentent les connections entre les auteurs, les organisations et les revues scientifiques. Les réseaux de citations ainsi générés sont analysés sous l’angle de la transdisciplinarité, de l’hétérogénéité et de la représentativité des parties prenantes du système de connaissance. L’analyse des systèmes de production de connaissance en tant que réseaux permet de mesurer comment les caractéristiques du réseau affectent la qualité du savoir dans différents systèmes de connaissance. Cela a pour effet de favoriser une meilleure compréhension des forces et des faiblesses du système et d’informer un futur agenda de recherche.

Mots-clés: production de connaissance ; sociologie de la connaissance ; analyse de réseau ; bibliométrie ; soins primaires ; santé rurale ; contre-insurrection

Introduction

In a knowledge society, such as the United States, each epistemic community is expected to produce valid, practically applicable and socially accountable research. Accepted validity and reliability checking mechanisms exist to establish the soundness of research at the construct level of an individual study (Cook and Campbell 1979; Shadish, Cook, and Campbell 2002). However, there is no mechanism that enables us to analyze and compare the properties of knowledge production systems. Assessing the structure and ensuing properties of knowledge production systems across disciplines is instrumental for any scholar concerned with the production and validity of social science research.

Drawing on the sociology of science (Merton 1968), the sociology of scientific knowledge (Barnes 2012: 24-25; Bloor 1976) and the evidence-based research literature, we argue that both individually constructed social realities and external social conditions shape scientific enterprises and knowledge production. Furthermore, the structure of this knowledge production can be empirically studied to draw conclusions about the quality and validity of the emergent research. We utilize Campbell’s conception of a disputatious community, comprised of researchers from multiple overlapping disciplines open to multiple perspectives and rival hypotheses, to characterize a high quality knowledge production system (Campbell 1985: 40). To define a disputatious community, we synthesize Campbell’s sociology of scientific validity with the knowledge evaluation literature to identify three indicator constructs: heterogeneity, multidisciplinarity, and diversity of sites.

The aim of the study is to develop a Knowledge Production Evaluation Index for assessing specific issue areas against the normative model of knowledge production across the three constructs (Porter et al. 2007; Gibbons et al. 1994). We introduce Citation Network Analysis (CNA) as a method for eliciting and mapping the knowledge production system over time with the system defined as a pool of all publications on a specific topic. We employ CNA and social network analysis centrality measures as a basis for operationalizing heterogeneity, multidisciplinarity and diversity of sites. We then create the Knowledge Production Evaluation Index where each construct is assigned a measurement scale. While the evaluation instrument developed in this study is generalizable to any knowledge production system in social sciences, this framework is applied to evaluate the knowledge system encompassing U.S. security strategy on Afghanistan and access to primary care services in rural U.S.

Theoretical Foundation

Knowledge societies expect that policymakers make decisions based on evidence-based research, and as such, a scholarly community has formed to evaluate knowledge creation and validity. More specifically, the sociology of knowledge focuses on all practices of knowledge creation in society: the process of the construction of social reality by individuals and collectives, the social structure that affects cognitive maps of individuals and cultures, and the interactions that exist between mental maps of individuals and the understanding of social reality. Constructivists focus on the meanings of knowledge to individuals and entire cultures (Mannheim 1952/1993, 1953; Berger and Luckman 1967), while functionalists investigate the structures and functions of knowledge systems (Parsons 1951, 1961; Holzner and Marx 1979; Dunn, Holzner et al. 1982). Constructivism, in its most basic terms, suggests that individuals produce society, which then becomes an objective and reified reality where the individual is a social product of his/her own creation. While this constructed social reality is maintained by techniques such as intimidation, propaganda, mystification, or symbolic actions, an individual has power to alter this reality. Therefore, if a knowledge structure generates poor quality knowledge, we can change this by altering our construction of social reality.

In contrast to the constructivist paradigm, proponents of functionalism view social organizations as a function of their constituent elements, operationalized as norms, customs, traditions, and institutions. According to Talcott Parsons (1951, 1961), to understand the social organization of science, it is imperative to assess each practice that affects the functioning of the system in the context of the society as a whole. While fundamentally agreeing with Parsons’ theory, Merton (1968) viewed the Parsonian theory of functional unity as overgeneralized and maintained that societies are too complex to be assessed in their entirety. In response to Parson Merton proposed the Middle Range Theory, suggesting the importance of an appropriate level of analysis without being excessively universal or abstract. Merton acknowledged the differences among individuals that comprise a society allowing the possibility for structural and functional alternatives to the institutions and structures currently fulfilling the functions of society (Merton 1968).

David Bloor and Barry Barnes criticized Merton’s approach for excluding the organization of science. Heading the sociology of scientific knowledge (SSK) movement, Bloor, Barnes, and their colleagues at the Science Studies Unit at the University of Edinburgh (the “Edinburgh School”) developed an approach that utilizes sociological explanations of scientific ideas. The aim of this approach is to investigate empirically how social conditions (psychological, social, and cultural) influence scientific beliefs. The main principle of SSK is that knowledge, including scientific knowledge, is a product of collectives and their shared cultural tradition. The distribution and use of knowledge within these collectives is socially structured, and, therefore, its use and application is socially contingent (Barnes, 2012: 24-25; Bloor, 1976).

The key difference between Merton’s sociology of science and Bloor’s sociology of scientific knowledge is often characterized as the “internalist/externalist” debate. Internalism views scientific knowledge as a product of cognitive and social factors regarded as “internal” to science (e.g., effects of organizational incentives) with an emphasis on the norms of modern science, such as disinterestedness and universalism. Externalism (Durkheim 1895/1938/1964; Hessen 1931; Barnes and Dolby 1970; Bloor 1976, 1981; Latour & Woolgar 1979), on the other hand, regards the development of science in terms of factors “external” to science, such as the structure of the larger society and the social interests and religious affiliations of scientists (Schmaus 1994: 18). Merton (1957; 1973) accepted the influence of external factors on science, but he maintained that while researchers may be inspired by extra-scientific factors, ultimately the researcher’s interests are driven by the internal history of the science in question.

Together the sociology of science and the sociology of scientific knowledge offer a comprehensive set of frameworks that allow us to investigate processes of knowledge production either by examining how individuals construct a social reality that then creates the structure of science or by examining the effect of social conditions on our beliefs about knowledge that shape the organization of scientific enterprises. By contrast, evidence-based research literature provides us with a methodology to evaluate the causal relevance of programs and policies to specific outcomes. To  our knowledge, no study has combined the two fields—evidence-based policy analysis and the sociology of knowledge—in order to investigate empirically the effects of social structures and processes on variations in the quality of evidence employed in policy analysis and program evaluation. Although earlier work by Bernstein and Freeman (1975) attempted to link an index of research quality to factors, such as the centralization of scientific and professional research centers, their work was not informed by the sociology of knowledge. As society grows more complex, policy-makers’ demand for high quality research that meets the “what works” standard increases. This creates a need for theory-guided empirical research on the effects of varying knowledge structures on the quality of evidence-based policy analysis.

In response to this need, at a 1987 conference at the Maxwell School at Syracuse University, Donald T. Campbell proposed a research program calling for an Epistemologically Relevant Internalist Sociology of Science (ERISS), which he and others named simply the sociology of scientific validity. Campbell’s sociology of science goes beyond description and seeks to explain how differences in the social structures and processes of sciences affect the degree of validity achieved by these sciences (Campbell 1986, 1987). The objective is to develop a social theory of how science works by looking at factors internal to science and how these factors correlate with improving scientific beliefs (Campbell 1985: 38).

The list of normative factors delineated by Campbell includes: rival hypotheses, overlapping research agendas, interdisciplinarity and the presence of independent decision makers. Advocating overlapping research agendas, Campbell calls for overlap between centers and diversity within agendas (Campbell 1987: 399-400). It is desirable to see vigorous debate and scholarship around fewer topics instead of a system with many topics but very little deliberation. The applied social sciences are, social system-wise, spread too thin and thus lack the critical mass to sustain mutually reinforcing, validity enhancing scholarly traditions focused on specific problems (Campbell 1987: 400-401). The diversity of decision makers is also critical because a system characterized by many decision makers has access to different funding sources, allowing the knowledge production system to increase rather than decrease the development of overlapping problems and expertise (Campbell 1987: 401; Campbell 1986). Finally, a normative knowledge production system must be interdisciplinary because much of social science research is intrinsically interdisciplinary. Ideally, scientists with interdisciplinary competence can draw on relevant aspects of several disciplines or several institutionalized and mutually isolated specialties within a single discipline (Campbell 1987: 409).

Campbell’s epistemologically relevant internalist sociology of science provides a theoretical platform for developing a model to evaluate current research practices. The objective of this article is to take Campbell’s concept of a disputations community and its characteristics, stated by Campbell in general terms, and convert these terms into operationalizable indicators. We view a disputatious community as a knowledge production system defined by heterogeneity, multidisciplinarity and a diversity of sites. Based on a review of available knowledge evaluation models, including Triple Helix (Etzkowitz and Leydesdorff 2000), academic capitalism (Slaughter and Leslie 1997), post-academic science (Ziman 2000), Mode 2 knowledge production (Gibbons et al. 1994; Nowotny et al. 2001),  post-normal science (Funtowicz and Ravetz 1993) and strategic research/strategic science (Irvine and Martin 1984; Rip 2004), current models support the normative standard that policy research must consider multiple heterogeneous perspectives. Moreover, in policy sciences the ability of a knowledge system to incorporate these perspectives is positively correlated with the quality of research the system produces (Allison 1971; George 1972). This literature suggests that to assess the extent to which a knowledge system corresponds with a disputatious community is to evaluate it for these three characteristics (Gibbons et al. 1994; Porter et al. 2007; Delanty 2001).

To define a knowledge system in general, we considered several definitions. For instance, it can be defined as an entire stock of knowledge generated by an academic field (Estabrooks et al. 2008; Dachs et al. 2001). Considering, however, the time and complexity required to collect and analyze all knowledge produced by a field, such as health care services, we confine a knowledge production system to an approximately complete collection of publications on an issue area representative of a selected academic field. At this system level, heterogeneity is represented by presence of independent research generating clusters (epistemic communities, research groups, etc.) linked to the core of the system by a bridge or author that connects otherwise detached communities. To be characterized as heterogeneous, clusters must remain in the form of structural holes or groups that are independent from the network and connected to it by one bridge (Burt 1992, 2004, 2007). Once clusters are internalized, the independent perspectives they represent cease to be alternative.

Multidisciplinarity characterizes a system where the production of knowledge entails mobilization of a range of theoretical perspectives and practical methodologies (Porter et al. 2007: 117; Hessels and van Lente 2008: 741). As social, political and economic environments become increasingly complex, practitioners and policymakers face issues that demand equally complex solutions, requiring expertise from a variety of disciplines. Therefore, to produce sound research, the system must incorporate expertize coming from different disciplines. Academic disciplines refer to a particular branch of learn­ing or body of knowledge, such as physics, psychology, sociology, or history (Moran 2010: 2). The term multidisciplinarity traditionally measures degree of collaboration among different disciplines. In this study, however, we use multidisciplinarity to capture presence of not only disciplines but also subject areas. Subject areas encompass research areas centered on a specific subject matter, defining a relatively coherent body of expertise and of research strategies (ex: Post‐genomics or Bioterrorism research) (De Rycke 2009: 14).

Diversity of sites refers to a system that is diverse in terms of the skills and experience of individuals engaged in knowledge creation. It is marked by an increase in the number of organizations or sites where high quality knowledge is created. It describes a process where not only academic institutions are participating in knowledge production, but also other organizations, such as think-tanks and NGOs, each bringing unique resources and connections to relevant stakeholders. These organizations are linked in a variety of ways through networks of communication (Gibbons et al. 1994: 6). The diversity of sites construct is further supported by a community of scholars asserting that valid research in strategic sciences must be adequately representative of all relevant stakeholders, including non-scientific actors (Hessles and van Lente 2008: 742). Funtowicz and Ravetz (1993) contend that issue-driven research necessitates engaging stakeholders in the quality assessment of scientific knowledge production to define problems, extend the knowledge base and provide quality assurance.

We propose heterogeneity, multidisciplinarity and diversity of sites as normative indicators for assessing the reliability and validity of research produced by knowledge systems and use the next two sections to convert them into an evaluation methodology.

Method: Citation Network Analysis

Citation Network Analysis (CNA) is used to elicit and analyze knowledge systems. CNA research is rooted in bibliometrics, an approach that employs citation data and quantitative analysis to map out published literature and evaluate the patterns of publication within a field. This approach evaluates scholarly fields as structures over time, using techniques such as co-citation, co-word, and author co-citation analyses (Estabrooks et al. 2008). The roots of “bibliometrics” extend from Alfred Lotka and his 1926 publication on frequency distribution of scientific productivity in chemistry and Samuel C. Bradford’s 1934 study on the frequency distribution of papers over journals. Eugene Garfield introduced citation analysis to bibliometrics in 1955 as a method to study the history of science (Garfield 1955). Further advancing the field, in 1960 Garfield created the Science Citation Index (SCI), laying the foundation for the application of citation indexing in science, technology, and humanities. Building upon concepts introduced by Garfield, Derek J. de Solla Price analyzed the system of science communication in his book Little Science – Big Science (1963), introducing the first systematic approach to the structure of modern science applied to science as a whole. In 1964, writing on the use of citation analysis in studying the history of DNA, Garfield and Sher first employed the concept of a citation network. Following this work in 1965, de Solla Price described the inherent linking characteristic of the SCI as “Networks of Scientific Papers.” 

Table 1. Citation Network Analysis Data

Attribute

Description

Citers (Citing Publications)

 

 

 

 

     Author(s)

Author(s) that published on a selected topic

     Title

Title of publication

     Source

Venue a paper is published in (name of the journal, newspaper, conference where the paper was published etc.)

     Year

Date  work  is  published  (necessary  for  temporal  analysis)

Citees (Cited Publications)

 

     Author(s)

All the authors each selected paper is citing

     Source

All the publication sources each selected paper is citing

     Year

Date cited  work  is  published

It is important to note that the primary focus of bibliometrics is the specific author or paper measurements, such as publication counts or the number of co-authors or the number of citations received by a set of publications. In citation network analysis, however, the focus is on the network, and bibliometric parameters of individual authors or papers are assessed in the context of the network. Drawing on Social Network Analysis (SNA), CNA uses centrality measurements to assess individual papers in the context of the network. For example, to assess direct citation networks, CNA uses in-degree centrality to rank all the cited papers from top cited to the least cited. Three types of citation networks can be created: direct citation network, co-occurrence citation network, and co-citation network (Belter 2012; Weingart et al. 2010). The primary focus in this study is direct citation networks.

To conduct CNA, we gather all the relevant publications on selected issue area from Scopus and Web of Science databases (see Table 1 for the type of information we collect) via a keyword search method. The data for our first case study, security publications on Afghanistan, was gathered from the Web of Science and Scopus databases because these two sources encompass a comprehensive list of security and social science journals and conferences. The Web of Science combines the three ISI databases SCI expanded (an SCI edition with broader coverage), the SSCI and the A&HCI in a unique on-line database. The SCIE includes over 5,900 journals and the ACHI covers more than 1,100 journals fully and about 7,000 journals selectively (Glänzel 2003: 16). For our second case study, rural access to primary care, we performed a search in PubMed and Scopus databases because they include clinical, health sciences, and social sciences journals. Finally, both Scopus and the Web of Science allowed us to collect direct citations relational data by linking publications with their references.[1]

Once the data is gathered, we review all publications’ titles or abstracts, as necessary, and filter them for relevancy based on an inclusion criterion developed for each issue area. Next, we export the relevant publications and their corresponding references into an Excel spreadsheet. The data is organized such that each citing paper (author, title, source, year) is matched with each of the references that it cites (author, source, year). After the data is cleaned and standardized, we process it through ORA to create two types of citation networks: an author-author network and a source-source network. Together these networks comprise the knowledge production system that we evaluate.

We analyze both networks on two levels: structural level and individual level (invisible college). At the structural level, we examine both the overall static network and its year-by-year evolution for clusters (groups connected to the network by one author) to assess heterogeneity. Next, we use Social Network Analysis (SNA) centrality measures on the overall static network to determine the most central authors and sources that make up the invisible college, defined here as a network of experts (Crane 1972; Miller and Mills 2009). The invisible college is evaluated in terms of the range of organizations (diversity of sites) and disciplines (multidisciplinarity) engaged in research production. The objective is to determine the extent to which the system is open to multiple perspectives at the level of the top experts. The ability to incorporate multiple perspectives and competing points of view is yet another indicator of a disputatious community, viewed as an inherent characteristic of a knowledge production system producing high quality research.

Knowledge Production System Evaluation Instrument

The instrument developed in this study is a three-point Knowledge Production Evaluation Index, consisting of three indicators – heterogeneity, multidisciplinarity and diversity of sites, each capturing one of the properties of the normative knowledge production system, shown in Table 2. Each of the indicators is measured on a rating scale, consisting of three categories: low, medium, and high. In line with standard procedure, we create categories that are well-defined, mutually exclusive, univocal and exhaustive (Guilford, 1965). In this sense, the three categories form a progression and exhaust the underlying variable.

Evaluating Knowledge Production Systems

The index is designed to evaluate any knowledge system in applied social sciences. In this study the model is employed to assess the quality of research produced in security studies and health sciences. There are varying definitions of a knowledge production system. For instance, it can be defined as the entire stock of knowledge generated by an academic field (Estabrooks et al. 2008; Dachs et al. 2001). Considering that it would be extensively time consuming to collect and analyze all knowledge produced by an entire field and the objective here is to create a manageable and generalizable instrument, a knowledge production system is confined to an approximately complete collection of publications on an issue area within an academic field. 

Table 2: Knowledge Production System Evaluation Instrument

Indicators

Citation Network

Scale

System Level

Heterogeneity

Author-author and source-source networks: number of clusters overall and over time

Low (≤ 3 clusters);

Med (4-6 clusters);

High (7-10 clusters)

Invisible Colleges (Top 20 Influencers by In-Degree Centrality)

Multidisciplinarity

Author-author network: range of disciplines measured by current affiliations of authors

Low (≤ 5 disciplines);

Med (6-10 disciplines);

High (≥ 11 disciplines)

Source-source network: range of disciplines measured by the sources’ disciplinary focus

Low (≤ 5 disciplines);

Med (6-10 disciplines);

High (≥ 11 disciplines)

Diversity of Sites

Author-author network: regional concentration by U.S. census regions (9)

Low (≤ 3 regions);

Med (4-6 regions);

High (7-9 regions)

Author-author network: number of countries in which the authors are located

Low (≤ 5 countries);

Med (6-10 countries);

High (≥ 11 countries)

Source-source network: number of countries in which the journals are located

Low (≤ 5 countries);

Med (6-10 countries);

High (≥ 11 countries)

Author-author network: types of institutions (e.g., academic, government, think tanks, non-profits, military etc.)

Low (≤ 4 organizations);

Med (5-6 organizations);

High (≥ 7 organizations)

Knowledge Production System 1: American Security Strategy in Afghanistan, 2001 to 2012

The first knowledge production system assessed in this study is comprised of the publications on the U.S. security strategy in Afghanistan from 2001 to 2012, one of the main current research trends in security studies field. Triggered by the September 11, 2001 terrorist attacks, Afghanistan became one of the principal focuses of U.S. foreign and national security policy. The response to the attacks transformed Afghanistan and its counterinsurgency, previously backwaters in U.S. foreign and defense policy, into a national priority. America’s war in Afghanistan has become protracted and costly and has spawned a body of literature, which is broad in both range and depth. Given the importance of this issue for the field of security studies and the cost for the U.S., it is critical to ensure that the research produced on Afghanistan is high quality, rigorous and valid. This study evaluates publications on American security strategy in Afghanistan from the initial ground war that began in October 2001 to the present post surge strategic environment of 2012. There have been several recent studies that evaluate terrorism research (Anwar and Al-Daihani 2011; Gordon 2011; Miller and Mills 2009; Reid and Chen 2007; Lum 2006) however, no work has reviewed scholarship published on U.S. security strategy in Afghanistan or evaluated it in terms of the three characteristics in our instrument. 

Data

For Afghanistan case study, all relevant publications from October 2001 through 2012 were collected. A keyword search method was employed to extract data from the Web of Science and Scopus. The results of the search are captured in a diagram below:

Figure 1. Literature Review Results


As Figure 1 illustrates, there are a total of 173 citing papers and 5,061 cited publications. Approximately 2,500 entries in the dataset fall under the category of journals or newspapers.  The dataset also contains websites, magazines and publications by government entities. Notably, the number of books and book chapters is low.

System Level of Analysis: Structural Properties

To assess the knowledge production system for heterogeneity, first the static author-author and source-source direct citation networks are examined (see Figure 2 and Figure 3). The output shows two highly uniform systems. Instead of being composed of several independent clusters linked to the core by one bridge, as prescribed by the normative standard, both author-author and source-source networks when viewed statically are characterized as an integrated epistemic community where papers cited more than once become an integral part of the mainstream research. Applying the Knowledge Production System Evaluation Instrument scale, the knowledge production system gets a “low” score for presence of clusters. The score denotes that by the heterogeneity parameter, the system fails to fully satisfy the requirements of being a disputatious community. This does not mean that the system is not open to multiple perspective or alternative points of view, but it does indicate that there are no independent scholarly communities that develop their own approach critical of the view accepted by the core. Instead, multiple perspectives (a property of the system that will be assessed more thoroughly in the next section) may come from individuals as opposed to groups. 

Figure 2. Author-Author Citation Networks over Time

Pendant Nodes and Isolates removed; Triangles denote Cited Authors; Circles Denote Citing Authors.

When the author-author and source-source networks are viewed dynamically, there are some years when the networks are not as uniform as in the case of static networks. Given that the year by year networks for both authors and sources are similar in terms of their structure and evolution over time, to avoid the redundancy we will discuss in detail only author-author dynamic network. From 2001 to 2006 the network does not have a core. Instead, it represents connected and disconnected clusters of authors and their references, which was expected for a research area that has been conceived only recently. In 2007, we notice the emergence of a core and one cluster connected to the core by Seth Jones (see Figure 4). Two primary contributors to the core are the citing authors that are seen in the network three times - Seth Jones (2006 to 2008) and A. Saikal (2001, 2006 and 2010). From 2009 onwards the network turns into a well-connected web and by 2012, although the network becomes less populated, it still remains in the form of one cluster (see Figure 2).

Considering the size of network over time, the author-author network in 2001 is sparse, which is expected because the inception point for all research on the U.S. counterinsurgency strategy on Afghanistan is October 2001. We then observe the system gradually growing into a bigger network of publications with a substantial jump in the number of cited and citing authors in 2007. The network becomes more populated between 2009 and 2011, reaching its peak in 2011. In 2012 the network’s size decreases. This occurs because the data for this case study was collected in October 2012 and does include the entirety of 2012. A decrease may also mean that the system is ready for re-evaluation and alternative approaches. In other words, the knowledge production system has a potential to emerge as a disputatious community, characterized by a core and independent clusters of scholars that are actively engaged with the core, challenging the mainstream approach

Invisible College Level of Analysis: Top 20 Influencers

For both networks the top influential citing authors and journals amount to twenty one by a natural cut-off point. In the case of the cited authors, we include all authors that have been cited nine times or more and in the case of journals we include those that have been cited ten times or more. First, the top cited authors are analyzed, as captured in Table 3.

Table 3.  Author-Author Network: Top Cited Authors

Rank

Citee

Times Cited

Rank

Citee

Times Cited

1

D. Kilcullen

21

12

United Nations

10

2

A. Rashid

21

13

T. E. Rick

10

3

S. Biddle

14

14

E. Schmitt

10

4

B. R. Rubin

14

15

P. Baker

10

5

A. Giustozzi

13

16

P. Bergen

9

6

S. Jones

13

17

R. Chandraskaran

9

7

G. W. Bush

12

18

J. Dobbins

9

8

A. Cordesman

12

19

D. Filkins

9

9

J. Nagl

12

20

C. Gall

9

10

B. Woodward

12

21

B. Hoffman

9

11

D. Galula

11

 

 

 

 

To be considered a system that produces valid research, publications that come from the invisible college and dictate policy in Afghanistan are expected to represent an array or authors and sources. Meeting the normative standard with regards to the diversity of sites criteria requires that the list of most cited authors and sources includes at least some scholarship from stakeholder countries, such as Afghanistan, other Middle Eastern countries and the NATO states. Furthermore, the top influencers are expected to be diverse in terms of organizations they represent, encompassing military complex, private sector, government and universities. The authors are categorized based on their current affiliations as well as the posts they held during the past five years, so that each author is attributed with up to three affiliations. The breakdown of organizations can be summarizes as follows:

  • University (8 total): Johns Hopkins University, George Washington University, New York University, London School of Economics, U.S. Naval Academy, Georgetown University, Harvard University, Fordham University
  • Newspaper/Magazine (6 total): Daily Telegraph, Washington Post, The New York Times, New York Times Magazine, CNN, The New Yorker
  • Government (4 total): Department of State, Department of Defense, President, Ambassador, Pentagon correspondent, White House correspondent
  • Think tank (3 total): Center for a New American Security, RAND, New America Foundation
  • Military (2 total): Department of Defense, U.S. Naval Academy
  • Consultancy firm (1 total): Caerus Associates
  • Intelligence community (1 total)
  • United Nations (1 total)

Altogether, the invisible college of top cited authors is comprised of eight different types of organizations, including the Department of Defense and U.S. Naval Academy as both government and military and the United Nations as its own category. While the system is dominated by universities and newspapers, there is a very strong presence of government. Most of the authors affiliated with universities hold or held a consultancy position within the government. Similarly, several journalists are in close connection with the government, holding positions such as Pentagon correspondent or White House correspondent. This means that even though the system ranks as “high” on the diversity of organizations scale, some of the organizations are very tightly linked, exerting profound influence on each other. However, the system has organizations such as the United Nations and RAND, which are reputed to produce objective research. There is also intelligence community present, which is known to maintain its own unique perspective.

In addition to performing well on the diversity of organizations indicator, the invisible college of top cited authors is sufficiently varied in terms of countries where the experts are based. The experts are mainly from and currently live in four countries, U.S., U.K., Pakistan and France. There are three additional countries where the U.S. journalists currently reside or resided in the past five years: Iraq, Afghanistan and Pakistan. As such, the system is characterized by a total of seven countries, which places it in the “medium” category. Assessing the top influencers, the system encompasses a range of subject areas with a total of eight disciplines, also reaching a “medium” score on the scale. The range of the disciplines is as follows: security studies/theory, military strategy, investigative journalism, foreign policy, political science, counterterrorism, national security and counterinsurgency. While these are separate areas of research, they do not constitute independent disciplines as far as epistemology and ontology is concerned. All of these research domains, with the exception of investigative journalism, can be grouped under the umbrella of political science. Hence the system can be deemed as somewhat multidisciplinary but with a caveat that all the different research areas are related.

Next we examine the source-source network. The top sources that were cited ten times or more are included in Table 4.

Table 4.  Source-Source Network: Top Cited Sources

Rank

Citee

Times Cited

Rank

Citee

Times Cited

1

New York Times

38

12

Times

13

2

Washington Post

29

13

New Yorker

13

3

Foreign Affairs

27

14

Communication

12

4

LA Times

17

15

International Security

12

5

Guardian

16

16

Military Review

12

6

BBC

16

17

RAND

12

7

Survival

16

18

Newsweek

11

8

Time

14

19

RUSI

10

9

Foreign Policy

14

20

American Pol. Sci. Rev.

10

10

Parameters

13

21

Washington Times

10

11

Wall Street Journal

13

 

 

 

The sources are categorized according to eight different categories:

  • Newspaper/public broadcasting (8 total): NY Times, Washington Post, LA Times, Time, Wall Street Journal, Times, Newsweek, Washington Times
  • Magazine on international relations (2 total): Foreign Affairs, Foreign Policy
  • Peer-reviewed academic journal (4 total): Survival, International Security Journal, RUSI, American Political Science Review
  • U.S. Army professional journal (2 total): Parameters, Military Review Journal
  • Magazine (2 total): Guardian, New Yorker
  • Communication (1 total): Bin Laden's Former Bodyguard
  • Public broadcasting (1 total): BBC
  • Think tank (1 total): RAND

In evaluating the source-source network we observe that there is some variety, although the system appears to be much more homogeneous then author-author network. In terms of multidisciplinarity we have four subject areas (foreign affairs, security, political science, military strategy), indicating that the level is “low.”  As in the case with author-author network, all of the subject areas present among the top cited sources are related to security. This is expected, however, since top cited sources represent the core of publications on national security. One of the points of concerns here is that one would expect at least one venue to be from the Middle East since these publications inform U.S. counterinsurgency strategy in Afghanistan.

The system performs better on the diversity of sites indicator. There are a total of six organizations present: public broadcasting/newspaper, academia, military complex, think tank, government (via communication); private sector (via communication and newspapers). As such, the system is ranked as “medium.” In rating the system for a range of countries in which the publishing venues are located, there are only two countries – the U.S. and the U.K. At the same time, it is known that newspapers, such as the New York Times and Washington Post, have reporters that reside in the Middle East. Considering that three of the top cited authors are U.S. journalists that live or have lived in Iraq (Rajiv Chandrasekaran, Dexter Filkins), Afghanistan (Dexter Filkins, Carlotta Gall), and  Pakistan (Dexter Filkins) while writing about one of this three areas the system could be considered to include up to five countries. It is still places the system in a “low” category.

When viewed as a whole, the invisible college comprised of top cited authors and sources ranks “high” to “medium” on the range of knowledge producing organizations scale. This indicates that the system approximates a normative criterion, incorporating most of the stakeholder organizations. Considering scholarship from stakeholder countries, the system ranks “low” to “medium.” Afghanistan and two other Middle Eastern countries are present, however, NATO is represented only by the U.K. The system performs the worst in regards to the multidisciplinarity indicator. There are a number of research areas that feed the system, but they are all closely knit. If the system is to become a truly disputations community and produce high quality research, the core of the system should be extended to connect to independent clusters of scholars that evaluate existing research and offer new approaches and insights to the issue. 

Knowledge Production System 2: Rural Access to Primary Care in the U.S. from1996 to 2012

The second case study uses the Knowledge Production System Evaluation Instrument to evaluate research in health sciences. Considering the breadth of the health sciences field, the focus is on the topic of the provision of primary care health services to rural populations in the United States. Primary care is construed as the main point of contact between an individual and the health system where clinicians provide a wide variety of services from screening through monitoring across a large range of physical and mental health conditions. The World Health Organization, Healthy People 2020, and the United States Institute of Medicine agree that access to primary care is of critical importance for the health of populations and many practice and policy interventions have occurred to promote access. Rural populations, which account for 20% of the United States population, have a continuing shortage of primary care providers: half the per capita amount compared to urban populations. Consequently, the health status of rural populations is significantly worse than their urban and suburban counterparts. This topic’s stock of knowledge, collected from all publications from 1996 to present, enables us to evaluate the knowledge production system over time. Systematic reviews have studied aspects of this topic; however, no work to our knowledge has studied scholarship or knowledge production in this area. Given the persistence of this issue and the national shortage of primary care physicians overall, it is of particular interest for research to explore innovative approaches.

Data

The data for the second knowledge production system includes all publications on rural access to primary care in the U.S. from 1996 through 2012. Using the same methodology as the first knowledge production system, we used a keyword search to identify publications in PubMed and Scopus and extracted citations in Scopus. 

Figure 3. Literature Review Results

As presented in Figure 3, a total of 1,210 papers cited a total of 23,012 publications. The majority of the cited publications were peer-reviewed journals (16,479) followed by paper (3,191), books (1,833), websites (1,268) and others. Already the sheer volume of this knowledge production system illustrates that it is a far more established system than the previous example and is far more concentrated in the academic literature, which has implications for the system level and invisible college analyses.

System Level of Analysis: Structural Properties

The analysis of heterogeneity in the knowledge production system begins with the direct citation networks of the authors and sources (Figure 6, Figure 7). Both of these networks are large networks with no discernible clusters separated from the core by a bridge. Notably, the author-author network does have twenty three isolated authors and many offshoots of one author, suggesting that there are some independent perspectives, but they come from individuals rather than groups. Both of these networks receive a “low” ranking for presence of clusters, suggesting that their characterization as a disputatious community, when measured in terms of heterogeneity, is questionable. The author network is visibly more interconnected, suggesting that the structure is rigid and unwelcoming to rival hypotheses as the connections grow tighter among top actors while the source network remains more loosely structured.

Next we consider cross-sections of these networks over time to understand heterogeneity more dynamically. Figure 4 shows the author-author network in 1996, 2001, 2006 and 2011. We exclude the source-source progression of networks because they are similar structurally to the author-author networks. 

Figure 4. Author-Author Citation Networks over Time

Pendant Nodes and Isolates Removed; Circles Denote Citing Sources; Triangles Denote Cited Sources.

Author names excluded, top influential authors included in the subsequent analysis

None of the network representations appear to have any significant clusters in any of the years, furthering the “lowheterogeneity by structure. However the author networks increase in number of isolated authors and maintain offshoot patterns, particularly the ones circled in 2011, indicating that independent authors and small groups of authors may form rival hypotheses and ideas relative to the core network.

Table 5.  Author-Author Network: Top Cited Authors

Rank

Citee

Times Cited

Rank

Citee

Times Cited

1

Center for Disease Control and Prevention

211

11

U.S. Dept. of Agriculture

73

2

U.S. Dept. of Health and Human Services

205

12

U.S. Public Health Service

69

3

Health Resources and Services Administration

195

13

Agency for Healthcare Research and Quality

68

4

Donald E. Pathman, MD, MPH

153

14

David Hartley, PhD

62

5

Howard K. Rabinowitz, MD

152

15

U.S. Government Accountability Office

61

6

Roger A. Rosenblatt, MD, MPH

141

16

Thomas C. Rosenthal, MD

60

7

Thomas C. Ricketts III, PhD, MPH

120

17

National Institutes of Health

60

8

Institute of Medicine

80

18

Stephen S. Mick, PhD, FACHE

59

9

Kathryn Rost, PhD

77

19

Ronald C. Kessler, PhD

58

10

Ronald M. Andersen, PhD

74

20

Lu Ann Aday, PhD

 

Invisible College Level of Analysis: Top 20 Influencers

For both the author-author and source-source networks, the most influential actors for the whole network period from 1996 to 2012 were assessed. For both networks we analyzed multidisciplinarity, and for the author-author network, we analyzed the diversity of sites. Beginning with the author-author network, Table 5 displays the top twenty most cited authors elicited based on their in-degree centrality. First, the authors’ primary discipline is determined to gauge the degree of multidisciplinarity in the system. Many of the agencies that comprise the knowledge system under discussion, such as the Department of Health and Human Services, are multidisciplinary by nature. To mitigate generalization, we analyzed the top cited articles and their focus to determine which disciplines characterize the core of the system and whether this core is uniform or diverse.

The top three disciplines in the invisible college are family medicine, health policy and management, and health services. The periphery authors represent the disciplines of epidemiology, mental health, sociology, and health disparities research. Altogether the network shows “mediummultidisciplinarity with seven disciplines represented. It should be noted that all of these fields fall within the categories of health, however, health policy and management, epidemiology, and sociology fall outside of the field of medicine, indicating that the system is multidisciplinary. We see no influence of the business community or political science, insofar that they are not represented by health policy and management. Additionally, the top influencers within the network do not adequately represent professions outside of medicine.

Next the diversity of sites of the top cited authors is measured to determine if they reflect both geographic and organizational type diversity. The authors represent five census regions with a strong dominance by the South Atlantic region at over 70% of citations. This region includes government agencies in Washington, DC and Maryland, as well as, the Cecil G. Sheps Center for Health Services Research at the University of North Carolina, which has a long standing program dedicated to rural health research. The second region is the Pacific, home of the University of Washington WWAMI Rural Health Research Center, which has also been working for several decades to address rural workforce concerns. Overall, the network is on the low end of “medium” for regional diversity of sites.

Table 6.  Source-Source Network: Top Cited Sources

 

Rank

Citee

Times Cited

Rank

Citee

Times Cited

1

Journal of Rural Health

1,067

11

Health Resources and Services Administration

250

2

Journal of American Medical Association

828

12

U.S.  Bureau of the Census

236

3

American Journal of Public Health

751

13

Social Science and Medicine

235

4

Medical Care

471

14

Morbidity and Mortality Weekly Report

230

5

Health Affairs

390

15

U.S. Department of Health and Human Services

225

6

Health Services Research

352

16

American Journal of Preventive Medicine

202

7

Pediatrics

349

17

Cancer

199

8

Academic Medicine

319

18

Family Medicine

176

9

New England Journal of Medicine

288

19

Public Health Reports

173

10

Center for Disease Control and Prevention

264

20

Preventive Medicine

171

It should be noted, however, that the geographical location of a researcher or research group may not fairly represent the full scope of the activities undertaken. Often research teams engage a variety of stakeholders from many regions, particularly rural regions, despite their institution’s location. One might argue, however, that location in the community being studied is crucial for stakeholder buy-in.

The diversity of sites is also considered from the perspective of types of organizations involved. Within the top twenty authors, eleven authors are from academic institutions, eight from government agencies, and one from not-for-profit institution for a total of three types of institutions. Although, this ranks the network as “low” for diversity of sites, in this case it might be prudent to consider our definition for types of institutions. Although military and NGOs make sense for some applications of political science, they would certainly be unexpected in this field of health care. We observe great variability among the types of government organizations involved from the expected Department of Health and Human Services and Health Resources and Services Administration to the less expected Department of Agriculture and Government Accountability Office. The involvement of numerous types of government entities along with academic researchers from a variety of disciplines indicates that a multitude of viewpoints maybe represented in this knowledge production system.

Next, the source-source network is analyzed to further understand multidisciplinarity in the network (See Table 6). The focuses of the journals in the top twenty are broad in range, and include the following thirteen subject areas: rural health, medicine, public health, health policy, health services, pediatrics, medical education, epidemiology, demographics, social science, preventive medicine, cancer, and family medicine. This suggests that the network has “highmultidisciplinarity and may reflect this measure better than the author version because it better represents the multi-faceted capabilities of the authors. Again we see a concentration in medicine and health policy, but greater representation by public health and social sciences and specialization in preventive medicine and chronic diseases. The disciplines notably lacking are nursing and mental health.

Altogether, this network shows attributes of mature, well-connected network dominated by strong central actors. The overall network structures reflected the lack of clusters, suggesting “lowheterogeneity. Furthermore, there is a notable regional concentration in the South Atlantic region due to the dominance of government agencies and academic research centers in the area, reflecting a “medium” to “low” diversity of sites. As such, the system does not have the attributes of a disputatious community and its ability to develop new innovative ideas might be low. However, analysis of the year by year author network shows signs of potential clustering and offshoots that could be indicative of newly forming knowledge groups. Furthermore, while the government agencies and academic research center leads were assigned a discipline, it does not reflect the full scope of their work, nor does their geographic location necessarily reflect their sphere of influence. Although we noted some excluded disciplines, we see promise in the disciplinary evolution of this knowledge production system.

Conclusion

The objective in this study was to develop a knowledge system evaluation instrument to evaluate two knowledge systems across three indicators - heterogeneity, multidisciplinarity and diversity of sites. These indicators are considered normative standards that characterize a disputatious community that creates valid research. Assessing knowledge system based on the normative indicators helps to determine the ability of social sciences to contribute to scientific progress and, more practically, to ensure that the research supporting policy agendas that affect human life is sound research. There are elaborate validity and reliability checking mechanisms ingrained in each epistemic community to establish soundness and reliability of an individual study at the level of constructs. However, in line with Merton’s functionalism, knowledge produced at the individual level is affected by the knowledge system the study is a part of. We also know from Donald T. Campbell that certain attributes of knowledge production systems are positively correlated with system’s ability to create optimally valid research. Citation Network Analysis allows to evaluate knowledge systems as networks of authors and journals, where these networks are simultaneously assessed at individual and system levels.

We chose to evaluate one of the issue areas in health sciences and compare it with an issue area in security studies. Considering that primary care and war in Afghanistan are among national top priorities, evaluating each of these knowledge systems is critical in itself. Revealing the weaknesses and strengths of each of the systems helps us to determine what works in regards to the knowledge production process and what needs to be changed. Comparing the two systems allows to draw conclusions not only about specific academic fields but also about research enterprise as a whole. The first conclusion reached in this study is that none of the systems performs satisfactory in regards to the heterogeneity standard. When we examine each of the systems year by year, we observe that there are years when systems are characterized by a core and one or two clusters connected to the core by a bridge. This indicates that there are times when groups of scholars challenge the mainstream approach, offering new insights and perspectives. According to the scale we developed for “presence of clusters” parameter, the fact that neither of the systems allows for more than two clusters means that both systems perform poorly in regards to heterogeneity criteria. At the same time, we conclude that both systems are open to the inflow of new ideas that come from individual authors.

In measuring multidisciplinarity, the publications on health perform considerably better than publications on the U. S. security strategy in Afghanistan. In the case of primary healthcare issue area, the top influencers represent a range of disciplines. While all of these disciplines are related to health they are different enough to qualify the system as multidisciplinary. Such multidisciplinarity can be attributed to the history of the rural access to primary care issue, which has been pervasive since the 1970s. Many funding sources and types of researchers have considered this issue from several perspectives, such as the classic family medicine workforce approach to more innovative preventive care models in recent years. In the case of security studies, although there are eight research domains, from the epistemological point, they are all rooted in political science and policy studies. One might argue that this is what should expected of top experts in security studies, since traditionally security strategy is informed by political scientists, international relations scholars, and military strategists.  However, if the system is to be a disputatious community the core should incorporate experts from such fields as development studies, human security, sociology and cultural studies. This suggests that the issue in question and its timeline, funding sources and research bases may influence the evolution of multidisciplinarity in an issue area.

Diversity of sites is a particularly important indicator when assessing the security literature for multiple perspectives, however, it was less telling for health sciences. U.S. security strategy in Afghanistan involves several apparent stakeholders, including Afghan population, Afghan Army, tribal leaders, Afghan government, governments of courtiers neighboring Afghanistan, and NATO states. In view of these stakeholders, one would expect that designing a strategy that aims to counter the insurgents in Afghanistan and make this state economically and social stable necessitates studying the perspectives of all the relevant groups. It is encouraging to see that Afghanistan, Iraq and Pakistan are represented among the top cited authors and that most of the U.S. stakeholder organizations are taking part in the knowledge production enterprise. We noted, however, that Afghan scholars are not included among the top cited publications despite U.S. journalists residing in Afghanistan. Another limitation is that NATO is represented exclusively by the U.K. In sum, although this knowledge production system is far from meeting normative criteria of a disputatious community, it is reasonably open to multiple perspectives. For the health sciences field, while the publications on primary health care in rural U.S. rank “low” for diversity of sites, this does not mean that the system is completely inadequate in representing all the relevant stakeholders. The authors are concentrated in the South Atlantic region, however, the research represented may involve more regions due to the collaborative nature of health services research and the national data available. Furthermore, the top twenty experts come from three types of institutions: academia (predominant), government (eight federal agencies), and one NGO. These institutions reflect diversity in and of themselves; however, one might expect that patient advocacy groups, rural state agencies and other think tanks and NGOs might play a larger role. Altogether, the diversity of sites measure reflects some complexity inherent in determining the geographic area a researcher represents and nature of complex research institutes and relationships.

This study has limitations that may affect the interpretation of the outcomes observed, and we plan to address several of these limitations in future work. First, the instrument was tested on only two systems, which is not representative of all social science knowledge systems. Our future work involves testing the instrument on other knowledge systems, comparing the results and further refining the measures. Second, the articles selected are not based on a full systematic review of all relevant databases. Our search was limited to PubMed, Scopus and Web of Science, which appear to represent the majority of relevant sources but may exclude some, particularly influential gray literature. We also limited the author-author analysis to first authors, which may underestimate the influence of prominent second and last authors. Finally, this analysis relies on the primary appointment of the authors to determine topical focus and does not consider the content of their work, which may be broader ranging. We plan to conduct semantic network analysis through content analysis of the titles and abstracts to better understand the nature of information flowing within the network.


Alla G. Khadka is a PhD Candidate at the Graduate School of Public and International Affairs at the University of Pittsburgh. Concurrently, she is a Dean’s High Merit Fellowship Researcher on a project led by Dr. William Dunn aimed at evaluating how policy-makers utilize knowledge created for them in academia. She is currently the principal investigator at Social Network Analysis Collaboration working on projects that employ social network analysis to evaluate knowledge systems, information networks and criminal networks

Johanna E. Steenrod, MS, CFA is currently pursuing a PhD in Health Services Research and Policy at the Graduate School of Public Health at the University of Pittsburgh. She is also a graduate student researcher at the University of Pittsburgh Health Policy Institute. Her research interests include organizational and policy level factors that affect the delivery of primary care with a particular focus on the use of interprofessional teams and care delivery in rural settings. Before attending the University of Pittsburgh, Ms. Steenrod worked as a consultant in the Healthcare Industry Group at Alvarez and Marsal. She consulted on a variety of projects, involving qualitative and quantitative analysis to achieve acquisitions, restructurings, and strategic realignments. Her major projects included a restructuring and system reorganization of a $250M system in Westchester County, an ambulatory services strategy at a $750M system in Western New York, an acquisition of a major teaching hospital and a strategic repurposing of a community hospital, both in New York City. Prior to Alvarez and Marsal, Ms. Steenrod worked at GE Commercial Finance in underwriting and portfolio management.

Jeffrey L. Carasiti is an MPIA candidate at the Graduate School of Public and International Affairs in Security and Intelligence Studies program. He is a graduate research assistant to the Director of Matthew B. Ridgway Center for International Security Studies.  His research focuses on information networks, security cooperation and China-US relations.  Prior to the MPIA program, Jeffrey worked at the James Martin Center for Nonproliferation Studies at the Monterey Institute of International Studies as a research assistant. He graduated from the University of Pittsburgh with Bachelor of Arts in Administration of Justice, Certificate in Global Studies, Certificate in Asian Studies, and Minor in Chinese.

Megan Thomas is an MA candidate in Security and Intelligence Studies at the Graduate School of Public and International Affairs, University of Pittsburgh. Her research interests include dynamic network analysis of knowledge systems and information networks. She is also interested in global security, counterterrorism and assessment of clandestine networks.  She received her undergraduate degree in International Studies from Penn State University. While at Penn State University, she worked at the International Center for the Study of Terrorism and at Pittsburgh Office of Emergency Management and Homeland Security. She is currently affiliated with University of Pittsburgh Ridgeway Center for International Security Studies.

Abigail R. Starkis a Master of Public Affairs student at the University of Pittsburgh, Graduate School of Public and International Affairs, specializing in Urban and Regional Development. She is working as a research Associate for the Social Network Analysis Collaboration on project aimed at evaluating knowledge systems in health sciences. Additionally, Abigail is a researcher at the University of Pittsburgh Center for Metropolitan Studies conducting comparative analysis of poverty in Pittsburgh. Before entering graduate program, she worked as an Open Space Intern for the City of Pittsburgh Department of City Planning.

Jonas Gamso is a PhD student at the University of Pittsburgh, Graduate School of Public and International Affairs, studying international political economy and international development.  Primary research areas are trade and redistribution, and Latin American political economy. His other interests include social network analysis and comparative analysis of knowledge networks.


Notes

1 For a detailed comparison of PubMed, Web of Science, Scopus and Google Scholar please see Falagas et al. 2008

References

Allison, Graham T. 1971. The Essence of Decision: Explaining the Cuban Missile Crisis. Boston, MA: Little, Brown and Company.

Anwar, Mumtaz A., and Sultan Al-Daihani. 2011. "Literature on Terrorism: A Bibliometric Analysis of Articles Published During 1981-1990." Malaysian Journal of Library & Information Science no. 16 (2):33-43.

Barnes, Barry. 2012. "Relativism as a Completion of the Scientific Project." In The Problem of Relativism in the Sociology of (Scientific) Knowledge, edited by Richard Schantz and Markus Seidel. Frankfurt: Ontos.

Barnes, Barry, and R.G. Alex Dolby. 1970. "The Scientific Ethos: A Deviant Viewpoint." European Journal of Sociology no. 11:3-25.

Belter, Chris. 2012. "Visualizing Networks of Scientific Research." Information Today, Inc no. 36 (3).

Berger, Peter L., and Thomas Luckmann. 1967. The Social Construction of Reality: A Treatise in the Sociology of Knowledge. Garden City, NY: Anchor Books.

Berstein, Ilene N., and Howard E. Freeman. 1975. Academic and Entrepreneurial Research. New York, NY: Russell Sage Foundation.

Bloor, David 1976. Knowledge and Social Imagery. London: Routledge & Kegan Paul.

Bloor, David. 1981. "Hamilton and Peacock on the Essence of Algebra." In Social History of Nineteenth Century Mathematics, edited by H. Bos and I. Schneider H. Mehrtens, 202-232. Boston: Birkhäuser.

Bradford, Samuel C. 1934. "Sources of Information on Specific Subjects." Engineering no. 137:85-86.

Burt, Ronald S. 1992. Structural Holes: The Social Structure of Competition. Cambridge, MA: Harvard University Press.

Burt, Ronald S. 2004. "Structural Holes and Good Ideas." American Journal of Sociology no. 110 (2):349-399.

Burt, Ronald S. 2007. "Second-Hand Brokerage: Evidence on the Importance of Local Structure for Managers, Bankers, and Analysts." Academy of Management Journal no. 50:119-148.

Campbell, Donald T. 1985. "Toward an Epistemologically-Relevant Sociology of Science." Science, Technology, and Human Values no. 10 (1):38-48.

Campbell, Donald T. 1986. "Science’s Social System of Validity-Enhancing Collective Belief Change and the Problems of the Social Sciences." In Metatheory in Social Science, edited by Donald W. Fiske and Richard Shweder, 108-135. Chicago, IL: University of Chicago Press.

Campbell, Donald T. 1987. "Guidelines for Monitoring the Scientific Competence of Preventive Intervention Research Centers: An Exercise in the Sociology of Scientific Validity." Knowledge: Creation, Diffusion, Utilization no. 8 (4 (March)):389-430.

Cook, Thomas D., and Donald T. Campbell. 1979. Quasi-Experimentation: Design & Analysis Issues for Field Settings. Chicago: Rand McNally College Pub. Co.

Crane, Diana. 1972. Invisible Colleges: Diffusion of Knowledge in Scientific Communities. Chicago, IL: University of Chicago Press.

Dachs, B., Roediger-Schluga, T., Widhalm, C., Zartl, A. 2001. Mapping Evolutionary Economics. A Bibliometric Analysis. In European Meeting on Applied Evolutionary Economics Conference (EMAEE). Vienna: University of Economics and Business Administration.

De Rycke, Jean. 2009. Bibliometric Mapping of Research Disciplines in the Field of Infectious Diseases of Animals. Paper read at EMIDA ERA-NET. Workpackage 2, Deliverable 2-7, at Institut Nationl de la Recherche Agronoamique (INRA), Animal Health Division. Nouzilly, France.

De Solla Price, Derek J. 1963. Little Science, Big Science. New York, NY: Columbia University Press.

De Solla Price, Derek J. 1965. "Networks of Scientific Papers: The Pattern of Bibliographic References Indicates the Nature of the Scientific Research Front." Science no. 149 (3683):510-515.

Delanty, Gerard. 2001. Challenging Knowledge: The University in the Knowledge Society. New York, NY: The McGraw-Hill Companies.

Dunn, William N., Burkart Holzner, Mary Jo Dukes, Evelyn M. Fisher, Alexander Weilenmann, Gerald Zaltman, Ralph L. Bangs, Anthony G. Cahill, Smita Kumar, Thomas R. McIntyre, Grant W. Nauman, Hassan Rahmanian, and Mohammed Shahidullah. 1982. Methodological Research on Knowledge Use and School Improvement. Measuring Knowledge Use: A Procedural Inventory. Vol. III. Washington, D.C.: National Institute of Education.

Durkheim, Émile. 1895/1938/1964. "The Rules of Sociological Method." In, edited by George E. G. Catlin. 8th edition, trans. Sarah A. Solovay and John M. Mueller. New York, NY: Free Press.

Estabrooks, Carole A., Linda Derksen, Connie Winther, John N Lavis, Shannon D. Scott, Lars Wallin, and Joanne Profetto-McGrath. 2008. "The Intellectual Structure and Substance of the Knowledge Utilization Field: A Longitudinal Author Co-Citation Analysis, 1945 to 2004." Implementation Science no. 3.

Etzkowitz, Henry, and Loet Leydesdorff. 2000. "The Dynamics of Innovation: From National Systems and “Mode 2” to a Triple Helix of University–Industry–Government Relations." Research Policy no. 29 (2):109–123.

Falagas, Matthew E., Eleni I. Pitsouni, George A. Malietzis, and Georgios Pappas. 2008. "Comparison of PubMed, Scopus, Web of Science, and Google Scholar: Strengths and Weaknesses." The FASEB Journal no. 22:338–342.

Funtowicz, Silvio O., and Jerome R. Ravetz. 1993. "Science for the Post-Normal Age." Futures no. 25 (7):739-755.

Garfield, Eugene. 1955. "Citation Indexes for Science." Science no. 122 (3159):108-111.

Garfield, Eugene, Irving H. Sher, and Richard J.Torpie. 1964. The Use of Citation Data in Writing the History of Science. Philadelphia, PA: United States Air Force Office of Scientific Research. Institute for Scientific Information.

George, Alexander L. 1972. "The Case for Multiple Advocacy in Making Foreign Policy." American Political Science Review no. 66 (3):751-785.

Gibbons, Michael, Camille Limoges, Helga Nowotny, Simon Schwartzman, Peter Scott, and Martin Trow. 1994. The New Production of Knowledge: The Dynamics of Science and Research in Contemporary Societies. London: Sage.

Glänzel, Wolfgang. 2003. Bibliometrics as a Research Field : A Course on Theory and Application of Bibliometric Indicators. Budapest: Magyar Tudományos Akadémia, Kutatásszervezési Intézet.

Gordon, Avishag. 2011. "Terrorism and the Scholarly Communication System." Terrorism and Political Violence no. 13 (4):116-124.

Guilford, Joy Paul. 1965. Fundamental Statistics in Psychology and Education. 4 ed. New York, NY: McGraw-Hill.

Hessels, Laurens K., and Harro van Lente. 2008. "Re-Thinking New Knowledge Production: A Literature Review and a Research Agenda." Research Policy no. 37 (4):740-760.

Hessen, Boris. 1931. "The Social and Economic Roots of Newton's Principia." In Science at the Crossroads, edited by Nicolai I. Bukharin, 151-212. London (Reprint New York 1971).

Holzner, Burkhart, and John H. Marx. 1979. Knowledge Application: The Knowledge System in Society. Boston, MA: Allyn and Bacon Inc.

Irvine, John, and Ben R. Martin. 1984. Foresight in Science: Picking the Winners. London: Frances Pinter.

Latour, Bruno, and Steve Woolgar. 1979. Laboratory Life: The Social Construction of Scientific Facts. Los Angeles, CA: Sage Publications.

Lotka, Alfred J. 1926. "The Frequency Distribution of Scientific Productivity." Journal of the Washington Academy of Sciences no. 16:317-323.

Lum, Cynthia, Leslie W. Kennedy, and Alison J. Sherley. 2006. "The Effectiveness of Counter-Terrorism Strategies." Campbell Systematic Reviews no. 2.

Mannheim, Karl. 1952/1993. "On the Interpretation of Weltanschauung." In From Karl Mannheim, edited by Kurt Wolf. New Brunswick and London, UK: Transaction Publishers.

Mannheim, Karl. 1953. "Essays in the Sociology of Knowledge." Edited by Paul Kecskemeti. London: Routledge & Kegan Paul.

Merton, Robert King. 1957. Social Theory and Social Structure. Glencoe, IL: Free Press.

Merton, Robert King. 1968. Social Theory and Social Structure. enl. ed. New York, NY: Free Press.

Merton, Robert King. 1973. The Sociology of Science: Theoretical and Empirical Investigations. With an Introduction by Norman W. Storer. Chicago, IL: University of Chicago Press.

Miller, David, and Tom Mills. 2009. "The Terror Experts and the Mainstream Media: The Expert Nexus and Its Dominance in the News Media." Critical Studies on Terrorism no. 2 (3):414–437.

Moran, Joe. 2010. Interdisciplinarity: The New Critical Idiom. 2 ed. New York, NY: Routledge.

Nowotny, Helga, Peter Scott, and Michael Gibbons. 2001. Re-Thinking Science: Knowledge and the Public in an Age of Uncertainty. New York, NY: Sage Publications Ltd.

Parsons, Talcott. 1951. The Social System. London: Routledge.

Parsons, Talcott. 1961. Theories of Society: Foundations of Modern Sociological Theory. New York, NY: Free Press.

Porter, Alan L., Alex S. Cohen, J. David Roessner, and Marty Perreault. 2007. "Measuring Researcher Interdisciplinarity." Scientometrics no. 72 (1):117-147.

Reid, Edna F., and Hsinchun Chen. 2007. "Mapping the Contemporary Terrorism Research Domain." International Journal of Human-Computer Studies no. 65:42–56.

Rip, Arie. 2004. "Strategic Research, Post-Modern Universities and Research Training." Higher Education Policy no. 17 (2):153-166.

Schmaus, Warren. 1994. Durkheim's Philosophy of Science and the Sociology of Knowledge. Chicago, IL: University of Chicago Press.

Shadish, William R., Thomas D. Cook, and Donald T. Campbell. 2002. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston: Houghton Mifflin.

Slaughter, Sheila, and Larry L. Leslie. 1997. Academic Capitalism: Politics, Policies, and the Entrepreneurial University. Baltimore, MD: The John Hopkins University Press.

Weingart, Scott, Hanning Guo, Katy Börner, Kevin W. Boyack, Micah W. Linnemeier, Russell J. Duhon, Patrick A. Phillips, Chintan Tank, and Joseph Biberstine. 2010. Science of Science (Sci2) Tool User Manual. Bloomington.

Ziman, John. 2000. Real Science: What It Is, and What It Means. Cambridge: Cambridge University Press.


This published article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 3.0 Unported License. Copyright for this article is held by McGill Sociological Review.