March 28 2024

DRAFT: AAR Guidelines for Evaluating Digital Scholarship

 

The AAR has established a task force charged with making recommendations to the AAR Board of Directors about various aspects of publishing and the ways the AAR promotes scholarship and publishing in a rapidly changing environment. One piece of this larger charge is to make recommendations for guidelines for evaluating digital scholarship. The task force has drafted the following set of recommendations, and would like to open it to comments and suggestions from the AAR membership before forwarding it to the Board of Directors. Please feel free to comment directly on the draft, or e-mail comments to Ted Vial (tvial@iliff.edu). Please make comments by August 31, 2017. Thank you so much for your input!


I. Defining Scholarship in the Academic Study of Religion

Scholarship in the study of religion is currently in a period of rapid change. Established modes of research, teaching, and scholarly communication have been augmented or transformed by new technologies and digital research methods. Books, journal articles, and other print-based works are now joined by a plethora of new genres of digital publication, while the proliferation of digital tools have redefined what it means to discover, interpret, and disseminate information.

Despite this flourishing of new work, however, the study of religion has yet to establish guidelines for the professional evaluation of digital scholarship. This disconnect is unfortunate. It hinders the American Academy of Religion’s ability to carry out its principal goals as stated in the organization’s mission statement, “To promote understanding of and critical reflection on religious traditions, issues, questions, values, texts, practices, and institutions” through “communication and exchange among teachers and scholars, and the public understanding of religion,” while also serving “the professional interests of members as students, teachers, and scholars.”  The lack of guidelines for the evaluation of digital scholarship can also impede the recruitment and retention of innovative scholars. In an age where both scholars and their many publics increasingly circulate knowledge about religion online, it is incumbent upon the AAR to propose guidelines that properly evaluate, promote, and share digital knowledge.

This document aims to guide researchers, teachers, and administrators in evaluating digital scholarship in the study of religion. Building upon the AAR’s Guidelines for Responsible Research Practices,” which notes that “Scholars may share the results of their research in multiple media, including peer-reviewed digital scholarship,” it seeks to promote the evaluation of scholarship in new and emerging media in ways that take into account the distinctive characteristics of digital projects and work. In contrast to print--which tends to be single authored, text-based, and completed upon publication--digital scholarship has several peculiar attributes that must be considered when assessing its merit. First, digital scholarship tends to be collaborative in that it not only can involve several scholars at multiple institutions but can also incorporate a variety of professions such as computer programmers, librarians, and even students. Digital scholarship also tends to be multimodal, utilizing multiple forms of media and even multiple platforms simultaneously. In some work, textual or narrative forms of analysis may take a back seat to more visual or auditory displays of knowledge whose contribution is perhaps made through juxtaposition or other nonlinear forms of argumentation. Finally, digital scholarships is also often open ended. In many instances, the launching of a digital project is the beginning of the work rather than its end, while digital scholarship frequently requires maintenance or regular upgrades. What’s more, exemplary digital projects also often make their findings, content, and data available to an open scholarly commons so that other researchers can build upon, remix, or augment prior work.

By offering these guidelines, we do not intend to suggest that digital scholarship should be held to a different set of standards. No matter the medium, scholarship should engage in “disciplined reflection” and “critical examination” as outlined in the AAR’s mission. This includes the rigorous analysis or exploration of primary source material or data; the sustained engagement with an established body of scholarly literature; and a clear articulation of the work’s argument, methodological choices, and original contribution. In other words, quality digital scholarship should also always be quality scholarship. The recommendations below aim to place digital scholarship within this broader conversation of scholarly merit. They should be taken as an invitation to a conversation, and not a fixed set of permanent requirements. Indeed, the rapidity with which digital technology changes will necessitate these guidelines be reevaluated at a future date.

II. Guidelines for Evaluating Digital Scholarship

To aid in the evaluation of scholarship produced in digital form, we offer the following overview of the central attributes of digital scholarship as a framework for evaluation. While not all of these areas will apply to all digital projects, and additional attributes may prove central for the evaluation of some projects, this list provides a description of core elements shared by many of the best scholarly digital works. Although the focus of evaluation is often within the context of tenure and promotion, this framework is intended as a guide for the creation and evaluation of digital scholarship at all levels of academic engagement, from undergraduate research to doctoral dissertations and professional scholarly work. As such, we see this framework as a guide to best practices in digital scholarship of religion, one that can be used for both teaching and evaluation. Because the contribution of digital scholarship is tied to the medium of creation and dissemination, we also outline parameters for how the evaluation of digital scholarship should be conducted, and note external evaluative sources which can help identify and articulate the quality and impact of a digital project.

II.a. Central Attributes of Digital Scholarship

1. Content

As with all scholarly work in the study of religion, it is vital that the content of digital scholarship in the study of religion is academically sound and engages with the current scholarship in the field. While that engagement may take many forms, it is incumbent upon the creators to clearly communicate their contribution to the field.

2. Use of digital media

For many digital projects, the digital medium is a central part of the scholarly work, enabling the creator(s) to advance an argument or mode of analysis that would not be possible in print. Creators of digital projects should include some explicit statement of why the digital medium was selected and how it advances the project goals. Digital projects should be evaluated on the effectiveness of the use of the medium to advance the intellectual goals of the project, considering what the medium enables that would not be possible in other formats.

3. Design

The design and organization of the interface of a digital project is a central part of the communication of its content. The design signals and enables the ways readers interact with the scholarly work. As a result, the design of digital scholarly works should be evaluated on elements such as the clarity and effectiveness of interface design, whether graphical or computational; ease of identifying and accessing information; ease of navigating the resources; adherence to established standards of accessibility and ease of use for all users; and coherence between the design and the argument of the project.

4. The Varieties of Digital Scholarship

“Digital scholarship” is an expansive terms that includes many types or genres of scholarly digital work. In evaluating such work, it is important to take into consideration the genre or genres with which the digital work is engaging. For the purposes of evaluation and review, we recommend the following list of genres of digital scholarship, adapted from the Journal of American History Digital History Review Guidelines, as examples of the current range of digital scholarly work. This list is not exclusive, and projects may span multiple genres.

  • Archive: a collection of digital primary sources.
  • Essay, Exhibit, Digital Narrative: something created or written specifically for the Web or with digital methods, that offers interpretation, narrative, or argument. This category can also include maps, network visualizations, or other ways of representing religion data.
  • Teaching Resource: a site that provides online assignments, syllabi, other resources specifically geared toward using the Web, or digital apps for teaching, including educational religion content for children or adults, pedagogical training tools, and outreach to the education community.
  • Tool: a downloadable, plugin, app, or online service that provides functionality related to creating, accessing, aggregating, or editing digital religion content (rather than the content itself).
  • Gateway/Clearinghouse: a site that provides access to other websites or Internet-based resources.
  • Journal/Blog/Publication: any type of online publication.
  • Professional/Institutional Site: a site devoted to sharing information on a particular organization.
  • Digital Community: online social spaces that offer a virtual space for people to gather around a common experience, exhibition, or interest.
  • Podcasts: video and audio podcasts that engage audiences on religion topics and themes.
  • Games: challenging interactive activities that educate through competition or role playing, finding evidence defined by rules and linked to a specific outcome. Games can be online, peer-to-peer, or mobile.
  • Data sets, APIs: compilations of machine-readable data, shared in a commonly-accessible format, possibly through a CSV file or an Application Programming Interface (API), or data files, that allow others to make use of this data in their own digital work.

5. Audience

In the evaluation of digital projects, it is vital to consider the intended audience of the project. This is often linked to the genre of the work, as projects such as blogs, podcasts, games, and teaching resources are most often aimed at a non-academic audience, whereas digital narratives, data sets, and tools are primarily resources for an academic audience. While insisting on a high scholarly standard regardless, a digital project should be evaluated in relation to the intended audience -- whether that scholarship is communicated appropriately for the intended audience and in ways that encourage readers to develop nuanced understandings of religion.

6. Sustainability and Preservation Planning

Unlike tradition monographs and articles, which have a standard form and for which there are established preservation processes, the variety of formats of digital scholarship and their reliance on ever-changing technological infrastructure, together with a lack of established practices around their preservation, necessitates that scholars working in the digital medium attend to the sustainability of their work. While no project can guarantee future stability or availability, digital scholarly works should follow best practices in the use of digital technologies and include a plan for the preservation of the various elements of the digital work, such as data, code, and interface.

7. Collaboration

While not all digital projects are collaborative, many rely on teams to bring together the technical infrastructure, create and manage data, design the interface, and craft the content. Some features of collaborative digital projects may be more labor-intensive than others. The scholar is responsible to explain the different roles each collaborator plays and estimate each collaborator’s contribution. However, evaluators should note that digital projects cannot be easily broken down into percentages that equal a whole. Therefore, collaboration should be evaluated as a process in which scholarship is generated, and not as deemed more or less valuable according to conventional assessment standards like title or author order.

II.b. Parameters for Evaluation of Digital Scholarship

One of the distinctive aspects of digital scholarship is the medium of production and dissemination. Because the medium is central to the construction and argument of the scholarly work, the evaluation of digital scholarship must be done in the work’s native environment and with consideration of the collaborative nature in which digital work is produced. We recommend the following parameters for the evaluation of digital scholarship:

1. Evaluation in the work's native environment

Since digital scholarship is designed for use and presentation in a particular medium, it is imperative to evaluate work in the specific native environment in which it was produced. As the CAA/SAH Guidelines put it succinctly, "it is crucial that digital work be seen in the environment for which it was designed. Scholars deserve to have their work taken seriously, including the digital contribution. Hence, all work of digital scholarship must be evaluated in its appropriate environment."

2. Media-specific expertise of committee members

When possible, evaluation committees should have members with expertise or experience in the particular forms of media technologies of which they are assessing. Committees that lack members with expertise in digital scholarship should contract colleagues as media-specific evaluators even if they do not possess content expertise in the project's subject field. In this way the committee can engage and evaluate the strengths and weaknesses work on its own terms.

3. Role of contributor in the creation of project

Digital projects are often collaborative products, the work of teams with a range of areas of expertise, from subject matter to technical organization and implementation. In the evaluation of the digital work of a particular scholar, evaluators should take into consideration both the overall success of the project and the role of the contributor in the creation of that project. It is the responsibility of the scholar engaged in digital work to articulate his or her contributions to the digital project, thereby setting up the parameters of evaluation.

II.c. Additional Evaluative Sources

As with print scholarship, the receipt of grants and awards for the scholarly work, as well as information about its use, are indicators of the reception of the work within the scholarly community. We recommend that those evaluating digital scholarship consider external sources, such as those mentioned below, in evaluating the impact of the scholarly work.

1. Grants received

Unlike traditional peer-reviewed materials that are assessed upon completion, digital scholarship is often continuously developed and improved. Therefore, grant funding is one clear indicator of the impact or potential of digital work. Securing competitive grants demonstrates that after formal peer-review by funding committees the digital work has shown significant potential and/or proven historical success.

2. External Awards

Awards and other recognitions by professional associations, or scholarly organizations, networks,  and collectives further confirms that digital work is making a significant contribution to research.

3. Data regarding use of and engagement with project

Digital scholarship often has metrics that demonstrate the type of engagement they have with audiences. For many projects, instruments such as page views, downloads, plays, number of users, engagement, code or data reuse, as well as other alternative metrics can be used as evidence of its impact. For other types of digital projects, influence can be established through the development of new work based on its technologies and/or design. Assessment should be based on both the quality and quantity of public access, scholarly engagement, and audience use since evaluative metrics will not be standardized across digital work. Altogether, data from a project’s standard measuring mechanisms should be clearly explained in order to evaluate how it generates stimulating technical and theoretical contributions to scholarship.

However, these metrics should not be relied upon as a sole indicator of a work’s importance. Just as book sales are not necessarily indicative of the importance of a scholarly work for the development of the field, a high quality work of digital scholarship may have low use metrics. In addition, as with all humanities scholarship, the importance of a particular work in the scholarly ecosystem often increases over time and is not easily seen in data that privileges current popularity. As such, we caution against the use of metrics as a shorthand for the value or impact of digital scholarly work, recommending instead that the primary mode of evaluation for digital scholarship be qualitative, and that use metrics be considered as providing a supplemental indicator of the current reach of the work.

III. Digital Scholarship for Promotion and Tenure

III.a. Scholar:

Before beginning and throughout the project, the scholar should consult established departmental and/or other institutional standards for evaluating scholarship for promotion and tenure, and should make plans to explain and document the project’s development, progress, and contributions to scholarship in light of those standards. These plans should be discussed with departmental and committee chairs at each stage to ensure everyone has the same expectations.

Scholars should seek support in preparing tenure or promotion portfolios and making a case for its importance, or in preparing to present “dissertations,” from the AAR, from departments, and from colleagues.

Scholars should be transparent, and keep department and/or committee chair informed of collaborations as they unfold.

Scholars should be aware of the processes of evaluation which their departments, schools, and committees will use in evaluating their projects. Scholars should be aware that their departments may weigh different varieties of digital scholarship differently. Scholars should be prepared to help draft those processes if they are non-existent or inadequate.

III.b. Institution:

Standards and procedures for evaluating scholarship in cases of appointment, tenure, and promotion vary widely among different institutions of higher education. In any case, those standards and procedures should explicitly address the evaluation of digital scholarship as distinct from traditional scholarship in print media. Specifically:

  • Evaluation of digital scholarship should be conducted within the work’s native digital environment rather than in the form of print-outs or scans included in review materials along with other print media.
  • Every reasonable effort should be made to identify and select external evaluators who are qualified to review the technological and design aspects of the scholarship (e.g., appropriateness of the medium to the project; choice of database, programming language, and web application framework; quality of back-end and front-end coding and design; digital continuity; documentation).
  • If possible, internal review committees should also consider including at least one member whose expertise is in the digital technologies of the candidate’s scholarly work, even if that committee member’s research expertise is different from that of the candidate.
  • Collaborators (internal or external) on the candidate’s project should provide evaluations of the candidate’s contributions to the larger process, including responsibilities in curating and negotiating the relationships of the project team.
  • There should be clear standards for measuring evidence of the scholarly influence of the project. Metrics might include: (1) awards, reviews, and other forms of academic peer recognition; (2) grants, fellowships, and other project funding; (3) quality and quantity of access and use by other scholars and students (incl. links to it in other projects); and (4) evidence of influence in terms of the development of new projects based on its technologies and/or design.
  • In addition to the work itself, the candidate’s statement on research should include details that describe and document the project as a means of enabling the committee to evaluate it as both product and process of academic research. This statement might include the following elements, as applicable: (1) the relevance of the project for scholarship, teaching, or research; (2) contributions of the project to scholarship in the field; (3) evidence of originality or innovation; (4) broader applications of the project in other fields; and (5) a project narrative of the process of preparing and completing the project, including benchmarks for evaluation and progress, descriptions of hardware and software requirements for viewing and participating in the project, and the names and contributions of collaborators.
  • At certain points in the course of a candidate’s progress toward tenure and/or promotion (e.g., annually or at the time of a third-year review), s/he might might offer a demonstration to department faculty of the project's development process. Reports on and/or recordings of these demonstrations could then become part of the candidate’s file. Such demonstrations could be especially important when evaluating projects that have no clear endpoint or launch date.

IV. Selected Resources and References

This selected list of resources below are either cited in the document above, or served as guides for the crafting of these guidelines.

Journal of American History Guidelines for Digital History Reviews:
http://jah.oah.org/submit/digital-history-reviews/

AHA Guidelines for the Professional Evaluation of Digital Scholarship by Historians:
https://www.historians.org/teaching-and-learning/digital-history-resources/evaluation-of-digital-scholarship-in-history/guidelines-for-the-professional-evaluation-of-digital-scholarship-by-historians

Cooper, D., Schonfeld, R. C., Adams, R., Baker, M., Bakkalbasi, N., Bales, J. G., Bedard, R., . . .Weiss, N. (2017, February 8). Supporting the Changing Research Practices of Religious Studies Scholars. https://doi.org/10.18665/sr.294119.

Modern Language Association’s Guidelines for Evaluating Work in Digital Humanities and Digital Media
https://www.mla.org/About-Us/Governance/Committees/Committee-Listings/Professional-Issues/Committee-on-Information-Technology/Guidelines-for-Evaluating-Work-in-Digital-Humanities-and-Digital-Media.

Cantwell, C. D., Rashid, H. (2015) Religion, Media, and the Digital Turn. Social Science Research Council.
http://www.ssrc.org/publications/view/religion-media-and-the-digital-turn/.

Guidelines for the Evaluation of Digital Scholarship in Art and Architectural History
http://www.collegeart.org/pdf/evaluating-digital-scholarship-in-art-and-architectural-history.pdf

Case Studies and Examples for Evaluating Digital Scholarship
http://www.collegeart.org/news/2016/02/23/case-studies-and-examples-for-evaluating-digital-scholarship/

Explaining Digital Humanities in Promotion Documents
http://journalofdigitalhumanities.org/1-4/explaining-digital-humanities-in-promotion-documents-by-katherine-harris/

Evaluating Scholarly Digital Outputs: The Six Layers Approach
http://journalofdigitalhumanities.org/1-4/evaluating-scholarly-digital-outputs-by-james-smithies/

Promotion and Tenure for Digital Scholarship
http://journalofdigitalhumanities.org/1-4/promotion-and-tenure-for-digital-scholarship-by-laura-mandell/

Short Guide To Evaluation Of Digital Work
http://journalofdigitalhumanities.org/1-4/short-guide-to-evaluation-of-digital-work-by-geoffrey-rockwell/

How to Evaluate Digital Scholarship
http://journalofdigitalhumanities.org/1-4/how-to-evaluate-digital-scholarship-by-todd-presner/

Living in a Digital World: Rethinking Peer Review, Collaboration, and Open Access
http://journalofdigitalhumanities.org/1-4/living-in-a-digital-world-by-sheila-cavanagh/

A Call to Redefine Historical Scholarship in the Digital Turn
http://journalofdigitalhumanities.org/1-4/a-call-to-redefine-historical-scholarship-in-the-digital-turn/

Evaluating Collaborative Digital Scholarship (or, Where Credit is Due)
http://journalofdigitalhumanities.org/1-4/evaluating-collaborative-digital-scholarship-by-bethany-nowviskie/

Digital Dissertation Guidelines, Department of History and Art History, George Mason University
http://historyarthistory.gmu.edu/graduate/phd-history/digital-dissertation-guidelines

Drafting Guidelines for Emerging Scholarship, Iliff School of Theology
http://library.iliff.edu/blog/draftingdigital


[Editor's Note, 9/1/2017: Commenting has been closed.] Religious Studies News welcomes comments from AAR members by August 31, 2017, and you may leave a comment below by logging in with your AAR Member ID and password. RSN reserves the right to remove any comment that is offensive, inappropriate, or defamatory. Comments reflect the views of the author and are not indicative of approval or endorsement by RSN or the AAR.

If you would like to share your comment privately with Publishing Task Force chair Ted Vial, please send an e-mail with your comment to tvial@iliff.edu.

Comments

Dear Task Force on Publishing,

Thank you for this excellent draft on evaluating digital scholarship! It is a commendable document in every respect.

I am particularly impressed by your insistence on recognizing and encouraging collaborative research, including the vital roles of other members of our academic communities (librarians, programmers, students). Too often, these collaborative contributions are relegated to the acknowledgements section of a monograph. As you state, researchers within these populations not only facilitate our work, but also produce scholarship of their own. It is refreshing to read such a straightforward recognition of their shared academic labors.

I offer a small set of additional comments below. Those of you who know me understand that my default tone is one of bluntness. That is a deficiency of my communicating on matters such as this, not of your work. Please know that these comments arise from my profound respect and admiration for the gravity and generosity of your labor. While my comments are ordered to reflect my own prioritization, I have tried to note their corresponding location within the draft statement.

 

The Value of Digital Scholarship in Tenure and Promotion (Sections I and III)

I failed to recognize in this draft a clear recommendation as to whether member departments ought to incorporate digital scholarship into their policies for tenure and promotion. While the draft notes a marked increase in digital scholarship, I fear that its delicate wording in Section III risks further obscuring the tangible value of digital research. In my reading, the draft defers such valuation to (a) the pre-existing policies and procedures of member institutions, and (b) the garnering of grants and awards.

The current wording leaves open for interpretation signifcant questions. Chiefly: Does the task force agree that it is a best practice of departments of religious studies to factor digital scholarship into their equations for promotion and tenure? Is there an inherent value in encouraging our member institutions to increase the value they place on digital scholarship? I would welcome clarification on these issues in the revised version.

Section I, “Defining Scholarship,” may be a more logical location for making such a recommendation.  However, I likewise found none there, save perhaps the understatement, “This disconnect is unfortunate,” which does not inspire the kinds of significant shifts in evaluative standards that would adequately reflect what has been, as you again understate, “A period of rapid change.”  What is needed here is a strong affirmation of the value of digital scholarship, not a gentle admonishment of its absence from extant AAR guidelines. In clarifying this document, I encourage you to be boldly constructive rather than passively critical.

 

The Specificity of Our Discipline (Sections I and II)

I worry, as well, that the task force’s statement relies too heavily on the credibility and insight of other fields, referring multiple times to the presence of such standards within other professional societies but without specifying the value of digital scholarship within religious studies itself. This is not merely an issue of arriving late to the party, nor is it moored to the narrower question of tenure and promotion. While I recognize that our community has a tendency to (mis)perceive of religious studies as a discipline that follows rather than leads, I think we owe it to both our society and our research subjects to specify the ways in which digital scholarship within religious studies may be different from digital scholarship in, for example, history.

Are religious populations potentially made more vulnerable when we use digital tools to analyze and share our research on them? Might there be opportunities for scholars to enhance and supplement the vitality and experience of religious communities through digital research? The draft’s silence on such issues may simply indicate that I am an outlier in finding them relevant to these guidelines.

 

Varieties of Digital Scholarship (Section II.a.4)

Two of the most common varieties are notably absent from the list: social media and online video. Surely, since Podcasts and Games are already included, the mere conjunction of video and social media with infotainment should not preclude their presence from this list. Although these two varieties are most commonly used as data, rather than methods for digital scholarship, both digital video and social media are certainly also mediums of publishing our digital research.

Social media is the more arresting omission. If it is the task force’s judgement that individual posts are too ephemeral to render a scholar’s professional social media accounts void of any evaluative relevance, that in itself might be worth stating. Such judgement would surprise me, however, because of the robust ways in which some members of this task force are already using social media to communicate scholarly developments, challenge research paradigms, and enhance scholarly communication across the AAR community. If, as I do, you cherish the best ways in which AAR members use social media, please add a note on the role(s) of social media in scholarly publishing.

Regarding video, I trust that the AAR already has guidelines for evaluating creative, documentary, archival, and analytical video productions. Assuming that is the case, online venues for creating and publishing digital video offer a number of features, including dynamic and interactive content, which still might warrant discussing video at greater length within these guidelines. Similarly, digital books and audio books have features that may distinguish them from their print counterparts in ways the current draft does not acknowledge.

(Parenthetically, I would note that “Podcasts,” the category under which video is currently listed, stands out within the document as a rare preference towards corporate branding. While steaming and downloadable audio files may now suffer from a “Kleenex” syndrome, this ubiquitous term is inconsistent with the nomenclature chosen for other categories and, at any rate, may eventually be outdated.)

 

Thank you for considering these comments, which again are offered in the spirit of great admiration for your service to the AAR.

 

Sincerely,

 

Brian J. Clites
Case Western Reserve University

Two points, one small and one which is perhaps more broad.

1.  In section 1, the following section needs revision: "Finally, digital scholarships is also often open ended."

2.  Perhaps it is not within the scope of these proposed guidelines, but I missed any reference to the nature, scope and methodology of digital research.