Curated Approaches

Take steps to implement a holistic review system that includes an assessment of applicant characteristics – such as academic metrics, attributes (skills, abilities, personal and professional traits), and life experiences (path traveled and challenges and opportunities) – in a manner that aligns with program’s mission and goals.

Steps to create this system could include: 

• Developing a Mission Statement for the program which articulates goals, including the composition of the admitted student pool
• Promoting faculty discussion in advance of each admissions cycle, as committee membership will typically change over time. Consider discussing the institution and program goals, the admissions process, application review criteria and implicit bias. Use an outside facilitator if necessary. You can make attendance in the conversational workshop mandatory for everyone who will participate in the admissions cycle
• Revising application materials and process in light of the discussions at the workshop and implement them as soon as is feasible
• Using a rubric for evaluating factors considered in the admissions process
• Using standard interview questions for all applicants to help determine strength of interest in the program, problem solving, decision-making, resourcefulness, resilience, understanding of program, etc.
• Having one group of people review the applications and a separate group of people conduct and summarize the findings from the interviews, so that a single set of people are not judging everything. Then both sets of people could discuss the takeaways from the applications and interviews and make the final admissions recommendations together.
• Conducting a post-admissions discussion to evaluate the process and outcomes and possibly make revisions for future admissions cycles

Asking departments to focus on at least three criteria for admissions consideration encourages faculty to consider at least one additional factor beyond GRE test scores and GPA (e.g., scored letter of recommendation). By including a minimum of three screen-in criteria (e.g., compensatory model), each applicant has at least three opportunities to be placed into the review pool. This ensures that qualified applicants aren’t overlooked simply because of one attribute, such as a test score or GPA.

One graduate school revised their application to:

• Request that applicants provide information about: specific educational experiences (which may have been obtained inside or outside of an academic setting), community involvement and/or service, leadership, overcoming social, economic, or physical barriers, personal and/or professional ethics, recognition of achievements over time, and research/scholarship. The applicants then indicate which of the experiences they have had and provide an explanation of their experience.
• Ask recommenders to rate applicants on communication skills, ethics and integrity, initiative, innovation and curiosity, planning and organization, and teamwork within the letter of recommendation. Additional space is provided on the form for the recommender to provide examples and additional comments.
• This information (i.e. educational experiences and faculty ratings on key knowledge, skills, and abilities) can be used in the screen-in process to create the applicant pool for review.

It is important for graduate programs to clearly define and articulate the application review and selection processes and ensure that all admissions committee participants:

• Are familiar with the admissions process and goals
• Clearly understand the role and importance of each component of the application (e.g. GPA, GRE scores, personal statement, research experience, and interview) and the order in which each component should be reviewed
  • Reviewing the same set of application materials but in a different order can have a meaningful impact on the resulting decision (i.e. framing bias)
  • Ensure reviewers are aware of appropriate inferences that can be made about each component of the application
• Understand the role that unconscious or implicit bias can play in the review and selection processes and how to mitigate it
• Examine each component of the application as defined in the application review process
• Adhere to the agreed upon process to resolve any disagreements about an applicant’s file that may arise within the review committee

A holistic application review process includes a combination of traditional academic materials (e.g.  GRE scores, GPA) and experiences (e.g. research, work experience), but also includes materials (e.g. personal statements, letters of recommendation, interviews) that demonstrate personal attributes (e.g., motivation, perseverance or “grit”, character, self-sufficiency) important to success in the program.

• To more directly obtain information about the personal attributes deemed helpful in evaluating applications, prompts for essays, interviews, letters of recommendations, etc. can be reviewed and revised, as necessary, to explicitly request that the candidates discuss or provide evidence (e.g. behavioral examples) of these attributes.
• Consider providing training so that personal attributes important to success in a graduate program can be identified and captured through the review of admissions materials.
• Rubrics can be used to ensure objective and consistent evaluation of materials related to personal attributes.

• Use GRE scores only in consultation with other admission materials
• If an applicant’s GRE score is not consistent with other pieces of information that reflect the applicant’s candidacy, reconsider the amount of emphasis placed on the applicant’s GRE score(s) and assess other components of the applicant file (e.g., transcript, research experience, personal statement, letters of recommendation, work experience). For example:
  • If the GRE score(s) is low but sufficient evidence is presented that demonstrates an applicant’s potential for success in graduate school, consider decreasing the weight given to the GRE score.
  • If the GRE score(s) is high but is not consistent with other information within the application, GRE scores may be a positive indicator of the applicant’s potential and indicate when other educational opportunities may not have been available.
• GRE scores provide the most valuable information for admissions decisions when they are required (vs. optional) by all applicants for a particular program because comparable information is then collected for all applicants. The same holds for other components of the application
  • GRE scores are especially useful when faculty members are unfamiliar with an applicant’s undergraduate institution

If using GRE scores as a cut score to manage application volume, consider including additional information.

Promising Practices to move beyond the use of GRE cut scores, include:

• Using a rubric with GRE scores not overly weighted
• Quantifying traditional qualitative components (e.g. Letter of Recommendation)
• Looking at other criteria first, placing GRE scores at the end of the review process so that faculty/reviewers do not overweight GRE scores at the neglect of other less quantitative information
• Using GRE scores to confirm the academic preparedness of applicants when GPA is borderline, or applicant is from an institution with which the faculty (admissions committee) are unfamiliar

Alternative review process models include:

• A multi-stage process:
  • Use an initial set of criteria (typically GRE scores and GPA) to narrow the pool of candidates
  • Identify candidates from target populations (e.g. women, underrepresented minorities, first generation, lower socioeconomic status, older or non-traditional students) who did not meet the initial set of criteria. Include these candidates with those selected based on the initial criteria
  • Review the resulting group holistically
• A two-pool process:
  • Pull files for target populations so they can be holistically reviewed first.
  • Given the competitive landscape, if these applicants aren’t reviewed quickly, it is likely they will receive and accept an offer from another institution. Consider not only accelerating their review, but also any offers for admission.
  • Engage separate committee members (faculty, staff, and graduate students) or groups of committee members to review different components of the application then convene to discuss the applicant.
    • For example, Committee Member A reviews components of the application that contain cognitive (GPA, GRE, transcript) factors and Committee Member B and Committee Member C review components of the application that contain non-cognitive (lab/research experience, Letters of Recommendation, statement of purpose, and/or CV) factors. Then the members reconvene to discuss the takeaways from the applications and interviews and together make the final admissions recommendations.  

• Providing information to help faculty understand why diversity is important, and why changes that help fulfill the institutional mission are beneficial for achieving program goals
• Identifying merit in broader terms than traditionally employed measures. Consider additional information that has been shown to be relevant when assessing diverse populations: leadership experience, community involvement, knowledge acquired in a field, or successfully negotiating the education system in spite of challenges  
• Making race/ethnicity/first generation an explicit consideration within your holistic review, if possible, defensible and appropriate
• Providing evidence to faculty regarding how looking at additional information can improve outcomes regarding diversity
• Explore reframing how your institution or program discusses and considers diversity, depending upon your institution or program’s mission and the legal landscape. This could include:
• Creating a public-facing data dashboard to showcase program and institutional demographics to encourage transparency

• It is important for institutional leadership to make the commitment to diversity and inclusion an institutional priority. Just as important is including representatives with a commitment to diversity on the admissions committee. Consider:
  • Including faculty/staff from diverse backgrounds or with accountability for diversity and inclusion on the admissions committee. If not from an underrepresented group, consider including faculty/staff from the diversity office or faculty/staff with a diversity role within the program or university
  • Including currently enrolled students in the recruiting process. If possible, include a mix of students from different backgrounds
  • Including currently enrolled students in the interview process, even if they are not formally on the admissions committee. If possible, include a mix of students from different backgrounds. It is important that feedback from the student interviews is considered by the admissions committee. Note: If structured appropriately, being involved in the interview process can serve as a professional development opportunity for the students
• Key to the process is providing admissions committee members with the context needed to evaluate students appropriately. Consider creating an institutional culture supporting diversity that provides faculty with opportunities to:
  • Learn about the quality of undergraduate education at various minority-serving institutions and institutions that enroll a large proportion of first-generation students
  • Build relationships with less familiar undergraduate institutions (particularly minority-serving institutions)
  • Work with underrepresented minority undergraduates (e.g. through summer programs)

It is important for graduate programs to articulate diversity objectives tied to the missions of their institutions, and communicate this information to administrators and faculty who will be involved in making admissions decisions. In addition, it’s key to review admissions processes annually.

• If appropriate and defensible, consider enrollment goals for specific populations desired by the program.
• In order to better understand the relation between admissions data and student success, analyze historical application and outcomes data specific to your program, such as:
  • Characteristics (e.g. race, ethnicity, gender, socioeconomic status, age, academic background) for those admitted and those rejected
  • Whether those with high standardized admissions test scores and GPAs are more likely to succeed, and whether there are other groups of students that are successful too
• To ensure that your admissions policies and practices are producing the desired student and program outcomes, it is important to periodically (e.g. every 3-4 years) evaluate outcomes data against admissions info/data and enrollment goals. Observed patterns of relations between admissions data and important student outcomes can be used to refine the admissions practices in subsequent cycles.

Graduate deans and program coordinators should work together to establish enrollment goals that will allow each program/faculty committee to create guiding principles for the entire admissions process (recruitment to enrollment), prior to the beginning of the admissions process.

• If there are specific populations or applicants’ attributes desired by your program, it’s important to be explicit on how to identify and recruit the desired populations and to assess those attributes.
• Using the program’s guiding principles and goals, establish the role and importance of each component of the applicant’s file (e.g. test scores, interview).

Consider establishing bridge programs for high-potential individuals who have a bachelor’s degree and want to pursue a graduate degree, but need more research experience, coursework, or education.

There are a variety of bridge programs that have been employed successfully.

• Feeder Programs: Collaborate with another institution to establish a bridge program/feeder system for high-potential individuals who have a bachelor’s degree and want to pursue a Ph.D. degree, but need more research experience, coursework, or education. Bridge programs, with integrated support systems, can be an effective means for increasing the diversity in graduate programs – particularly science, technology, engineering, and mathematics (STEM) programs.
  • Example:
    • Fisk-Vanderbilt Master’s to Ph.D. Bridge Program

• Orientation and Enrichment Programs: Offer an orientation and enrichment program to allow undergraduate students who plan to pursue a Ph.D. to become acclimated to the rigors of graduate-level courses. Program activities can include academic advising, coursework in math, science, and humanities, training in analytical problem-solving, research experiences, social events, and assistance with internships and graduate school applications. These programs could be designed to take place during the summer of a student’s undergraduate studies or as a post-baccalaureate program.
  • Examples:
    • Meyerhoff Graduate Fellows Program Summer Bridge Program
    • Minority Access to Research Careers (MARC)-U*STAR (Undergraduate Student Training in Academic Research)
    • Hot Metal Bridge Program at the Dietrich School of Arts and Sciences at the University of Pittsburgh
    • National Institute of General Medical Sciences Post baccalaureate Research Education Program

• Alliance Pipeline Nurturing: Collaborate with disciplinary societies, professional organizations, or foundations that identify diversity as an organizational goal. Work together to identify, enroll and nurture a pipeline of students that have demonstrated an interest in pursuing a Ph.D., but may need additional transitional support to be successful in a Ph.D. program.
  • Examples:
    • American Physical Society Bridge Program
    • Louis Stokes Alliances for Minority Participation (LSAMP) program