Select a preliminary set of variables
Once you've determined that a peer group needs to be created and considered some initial parameters to help limit the types of institutions to include, submit a request to the office of Planning, Analytics, and Institutional Research (PAIR). The PAIR team will work with you to identify variables the university (or your unit/group) wants to use to develop its peers. These variables are the characteristics you want to use to establish your peers.
The process for selecting variables can be lengthy and involves thoughtful consideration on the part of your group/the institution. Peer groups will vary widely depending on the variables used in the analysis, so it’s important that you consider the following when determining the characteristics used to develop a set of peers:
How does this variable contribute to a useful benchmarking on the outcome in question?
The variables you use to develop your peer group should be relevant to the reason you’re trying to develop a benchmarking group in the first place. For instance, if you want to develop a peer group to benchmark JMU’s endowment against other institutions, then the variables you choose to include should reflect institutional characteristics such as average family income, number of degrees awarded, and types of programs offered. A combination of these variables may illustrate institutions that should have similar factors influencing an endowment’s size. Choosing other variables such as percent of full-time faculty, institutional location, or graduation rates may produce a much different list of institutions that will yield highly different endowment sizes, making the ability to benchmark appropriately more difficult.
Is data for the variable readily available for ALL institutions in your population of interest? (see Step 3)
Often, you may want to include a variable that we track at JMU, but that isn’t available in the limited number of publicly available datasets. For instance, you may want to compare JMU against institutions with similar enrollment of first-generation students. However, first-generation status is not a variable that exists in a comparative dataset and thus, we cannot compile data across all possible institutions.
Action Item: Contact PAIR to assist you with answering this question.
Are any of the variables you want to use similar to each other or to the outcome in question?
Example: You want to benchmark JMU against peers to see how our graduation rate stacks up. You want to include retention rates as one of your variables. Because retention rates are highly correlated with graduation rates, including this variable may influence your ability to benchmark effectively.
Similarly, using two correlated variables may skew your peers towards those with these characteristics. If you want to include both retention and graduation rates as variables, you may skew the institutions included to those with high student success outcomes. In this case, you may want to choose either retention or graduation but not both.
Action Item: Involve constituents to select variables.
When determining variables to include in the analysis, it’s important to consider the constituents that have a vested interest in the peer group or outcome being measured. To that end, this is the first of two points in the peer group development process where stakeholder feedback should be sought.
We recommend the following strategy to collect feedback on variables:
- Following the process outlined in Step 4, generate a list of any possible variable you might want to include. At this point, do not worry about whether data exists, or a variable might be correlated with the measure of interest. Although you don’t want 100 variables, it’s okay if this list includes 25-30 variables.
Separate variables that offer concrete categories like the ones you identified in Step 3 (Private vs Public, Medical School vs No Medical School) from those that are continuous in nature (enrollment, endowment size, etc.). - Share the list with any core team, task force, or leadership group who has a vested interest in the peer group. For instance, if the measure of interest is faculty salaries, then share the variable list with the Compensation Advisory Council, Deans, Faculty Senate, and other leadership teams from academic affairs.
In some cases, this may have already been done in part A. Ask each member of the group to choose no more than 10 continuous variables for inclusion. In other words, of the 25 variables shown, what are the most important variables that each member thinks should be included? As a separate item, ask them to identify the top three categorical variables that might be used as filters (see Step 3 for description). This survey can be done on paper, or you can generate a short survey using Qualtrics or QuestionPro. See a screenshot below for an example of how you might structure this survey. - Once each group member provides their list of the top 10 continuous variables and top three categorical variables, tally the results to see which variables receive the most consensus support. It is likely that everyone will agree on some variables, while others will be less popular. Ultimately, identify the top 10-15 continuous variables that received the most support and the top three categorical variables that the group thinks should be used to filter out institutions.
- Share the list of consensus variables with whatever team or group is leading the peer group development effort. We recommend sharing the list of all variables that received votes and allow members to add support for variables that didn’t make the top list of consensus variables. Require group consensus, though, before adding or removing any additional variables at this point. You still want to limit your final list to 12-15 variables, as some may need to be removed for reasons outlined in Steps 5 and 6.
At this point, you would ideally have around 12-15 preliminary continuous variables to analyze, and three to four variables used to filter the initial population or the measure of interest. It is not recommended to use more than three to four categorical variables because otherwise, the population of possible institutions will become too small. It is better to have a few additional variables identified at this step as some variables may be excluded in the preliminary analysis (steps 5 and 6) for the following reasons:
- Data don’t exist for other institutions (see Step 5)
- Data are missing for some institutions (see Step 5)
- Variable is too similar to another variable (see Step 6)
- Variable is too similar to the outcome or measure of interest (see Step 6)