Developing 360-degree Competency Model

Here is a road map that helps our clients and friends create their own competency-based 360 model. We hope you find it useful.

There are 5 steps for collecting and analyzing data when developing a 360-competency model from scratch:

1. Determine data collection methodology.

2. Conduct interviews and focus groups.

3. Perform job observations.

4. Analyze data and develop an interim competency model.

5. Validate the model.

Challenge: Ensure the accuracy and usefulness of the information collected by formulating and testing assumptions and ideas about the key factors required for success in the role. The challenge is to maintain a balance between data collection and data analysis and allow each activity to drive the other in equal measure.

Step one: Determine Data Collection Methodology

The purpose of data collection is to learn about critical incidents/stories that reflect effective performance of the job or role targeted in the study.

  • Decide who will be interviewed (population or sample size).
  • Determine data collection format (one-on-one interviews/focus groups).
  • Choose data recording methods (taping, using two interviewers, extensive note taking by one interviewer).

Step two: Determine Data Collection Methodology

Provide the interviewee with background on the project and a preliminary set of questions to ensure the discussion is focused and succinct.

  • Use open-ended questions (allows interviewee to respond in a full and complete manner).
  • Ask for stories and examples (elicits what they do, how they do it, and why they do it).
  • Probe for specifics (ask follow up questions to uncover the thought processes, specific behaviors used to resolve a situation).
  • Avoid leading or directing the interviewee (by using empathetic responses or prompting them).
  • Establish a comfortable, open environment (begin with a few words of welcome; have people introduce themselves etc.)
  • Let the interviewee talk (90% of the time).
  • In regard to focus groups, have an agenda and stick to it (ensure consistency of data collection by making sure all the questions are covered in the time allotted).
  • Ensure that everyone participates (individuals should not dominate the conversation and facilitator should engage the quiet individuals).

Step three: Direct Observation of Incumbents

What people say they do and what they actually do sometimes differ because they may describe the ideal behavior or expected response.

  • Direct observation offers a reality check against the information gathered during the interviews and creates a realistic picture of effective job behaviors.
  • Observe at least three people who have not been previously interviewed from each performance category – exceeds, meets and falls below criteria.
  • Observations need to take place on a typical day – no special events scheduled.
  • Observer should sit away from the main interaction but close enough to hear the conversation.
  • Observer should avoid making eye contact with incumbent or others present, and avoid inappropriate cues i.e. raised eye brows, heavy sighs, or slumping).

Step four: Develop an Interim Competency Model

Examine the data collected during interviews, focus groups or observations for themes and patterns, which are then analyzed to identify relevant competencies.

  • Identification of themes and patterns (individual work). The interviewer combs through the information gathered and identifies common themes and patterns.
  • Start with a general idea (based on repeated behaviors and skills., and patterns emerging in the interviewees descriptions of how they handled specific situations).
  • Start with a blank slate (data analyst records any quote that refers to particular skill, knowledge, or characteristics. Using the complete list, the frequency of similar comments is noted).
  • Identification of themes and patterns (project team meeting). All individual interviewers review the themes and patterns emerging from the interviews and the supporting evidence of such themes and patterns.
  • As a result of the group review some themes will be reshaped, others eliminated and interim competency model will be produced.
  • The group should agree in advance on a format to present its members’ findings allowing them to compare “apples to apples.”

Step five: Finalize Your Model

For the model to be used effectively it needs to have face validity (the competencies described in the model must make sense to those performing the job) and it must be validated as a predictor of successful performance (competencies must be demonstrated by the top performers in the job). To assure you’re there, consider the following:

1. Test the competency Model

The purpose is to test the accuracy and relevance of the interim model with a broader cross section of incumbents and stakeholders than that included in the initial data collection phase.

  • Circulate the model to a wider audience (ensures all relevant competencies have been captured and uncovers meaningful demographic or functional differences in the applicability of the model).
  • Collect data (survey or focus groups) to determine how well people currently in the position or role feel that the competencies reflect the skills, knowledge, abilities,; and personal characteristics necessary to succeed.

2. Analyze the new data and refine the model

New data will enable the interviewer to determine if their assumptions about needed competencies are shared by most incumbents and informed observers. Eliminate items considered unimportant, add or modify items viewed as more relevant.

  • Analyze survey data to determine which competencies are perceived as most essential (do managers rate a certain competency more relevant than non-managers?).
  • Analyze focus group data (sort through attendee responses and identify common themes/patterns and review them with entire team)
  • Based on qualitative and quantitative data refine the interim model to reflect the new information. Competencies may be added, deleted or redefined.
  • Finally, circulate the model to a cross section of people who participated in the study. This ensures that the key elements and critical success factors of the job have been captured.

3. Validate and Finalize the Competency Model

By now the interim model has been tested and refined and determined to have a high degree of face validity. If the model is to be used for training and development only, face validity may be sufficient. However, if the model is to be used as a basis for selection, appraisal, or compensation systems the additional step of validating the model is essential.

  • If the model is used for hiring decisions, it needs to show how well the competencies predict success.
  • If the model is used for performance appraisal decisions and recommendations, it needs to show how often the most effective managers use the competencies and when they correlate with measures of performance such as productivity, increases in sales or profit, employee satisfaction etc.

A validation study:

  • Convert competencies into a 360 questionnaire. Select four to six specific and observable behaviors that illustrate competency on the job. Arrange the items so that they illustrate each competency in a survey format. A rating scale accompanying such a list allows bosses, direct reports, colleagues, and customers to rate the importance of the competency, how often it has been used, and how often it should actually be used.
  • Distribute the questionnaire to managers (appropriate sample size) who are above, at and below the performance criteria developed during the first stage of the project (keep performance levels confidential).
  • Instruct them to distribute the questionnaire to five or seven direct reports, five to seven colleagues, and all of their supervisors.
  • The raters complete the questionnaire and return it. Use a third party outside the organization to conduct perform the statistical analysis, and to ensure confidentiality.
  • Tabulate the scores for each individual and analyze the strength of the relationship between the competencies and the aggregate ratings of the different performance groups. If there is a significant correlation between the competency and the high- performance group, the competency is said to have concurrent validity.
  • Significant differences between competency scores of high performers and low performers are also considered.
  • If there are weak relationships between a competency and the high performance group, consider reviewing the clarity of the behavior descriptions. The soundness of the performance criterion used to select the groups could also explain the weak relationship.

4. Finalize the model

Using the analysis of the 360-degree data, finalize the model by eliminating any categories or items that do not correlate with effectiveness. At this point the competencies and their associated behaviors can be used as the basis for developing or enhancing tools used for human resource decision-making and establishing performance coaching priorities.