Simply the best,better than all the rest? Validity issues in selections in elite sport

Selecting sports and informants

One individual sport (alpine skiing) and one team sport (football) were chosen as study objects. Both sports are well-recognized sports with well-developed competition programmes at all levels. They are both Olympic sports and have many competitors of both genders world-wide vouching for that selection to competitions and tournaments are of significant public interest. The factor of public interest together with the importance of the outcome of selections made the elite level of each sport specifically interesting. Coaches at the elite level were chosen as respondents and regarded as experts due to their extensive experience and previous selection experiences (cf. 39 ‘expert judgements’).

In elite-level alpine skiing, athletes mainly compete within their respective national teams. Thus, coaches from national teams were selected as respondents. In elite football, athletes mainly compete in club teams, but some players also play a significant number of games in national teams. Hence, coaches from both club teams and national teams (coaches from seven women’s teams, five men’s teams and two mixed teams) were selected. In total, 14 top-level coaches were interviewed; six coaches from the most successful alpine skiing teams based on the world ranking (at the time of data collection), seven coaches from the most successful football teams based on the ranking (at the time of data collection) in the highest football league in Sweden and one national football team coach.

Collecting data

Interviews were chosen to collect data since written documentation about the selections was limited. In addition, it was deemed unfeasible to study selections through observations because the characteristics of selection processes were assessed as extensive and secret. In choosing between interviews and questionnaires, interviews were chosen because they were presumed to offer a deeper understanding ⁴⁵ of the selectors’ thoughts about the advantages, limitations and consequences of the selection methods than a questionnaire would have been able to provide.

An interview guide based on central aspects of validity theory was developed. It contained the following seven areas (items are exemplified in parentheses): (1) Background/context (Coaching background, position and responsibility/power of the coach?); (2) The idea of the selections (The goal/purpose of the selection?); (3) The selection, divided into three sub-areas: (a) The basis/criteria (What are the basis for the selections? Are selection criteria defined? What abilities/skills are judged/measured? Who has knowledge about criteria and similar?); (b) Values/philosophy behind the selections (What factors are most important to consider, if you have to choose between two similar/equally good athletes?); and (c) The process (Who are involved? Who has most power? How long is the selection period? Presence of pressure and bias?); (4) Difficulties in team selections (Are some selection situations more difficult? Possibilities to appeal? Protests/discussions about selections?); (5) Outcome and consequences (Are the ‘right’ athletes selected? Are selections evaluated?); (6) Reactions from the athletes (Fairness issues? Possibilities for athletes to influence selections?); (7) Other things (Gender issues? Advice and other thoughts about selections?).

Knowledge about the idea of the selections, what the selections was based on, criteria or similar and the selectors’ perceptions about these issues were constructed as ‘validity evidence based on test content’. ‘Validity evidence based on other variables and internal structure’ were constructed through the selectors’ views on how different criteria were measured and judged in relation to each other and whether they felt other variables also had an effect on the selections. The selectors’ view on the athletes’ reactions in the selection process were constructed to support the understanding of ‘validity evidence based on response processes’. A pilot interview was carried out and after some minor adjustments, mostly concerning the structure of the questions, the guide was used in the interviews. All interviews were carried out and recorded by the first author when meeting with the coaches in person.

Analysing data

The interviews were transcribed verbatim. Listening through the recordings of the interviews and reading the transcribed interviews over and over was an important first step of the analysis. The seven question areas were used as the main frame for the second step of the analysis, utilizing the meaning condensation technique. ⁴⁶ In the third step, data were categorised according to different perceptions and phenomena by way of meaning categorisation. ⁴⁶

The idea of the selections

The idea of the selections, the reasons for the selections, is connected to selecting the athletes that would perform achievements that would realize the team’s goals. To clarify and define what the selectors see as the goals are therefore a basic condition for validity discussions. These goals were by the coaches mostly connected to top-level outcome goals such as ‘winning as many medals as possible in the World Cups, Olympics and World Championships’, ‘winning the league’, ‘making the podium’ and the like. As both short and long term outcomes were of importance the coaches had to adapt both the selections and the work with the athletes to secure for both immediate and long term success. This was referred to as the ‘split philosophy’ and to include athletes from different levels was emphasised as being especially important in the National teams where the coaches had to secure continuous success for their country.

The goals for the teams were often set by the boards of the clubs or the federations and not by the coaches who were performing the actual selections. This was perceived by some of the coaches to create ambiguities between the coaches’ own goals and the goals delivered to them. More specifically, they expressed how they preferred to think more in terms of and work with performance and process-oriented goals rather than the outcome goals set by their boards. In trying to solve this dilemma, many of the coaches adjusted to use outcome goals as goals for the selections but performance and process-oriented goals when performing actual coaching activities. Some coaches expressed that if the board provided the goals for the team this would increase possibilities for more strategic long term planning than if the coaches, who might be in the team only for a short time period, would set the goals. On the other hand, some coaches revealed that if the goals set by the boards felt unrealistic, they secretly reformulated them. The latter strategy was claimed to have an impact on both the idea of the selection and the selections performed.

The criteria for the selections

Sport-specific rules limit the numbers of athletes who are allowed to participate in competitions and games. Thus, selections of athletes to teams are necessary when there are more athletes available than spots in the teams or competitions. In football, 11 players constitute a team, and World Cup alpine skiing competitions have pre-determined quotas (divided in a basic quota and a national quota) depending on the rankings of each country’s skiers and whether the country in question is hosting the World Cup. The maximum number of skiers allowed on a national team is nine (one from the basic quota and eight from the national quota) and the minimum is one (basic quota). The selection ratio varies as the total number of athletes available for the teams varies over time, nation, club and competition.

The criteria, guidelines or other grounds for selections show what abilities and factors that are valued by the coaches. The coaches had very different opinions about and ways of working with criteria. In some of the teams the criteria for the selections were very well defined and decided upon by the federation boards and are also communicated officially through web pages and other written material. Some of the coaches working with well-defined criteria saw this as a necessity in making fair selections as well as being able to justify their decisions to athletes, parents and federations. One disadvantage of using well-defined and officially communicated criteria (brought up by a coach who in general was positive towards criteria) was the risk that athletes used the selection criteria as their personal goals instead of focusing on reaching their own maximum level of performance. In other teams, the bases for the selections were not made explicit at all, nor were they communicated officially. The coaches not using well-defined criteria saw the benefits of this as it allowed them to be more flexible in their decisions.

A mix of current and past performances together with predictions of future performances was stated to form the bases for the selections, but with an emphasis on previous results, especially in alpine skiing. Some alpine skiing coaches used ranking lists as a very important selection tool, although they also reflected on the validity of these lists. Especially, the validity in one of the International skiing federation ranking lists, the FIS-point list (where athletes who have not participated in the same competitions are ranked in the same list), was seen as problematic.

In football, previous results were estimated using more subjective assessments of athletes. None of the football coaches had made their selection criteria official and the reason for this was that the opposing teams otherwise could get too much knowledge about their team. The idea of the selections in football was often connected to the team’s game plan. Some of the football coaches had developed relatively clear criteria describing the skills needed in relation to the demands of different positions in the team and their game plan. Others claimed that it worked the other way around – choosing their game plan depended more on the abilities of the players in the team. One coach stated that he used the game plan to support his ideas and selected the players based on intuition.

In alpine skiing, this subjective assessment was often called ‘the coaches’ discretion’. In some alpine skiing teams, the coaches’ discretion was defined more clearly to include factors such as skiers’ attitude, skiers’ potential, and technique. In other alpine skiing teams these more subjective assessments were not officially communicated but the coaches meant they still shared a common understanding about the bases for the selections.

According to the football coaches, selecting the best team was not necessarily about finding the best players but about finding the players who worked best together as a team. Balance between players with different skills, experiences and ages was also considered important. A common consideration among all the football coaches was the importance of the players’ behaviour and personality. They were all very firm on the notion that the players had to have a good character to be selected to the team. The behaviour and personality of the athletes was also mentioned as important factors in alpine skiing since the athletes travel together as a team for several months per year even though they compete individually. A statement from one of the alpine skiing coaches provides an example: ‘Sometime you make a discretion call because you need a guy in the team for morality and to push the other guys…’ (Alpine skiing coach men’s team 1).

Interestingly enough, when the coaches discussed technical skills, they seemed to look for athletes they assessed as having a good chance of developing the necessary skills. But when it came to behavioural and psychological factors, the tendency was more on finding athletes who already possessed those skills.

The above reasoning shows how selections are connected both to the formation of a team for the season and to a specific game or competition. In addition, players’ current form, injuries and the opponents were emphasised as important factors when teams for specific football games were selected. In alpine skiing, selections to specific competitions (World Championships, Olympic Games and similar) seemed to be based to a high degree on former results and ranking lists (independent on how much impact these lists had on the selections to the teams).

Weighing these different rationales, criteria, abilities and skills towards each other clearly impact on selection outcomes. However, it was in some teams unclear how these different grounds were weighed and perceived by the coaches. In most of the teams, data from physical tests and different kinds of statistics were used more as ‘tools for quality control’ rather than as parameters in the selections. The ranking lists in alpine skiing were an exception though, as some coaches based most of their selections on them. In general, all of the coaches seemed to be of the opinion that despite test results and ranking lists, a lot more had to be considered in determining whether an athlete would become successful or not. They mentioned how the experience of analysing athletes they have gathered over the years was key in predicting successful athletes. The coaches’ eyes were highlighted as the most important tool in the selection process and a highly valued complement to (and by some even a replacement for) statistics and tests. One of the respondents elaborates on this:

Yes, we do make some tests but I am not so much in favour of tests, because a coach’s most important tool is the eyes. You can never, never forget that the more tests you make, the further you get from your most important tool [the eyes] and I think it is priority number one… (Soccer coach women’s team 1)

The selection process

Data show how the handling of pressure and bias, the communication of selection grounds, the power structure among the selectors and whether the selections were based on interaction or on independent judgements by individual selectors, impact on the validity in the selections. The actual process of selecting athletes was performed quite differently between the teams in the study. The impression was that the coaches who used more set and well-defined criteria also had more formal meetings to select athletes. Some selections took months to decide while others were processed over a few days or even hours. Since the alpine skiing coaches are busy traveling with the national team, they have little chance of discovering new talents. Instead, they have to trust judgements from other coaches involved in these athletes’ careers.

In both sports, the head coach was the one making the final selection decisions and in many teams those decisions then needed to be approved by the board of the club or the federation. In all except for one team the selection decisions were discussed among the coaches, even if the head coach had the final word. One team differed a lot from the rest, as the coach for this team made all the decisions himself. He talked and listened to the other coaches prior to the decisions, but when it came to the final decision he wanted to make it by himself. His rationale for the decisions was intuition and a gut feeling, although he seemed to have a rather structured way of thinking in terms of different sets of formations of players and weighing them towards each other.

None of the coaches expressed being pressured by their boards to select certain athletes, but it was mentioned how investments in certain high-profile football players to some extent affected who should play. One football coach exemplified this by describing how choosing between two equally skilled players in the team, one of whom was bought for a lot of money, could be affected by the importance for the club to get value for its money. Also, when aiming to sell a player it was perceived important to give this player some playing time to attract other clubs. Overall, the football coaches felt pressure from the media, agents, the general public, sponsors and even parents trying to influence the selection. It was also mentioned that spending a lot of time with the athletes could affect the decisions if well-defined selection criteria were lacking. Most coaches perceived the selection process as difficult but two coaches thought it was fairly easy to select a team. One of the respondents expands on the difficulties in making decisions that could have great implications for an athlete: ‘Of course it is hard, you pretty much decide about the future of a young person’ (Alpine skiing coach women’s team 1).

Selecting the majority of the athletes was not perceived as difficult as selecting the very last ones. It was also perceived as difficult to tell someone he or she had not made the team this time, especially if this athlete had been part of the team for a long time. Still, all coaches clearly expressed that this was part of their job and something they had to be prepared to do. One of the coaches (who did not work with well-defined criteria) expressed difficulties in determining how to tell an athlete that he or she was not going to be selected, especially since the decision most often was based on intuition or a gut feeling. To make up for this, the coach felt he had to come up with some other more valid explanation to the athlete, even if this was not the real reason. It is interesting to note that none of the coaches had any formal education for making selections, even though all of them had completed several levels of coaching education. The knowledge they had was acquired from their experiences during the selection processes they had been involved in.

In alpine skiing, the majority of the teams had some kind of official procedure for announcing the selections. New athletes and athletes who were cut from the teams were personally contacted by the coach or a team manager before the selections were made official. In football, the teams were usually presented to the players one or two days before the game so that they could adjust to being selected or not. If a player was supposed to be dropped from the team, the coaches usually spoke to the player before the announcement of the team. Although the football coaches considered communication about selections important, some of them were also hesitant about explaining too much, as that could make the athletes to focus on negative things about themselves. Noticed here should be that all but one coach had personal contact with the athletes to let them know if they were selected or not selected.

The outcome and consequences of the selections

So are the ‘right’ athletes selected, then? When it comes to alpine skiing, the coaches expressed confidence in selecting the ‘right’ skiers even if not all of the chosen athletes achieved the results expected from them. In football, on the other hand, all coaches admitted they had selected players whom they in hindsight realised were not the ‘right’ ones. Explanations provided by the coaches were overestimation of players’ capacity and having too many feelings involved in the selections instead of being consistent and following their coaching philosophy. This is expressed in the following way by one of the respondents: ‘Well, it has happened a couple of times, it happens regularly, ooh… now I am acting wrongly… and then it is because you act too much on feelings and are not being consequent in your leadership’ (Soccer coach women’s team 1).

Failing to be selected was taken seriously by the athletes, in the coaches’ experience. The alpine skiing coaches talked about being selected to a national team as a career-determining event for an athlete, since being selected is connected to access to better training opportunities (qualified coaches, competitive teammates, good training facilities) but also to getting access to advanced equipment, sponsors, insurance and other perks. Being close to the coaches’ eyes might also be of importance for upcoming selections. Some of the football coaches thought that some players took the selections too seriously. One of the coaches expressed that he felt the selections had become too connected to the players’ identity and that being selected involved being pointed out not only as a good footballer but also as a good person.

There were also consequences for the coaches connected to the selections such as if they make bad selections and not reached the expected goals they might lose their jobs as well as their respect. However, evaluations of the selections were not performed in a systematic way and were seen as a challenging task.

Evidence based on test content

From a validity perspective, it is important to identify the goal of the selection. However, the findings show some discrepancies concerning the goals. Partly this had to do with how developmental goals versus outcome goals was treated. This problem was solved by some of the coaches through (secret) modifications of official goals to make them suitable for their practices and this might have effect on the validity in selections. Previous research has pointed to how the use of different goals in performance rating tends to result in different ratings even though the same performances are observed. ⁴⁷ Another observation we make is how large the differences are in how explicit the selection criteria are constructed and in the coaches’ views on the importance of having and communicating these criteria. This also has implications for the validity of the selections with regard to what is actually used as the bases for the selections and how stable and predictable these bases are. Yet other implications for validity issues are the statements provided by some coaches (who used explicit criteria) that they felt their existing criteria prevented them from selecting the athletes they wanted. The complexity of the sport and the difficulties to develop useful criteria was expressed as a reason to not use well defined criteria. The differences seem in part being connected to the context and the variability of athletes to choose from.

Evidence based on internal structure and relations to other variables

How much weight different bases are given in relation to each other impact the selection outcome. In the teams where the weight between criteria was unclear the risk increases that the selectors’ individual experiences and preferences as well as culture and situational factors prevent consistent selections. Interestingly, some coaches from teams where criteria were communicated clearly, partly referred to this as a limitation. For example, when performance criteria such as ranking lists (the more objective criteria) were considered more important than coaches’ discretion (the more subjective criteria). Unclear agreements about the relation between different criteria as well as the coaches’ disapproval of how different criteria were valued highlight concerns related to the internal structure of criteria.

The emphasis placed by the coaches on their own judgements as experts in their fields are very similar to how professionals in other fields value their judgements in comparison to more objective criteria. Highhouse, ⁴⁸ for example, has shown that even if the effectiveness of selection aids such as paper-and-pencil tests, structured interviews and mechanical combination of predictors have proven to outperform informal discussions, employers still argue that they learn more from an informal discussion. As an explanation, Colbert et al. ⁴⁹ argue that managers will not be motivated to put research into practice if they do not agree with the research findings. Highhouse ⁴⁸ also argues that an implicit belief among managers that experience will improve prediction of human behaviour is just a myth based on an overreliance on intuition. In that specific regard, the literature has for a long time been unanimous that structured analysis outperforms intuition.^49,50 However, this finding has recently been questioned based on a meta-analysis showing that mechanical predictions (statistical predictions, computer programmes, etc.) are typically as accurate as clinical predictions. In half of the studies, clinical judgements (by subjective methods) proved to be almost as good as mechanical predictions. At the same time, it has also been argued that relying too heavily on analytical methods for assessing performance might lead to a loss of a more holistic view as a result of overlooking certain important characteristics. ⁵¹

The perceived importance of athletes’ behavioural skills and personality is also in line with findings in the sport-related literature on the value of psycho-behavioural skills.^25–27 This finding also fits well with findings in the personnel selection literature, in which general mental abilities (GMA) are found to be the strongest predictor of future performance and learning.^52,53 However, it is highly uncertain whether coaches have sufficient knowledge to validate psychological skills in a manner requested by the literature. ⁵⁴

As our findings show, some of the coaches in this study expressed that they prefer to select athletes who have already developed the desirable behaviour, which they perceive to be fixed and stable. It is argued in the literature, however, that personality is both flexible and dynamic ⁵⁵ and changes over time. ⁵⁶ Whether the coaches feel it is their job to develop these skills versus sport specific skills and what competence they have for doing this probably affect their answers. Arguably, it could also be questioned what skills athletes at this level should have developed already before entering top level teams.

Evidence based on response processes

Evidence based on response processes highlighted how many of the coaches experienced how the athletes who were selected perceived the selections as being fair and how the athletes not being selected perceived the selections as unfair. Findings also show how the coaches, independent of if selection criteria were made explicit or not, felt that the athletes had an understanding of what it takes to be selected. In the literature on personnel selection it has long been argued that applicants’ reactions to the methods used for hiring should be considered ⁵⁷ and that organisations should pay attention to the effects of selection procedures on the applicants. ⁵⁸ In a study of selections to sport teams, Stevenson ⁵⁹ showed how the athletes’ perceptions of the fairness in the selections were related to the selection procedures. Selections are often discussed from the perspective of the person who is not selected, but it is equally important to realize how selections also affect the athletes that are selected. It has been shown that being hired under conditions perceived as unfair may have negative effects on the applicants’ self-perception. ⁶⁰

Although the boards of the federations and clubs had to approve the selections in some cases, the head coaches claimed to have the final word in the selection processes. Even so, it appears quite unclear how the mandate to select athletes was distributed. In addition, since no explicit tools for preventing biased decisions seemed to exist, the location of the actual power to decide is even more difficult to pinpoint. But even if discussions about selections took place in an open and searching climate, findings on the vagueness of selection criteria indicate that the space for interpretation, and thus for bias, was considerable.