Which of the following is a disadvantage of using the critical incident technique for job analysis?

Self-Report

One version of the self-report approach assumes that people are aware of, and understand the implications of, the strategies they apply as they execute tasks lying in different domains. Of course, given this assumption, it is possible to identify heuristics simply by asking people about the heuristics they apply in different endeavors. In accordance with this proposition, in one study, undergraduates were asked to indicate whether they applied heuristics such as visualization, step-by-step analysis, and analogies when working on interpersonal, academic, or daily life tasks. It was found that people could describe where, and how frequently, they applied these general heuristics. This direct reporting approach, however, has proved less effective when specific heuristics, particularly heuristics applied nearly automatically, are under consideration.

When it is open to question whether people can say when, and how, they apply heuristics, an alternative strategy is often used. In an approach referred to as the critical-incidents technique, people are not asked to report on strategies applied. Instead, they are asked to indicate the behaviors they exhibited when performing certain tasks. Specifically, they are asked to describe the setting, the event prompting performance, the actions taken in response to the event, and the outcomes of these actions. Content analysis comparing good and poor performers, or good and poor performances, is used to identify more or less effective performance strategies and the behaviors associated with application of these strategies. Assessment occurs by having people indicate either the frequency or extent to which they manifest behaviors associated with an identified heuristic.

Testing

In testing, people are not asked to describe heuristics or the behaviors associated with these heuristics. Instead, the capability for applying heuristics is inferred based on peoples' responses to a series of test items. In one variation on this approach, the objective scoring approach, test items are developed such that the problems presented call for certain processing activities. Response options are structured to capture the application of heuristics linked to effective process application. In one study, test items were developed to measure selective encoding, selective comparison, and selective combination. It was found that gifted students differed from nongifted students in that they were better able to identify relevant information (selective encoding).

In the subjective scoring approach, problems (typically open-ended problems) are developed in a way that a variety of responses might be used to address the problem. Judges are then asked to review peoples' responses and assess the extent to which they reflect the application of certain heuristics. This approach is applied in scoring the divergent thinking tests commonly used to measure creativity by having judges evaluate responses with respect to three creative strategies: generating a number of ideas (fluency), generating unusual ideas (originality), and generating ideas through the use of multiple concepts (flexibility). In another illustration of this approach, army officers were asked to list the changes in peoples' lives that might happen if certain events occurred (e.g., What would happen if the sea level rose?). Judges scored these responses for heuristics such as use of a longer time frame, application of principles, and a focus on positive versus negative consequences. Use of longer time frames and application of principles were found to be related to both performance on managerial problem-solving tasks and indices of real-world leader performance, yielding multiple correlations in the 0.40s.

4.12.5 Forced distribution

Here appraisers rate people on a forced distribution of categories – for instance, 10 per cent low; 20 per cent low average; 40 per cent average; 20 per cent high average; 10 per cent high. This can be seen in Table 4.10. Again, the appraisers’ discretion is constrained with this method, so it is less used by Asian organisations, although international employers regularly use this system to compel local managers to make judgements about performance.

Table 4.10. Example of forced distribution

4.12.6 Rating scales

Here various attributes of performance are listed (for example, accuracy, knowledge, quality of work) and the person is evaluated on each of these dimensions (usually from the job description) individually (see Chapter 2). A scale is often used – for example, 1 poor; 2 below average; 3 average; 4 above average; 5 excellent. An overall score is then calculated, so there is some ease of interpretation. An example of a 5-point rating scale can be seen in Table 4.11. When formal appraisal systems are used in Asia, this is one of the oldest and most popular methods.

Table 4.11. Example of rating scale

4.12.7 Critical incidents

This method is a procedure for collecting observed incidents that are seen as important or critical to performance. A list (or log) of incidents is compiled, with details of examples of positive and negative employee performance recorded and kept. High-performing employees are identified as those performing well during many critical incidents. It is considered that adequate, but not exceptional, employees are those involved in only a few critical incidents.

4.12.8 Behaviourally anchored rating scales (BARS)

These appraisal methods specify definite, observable and measurable behaviour. The format uses critical incidents to serve as ‘anchor statements’ on a scale. The form contains defined performance dimensions, each with critical incident anchors (examples of actual behaviour on the job, not general descriptions or traits). The appraiser then rates the person against these predetermined factors. An example can be seen in Table 4.12. These methods are used less often in some parts of Asia because of their complexity and need for judgement. However, the advantages of such methods include developing the following.

Table 4.12. Example of BARS performance dimension

Performance dimension scale development under BARS for the dimension ‘Ability to absorb and interpret policies for an employee relations specialist’ (Rated 1–9).

Source: Adapted from DeCenzo and Robbins (1999)

Validity of each of the main duties (obtained from job descriptions)

Agreement over suitable descriptions for each category of behaviour

Economies of scale if many people have the same job description

4.12.9 Behavioural observation scales (BOS)

Like the above technique, this method uses critical incident techniques to identify a series of behaviours in the job. However, the format is different in that, instead of identifying behaviours exhibited during the rating period, the appraiser needs to indicate on a scale how often the person was actually observed engaging in the specific behaviour under review. An example of this method can be seen in Table 4.13. Because of its complexity this method is less common in some parts of Asia.

Table 4.13. Example of BOS performance dimension

Sample BOS items for the performance dimension ‘Communicating with subordinates’ (Rated 1–5).

Source: Adapted from Fisher et al. (1999)

4.12.10 Peer

With this method, colleagues and co-workers at the same level assess each other’s performance. This is increasingly popular in the West where team working has been encouraged. One advantage of this method is that it is based on actual experience in the workplace. In some Asian cultures, however, seeking the views of colleagues and co-workers at the same level risks losing power and respect. Asian group loyalty would lead almost all peers to rate each other highly.

4.12.11 Subordinate

With this method, employees are asked to rate their bosses. There have been experiments with this method at DuPont, Nabisco, Mobil, GE and UPS in the US, and BP in the UK. This method is seen as more ‘democratic’ and useful in improving channels of communication. In much of Asia there would be not only loss of ‘face’, but also loss of power, respect and loyalty in using such a system. Therefore, this method is rarely acceptable in parts of Asia.

4.12.12 Self-appraisal

One of the more recent methods to take off has been self-appraisal. To try to seek more differentiation in rankings with such formats, people can be asked to rank different aspects of their performance in relation to other aspects. In Asia it is found that there is a range of responses to self-appraisal so that it is expected that, for example, the Malaysian or Taiwanese would be modest in their assessment, while the Pakistani or Indian would be more generous in their self image. There are also variations by sector and in organisations so that, for example, investment bankers will be more inclined to rate themselves highly, while social workers are more inclined to say that they could do better.

4.12.13 360-degree appraisal

This is one of the latest trends in this area in the West. This method involves as many different people as possible in performance evaluation. This can range from subordinates to peers to managers, customers and clients. As most Asians are unwilling to be rated by peers or subordinates, this technique is less common in parts of Asia, although some international organisations attempt to impose the system.

4.12.14 Management by objectives (MBO)

This method was traditionally used more for professional and managerial grades, but is still found in many organisations at lower levels. This is seen particularly when performance management and assessment (PMA) is used. There is commonly a cycle, which may include the following stages.

Discussion – forms are completed as the basis for initial discussions.

Agreement – reached on objectives/goals to strive to achieve during the period.

Training and development – needed for achievement of objectives.

Modification – as a result of changed circumstances (corporate policy, environment).

Review – at the end of the period to see if goals have been met and fresh goals set.

Better managed Asian organisations tend to use some form of MBO; those most concerned with improving performance use PMA.

4.12.15 Interviews

While we have seen that there are many types of performance appraisal, those that involve an interview at some stage are common. Interviews are an important part of the process, not least as they are integral to the communication and feedback that is often involved. Some of the same points that apply to employee selection interviews (see Chapter 2) can be followed as ground rules for performance appraisal interviews. These can be seen in Table 4.14. Problems are encountered, however, when evaluating the performance of an individual, irrespective of the interview style.

Table 4.14. Interview structure

Source: Adapted from Torrington and Hall (1998)

2.1 Unit of Analysis

The job-analysis unit is the job itself and the tasks that comprise it. For this reason, many job-analysis systems relate directly to the tasks and the immediate work conditions. More comprehensive job-analysis systems (see Dunckel 1999, Gael 1988) take into account the fact that essential features (e.g., the potential of the individual for defining goals self-reliantly) and results of the tasks become comprehensible only if the overall organizational conditions and the knowledge, skills and attitudes of the workers are taken into consideration. They therefore extend the analysis to include both the organizational conditions (e.g., degree of responsibility, codetermination potential, values and norms, leadership climate) and the working individuals with their respective knowledge, skills and attitudes.

Methods can be classified, based on widely accepted criteria, according to whether they are more job/task-oriented or worker/person-oriented. Job-oriented analysis is concerned with analyzing tasks, job conditions or job features, without regard for the concrete individuals involved and their different knowledge, skills, abilities and attitudes. Person-oriented analysis, on the other hand, is centred on the person and specifically concerned with differences between individual workers in the perception, interpretation and performance of the job.

Typical job-oriented approaches are: Functional Job Analysis, Task Inventories, Health Services Mobility Approach; typical person-oriented approaches are: Position Analysis Questionnaire, Critical Incident Technique, Ability Requirement Scales (see Gael 1988, Ghorpade 1988).

The distinction between job-oriented and person-oriented analysis is of conceptual and practical significance. In stress research, for example, it is conceptually important to begin by determining stress factors independently of the person involved (and their individual perceptions and coping behavior) in a job-oriented manner, then going on to examine how objectively identical stress factors are perceived and dealt with differently by different individuals and how they affect different people. However, the (person-oriented) analysis of these inter-individual differences means first determining objective stress factors, because we are concerned here with inter-individual differences in relation to objectively identical stress factors. This distinction is of practical significance when planning future workplaces for which there are not yet any workers. Here, the job-oriented approach is the only way of obtaining job analysis information.

2.2 Intended Applications

Techniques and methods have been and are being developed for different goals and potential applications. Job-analysis literature contains numerous different lists of goals and proposed uses for job-analysis information (see Ash 1988, Lees and Cordery 2000). Basically speaking, two goals complexes can be distinguished: work/job and organization design, and personnel development. More concrete aims of job analyses are, for instance, comparing work activities, changing and planning the work situation and organization, determining skill requirements and factors defining aptitude requirements, technology assessment, maintenance of industrial health and safety standards, and job evaluation.

Job-analysis methods and techniques play an important part in human-resources management and personnel selection (see Algera and Greuter 1998). There are used to determine more precisely the requirements a person must meet to perform their work tasks. These requirements may be specified as tasks to be accomplished, as behavioral requirements (e.g., required behavior or behavioral repertoire), as eligibility requirements (e.g., knowledge and skills) or as trait requirements (e.g., abilities and interests). For each of these requirement types, there are a number of techniques available (see Gael 1988).

If the results of such analyses are combined, job analysis can also be used to describe (work) roles as defined by Katz and Kahn (1978). According to these authors, roles are standardized behavior patterns demanded of all persons involved in a given functional relationship. Here, job analysis can also help identify when and under what conditions role expectations (of the different organization members with respect to individual workplace occupants) can lead to conflicts and job stress.

The intended applications of job analysis are not necessarily independent of one another. If, for example, the goal is to change the work organization, it will often be necessary to analyze the technological implications and the consequences for the workers involved as well.

Besides the application purpose, the application area also plays an important role. Job-analysis techniques and methods consider different levels and units of an organization. A distinction must be made between:

(a)

sectors (e.g., industrial, administrative, service);

levels of an organization (e.g., enterprise as a whole, business division, department, workplace group, workplace, work task);

professional groups (e.g., executives, specialized professional groups); and

activity classes (e.g., assembly, control and monitoring activities, administrative activities, service activities).

Depending on the specific concern, the emphasis will be on different information. Job analysis, too, faces the problem of breadth vs. depth. The more detailed the information, the more limited the application purpose. It may therefore be a good idea to proceed in several steps, starting with rough analyses to determine the key analysis areas, which are then analyzed in greater detail.

2.3 Theoretical Foundation

The theoretical foundation largely determines which information is captured at which level or with which analysis unit.

Work studies as defined by Taylor or Gilbreth (see Ghorpade 1988) are based on an additive movement (or motion) model. Techniques working on this basis thus attempt to define elementary movement units (e.g., grasping with the hand), combining these additively in order to then determine, say, the standard time required to perform the work.

Approaches rooted in behaviorism also attempt to identify elementary units (of behavior) (e.g., processing materials, recognizing optical differences). Their units are bigger, though; they are therefore not movement- but behavior-oriented.

These approaches, however, fail to take into account the fact that movements merge to form ‘wholes,’ are integrated in complex webs of activity and the regulating mental processes and representations and are codetermined by these. This is why many recent developments, especially in Germany, are based on the ‘action regulation theory’ (see Frese and Sabini 1985, Oesterreich and Volpert 1987), thus giving priority to questions relating to the psychological regulation of action, the level of psychological regulation, the completeness of actions, the degrees of freedom (Hacker 1998) or the scope for action or decision. The guiding idea here is that of humane work, i.e., work geared to human strengths and enabling individual workers to perform their job under permanently tolerable conditions, without impairment of their well-being and in a manner conducive to their personal development.

In the emphasis they place on characteristics such as scope for action, variability, identity and importance of the task, action-theory approaches are in keeping with the traditions of industrial sociology, e.g., the work of Turner and Lawrence (1965) and the work of Hackman and Oldham (1975) that builds on this. Since the latter approaches are specifically concerned with questions relating to the ‘motivation potential of work,’ they can also be classified as oriented to motivation theory.

Besides drawing on approaches based on behavior, action, and motivation theory, job-analysis systems also have recourse to concepts of stress theory, ergonomics and human engineering; in addition, worker-oriented approaches draw on concepts of personality theory.

2.4 Method Users

The main users of job-analysis methods are the workers themselves, first-level supervisors, higher-level supervisors, job analysts, technical experts and other company experts, but also works and staff councils.

Whether and to what extent a method can be used depends, among other things, on the application requirements that must be met by the users. Important factors here, besides formal qualifications, are the amount of experience the users need in analysis techniques, whether they can teach themselves how to use such techniques or whether special training is required.

2.5 Methods of Data Collection

There are also fundamental differences between techniques in terms of the data-collection methods used, e.g.:

(a)

interview methods (e.g., individual and group interviews, technical conferences with experts, more or less structured questionnaires and check lists);

observational methods (e.g., direct and indirect observation, continuous observation, work sampling);

analyses of company data (e.g., working hours lost, accident statistics, workplace descriptions);

analyses of documents (e.g., file analyses, form analyses);

work activities performed by job analysts.

Each of the methods has its advantages and disadvantages. It is therefore a good idea to combine several methods (see Gael 1988). These advantages and disadvantages can be highlighted by comparing interview and observation methods.

2.6 Methodological Standards

Job-analysis methods differ, among other things, in their degree of standardization—ranging from non-standardized ‘free’ descriptions to semi-standardized interviews to observations and interviews following exactly prescribed rules. They differ in the amount of time they take (from 30 minutes to several hours), their psychometric quality, the number of dimensions captured, etc. Which method to choose is not generally decidable; this depends largely on the intended application and on available knowledge and theory building with respect to the problems under investigation.

Scientifically based methods should be reliable and valid:

(a)

The reliability of a method shows whether and to what a extent a method can be used to obtain ‘stable,’ ‘reliable,’ or ‘replicable’ results. Ideally, repeated measurements of the same object should show as little deviation as possible (see Oesterreich and Bortz 1994).

The validity of a method indicates the extent to which it actually measures what it is supposed to measure.

Examining these quality criteria is not only of scientific interest, but also of utmost practical importance. If the results of the analysis are to have practical consequences, e.g., for job design, decision-makers must be able to depend on the fact that the results are reliable and valid.

2.7 Results

Job-analysis methods provide quantitative and qualitative results. Qualitative results are mostly verbal, narrative descriptions of the job or the tasks; quantitative results are presented in numerical—and in some cases graphical—form. Complex methods generally present results in both qualitative and quantitative form.

Unit of Analysis

The job analysis unit is the job itself and the tasks that comprise it. For this reason, many job analysis systems relate directly to the tasks and the immediate work conditions. More comprehensive job analysis systems (Dunckel, 1999; Gael, 1988) take into account the fact that essential features (e.g., potential for defining goals self-reliantly) and results of the tasks only become comprehensible if the overall organizational conditions and the knowledge, skills, and attitudes of the workers are taken into consideration. They therefore extend the analysis to include both the organizational conditions (e.g., degree of responsibility, codetermination potential, values and norms, leadership climate) and the working individuals with their respective knowledge, skills, and attitudes.

Methods can be classified, based on widely accepted criteria, according to whether they are more job/task-oriented or worker/person-oriented (Brannick et al., 2007). Job-oriented analysis is concerned with analyzing tasks, job conditions, or job features, without regard for the concrete individuals involved and their different knowledge, skills, abilities, and attitudes. Person-oriented analysis, on the other hand, is centered on the person and specifically concerned with differences among individual workers in the perception, interpretation, and performance of the job. Typical job-oriented approaches are: functional job analysis, task inventories, health services mobility approach (Gael, 1988; Ghorpade, 1988); typical person-oriented approaches are: position analysis questionnaire, critical incident technique, ability requirement scales, cognitive task analysis (Gael, 1988; Ghorpade, 1988).

The distinction between job-oriented and person-oriented analysis is of conceptual and practical significance. In stress research, for example, it is conceptually important to begin by determining stress factors independently of the person involved (and their individual perceptions and coping behavior) in a job-oriented manner, then going on to examine how objectively identical stress factors are differently perceived and dealt with by different individuals and how they affect different people. However, the (person-oriented) analysis of these interindividual differences means first determining objective stress factors because we are concerned here with interindividual differences in relation to objectively identical stress factors. This distinction is of practical significance when planning future workplaces for which there are not yet any workers. Here, the job-oriented approach is the only way of obtaining job analysis information.

Intended Applications

Techniques and methods have been and are being developed for different goals and potential applications. Job analysis literature contains numerous different lists of goals and proposed applications for job analysis information (Ash, 1988; Lees and Cordery, 2000). Basically speaking, two complexes of goals can be distinguished: work/job and organization design, and personnel development. More concrete aims of job analyses are, for instance, comparing work activities, changing and planning the work situation and organization, determining skill requirements and factors defining aptitude requirements, technology assessment, maintenance of industrial health and safety standards, and job evaluation.

Job analysis methods and techniques play an important part in human resources management and personnel selection (Algera and Greuter, 1998; Brannick et al., 2007). They are used to determine more precisely the requirements a person must meet to perform his or her work tasks. These requirements may be specified as tasks to be accomplished, behavioral requirements (e.g., required behavior or behavioral repertoire), eligibility requirements (e.g., knowledge and skills), or trait requirements (e.g., abilities and interests). For each of these requirement types, there are a number of techniques available (Gael, 1988).

If the results of such analyses are combined, job analysis can also be used to describe (work) roles as defined by Katz and Kahn (1978) (see also Dierdorff and Morgeson, 2007). According to these authors, roles are standardized behavior patterns demanded of all persons involved in a given functional relationship. Here, job analysis can also help to identify when and on what conditions role expectations (of the different organization members with respect to individual workplace occupants) can lead to conflicts and job stress.

The intended applications of job analysis are not necessarily independent of one another. If, for example, the goal is to change the work organization, it will often be necessary to analyze both the technological implications and the consequences for the workers involved.

Besides the application purpose, the application area also plays an important role. Job analysis techniques and methods consider different levels and units of an organization. A distinction must be made between

sectors (e.g., industrial, administrative, service);

levels of an organization (e.g., enterprise as a whole, business division, department, workplace group, workplace, work task);

professional groups (e.g., executives, specialized professional groups); and

activity classes (e.g., assembly, control and monitoring activities, administrative activities, service activities).

Depending on the specific concern, the emphasis will be on different information. Job analysis, too, faces the problem of breadth versus depth. The more detailed the information, the more limited the application purpose. It may therefore be a good idea to proceed in several steps, starting with rough analyses to determine the key analysis areas, which are then analyzed in greater detail.

Theoretical Foundation

The theoretical foundation largely determines which information is captured at which level or with which analysis unit.

Work studies as defined by Taylor or Gilbreth (see Ghorpade, 1988) are based on an additive movement (or motion) model. Techniques working on this basis thus attempt to define elementary movement units (e.g., grasping with the hand), combining these additively in order to then determine, say, the standard time required to perform the work.

Approaches rooted in behaviorism also attempt to identify elementary units (of behavior) (e.g., processing materials, recognizing optical differences). Their units are bigger and hence they are therefore not movement-oriented but behavior-oriented.

These approaches, however, fail to take into account the fact that movements merge to form ‘wholes,’ are integrated in complex webs of activity and the regulating mental processes and representations, and are codetermined by these. This is why many developments, especially in Germany, are based on the ‘action regulation theory’ (Frese and Sabini, 1985; Oesterreich and Volpert, 1987), thus giving priority to questions relating to the psychological regulation of action, the level of psychological regulation, the completeness of actions, the degrees of freedom (Hacker, 1998), or the scope for action or decision. The guiding idea here is that of humane work, i.e., work geared to human strengths and enabling individual workers to perform their job under permanently tolerable conditions, without impairment of their well-being and in a manner conducive to their personal development.

In the emphasis they place on characteristics such as scope for action, variability, identity, and importance of the task, action theory approaches are in keeping with the traditions of industrial sociology, e.g., the work of Turner and Lawrence (1965) and the work of Hackman and Oldham (1975) that build on this. Since the latter approaches are specifically concerned with questions relating to the ‘motivation potential of work,’ they can also be classified as oriented to motivation theory (see Work Motivation).

Besides drawing on approaches based on behavior, action, and motivation theory, job analysis systems also have recourse to concepts of stress theory (see Workplace Stress), ergonomics, and human engineering (see Human Factors and Ergonomics); in addition, worker-oriented approaches draw on concepts of personality theory.

Method Users

The main users of job analysis methods are the workers themselves, first-level supervisors, higher-level supervisors, job analysts, technical experts and other company experts, but also works and staff councils.

Whether and to what extent a method can be used depend, among other things, on the application requirements that must be met by the users. Important factors here, besides formal qualifications, are the amount of experience the users need in analysis techniques, whether they can teach themselves how to use such techniques or whether special training is required.

Methods of Data Collection

There are also fundamental differences between techniques in terms of the data collection methods used, for example:

interview methods (e.g., individual and group interviews, technical conferences with experts, more or less structured questionnaires and checklists);

observational methods (e.g., direct and indirect observation, continuous observation, work sampling);

analyses of company data (e.g., working hours lost, accident statistics, workplace descriptions);

analyses of documents (e.g., file analyses, form analyses); and

work activities performed by job analysts.

Each of the methods has its advantages and disadvantages. It is therefore a good idea to combine several methods (Brannick et al., 2007; Gael, 1988). These advantages and disadvantages can be highlighted by comparing interview and observation methods.

Methodological Standards

Job analysis methods differ, among other things, in their degree of standardization – ranging from nonstandardized ‘free’ descriptions to semistandardized interviews to observations and interviews following exactly prescribed rules. They differ in the amount of time they take (from 30 min to several hours), their psychometric quality, the number of dimensions captured, etc. Which method to choose cannot be decided in a general manner; this depends largely on the intended application and on available knowledge and theory building with respect to the problems under investigation.

Scientifically based methods should be reliable and valid:

The reliability of a method shows whether and to what extent a method can be used to obtain ‘stable,’ ‘reliable,’ or ‘replicable’ results. Ideally, repeated measurements of the same object should show as little deviation as possible (Oesterreich and Bortz, 1994).

The validity of a method indicates the extent to which it actually measures what it is supposed to measure.

Examining these quality criteria is not only of scientific interest, but also of utmost practical importance. If the results of the analysis are to have practical consequences, e.g., for job design, decision makers must be able to depend on the fact that the results are reliable and valid.

Results

Job analysis methods provide quantitative and qualitative results. Qualitative results are mostly verbal, narrative descriptions of the job or the tasks; quantitative results are presented in numerical – and in some cases graphical – form. Complex methods generally present results in both qualitative and quantitative forms.