Common factors are that these machines consist of at least two work- ing systems that are ... Furthermore, Figure 3 also shows how in the outer loop the primary power from the diesel engine is .... The main parts of the calculations were executed in MathCad worksheets that link to data files .... commands? The answer is no.
Study of a method for assessing operability of working machines in physical and virtual testing Reno Filla VOLVO CONSTRUCTION EQUIPMENT AB, ESKILSTUNA, SWEDEN
Abstract In this study of eighteen wheel loader operators, test-driving a machine in three different traction force settings, we found strong support for the hypothesis that the operator’s control commands can be used to assess the machine’s operability, at least in form of ease of bucket filling. The methods chosen to derive the control effort worked well and were computationally efficient. Keywords: operator; control effort; mental workload; operability; human-machine interaction; operator model; simulation
Whatever they say about reality, it's the only place where you can get a good steak. (Woody Allen)
This paper has been published as: Filla, R. (2012) “Study of a Method for Assessing Operability of Working Machines in Physical and Virtual Testing”. International Journal of Vehicle Systems Modelling and Testing, vol. 7, no. 3, pp. 209-234, 2012. http://dx.doi.org/10.1504/IJVSMT.2012.048939
Study of a method for assessing operability … 3
1 Introduction The trend of replacing traditional testing of physical prototypes with virtual testing of simulated vehicles and machines is continuing. Thanks to increased power and capability of both computer hardware and software, simulation models are becoming larger and more detailed, and integrated simulation of subsystems of different technical domains is now common practice. Working machines in construction, mining, agriculture, and forestry are complex in architecture with various subsystems that are used simultaneously. Optimisation with traditional development methods is difficult; the trend towards increased virtual product development is especially apparent. With the help of co-simulation, complete working machines have been simulated for about ten years . X
However, problems arise when a simulation needs to account for the actions of a human driver or operator. For example, the operator of a working machine is essential to the performance of the machine in its working place and can influence productivity and fuel efficiency to a large degree, positively or negatively, depending on the way of using the machine. It is therefore essential to include the operator’s behaviour in the simulation. Since such an operator model would be used in simulation in conceptual design, i.e. before any physical prototype is available, it is important that the model not be hardcoded in any way. Using traditional methods like fixed time, speed or position references and predefined trajectories, there is a significant risk that these references will only be valid for the machine that was used during development of the operator model, but result in large deviations for a new machine with as yet unknown properties (for example a hybrid machine with a new drive train that gives the machine a different characteristic behaviour). Therefore, any such references in an operator model must be weak ones and either constant for all machines of any size and architecture, or possible to formulate parametrically, i.e. as a function of bucket length, loading capacity, wheel base or similar. Using a wheel loader as an example, such models were reported in , using one approach similar to fuzzy-logic and another one with discrete events. Later, these were combined and further explained in , which presented a methodology for modelling the influence of construction machinery operators on productivity and fuel consumption. Others, inspired by this work, either copied and again validated the discrete-event approach  and , or presented less flexible models using a blend of path-tracking and velocity-tracking . Still others use a fuzzy-rules based approach . X
Nonetheless, one aspect had been neglected: the operator’s physical and mental workload which strongly influences how the machine is used. This aspect is part of the machine system’s inherent operability, which in  is defined as “the ease with which a system operator can perform the assigned mission with a system when that system is functioning as designed”. In  examples are given of operating situations of a wheel loader where knowledge of the operator’s workload is of importance when testing of both physical and virtual prototypes. If this cannot be simulated due to a lack of fully X
4 Paper 08 cognitive and information-processing models of human beings, then at least the control effort in the operator models must somehow be correlated to the workload a human operator would have had, performing the same actions. Once established, such measure would not only be useful in virtual testing, but also enable a more objective assessment of operability in physical testing by offering a complement to test operators’ subjective evaluation. It is thus desirable to have some method of quantifying workload other than by asking the operator for a subjective assessment after test-driving. This article will report on the setup and results of a larger study conducted to find such method for workload quantification. The method found is relatively simple and can be applied in both physical and virtual testing.
2 Wheel loaders and short loading cycles In this ongoing research, a wheel loader was chosen as the object of study, although working machines of similar complexity can be found in construction, agriculture, forestry and mining. Common factors are that these machines consist of at least two working systems that are used simultaneously and that the human operator is essential to the performance of the total system. Wheel loaders are versatile machines and each working place is unique, yet common features can nonetheless be found. The short loading cycle shown in Figure 1 is highly representative of the majority of applications.
Figure 1. Short loading cycle (also called V- or Y-cycle)
Study of a method for assessing operability … 5 Typical for this cycle is bucket loading of granular material (for instance gravel) on an adjacent dump truck (or other load receiver, mobile or stationary) within a time frame of 25-35 seconds, depending on working place setup and how aggressively the operator uses the machine. A detailed description with identification of all phases can be found in . X
Visualising test results obtained in earlier studies, Figure 2 shows that the fuel consumption rate (expressed in volume or mass per time unit) is approximately 60% higher during bucket filling than the cycle average. Expressed in absolute values, bucket filling accounts for 35-40% of the mean total fuel consumption per cycle, yet the time spend for filling the bucket is only 25% of the average cycle time.
Figure 2. Fuel consumption during short loading cycle
After bucket filling (phase 1 in Figure 1), the operator drives backwards towards the reversing point and steers the wheel loader to achieve the characteristic pattern of a short loading cycle. The lifting function is engaged the whole time. The operator chooses the reversing point such that having arrived at the load receiver and starting to empty the bucket (phase 6 in Figure 1), the lifting height will be sufficient to do so without delay. Figure 2 shows that the fuel consumption rate for phases 2 to 6 is approximately constant and close to the average fuel consumption rate of the complete short loading cycle. In the remaining phases, the bucket needs to be lowered and the operator steers the wheel loader back to the initial position in order fill the bucket again in the next cycle. Phases 7-10 are less energy-demanding and the fuel consumption rate is therefore lower than the average for the complete short loading cycle.
6 Paper 08 The higher fuel consumption rate during bucket filling warrants a closer look at this phase. The inner loop in Figure 3 shows how the human operator interacts with the wheel loader. In order to fill the bucket, the operator needs to control three motions simultaneously: a forward motion that also exerts a force (traction), an upward motion (lift) and a rotating motion of the bucket to fit in as much material as possible (tilt). This is similar to how a simple manual shovel is used. However, in contrast to a manual shovel, the operator of a wheel loader can only observe, and cannot directly control these three motions. Instead, he or she has to use different subsystems of the machine in order to accomplish the task. The gas pedal controls engine speed, while lift and tilt lever control valves in the hydraulics system that ultimately control movement of the linkage’s lift and tilt cylinder, respectively.
Lifting + Breaking/Tilting
Figure 3. Simplified power transfer and control scheme of a wheel loader
The difficulty lies in that no operator control directly affects only one single motion. The gas pedal controls engine speed, which affects both the machine’s longitudinal motion and via the hydraulic pumps the speeds of the lift and tilt cylinders. The linkage between the hydraulic cylinders and the bucket acts as a non-linear planar transmission and due to its design a lift movement will also change the buckets tilt angle and a tilt movement affects the bucket edge’s height above the ground. Furthermore, Figure 3 also shows how in the outer loop the primary power from the diesel engine is split up between hydraulics and drive train in order to create lift/tilt movements of the bucket and traction of the wheels, but is connected again when filling the bucket in e.g. a gravel pile. Figure 4 shows that in this situation, the traction force from the drive train, acting between wheels and ground, creates a reaction force between gravel pile and bucket edge, which in turn counteracts lift and tilt forces from hydraulics, and vice versa.
Study of a method for assessing operability … 7
Figure 4. Force balance during bucket filling
The normalised diagram to the right visualises that depending on lifting height and applied traction force, the lifting force can be cancelled out totally, and the bucket cannot be moved upwards any further. The operator then has to either reduce the traction force by reducing the engine speed or apply the tilt function. In summary, there are many interdependencies and it thus takes a certain amount of training to be able to use a wheel loader efficiently. It is therefore of great interest to be able to quantify operability as precise and as early in the development cycle as possible.
3 Method 3.1 Working cycle The aforementioned challenges are one reason for choosing the short loading cycle as a kind of standard test cycle when productivity, fuel consumption and operability are to be assessed. In our study it would therefore have been natural to do the same. However, with the intriguing facts visualised in Figure 2, Figure 3, and Figure 4 in mind, we specifically wanted to study the bucket filling phase. We therefore chose to use a modified short loading cycle, where the operator was instructed to still go through most of the motions, but not to use steering and thus empty the bucket at the same spot where it was filled. Since the material used was sorted gravel which does not stick together like e.g. clay and is thus fairly easy to handle, this procedure did not introduce any skewing of the test results as far as the bucket filling phase is concerned. The study also included another part, not further presented here, in which most of the psychophysiological signals that are of interest with regard to the operator’s workload, i.e. finger temperature, heart rate, respiration rate, CO2 concentration in exhaled air, and galvanic skin response (skin conductance) were recorded. The attachment of the sensors required the operator to use the joystick for steering, instead of the steering wheel. Inexperienced operators are unused to this, which in turn means that forcing them to do so
8 Paper 08 would introduce additional mental workload, thus potentially skewing the test results. The solution was not to use the steering function at all.
3.2 Machine setup The wheel loader used was a Volvo L70F equipped with a general purpose bucket with a load capacity of 2.3m3. The total operating weight of the machine is approximately 13.4t, according to . X
In order to study the effect of the previously mentioned interdependencies during bucket filling, the wheel loader software used in this study was modified to limit engine speed, and thus traction force, whenever the transmission’s 1st gear was automatically engaged, which occurs only during bucket filling (all machines normally start in 2nd gear). The first limitation of engine speed was chosen so that maximum traction force was reduced to 62% in 1st gear (see the top left diagram in Figure 5), which resulted in a maximum engine speed that was comparable to what expert operators on average used the wheel loader in. This setting also avoided wheel spin. There was a slight impact on speed of the hydraulic functions (due to the rotational speeds of the engine, torque converter and hydraulic pump being connected, see Figure 3 and the explanation in an earlier section). In the other condition, engine speed was limited such that maximum traction force was reduced to 47% in 1st gear (see the bottom left diagram in Figure 5). We deliberately used an engine speed limit below what was needed for the working task planned, which also affected the speed of the hydraulic functions more noticeably. Furthermore, the maximum obtainable traction force in 1st gear was even lower than what was available in 2nd gear, all of which an operator should experience as clearly negative.
Figure 5. Effect of engine speed limitation on traction force and on resulting lifting force
Study of a method for assessing operability … 9 The 3D diagram to the right in Figure 5 shows that the counter-acting effect of the traction force is clearly less pronounced for the 62% and 47% settings, and that even at maximum use of the traction force available the lifting force is never cancelled out completely. In this article the three tested machine variants will be referred to by their respective limitation of maximum traction force as “100%”, “62%”, and “47%”.
3.3 Apparatus During all sessions various data were recorded off the wheel loader’s CAN bus, using Vector Informatik’s CANcaseXL equipment and CANalyzer software. Not all signals of interest were readily available, which in some cases required modification of the software for engine and transmission control, while in other cases an external ECU (Parker IQAN MDL) was used to place calculated signals on the CAN bus. Additional data from externally mounted sensors for lift and tilt cylinder stroke and lever position for lift and tilt were acquired and placed on the CAN bus. A modified LoadTronic system from AADI was used to measure the net weight of the bucket. The data were transmitted to the machine controller via SAE J1587 protocol and then merged into the data stream on the CAN bus. All tests were recorded on video using an externally placed digital video camera and later synchronised with the acquired data from the CAN bus.
3.4 Participating operators In all, eighteen people, all male, agreed to participate in this study; all of them close colleagues of the author. Their data have been recorded and handled in a way that prevents drawing conclusions as to their identities (this was especially important for the part of the study not reported here, where psychophysiological body signals such as heart rate and respiration rate were recorded). In all documents, as in this article, an operator is referred to by the day of his test driving, from “04” (tested on October 4th, 2010) to “02” (tested on November 2nd, 2010). The people asked to participate as test operators in this study are all male, mostly in order not to subject the other part of the study to possible skewing due to gender differences in the way the human body reacts to mental workload. The potential operators were not randomly chosen, but pre-selected according to the author’s preliminary judgement of their wheel loader operating skills. However before starting the test each operator was asked to make a self-evaluation, using a visual analogue scale without values, but with some helpful guiding statements, see Figure 6 to the left. One operator commented that the last two skill statements should change places, so that “professional” is the highest ranking. The author’s intention with the scale as shown in Figure 6 was to beat the innate modesty of most people, which leads to the phenomenon that almost nobody ever chooses 100% (or close to) as a skill level. Also,
10 Paper 08 there might be several people who have machine testing as their profession, yet they know of others who they perceive to be even better at operating a wheel loader.
Figure 6. Sorting of the participating operators into skill groups
To the right, Figure 6 shows the division of the scale into four skill groups and how the individual operators are placed in these groups, using their self-evaluated skill ranks. These turned out to be similar to the author’s preliminary judgement, resulting in nine operators in skill group 1 (inexperienced), six in skill group 2 (experienced) and three operators in skill group 3 (expert). This also gives the possibility to combine groups 2 and 3 at a later stage into one “experienced and above” group that matches the “inexperienced” group in size, which would benefit the statistical power of direct comparisons. In our pre-selection of potential test operators for this study we chose not to focus on expert operators alone, but to include operators of all skill groups in order to test whether there is a difference in how operators of these groups handle and evaluate the different machine variants. However, we deliberately avoided skill group 0 (newcomers) in order to guarantee at least a theoretical knowledge of how to operate a wheel loader, and in order to exclude practice effects as much as possible. The ten minutes of selftraining before each live testing session would not have sufficed for newcomers, but were deemed sufficient for operators of skill group 1 and above.
Study of a method for assessing operability … 11
3.5 Design of the study and procedure Each operator was given an exclusive 2.5 hours session after lunch, during which all testing of all three machine variants was performed. In this study, the traction force setting was the independent variable. The two limited settings of 47% and 62% were tested against the unmodified software version with 100% maximum traction force, in random order, unknown to the operator. The operators were not told what had been modified and how, but most of them, especially the more experienced ones, were able to deduce this fairly quickly. In order to minimise the skewing influence of learning effects, before testing a machine version each operator was given ten minutes’ self-training just to familiarise himself with the specific characteristics of the current machine version. After the training followed five minutes’ live test-driving, where the operator was asked to make sure to fill the bucket completely and use the machine at a normal production tempo, resulting in a cycle duration of approx. 25 seconds (both of which most of the inexperienced operators did not manage to achieve completely, but at least this demand created a certain pressure). The session ended with a subjective evaluation where, among other things, the operator was asked to judge the tested machine version’s ease of bucket filling and perceived power on a visual analogue scale (without printed values). The operator was also asked to rank his overall impression of the machine in comparison to all previously tested variants on a nominal scale (“better”, “worse” or “same”). Since the study was performed during the autumn, comparable conditions for the operators were ensured by performing tests only on days with good weather (no rain) and at the time of the highest ambient temperature, i.e. immediately after lunch. The latter also guaranteed that the operators were rested.
4 Analysis The measurements conducted during this study resulted in several gigabytes of data, making automated analysis necessary. The main parts of the calculations were executed in MathCad worksheets that link to data files originating from CANalyzer (machine data) and cStress (psychophysiological data), exported in Matlab and Excel format, respectively.
4.1 Detection of the bucket filling phase Among other things, it was necessary to develop a function for automatic identification of bucket filling. While this phase of a short loading cycle clearly ends with the operator tilting back fully and lifting the bucket out of the pile, then putting the transmission into reverse and backing away from the gravel pile, the question of when this phase actually begins is almost philosophical. In earlier papers  it was stated that bucket filling begins when the bucket’s cutting edge meets the gravel pile. But how can one measure this without involving some comX
12 Paper 08 plicated measurement of absolute position or video analysis? Merely integrating the rotary speed of the wheels is not sufficient, because the wheels can spin, fully or partially, in both cases, which changes the relationship between machine speed and wheel speed. Also, the position of the front edge of the gravel pile will change with each loading cycle. This definition is therefore not practical. It has also earlier been stated that the beginning of the bucket filling phase is marked by the operator with putting the transmission into 1st gear. While such manual kickdown is still available, most operators of modern wheel loaders utilise the automatic function, which among other parameters uses the torque converter slip to calculate when 1st gear is to be activated. But by then the bucket has already begun to penetrate the gravel pile, making also this definition impractical. Some operators use the momentum of the moving machine to put the bucket into the material; some even deliberately accelerate the machine to higher speeds on their way towards the place of loading. The energy consumed during this phase is of course not free. If fuel consumption during bucket filling were to be compared, we would need to consider this in order to avoid skewed results. Therefore, we can use the shift from 2nd to 1st gear as a sign that bucket filling is probably ongoing, but the real start might already have occured, so we have to look back in time in order to include some of the earlier events. If the operator is using the machine in continuous cycles, then the earliest the bucket filling phase can begin is when the machine reverses direction (see phase 9 in Figure 1). But the operator might also have used the machine from stand-still. Common to both is that at some point in time the operator steps on the gas pedal in order to accelerate the machine towards the place of loading. Relying on this, however, can give erroneous results, in particular for longer return distances. The method used in this study is based on the experience that the operator, returning to the gravel pile from the last reversing phase, always lowers the bucket to the ground only immediately before the beginning of each filling phase. This is utilised in the following algorithm that has been found to be of practical use (Algorithm 1): Algorithm 1. Detect start of bucket filling # 1 2 3
Description Begin at the point in time where the transmission is shifted from 2nd gear to 1st Go back in time to where the lift and tilt function begin to be used (before that, they were not activated) Go back in time even further to where lift and tilt function are just beginning to not be used any more
Figure 7 shows how the algorithm performs on a typical sequence. The grey areas do not belong to the identified bucket filling phase, which can be seen to start some time before the transmission is shifted into 1st gear.
Study of a method for assessing operability … 13
Figure 7. Automatically identified bucket filling phase (white area)
Some operators clean the working place by reversing farther away and then putting the bucket on the ground to scrape off excess material all the way until right in front of the gravel pile. The algorithm above therefore needs to also specify some maximum distance from the gravel pile. This can be done by taking the point in time where 1st gear is activated and integrating wheel speed backward in time. If the resulting distance reaches the pre-determined threshold, then the algorithm will be stopped. Also, if while going back in time a gear shift from reverse to forward is encountered, the algorithm also needs to be stopped. This algorithm can even be used in real-time by utilising a buffer that after reversing starts counting from when lift and tilt lever are both no longer used until the transmission shifts into 1st gear, at which point the bucket filling is confirmed and the previously buffered time is used. Possibly the stroke of lift and tilt cylinder could also be taken into account to increase the detection quality, because in rare cases an operator may start to use the lift function and then stop it again, all before the automatic kick-down function activates 1st gear.
4.2 Operability measure from machine signals The main idea in this part of the study was to find some measure that can also be recreated in a simulation and that in the tests conducted in this study is shown to correlate well with the test operators’ subjective evaluations of the machine versions’ operability, with the focus on the bucket filling phase. There are some seemingly obvious choices for such measure, one being time on task, i.e. how long the operator needs to fill the bucket. But two simple thought experiments disprove this hypothesis. To begin with, merely by reducing the speed of the lifting function we could accomplish longer bucket filling phases. However, this would not
14 Paper 08 mean that the wheel loader would have been harder to operate; it would just have taken longer to lift the bucket out of the gravel pile until the transmission can be set to reverse, thus concluding the bucket filling phase. Secondly, it should mean a difference in operability to a human operator if a bucket can be filled in for example ten seconds just by activating and holding the levers or if during the same period of time the positions of the levers require frequent readjustment. Another seemingly obvious choice would have been task completion, i.e. how much material the operator manages to load into the bucket. However, up to a certain point the operator always manages to fill the bucket completely; it is just a matter of effort. This also invalidates bucket load as a measure of operability. Another easily acquired measure would have been fuel consumption, which indeed should correlate fairly well with operability, since it is indicative of how much energy was required to fill the bucket. However, in the future we also want to compare operability between different machines. Fuel consumption is impacted to a very large degree by the component and system losses, which are affected by size and the state of technology. For instance, fuel consumption for the same size of an engine differs when comparing versions that comply with different exhaust emission legislations (Tier II vs. Tier III vs. Tier IV). Also, the drive train can feature similarly strong transmission but of vastly different technologies (e.g. hydrodynamic or hydrostatic, planetary or countershaft) and therefore different internal power losses. Finally, different hydraulic systems can be employed that deliver similar power but with different internal losses (e.g. open centre or load sensing systems, employing fixed or variable displacement pumps). Fuel consumption during bucket filling is therefore also not a usable measure of a wheel loader’s operability.
4.3 Operability measure from operator control commands All of the above measures derived from the machine itself, bucket filling time, bucket load and fuel consumption during bucket filling, are just secondary effects of the operator’s way of using the wheel loader. All of them are also profoundly impacted by the machine’s system design and the choice of components. A much better way of deriving a measure indicative of operability is to measure close to the operator. In one part of this study we therefore measured signals on the operators’ bodies: heart rate, finger temperature, respiration rate, CO2 concentration in the exhaled air, and galvanic skin response. None of these signals are (today) possible to generate in a dynamic simulation; we need instead to involve human operators in the process. In the future, this could be done by performing studies in using human-in-theloop simulators, for instance like reported in . X
In the following we will focus on the operator’s three most important control commands: the way the gas pedal, lift lever and tilt lever are used during bucket filling. If by using the data from the physical tests performed in this study a measure could be established that shows good correlation with operability, then this would also enable virtual operability testing by means of operator models, as reported in  and other works. X
Study of a method for assessing operability … 15 Some variants of such measures can be dismissed by a group of simple constructed examples such as in Figure 8. 3
Figure 8. Constructed examples for operator control commands
To begin with, one idea similar to measuring time on task was to measure the arc length of each control command, perhaps also normalised with respect to duration. Using the examples in Figure 8, this would lead to different values in examples 1 and 2, which could be acceptable, since in example 1 the operator does not have to do anything at all. However, though comparable in terms of difficulty, example 3 and 4 would result in different values, but both lower than example 2, which is less difficult to perform than both 3 and 4. Furthermore, even though the effort in example 5 seems quite high, the calculation would give the same value as example 2. On the other hand, even though less difficult to accomplish, example 6 would result in a higher value than example 5. Arc length, absolute or normalised to duration, can therefore not be considered useful. Another idea was to use the area below the control command curve, also this possibly normalised to duration. Using the same examples from Figure 8 we can show that this is not useful either: example 2 is easiest to accomplish (other than not doing anything, as in example 1), but will result in the highest value possible. Examples 5 and 6 are both more difficult to perform, but will result in lower values than 2. The more difficult to accomplish, rectangular pulses are added to example 6, the more its calculation result will approach example 2’s value. Also, the calculation for example 3 and 4 again will provide different values, even though the degree of difficulty is obviously similar. What is it then that can be used to describe the difficulty of an operator’s control command in the time domain? In earlier work, referenced in , it was speculated that calculating an RMS-based “control work dose” similar to the dose of whole body vibrations could be one way forward. In this study we extended this line of thinking and examined whether the control commands’ variance, skewness or kurtosis might correlate with the operators’ subjective evaluation of the ease of bucket filling. As it turned out, none did. X
The question above and the mentioning of the time domain also trigger the idea of using Fourier analysis. Example 2 in Figure 8, while easy to accomplish obviously contains an infinite amount of frequencies. A perfect half sine wave would be difficult to perform for the operator, but only result in one frequency being detected. In addition to this mismatch, the question arises of how to weigh the coefficients for the various fre-
16 Paper 08 quencies. The idea of somehow using the frequency spectrum of a signal is indeed intriguing, yet definitely not a trivial task. Another interesting idea comes from the above frequently used variants of the word “difficulty”, which leads to musing on whether “complexity” would not have been a better choice. In computer science, the Kolmogorov complexity (or algorithmic entropy) of an object is the least amount of information needed to accurately describe the object. The optimal description of an object may be virtually impossible to find without prior knowledge of the object itself. For instance, saving a rendered image of the Mandelbrot set fractal in PNG file format will result in the use of a compression algorithm that reduces the file size considerably compared to a raw image file. But smaller yet would be the representation of the Mandelbrot with the iterative equation that leads to the image. However, without the knowledge that one simple equation produces such an intricate image, the most efficient description might have been the PNG file mentioned above. Is then algorithmic entropy usable to describe the intrinsic difficulty of control commands? The answer is no. This is not due to the possible failure to find the most compressed representation, but due to the fact that most of the descriptions will be too efficient. If we take example 6 in Figure 8, then one way of representing this would be “rectangular pulse from 0 to 1 with duration t1, repeated after t2.” This would already be too compressed, because a human operator works serially and cannot just save a pattern and easily retrieve it later for reuse. Another example would be the aforementioned half sine wave, hard for an operator to perform but quite efficiently described as “half sine wave from 0 to 1 with duration t”. Not wanting to rely on theoretical considerations alone we have nevertheless tried to examine this in practice by measuring the size of ZIP-files containing control commands exported to text files and binary files. No good correlation was found. X
In all the reasoning above we human beings somehow seem to be able to intuitively evaluate the performance difficulty of the simple examples in Figure 8. Example 2 seemed easiest of all the time series where the operator control actually was activated. Examples 3 and 4 seemed comparable and both quite easy to perform. Example 6 was not too difficult either, while example 5 involved many steps and was therefore hardest. It thus seems that the number of steps or more precisely the number of break points in the graph correlates with our understanding of performance difficulty. This measure will in the following be called control effort. In reality, time series data of recorded operator control commands will not be composed of long straight lines such as in Figure 8. Rather, measurement noise and shaking of the operator’s seat will introduce many small variations that are not intended by the operator. We cannot therefore merely count the number of break points, but must first apply some algorithm that simplifies the time series data by approximating the control command with a minimum of straight lines without too great an approximation error. Because it is more difficult for humans to control movements with variable speed, using polynomials of higher order than one (i.e. lines) is not proposed. Such algorithms will be presented in the next section. Having applied these, each cycle results in three figures: the number of break points for the simplified control commands for gas pedal, lift lever and tilt lever. As the control commands are essentially orthogonal, a natural way of combining their associated control effort would be to calculate the Pythagorean sum.
Study of a method for assessing operability … 17
nsum = n gas + nlift + ntilt
where n is the number of break points, also called control effort.
4.4 Approximation of control commands with graphs composed of straight lines The goal is to approximate the operators’ control commands as shown in Figure 9, here using the gas pedal signal from Figure 8 as an example. X
Figure 9. Gas pedal signal, measured and approximated
Researching this sub-problem, we found that a similar task is typical within the field of mobile robotics: line extraction from a cloud of points resulting from a 2D laser scan. Several algorithms have been developed and presented, one of the most common being versions of Split and Merge, as for example presented in . X
Algorithm 2. Split and Merge (SM) # 1 2 3
4 5 6 7
Description Start with one line, connecting first and last point in data set Create list of all points in-between, in descending order by vertical distance from the line currently under examination Check list of potential break points, beginning from top until a valid break point is found, that splits the original line into two lines, whose combined residual sums of squares plus a pre-defined threshold is smaller than the residual sum of squares of the original line If valid break point found, then split up original line into two Repeat 2 – 3 recursively for all line segments until no more valid break points can be found for any line segment Start with break point between first and second line Check if break point is invalid and allows for merging the lines to the left and right of it to one line. Invalid if the original lines’ combined residual sums of squares plus a pre-defined threshold is larger than the residual sum of squares of the merged line If invalid break point found, then merge the two line segments into one Repeat 7-8 recursively for all line segments until no more invalid break points can be found
18 Paper 08 In addition to a variant with least-squares line fitting, which proved to be superior in terms of speed and accuracy, five other algorithms are presented and benchmarked in . In our work we used the SM variant shown in Algorithm 2. X
Steps 1 – 5 in Algorithm 2 are the Split phase, where all line segments are examined as to whether they can be split up into two segments. A pre-defined threshold added to the combined residual sums of squares of the two new lines regulates this process. A split is only allowed when this sum is smaller than the residual sum of squares of the original line segment. After this, steps 6 – 9 perform the Merge phase where all previously found line segments are examined to determine whether they can be merged into one line. The same pre-defined threshold regulates this process in the same way as before. We used different values for the operator’s gas pedal and lever commands: 1000 and 500, respectively. These settings were manually derived by experimenting with some sample signals and visually inspecting the overall quality of the approximation. In our work we also developed and tested an algorithm of our own, called Stretch and Relax. In contrast to our version of the Split and Merge algorithm (Algorithm 2), the line-fitting in Stretch and Relax makes use of the least squares method. The principal steps for the SR-A variant can be seen in Algorithm 3 (which actually performed slightly faster than SM in our testing). Algorithm 3. Stretch and Relax – Absolute (SR-A) # 1 2 3 4 5 6 7 8 9 10 11
Description Start with first data point (S1) Use next point as break point candidate BP1 Fit line L1 between S1 and BP1 with least squares method Use BP1 as preliminary start point S2 for next line Use next point as future break point candidate BP2 Fit line L2 between S2 and BP2 with least squares method Repeat steps 5-6 until residual sum of squares for L2 exceeds predefined threshold or until end of data set is reached If index of BP2 ≥ BP2max then BP1max = BP1 and BP2max = BP2 Repeat steps 2-8 until residual sum of squares for L1 exceeds predefined threshold or until end of data set is reached Use BP1max as break point for line L1 and as new start point S1 for next line segment L1 Repeat steps 2-10 until end of data set is reached
Also for this algorithm, the threshold values used for gas pedal and lever commands of the operator differed. We used 2000 and 500, respectively. These settings were derived in the same way as before, i.e. by experimentation with samples and visual inspection. We further examined two additional variants of the Stretch and Relax algorithm, called SR-RL (Relative, Linear) and SR-RS (Relative, Square root). The idea here was to encourage the algorithm to create longer line segments by normalising the residual sum
Study of a method for assessing operability … 19 of squares to the length of the line, either proportionally (SR-RL) or to the square root of the segment length (SR-RS). However, this invalidated some optimisations that were included in the original SR-A version and resulted in very long execution times, with only minor improvements in the quality of the approximation. While all operator control commands had still been processed also using these algorithm variants, the results are not essentially different from SM and SR-A, and are therefore not presented in this article. After finishing all calculations and signal processing, during writing of this article, we also discovered another variant of achieving straight-lines approximations of time signals , but limited to break points being stationary points. In their work, the authors introduced the steering wheel reversal rate as a measure for assessing the impact of secondary tasks (e.g. the interaction with in-vehicle information systems) on the lateral control performance of vehicle drivers (e.g. lane keeping). The proposed measure captures the number of steering wheel reversals per time unit and is derived by first low-pass filtering the signal and then applying algorithms to find stationary points and then reversal points. Yet another possibility would have been to use the residuals of the Rainflow Counting Algorithm, as for example described in . X
4.5 Statistical analysis A brief summary of the setup of our study reveals the possibility for both sub-studies with between subjects design and sub-studies with within subjects design:
Eighteen operators participated.
Each operator can be sorted into one of three skill groups according to his self-evaluated skill level.
If only two skill groups are to be used then there is the same number of operators in skill group 1 and the combined skill group 2+3
Each operator was tested under similar conditions.
There was no particular order between the operators; they were booked according to availability.
Each operator successively tested three different software settings of the otherwise unchanged machine.
The order of the tested machine versions was randomised.
Each machine version was first tested for ten minutes to minimise practice effects, followed by five minutes of live test-driving.
Five minutes of live testing resulted in at least six bucket fillings, often more, depending on operator
As practice effects are excluded by the ten minutes of self-training, there is no particular order for the cycles, i.e. the measurements taken can be seen as random.
20 Paper 08
A normal distribution can be assumed due to good repeatability for the same combination of machine variant and operator.
Subjective evaluations were captured on visual analogue scales with free comments, performed directly after each live test-driving session.
One interesting aspect to study was how time on task (i.e. bucket filling time), task completion (i.e. bucket load) and fuel consumption vary for different operators, but also for different skill groups (a typical between subjects design). However, this article is concerned with finding a measure indicative of operability, with the hypothesis that the operator’s control commands can be used. For this, we regard each operator as a sub-group and do not compare between operators, but only between machine variants test-driven by the same operator. It is thus a within subjects design with repeated measurements. At first glance, this appears to be a case for a Repeated Testing ANOVA. But then again, we have only three conditions (100%, 62% and 47% traction force), and we would need to perform a post-hoc analysis anyway, so it seems to involve less work to just calculate t-tests for the three combinations 100%-62%, 100%-47% and 62%-47% for each operator. One could then assume that a paired (related) t-test would be appropriate, but the design of our study does not really fit the usual examples from medicine with “pre treatment” and “post treatment”, or from economics/agriculture with the amount of goods produced per factory/field in successive years. A paired t-test would operate on the difference between sample 1 of condition 1 and sample 1 of condition 2, then the difference between sample 2 of condition 1 and sample 2 of condition 2, and so on. But in our study the measurements within subject (operator) and for the same test condition (machine variant) are not related and not in any particular order; there is no practice effect nor any other causality between the cycles. It was therefore decided to perform unrelated t-tests for all three combinations of machine variants per operator. The goal was not to derive a regression function with the operators’ skill level and control efforts as independent variables and the operators’ subjective evaluation results for ease of bucket filling etc as dependent variables, but rather to show that there is a monotone relationship that can be used for comparative studies. This can be done visually or as simple comparison, without the need to model the influence of operator skill. As statistical analysis it is sufficient to show that the data for the different conditions within the subjects do not share the same mean – which a common t-test provides. As mentioned before, the operator’s control commands, and thus the control efforts, can be considered orthogonal, comparable to Cartesian coordinates. If for example two three-dimensional locations are to be compared against each other, then three separate comparisons can be made for each coordinate, and in order to prove that a difference exists between the locations it is sufficient to show a difference in one such comparison. The same reasoning applied to the data from our study means that two sets of operator control commands (each set consisting of three signals: gas pedal, lift lever and tilt lever) can not be considered to be different only if we fail to reject the null hypothesis for all three individual t-tests. The combined value for α, the probability of having made at least one Type I error (a false positive; having declared some difference being signifi-
Study of a method for assessing operability … 21 cant (unequal means), when in fact there was no difference) is the product of all three individual probabilities:
α sum = α gas ⋅ α lift ⋅ α tilt
Since the decision was made to calculate t-tests for the three combinations 100%62%, 100%-47% and 62%-47% for each operator, the problem of repeated measurements needs to be addressed. Generally, performing m t-tests on the same data set will give a family-wise error rate of m times the error rate per comparison. If we test three times for α = 0.05 on the same set of data, then in total there is a 15% chance of having made at least one Type I error, provided comparisons are independent; if they are not independent, then 15% would be the upper boundary. In conclusion, we need to prove a lower α in the individual t-tests. Instead of checking against a pre-decided confidence level we used an explorative approach and inversely calculated the α value in each individual t-test. We combined these values according to equation (2) and added these combined probabilities in order to derive the upper-most boundary for the probability of a Type I error anywhere in the result set of the examined sub-group. 3
α total = ∑ α sum i =1
where i is the index of combination (100%-62%, 100%-47% and 62%-47%) We set the upper limit for αtotal to 0.01, which gives a 99.9% statistical confidence per sub-group of this study.
5 Results This article is concerned with finding a measure indicative of operability, hypothesising that the operator’s control commands can be used and that a monotonic relationship between the operators’ total control effort and their subjective evaluation of ease of bucket filling (etc) can be shown. This section will therefore not discuss other interesting results, for instance how bucket filling time, bucket load, and fuel consumption vary between operators depending on their skills.
5.1 Subjective machine evaluations It is interesting to look at the operators’ subjective ranking of the tested machine variants, especially in light of their operating skill. Figure 10 shows that the operators in group 3, i.e. the experts, share the clear opinion that the original machine (100% traction force) is the best, the wheel loader limited to maximum traction force of 62% next best, and the extremely limited machine worst. This opinion is predominant even in
22 Paper 08 skill group 2, the experienced operators, but two had already ranked the 62% variant as same or better than the 100% machine.
Best Medium Worst
2+3 Best Medium Worst
62% 2 5 2
47% 2 1 6
Machine variant 100% 8 1
62% 2 7
Machine variant Best Medium Worst
100% 5 1
1 2 3 2+3
62% 2 4
Machine variant 100% 6 2 1
Machine variant Best Medium Worst
Matrices show operators’ added votes for ranking of the machine variants.
9 6 3 9
Machines could be voted for the same place.
Figure 10. Results of the operators’ machine ranking, skill groups 1, 2, 3, and combined 2+3
The operators in skill group 1, the inexperienced operators, also follow the same voting pattern to a certain degree, but deviate more often. Two operators in this group actually ranked the 47% machine as best, and one ranked the 100% variant as worst. In this context it is of interest to analyse whether a relationship exists between the ranking a machine is voted for and the score given by means of a visual analogue scale for ease of bucket filling. Figure 11 shows that such a relationship is given most of the time, but is somewhat less distinct for operators in skill group 1.
Highest Medium Lowest
Ease of bf
2+3 Highest Medium Lowest
Best 8 1 1
Med. Worst 2 5 2 1 7
Machine rank Best 9
Med. Worst 8
Machine rank Highest Medium Lowest
Skill group 1 2 3 2+3
Med. Worst 5
Participants 9 6 3 9
3 Ease of bf
Ease of bf
Ease of bf
Machine rank Highest Medium Lowest
Med. Worst 3 3
Matrices show operators’ added assessments of ease of bucket filling for ranked machines. Machines could be voted for the same place.
Figure 11. Comparison of operators’ assessment of ease of bucket filling vs. machine ranking
The same analysis made for machine power, also evaluated by the operators using a visual analogue scale, is shown in Figure 12. The same picture emerges as before; the expert operators in skill group 3 are most distinct in their judgements, the experienced operators in group 2 are quite distinct with the exception of one, while the inexperienced operators in skill group 1 show more deviations from the expected outcome of the best ranked machine also having the highest score for machine power.
Study of a method for assessing operability … 23
Highest Medium Lowest
2+3 Highest Medium Lowest
Best 7 2 1
Med. Worst 1 1 6 1 1 7
Machine rank Best 9 1
Med. Worst 8 9
Machine rank Highest Medium Lowest
Skill group 1 2 3 2+3
Best 6 1
Med. Worst 5 6
Participants 9 6 3 9
3 M. power
Machine rank Highest Medium Lowest
Med. Worst 3 3
Matrices show operators’ added assessments of machine power for ranked machines. Machines could be voted for the same place.
Figure 12. Comparison of operators’ assessment of machine power vs. machine ranking
From the spontaneous comments, noted during the study, we know that several operators were unsure how to rank a machine variant. Several, especially inexperienced operators seem to correlate machine ranking with their impression of the machine’s power, even though they gave a lower score for ease of bucket filling, presumably blaming themselves for not being able to utilise the power available. One operator commented on this and reasoned that bucket filling is an important phase in a loading cycle and that he will therefore give one particular machine variant a lower rank due to him finding it more difficult to fill the bucket, even though the machine received the highest score for power. Three operators in skill group 1 also made the interesting comment that the machine with the least power, the 47% variant, seemed to be more powerful than the other variants in the moment right after finishing bucket filling, i.e. when backing away from the gravel pile. Analysing the test data, it can indeed be seen that both machine speed and engine speed ramp up faster for the 47% machine than for the others. But we can also see that in these cases the operator activates the gas pedal sooner, while activating the lift lever later than for the other machines. In some cases the operator never releases the gas pedal, which is understandable given that the traction force of the 47% variant is so low that wheel spin and total cancellation of lifting force are virtually impossible, thus eliminating the need to modulate engine speed. It is plausible that the operator in this machine learns to be less sensitive with the gas pedal during bucket filling, thus using it more aggressively also in the moment immediately following. Furthermore, with less traction force available the bucket load is usually also the lowest in the 47% machine, thus requiring less engine torque to lift, and therefore having more torque available to accelerate the engine and the whole machine.
5.2 Control effort Earlier we discussed measurable signals that might correlate to the operators’ subjective assessments of the tested machine variants’ operability. Their usefulness in this context was argued against by theoretical reasoning; however we still examined most of them more closely, using the test data to prove the reasoning correct, i.e. the correlation was indeed limited or non-existent. In this section we will not present these results but instead focus on the method that actually provided the monotone relationship we were seeking, the use of control effort according to equation (1) and application of Algorithm
24 Paper 08 2 (SM) or Algorithm 3 (SR-A). Both of these algorithms gave similar results, we will therefore focus on SR-A. Figure 13 shows that a strong monotone relationship between control effort and the operators’ subjective evaluation of the ease of bucket filling could be found in five cases, all of them being operators with higher skills. In five more cases, most of them operators with medium skills, we were able to find at least a partial monotone relationship, meaning that the control effort data for the machine ranked in-between highest and lowest ease of bucket filling were not significantly different from one of their neighbours.
Partial monotone relationship (e.g. only between best and worst machine or only for machine rank but not for ease of bucket filling)
No monotone relationship 04
Strong monotone relationship between control commands and self-assessed ease of bucket fill or machine ranking
Skill group 0
Skill group 1
Skill group 2
Skill group 3
Figure 13. Existence of a relationship between operator control effort and ease of bucket filling
Figure 14 shows details for all operators where a strong monotone relationship between control effort and the operability-related results of the subjective evaluations was found. The statistical analysis reveals a significance of above 99.9% (α< 0.001). Figure 15 shows details for all operators where a partial monotone relationship between control effort and the operability-related results of the subjective evaluations was found. In these cases we either see that the control effort data from one machine variant is too close to another, thus giving less statistical significance in that specific comparison (e.g. operator 20), or we see a monotone relationship between control effort and machine ranking and machine power, but not ease of bucket filling (operator 08). While not perfectly good, there are nevertheless signs of the same type of relationship that the operators shown in Figure 14 exhibit fully.
Study of a method for assessing operability … 25
Figure 14. Operators showing a strong monotone relationship btw. control effort and operability
26 Paper 08
Figure 15. Operators showing a partial monotone relationship btw. control effort and operability
Study of a method for assessing operability … 27
We chose to look for a relationship between control effort and primarily ease of bucket filling, secondarily to machine rank, because these are closest connected to operability. In some cases control effort shows a clearer monotone relationship with machine power, but not so in other. Bearing Figure 10, Figure 11, and Figure 12 in mind, we were also curious as to whether we could see a clearer relationship between control effort and traction force, which would show that several operators were making erroneous subjective assessments, but their bodies responded “correctly”. Figure 16 shows the results of this analysis: two operators from skill group 1 (operators 05 and 13) now show a partial monotone relationship and one operator from group 2 (operator 02) even exhibits a strong monotone relationship between control effort and traction force, while earlier there was none at all between control effort and subjectively evaluated ease of bucket filling. While interesting, this presumably only shows that some operators are not really accustomed to assessing machine properties via visual analogue scales.
Strong monotone relationship between control commands and traction force 26
Partial monotone relationship
(only between best and worst machine)
No monotone relationship
04 12 Operator skill
Skill group 0
Skill group 1
Skill group 2
Skill group 3
Figure 16. Existence of a relationship between operator control effort and traction force
28 Paper 08
6 Discussion of possible sources of error During the testing phase of this study we tried to keep the environmental conditions as similar as possible. We only tested on days with good weather (no rain) and at the time of the highest ambient temperature, i.e. immediately after lunch (which also ensured that the operators were rested). But since this study was performed in the autumn, temperatures were generally lower; during the testing around 5-10°C. The machine was parked outside overnight and had therefore cooled quite substantially by the morning. Together with 20-30 minutes’ engine warm-up each day (about two hours before testing) and a 1.5km drive to the testing site, the first ten minutes of self-training should provide enough waste heat to warm up the machine (engine, transmission, axles, and hydraulics), so that testing conditions were similar for all cycles, independent of order. The test machine’s automatic functions were fixed at gear-dependent BSS (Boom Suspension System), automatic APS (Automatic Power Shift), and with Auto Kickdown enabled. Of course, the BSS is only deactivated when in 1st gear, which means it is activated during the first seconds of the bucket filling phase, until the transmission automatically shifts from 2nd to 1st gear. This is done via an adaptive algorithm, which might differ between the operators. Perhaps one of the pre-defined fixed APS settings L (low power), M (medium power) or H (high power) should have been used instead. Or perhaps the automatic kick-down function might have better been disabled altogether, in order to give the operator the freedom to choose when to shift gear. The operators were asked to use the wheel loader for three times 10 + 5 minutes, in total 45 minutes. There is a risk of fatigue, especially for the less experienced operators. However, the most extreme evaluations of choosing the 100% machine as worst or the 47% machine as best are hard to explain by this. Of course, in spite of the ten minutes of self-training before each live testing session, a practice effect might still be present, especially for the less experienced operators. As mentioned earlier, it could also be the case that some of the inexperienced operators were also unaccustomed to using visual analogue scales. Their memory of the other tested machine variants might not have been correct and they could thus have skewed their own evaluation results. Though operators were allowed to look at their previously assessments for reference, hardly anyone asked for this opportunity. While all the possible sources of error listed should be avoided next time, their total impact on the results of this study is considered to be minor.
7 Conclusion and outlook In this study of eighteen wheel loader operators, test-driving a machine in three different traction force settings, we found ten individual cases that support the hypothesis that the operator’s control commands can be used to assess the machine’s operability, at least in the form of ease of bucket filling. The methods chosen to derive the control effort, our Split and Merge variant and the Stretch and Relax, Absolute algorithm, worked
Study of a method for assessing operability … 29 well and were computationally efficient. The statistical analysis gave a confidence of above 99.9%. This provides us with a tool to evaluate test data not only from physical testing, but also from virtual testing by using the control commands of an operator model similar to the ones presented in ,  and . Future studies will examine this in more detail. X
The findings of this study are not limited to wheel loaders, but should be generally applicable, in particular for working machines.
Acknowledgments It is gratefully acknowledged that this research has been supported by the Energy & Environment programme within the Swedish Vehicle-Strategic Research and Innovation programme FFI, administered by the Swedish Energy Agency, Energimyndigheten.
Larsson, J. (2001) “Concepts for Multi-Domain Simulation with Application to Construction Machinery”. Licentiate thesis, Division of Fluid and Mechanical Engineering Systems, Department of Mechanical Engineering, Linköping University, Sweden.
Filla, R. (2005) “Operator and Machine Models for Dynamic Simulation of Construction Machinery”. Licentiate thesis, Department of Mechanical Engineering, Linköping University, Sweden. http://urn.kb.se/resolve?urn=urn:nbn:se:liu:diva-4092
Filla, R. (2009) “A Methodology for Modeling the Influence of Construction Machinery Operators on Productivity and Fuel Consumption”. Proceedings of HCII 2009: Digital Human Modeling, LNCS 5620, pp 614-623, 2009. http://dx.doi.org/10.1007/978-3-642-02809-0_65
Elezaby, A.A., Abdelaziz, M. and Cetinkunt, S. (2008) “Operator Model for Construction Equipment”, Proceedings of IEEE/ASME 2008 International Conference on Mechatronic and Embedded Systems and Applications, MESA 2008, pp 582585. http://dx.doi.org/10.1109/MESA.2008.4735708
Elezaby, A. A. (2010) “Virtual Autonomous Operator Model for Construction Equipment Applications”. Doctoral thesis, University of Illinois at Chicago, Chicago, IL, USA. http://gradworks.umi.com/34/17/3417331.html
30 Paper 08 
Thiebes, P. and Vollmer, T. (2011) “Modellierung des Fahrers zur Untersuchung von Antriebssträngen in der 1D-Simulation am Beispiel eines Radladers mit Hybridantrieb”, Proceedings of 3. Fachtagung Hybridantriebe für mobile Arbeitsmaschinen, pp 47-59. http://goo.gl/f7WHa
Norris, W. R., Screenivas, R. S., Zhang, Q. (2002) “A Novel Real-Time Human Operator Performance Model for Performing Adaptive System Design”. Proceedings of the 2002 Conference of American Society of Agricultural Engineers, Automation Technology for Off-Road Equipment, pp 287-306. http://asae.frymulti.com/abstract.asp?aid=10019&t=1
Uwohali Inc. (1996) “Operability in Systems Concept and Design: Survey, Assessment, and Implementation”. Final Report, Website of Kennedy Space Center, NASA, USA. http://science.ksc.nasa.gov/shuttle/nexgen/Nexgen_Downloads/ Ops_Survey_of_Tools_Report_by_MSFC_1996.pdf
Filla, R. (2009) “Hybrid Power Systems for Construction Machinery: Aspects of System Design and Operability of Wheel Loaders”. Proceedings of ASME IMECE 2009, vol. 13, pp 611-620, 2009. http://dx.doi.org/10.1115/IMECE2009-10458
 Volvo CE (2009) “Product brochure L60F, L70F, L90F”. http://www.volvoce.com/constructionequipment/europe/engb/products/wheelloaders/wheelloaders/L70F/Pages/specifications.aspx
 Löfgren, B. (2009) “Kinematic Control of Redundant Knuckle Booms with Automatic Path-Following Functions”. Doctorial thesis, Royal Institute of Technology, Stockholm, Sweden. http://urn.kb.se/resolve?urn=urn:nbn:se:kth:diva-11495
 Zhang, L. and Ghosh, B. K. (2000) “Line Segment Based Map Building and Localization Using 2D Laser Rangefinder”, Proceedings of 2000 IEEE International Conference on Robotics & Automation, pp 2538-2543. http://dx.doi.org/10.1109/ROBOT.2000.846410
 Nguyen, V., Martinelli, A., Tomatis, N. and Siegwart, R. (2005) “A Comparison of Line Extraction Algorithms using 2D Laser Rangefinder for Indoor Mobile Robotics”, Proceedings of IROS 2005: IEEE/RSJ International Conference on Intelligent Robots and Systems. http://dx.doi.org/10.1109/IROS.2005.1545234
 Markkula, G. and Engström, J. (2006) “A Steering Wheel Reversal Rate Metric for Assessing Effects of Visual and Cognitive Secondary Task Load”, Proceedings of the 13th ITS World Congress. http://trid.trb.org/view.aspx?id=847195
Study of a method for assessing operability … 31  Deiters, H. (2009) “Standardisierung von Lastzyklen zur Beurteilung der Effizienz mobiler Arbeitsmaschinen”. Doctoral thesis, Technische Universität Braunschweig, Braunschweig, Germany. http://www.shaker.de/de/content/catalogue/index.asp?lang=de&ID=8&ISBN=978-3-83228111-3
(Internet links updated and verified on January 10, 2012)