Modeling a Prototype Optical Collision Avoidance Sensor For Unmanned Aerial Vehicles Cyrus Minwalla∗ , Mussie Tekeste∗ , Kyle Watters∗ , Paul Thomas† , Richard I. Hornsey∗ , Kristopher Ellis‡ and Sion Jennings‡ ∗ Computer
Science and Engineering, VISOR Lab, York University, Toronto, ON. Canada Email:
[email protected] † Topaz Technology Inc., Toronto, ON. Canada ‡ Flight Research Laboratory/Institute for Aerospace Research - NRC, Ottawa, ON. Canada
Abstract—Sense and avoid systems for civilian unmanned air vehicles (UAVs) are essential in controlled airspace under visual flight rules (VFR). A prototype optical sensor accomplishes the task with attractive performance specifications. Key requirements include long-range detection (up to 10 km), wide field of view, discrimination of small threats against the background and tolerance of direct solar illumination. We demonstrate a prototype system based on a network of independent camera modules equipped with local processing. Availability of a flyby-wire helicopter configured as a UAV emulator allows for realistic field tests with consumer components. Aspects of the design, implementation and evaluation of the prototype sensor are presented here, as are preliminary measurements to clarify the roles of platform motion, system optical point-spread, noise, direct sunlight and target highlighting.
I. I NTRODUCTION UAVs are extensively used in the military, government and private sectors. In order to operate in uncontrolled airspace, such aircraft should be able to autonomously “sense and react” to avoid other aircraft [1] [2]. The intent is to provide the level of safety equivalent to or exceeding that of an onboard pilot. With respect to collisions, this capability can be met by a “sense and avoid” (SAA) instrument. Numerous versions of the SAA instrument based on radar, LIDAR and passiveoptical, among other technologies, have been envisaged [3]. Visual sensors are suitable for this application due to their mass, size, frame rate, power, angular resolution and coverage [4] [5] [6] [7]. Some of the shortcomings of passive optical technology include degraded performance under low atmospheric visibility, the indirect nature of range information, scene clutter and susceptibility to damage from sunlight. CMOS image sensors, however, overcome these shortcomings by being readily available, low cost, and rapidly evolving with advances in optical, computing and imaging technologies that are pushed by sundry applications [8]. II. BACKGROUND AND M OTIVATION The critical factor in collision avoidance systems is the timely recognition of potential collisions. Sufficient time must be given to perform evasive actions that will maintain the minimum separation distance of 165 m (500 feet). A buffer of at least 12 seconds between detection and any collision is widely used [1]. Many sensor, target and environment
978-1-4244-8168-2/10/$26.00 ©2010 IEEE
parameters affect this time buffer, including target velocity, angular resolution, frame rate, optical field-of-view (FOV) in azimuth and elevation, atmospheric visibility, camera pointspread function, platform jitter and availability of processing capability. Customized CMOS imagers can deliver dramatic improvements in frame rate, dynamic range, digital/analog integration, power dissipation and unit cost - but not necessarily simultaneously. Enunciation of the specifications for a custom imager needs abundant data on the sensitivity of system performance to these parameters. Suitable flight data is valuable in modeling realistic or worst-case scenarios, the acquisition of which is facilitated by the National Research Council of Canada (NRC) Bell 205 airborne UAV emulator. Use of the emulator relaxes the mass, power and size constraints of the prototype, allowing commercial off-the-shelf (COTS) components to be employed to collect data essential for the customized imager specification. It can be expected that some of the requirements will change with the UAV platform and the mission. A common metric is required to simplify the system engineering of seemingly independent parameters. Relative importance and sensitivity of each parameter can be judged with this metric. The “range of first detection”, R0 , is an intuitive figure of merit that is platform-independent, and can be readily determined by ground-based measurements. Conversely, the time buffer (tbuf f ), although a more fundamental quantity, is scenario- and platform- dependent, where the link between the two is provided by modeling. Key parameters limit R0 and therefore the performance of the system. They include the physical metrics of mass, power, size and cost, signal-to-noise ratio due to local contrast between background and target, optical performance due to point-spread function effects, image processing to raise local contrast and apparent height above the horizon. The end objective is the design and development of an optical sensor that satisfies the requirements of a SAA instrument. We present here the design decisions and preliminary measurements towards achieving this goal. III. S CENARIO The baseline scenario for the present discussion is operation under VFR conditions, no traffic information(e.g. transponder not operating) and non-cooperative targets [9]. A “platform”
1657
IEEE SENSORS 2010 Conference
Optics
Optics
Optics
Optics
Optics
Sensor
Sensor
Sensor
Sensor
Sensor
PreProcessor
PreProcessor
PreProcessor
PreProcessor
PreProcessor
Managed Network Switch PostProcessor
Camera Group
Camera Group
Fig. 2.
Camera Group
To Helicopter
Fig. 1.
Block diagram of the sensor system
houses the sensor – a task accomplished here by a fly-bywire Bell 205 helicopter posing as a surrogate UAV [10]. For modeling convenience, both platform and target aircraft are assumed to be in level flight. Diffuse sunlight illuminates both the background, comprised of sky, clouds and ground, and the target. Detection needs to occur at ranges beyond a kilometre - perhaps as far as 10 km - in order to allow for evasive action by the platform. At such ranges, the angular size of the target is small and signal from the target is generally mixed with the background through the optical point-spread function of the sensor system. Reduced atmospheric visibility compounds this problem. Target aircraft fall under the FAR 23 general aviation standard; a Cessna 117 being a typical example with a wingspan of ' 10 m and maximum airspeed of 250 km/h (70 m/s or 136 knots). Target altitudes and velocities are not expected to exceed 3000 m (10000 feet) and 360 km/h (100 m/s or 195 knots) respectively. IV. I NSTRUMENT D ESCRIPTION In the present work, a prototype collision avoidance sensor has been created to operate on the helicopter UAV emulator. Aside from the mass and power benefits, the helicopter has extensive image storage capability and a suite of navigation instruments (attitude, GPS, altimeter) that can be read by the sensor. The sensor was configured as a network of many camera modules [11], which allows the use of COTS optics for coverage of the field of view, reduces communication and processing bottlenecks and facilitates the introduction of new hardware and software components. Several camera modules are clustered into a camera “group” that may have additional control and processing capability. A number of camera groups comprise the complete instrument (Fig. 1). This hierarchy means that most evaluations can be performed with a reduced instrument having only one or two camera groups. The total field of view needed for collision avoidance is ±110o azimuth by ±15o elevation [2], which can be achieved with about 20 camera groups (i.e. 102 camera modules in total). Total dimensions of the instrument are not specified, but are expected to be near 60 × 20 × 10 cm3 .
Laboratory prototype instrument
Elements of a camera module are shown in block form within each module in Fig. 1. The optics consist of a lens, spectral filter and shutter. A typical detector array used here is a commercial digital camera module, including readout circuitry, digitizer, some pixel-level processing and a USB communications interface. The local computer receives the imagery in USB format, selects a region of interest and performs appropriate image processing. The image processing is intended to identify ‘events of interest’ that need to be characterized and reported up the hierarchy. Fig. 2 is a photograph of a laboratory prototype of one camera group of the instrument. Standard C-mount lenses and processor boards dominate the photo. The camera modules (five in total) are mounted on a common baseplate to form a camera group with a total field of view of about 45o azimuth by 8o degrees elevation. A reduced-mass version of the camera module is the basis of the helicopter-mounted prototype. In the photograph, all of the camera modules have identical optics, detector arrays and preprocessors. Interchangeable f /1.4 lenses with variable aperture were used to measure the effect of aperture. A colour detector array was the most readily available, even though a monochrome sensor should improve the signal-to-noise ratio (SNR). Likewise, the local preprocessing computer was selected for low cost and rapid delivery. Ethernet links the camera modules to a camera-group computer that handles post-processing and communications. Imagery is usually processed only at the camera-module level. Table I lists selected specifications of a camera group. Mass and power are 10x over endpoint in this prototype. V. D ESIGN D ISCUSSION The development plan is to investigate range performance under various conditions (atmosphere, target and clutter) before sensor optimization. Table II lists representative features of the collision avoidance sensor/scenario. To illustrate the complexity of design tradeoffs, Fig. 3 shows three system parameters (mass, power and frame-rate), and the major subsystems (optics, detector array and processing), along with the approximate fractional contribution of each subsystem to these parameters. Various designs can lead to the same range of first detection. A tradeoff between power and mass, for instance, could keep the range constant while changing the processing and optics. The concentric circles indicate R0 in a
1658
TABLE II E NDPOINT F EATURES OF THE C OLLISION AVOIDANCE S ENSOR
TABLE I P ROTOTYPE S ENSOR PARAMETERS
Camera group
Single module
Features Range of First Detection Time to evade Angular Resolution Azimuth Field-of-regard Elevation Field-of-regard Overall dimensions Scene Dynamic Range Cost Mass Power Frame rate
Nominal Value 10 12 0.2 (' 1) ±110 ±15 60 × 20 × 10 105 104 2 50 10
G
IN
S ES OC
DE TE AR CTO RA R Y
ER
Km
10 m
5K PR
qualitative way. Larger ranges generally require greater larger mass/power/frame-rate.The modular design of the instrument aids in the evaluation of design alternatives. Highlights of the design include the use of distributed camera modules, local image processing, instrument pointspread function and the hazards of direct sunlight illumination. System design enhancements, which are ongoing, generally address such issues by a combination of simulation and experiment. Only a few camera modules need to be modified for evaluations, and tests are first performed in the laboratory and on the ground.
Units km s mrad (pixel) degrees degrees cm3 $ kg W Hz
Km
Units degrees degrees kg W Gbps pixel bit fps µm mm degrees MIPS MB GB Mbps
20
Nominal Value 5 45 × 8 10 4 30 1 1600 × 1200 8 5 2.0 25 1.4–16 10 × 8 500 256 8 100
W PO
Processing
Parameter Number of modules Angular field coverage Module separation Mass Power Communications Resolution Dynamic Range Frame rate Pixel pitch Focal length (f ) F-number (f/#) Effective FOV CPU (ARM Cortex 8) Memory (SDRAM) Storage Ethernet
FRA M RAT E E
Component
OPTICS
MASS
A. Distributed Cameras Although there can be mass/power and complexity penalties for using a distributed sensor network [11], the large and asymmetrical optical field of view of the collision avoidance sensor benefits greatly from partitioning of the optics into a large number of separate lenses with overlapping fields. Further benefits of a distributed sensor include enhanced reliability and parallel image processing capability. Custom CMOS fabrication will allow the imager and suitable preprocessing to be combined on a single chip. Finally, the cost of the sensor is linearly related to the total field of view (the field of view is augmented by simply adding more camera modules), whereas the single-sensor costs can rise quadratically and at the expense of frame rate. Stray light increases when the sun is close to the field of view of a camera module. If the stray light is excessive or direct sunlight imminent, the affected detector arrays are powered down. Potential sensor damage is reduced by integrating mechanical aperture shut-off capability. Because the sensor has 10 - 100 camera modules (depending on field of view and other design considerations), the fractional loss of coverage by the (temporary) deactivation of a camera module is small, a boon compared to single-sensor devices. Spurious specular reflections can also supply direct-sunlight-equivalent
Fig. 3. Engineering constraints for design tradeoffs. Note that this is qualitative in nature, with the objective of providing some intuition into the interplay among subsystem parameters and system specifications.
irradiance on a subset of the detector array, but are initially considered rare and transient. B. Image Processing The point spread function (PSF) of the sensor system determines the mixing of point-target signals and the optical background. Contributions to the PSF include optical aberrations and diffraction, detector pixelation, platform motion and atmospheric scattering. Mitigating the effects of the PSF is the first task of image processing. Reduction of signals that are not from the target of interest is important. The image of a small target is generally viewed against sky or other background (clouds, ground) whose perpixel signal can be comparable to that of the target. Removal of this ’background clutter’ is another task of the image processing. Further tasks include discrimination of signals of interest from random noise by a combination of noise estimation, signal deconvolution and thresholding [12]. Secondary image processing activities are expected as the instrument evolves.
1659
Detector Element Footprint Lens
80
Module Field of View
Pixel Viewing Direction
SNR per pixel
Detector Array
60
Small Target
Icam
1/4 1/2 1 2 4
Background/target signal ratio
70
Afoot
(Pixel Footprint) Range to Target
50 40 30 20 10
Fig. 4.
Optical model of the detector element footprint 0 2
4
VI. S IMULATIONS AND E XPERIMENTS Measurements were compared to MATLAB models embedded in a system-level simulation that could reproduce experimental results and predict sensor performance. Initially, a library of in-flight imagery was acquired from the surrogate UAV platform. Image sequences were generated as a series of fly-bys with target and platform at level altitudes and with varying trajectories. Most images did not contain targets and were used to create realistic backgrounds in the raytracing simulator. Both measured and synthetic pointlike targets were then combined with the background within the simulator. Performance predictions for selected laboratory conditions were used to validate the simulator. Field tests were used for simulator validation of realistic PSF and platform vibration. The simulator was able to predict performance under extreme field conditions as well, such as low visibility (due to inclement weather) and head-on collisions. A. SNR Variation with Target Range Each pixel of the detector array projects out onto a “virtual area” or “footprint”, Af oot , at the range of the target, as shown schematically in Fig. 4. Light from the scene elements within the footprint can be imaged onto the pixel, while light from outside the footprint hits other pixels. For this discussion, it is assumed that light diffusely reflected from the target has a clear line of sight to the sensor, and that light from the background scene can be considered to arise from a ‘virtual scene’ that is at the same range as the target but is partially obscured. In the presence of optical point spread, the footprint boundaries of each detector element blur and expand. This causes scene features in Af oot to spread among nearby pixels, and background to bleed into Af oot . Fig. 5 shows the variation of SNR for a single pixel as a function of target range for different background reflectivities, as per Eqs. 1 and 2. The images used were generated via raytracing synthetic targets at realistic distances embedded into flight imagery. Ambient lighting was controlled to simulate different background intensities. SN R =
| dnsig − dnback | dnnoise
(1)
6
8
12 10 Range [km]
14
16
18
20
Fig. 5. Variation of SNR with range for a sequence of background reflectivities
nenoise =
q ne2read + neshot
sig
+ neshot
back
(2)
dnnoise = nenoise ∗ cv Here, dnsig and dnback refer to target and background intensities in digital number respectively. The noise is modeled as a combination of shot and read noise measured in electrons, and converted to digital number by the conversion gain cv. Returning to Fig. 5, the noise is primarily the shot noise from the combined signal-plus-background. The SNR values require the selection of numerous instrument parameters (integration time, aperture, quantum efficiency and so forth), and so can best be considered as relative. These parameters affect both signal and background equally. Note that the signal can be brighter or darker than the background. When signal and background have the same reflectivity, the signal disappears. In this situation, the use of spectral or polarization features of the target and background may be important. An effective target area of 1.0 m2 has been used. At sufficiently long range, the target is small compared to the pixel footprint (Af oot ), and most of the signal (and noise) comes from the background. For nearby targets, the entire pixel is filled with light from the target. At intermediate ranges, the signal depends on the difference in reflectivity between target and background. The variation of signal with range arises from the increase in occlusion of the background by the target, until Af oot is completely filled by the target. B. Effects of the point-spread function (PSF) The SNR of a pixel containing the target image is shown in Fig. 6, with various point-spread functions added. Here, σ represents the standard deviation of the PSF in units of pixels. The impact of PSF on instrument performance can be seen from its effect on R0 (Fig. 7), with the detection threshold from Fig. 6. There is a large reduction of R0 for large PSF width, which can be caused by reduced atmospheric visibility [13], providing, as expected, a strong link between visibility and first detection. Some over-design of the sensor under normal conditions (i.e. R0 greater than 10 km) would
1660
12
10
8
Detection Threshold
6
4
4
2
2
0 2
4
Fig. 6.
6
8
12 10 Range [km]
14
16
18
Non-PSF SNR PSF Convolved SNR LoG Filter
10
SNR per pixel
SNR per pixel
8
6
12
σ = 0 [pix] σ=2 σ=5 σ=7 σ = 10 σ = 20
Detection Threshold
0
20
6
Impact of point-spread function on SNR
Fig. 8. 0.4
14
0.35
Range at First Detection (R0) [km]
15
13
Relative signal energy
0.3
12
11
10 9 8
10
12 Range [km]
14
16
18
Impact of image processing on R0
Measured Opt. PSF Fitted Opt. PSF Opt. PSF w/motion Combined PSF (atm., opt. & mot.)
Opt. PSF fit FWHM = 2 pix Opt.+Mot. PSF fit FWHM = 5 pix Combined PSF fit FWHM = 7.5 pix
0.25 0.2 0.15 0.1 0.05
7 6 0
8
5 15 10 Standard Deviation (σ) of point-spread function [pixel]
Fig. 7.
0
20
-10
Impact of point-spread function on R0
allow for adequate performance at the low-visibility end of VFR. Image processing is expected to recover some of the performance reduction caused by the PSF. In Fig. 8, the SNR versus range is shown for low atmospheric visibility, for the uncorrected and corrected cases. Also shown is the curve with negligible PSF. Here, a PSF σ = 10 [pixel] was used, as the worst-case combined PSF as per the full-width at half maximum value of 7.5 [pixel] estimated in Fig. 9. The correction was performed by applying a simple Laplacian of Gaussian (LoG) based point-detection algorithm [12]. In this case, R0 has increased by 40% due to image processing i.e. the range of detection shifts from 8 km out to nearly 12 km. Advanced processing techniques will improve the performance. Fig. 9 displays the representative point spread functions. The PSF of the optics/detector has a full width at half maximum of 2 pixels, which includes diffraction, aberrations, camera focus and detector array pixelation. As mentioned previously, atmospheric scattering broadens the PSF by an amount related to the atmospheric visibility. In addition, the motion of the sensor platform adversely affects the PSF at long integration
-5
0 5 Scaled pixel position [pixel]
10
Fig. 9. Optical and motion PSF of a helicopter flying at 90 knots at level altitude. Displacement of a point in the image for an integration time of 75 ms is averaged to recontruct the new PSF (dark grey). The motion PSF is then convolved with an estimate of atmospheric scatter for a 632 nm laser at 6.4 km, with the result in light grey [13]. Camera settings: f = 25mm, f/# = 2.8, pitch = 3.5 µm.
times. The motion broadening in Fig. 9 has been determined using the measured jitter in the yaw axis of the surrogate UAV in level flight (speed of 90 knots) using recorded accelerometer, altimeter and GPS data. An integration time of 75 ms was used, which is at the upper end of expected integration times and so provides a worst-case approximation of platformmotion jitter. C. Horizon modeling Apparent target height above the horizon, htgt , is a key identifier of targets on a collision course. Note that ∆htgt can be measured on each image that contains a target, since the location of the horizon line can be estimated from prior measurements (from the same and different camera modules) or modeling if it is not present on the image. The utility of htgt can be best illustrated by a ‘fence’ scenario, which assumes that the first detection of the target occurs at the outer fence boundary, placed at R0 . Targets
1661
4
Target Miss Distance
Target Shift - ∆htgt [mrad]`
3.5 3 2.5 2
ACKNOWLEDGMENT
-90 m 0m 150 m 400 m 650 m 900 m 1900 m
The authors would like to thank the National Research Council (NRC) of Canada and the Ontario Centres of Excellence (OCE) for their financial support. R EFERENCES
1.5 1 0.5 0
-0.5 6
6.5
7 7.5 8 Range To Target [km]
8.5
9
9.5
Fig. 10. Apparent above-horizon displacement in pixels for different target altitudes with both platform and targets on level flight-plans. Angular resolution of one pixel = 0.2 mrad.
piercing the fence are nominally detectable and, assuming level flight, follow a predictable trajectory. Calibration and other systematic errors are not expected to be significant. Fig. 10 shows a number of near-collision trajectories. Preliminary flights with a Bell 205 platform and a Harvard Trainer aircraft target validate Fig. 10 for selected (safe) trajectories. It is expected that targets on a collision course will maintain an altitude coincident with that of the platform, and therefore exhibit a ‘stuck pixel’ profile in the imager, barring noise. Platform motion is common to target and horizon. Such motions may shift the target position, but they also shift the horizon (or any other scene feature) by an equal amount. Hence, measurements of ∆htgt can be averaged over many frames, regardless of the platform motion. For targets on a collision course i.e. matching altitude (100 m in the figure), the horizon-height profile is relatively constant. For all other target elevations, ∆htgt grows nonlinearly with decreasing closing distance. In order to track htgt , the angular positioning precision of the sensor needs to be less than about 1.0 mrad. Because there is only a single target, angular resolution, in the sense of the Rayleigh criterion [14] and the PSF, is not important. For the modeled sensor choices, target miss distance is discernable out to at least 7.5 km. Although the variation may be within the platform PSF, appropriate image processing to deconvolve the point-spread is expected to reveal the target at this range.
[1] D. Ebdon and J. Regan, “Sense-and-avoid requirement for remotely operated aircraft (roa),” Langley Air Force Base, USAF, Tech. Rep., June 2004. [2] A. S. F2411-07, “Standard specification for the design and performance of an airborne sense-and-avoid system,” ASTM International, Tech. Rep., 2004. [3] B. C. Karhoff, J. I. Limb, S. W. Oravsky, and A. D. Shephard, “Eyes in the domestic sky: An assessment of sense and avoid technology for the army’s“warrior” unmanned aerial vehicle,” in Proc. IEEE Systems and Information Engineering Design Symposium. IEEE, April 2006. [4] J. F. McCalmont, J. Utt, M. Deschenes, and M. J. Taylor, “Sense and avoid technology for global hawk and predator uavs,” in Proc. SPIE Conference on Infrared Technology and Applications XXXI. SPIE, 2005. [5] D. Dey, C. Geyer, S. Singh, and M. Digioia, “Passive, long-range detection of Aircraft : Towards a field deployable Sense and Avoid System,” in Proc. Field and Service Robotics, 2009, pp. 1–10. [6] J. McCalmont, J. Utt, M. Deschenes, M. Taylor, R. Sanderson, J. Montgomery, R. S. Johnson, and D. McDermott, “Sense and avoid technology for unmanned aircraft systems,” in Proc. SPIE Conference on Automatic Target Recognition XVII, vol. 6566, no. 1. SPIE, 2007. [7] R. Carnie, R. Walker, and P. Corke, “Image processing algorithms for uav “sense and avoid”,” in Proc. IEEE International Conference of Robotics and Automation ’06. Orlando, Florida: IEEE, May 2006, pp. 2848–2853. [8] “Process integration, devices and structures,” ITRS, Tech. Rep., 2009. [Online]. Available: http://www.itrs.net/Links/2009ITRS/2009Chapters 2009Tables/2009 PIDS.pdf [9] W.-Z. Chen, J. M. DeLuca, J. D. Koeller, W. F. O’Neil, and I. H. Wong, “Autonomous unmanned aerial vehicle (uav) airspace operations sensing requirements. volume 1 - performance,” Northrop Grumman Corporation - Air Combat Systems, El Segundo, CA, Tech. Rep., July 2002. [10] A. J. Smerlas, D. J. Walker, I. Postlethwaite, M. E. Strange, J. Howitt, and A. W. Gubbels, “Evaluating h∞ controllers on the nrc bell 205 fly-by-wire helicopter,” Control Engineering Practice, vol. 9, no. 1, pp. 1 – 10, 2001. [11] T. D. Raty, “Survey on contemporary remote surveillance systems for public safety,” IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews, vol. PP, no. 99, pp. 1–23, March 2010. [12] R. C. Gonzales and R. E. Woods, Digital Image Processing, 2nd ed. Prentice Hall, January 2002. [13] N. S. Kopeika, I. Dror, and D. Sadot, “Causes of atmospheric blur: comment on atmospheric scattering effect on spatial resolution of imaging systems,” Optical Society of America, vol. 15, no. 12, pp. 3097– 3106, 1998. [14] E. Hecht, Optics, 4th ed. Reading, Mass.: Addison-Wesley, 2002.
VII. C ONCLUSION In conclusion, we present a system design informed by simulation and real-life image and platform data. The range of first detection R0 was used as a key experimental figure of merit, and preliminary models and measurements show the importance of local background illumination, system and atmospheric PSF, image processing, and apparent horizon position for the long-range collision detection scenario.
1662
