image processing

3 downloads 0 Views 4MB Size Report
Now, nearest lane will be that whose theta is more negative. So, while on going scanning lines, if above threshold value length, white length comes and if its ...
IMAGE PROCESSING APPLICATIONS IN AUTONOMOUS VEHICLES Dhaval Kadia | sites.google.com/view/DhavalKadia

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

Index    1.

About Me Motive Modules

Lane Detection & Departure Warning System 1.1. Lane Detection 1.1.1. Common Approach 1.1.1.1. Conversion to Gray 1.1.1.2. Blurring 1.1.1.3. Extracting Lines 1.1.1.4. Understanding Lines in Image 1.1.1.5. Thresholding 1.1.1.6. Line Map Compaction 1.1.2. Simple Lane Detection  Details  Output Images  Advantages & Disadvantages 1.1.3. Color Based Lane Detection 1.1.3.1. Converting Image from RGB to HSV/HSB  RGB color model o Advantages o Disadvantages  HSV/HSB color model o Analysis o Advantages o Disadvantages o RGB to HSV Conversion [Code]  Detection of White color o On RGB scale  Analysis  Advantages  Disadvantages o On HSV/HSB scale  Analysis  Advantages  Disadvantages  Detection of Yellow color o On RGB scale  Analysis  Advantages

PAGE 1

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

 Disadvantages o On HSV/HSB scale  Analysis  Advantages  Disadvantages 1.1.3.2. Solid Yellow between two Solid White Lanes  Explanation  Right Side o Output results o HSV/HSB thresholds  Wrong Side o Output results o HSV/HSB thresholds 1.1.3.3. Dashed White between two Solid Yellow Lanes  Explanation  Right Side o Output results o HSV/HSB thresholds  Left Side & Multiple Yellow solid lanes o Output results o HSV/HSB thresholds 1.1.3.4. Dashed Yellow between two Solid White Lanes  Explanation  Output results  HSV/HSB thresholds 1.2. Departure Warning System 2. Object Recognition & Depth Perception  Depth Perception o Definition o Types o Stereopsis o Binocular Disparity o Equation Derivation o Outputs of Stereopsis  Explanation o Object Recognition o Depth Map  StereoSGBM (Stereo Global Mapping) o Parameter details  Depth Map Outputs o Explanation o Parameters’ values

PAGE 2

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

About Me Dhaval Kadia Student, Department of Computer Science and Engineering Faculty of Technology and Engineering, The Maharaja Sayajirao University of Baroda Email- [email protected]

Motive Image Processing and Computer Vision has been playing leading role in Computer Science. Their selected applications mentioned in above Modules defines Autonomous System and Robotic Vision. Image Processing and Computer Vision has been playing leading role in Computer Science. Their selected applications mentioned in above Modules defines Autonomous System and Robotic Vision. Thinking for Autonomous Vehicle technologies, though LiDAR (Light Detection and Ranging) is well developed, but it is Energy-wise and Economically Costly. Cheaper solution is to do same thing using Image Processing. Understanding images is key point of whole project. This work can be further enhanced by introducing & analyzing 3D environment. Furthermost, final level of enrichment can be reached by Object Recognition.

Modules

Modules of Inclusion Lane Detection & Departure Warning System Depth Perception by Stereopsis (Binocular cues) Object Recognition

Difficulty Level (for me) Easy (Medium for Dynamic environment) Medium Hard



Lane Detection will be done by identifying Lane of various colors and their relative positions with respect to reference camera. RGB values images will be transformed and understood by applying Mathematical techniques. Next phase will be made up of Data & its Probability to filter Data that is necessary for further Analysis and making Decisions. Here Decisions will point to Lane Departure Warning System that makes important notifications for safety of vehicle & surroundings.



Depth Perception will be improvement for this project as it adds extra level for Safety, Detection & Decision making. It will be based upon Stereoscopic Vision using two Cameras. Basic work will be based upon depth of single object for detecting its co-ordinates (x, y, z) with respect to two Cameras.

PAGE 3

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES



Object Recognition will be added for recognizing frequently appearing objects like Street Signs. Combining it with Depth Perception will result in Accident Avoidance for Autonomous Vehicle.

1. Lane Detection & Departure Warning System Both modules are implemented in parallel. It depends upon how Road is designed and how Lanes are arranged. Pattern of the road is also considered here.

1.1

Lane Detection First necessary module is the Lane Detection to make decision. For that, approach is divided in some sub-modules in which input is passed and transformed in between.

1.1.1

Common approach

Images are scanned from Image file or from frames of a video. These scanned images are RGB scaled* images. Resolution of it depends upon Resolution of Image or Video from which it is scanned.

1.1.1.1 Conversion to Gray If Line detection is performed irrespective to Color, before applying HoughLine function, converting RGB scaled* Image to Gray scale will be optimization based on memory occupation because Gray scale requires third part of memory occupied by RGB scale and will be most appropriate step if we are detecting line irrespective to Color.

1.1.1.2 Blurring There are chances of detection of many small raw lines those are obviously not necessary for this approach. For handling it, I am blurring the image that how sharpness of image can be reduced and those small raw lines will be disappear. There is no issue for the lines those we are intended to

PAGE 4

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

find because they will always come over the threshold value that is chosen as that line has effective portion of area having single color.

1.1.1.3 Extracting Lines Blurred image is put in Canny transform and then it is processed in HoughLine function provided by OpenCV Library. It will give array of points representing end points of all lines detected in transformed image. Threshold value has significant role in detecting lines those are much necessary.

1.1.1.4 Understanding Lines in Image Though lines are detected in the form of end points, but all lines are not necessary. We cannot state directly that this is the line indicating proper Lane. That’s why I designed a procedure that decides line(s) indicating proper Lane detection. Procedure is quite simple. This procedure is based upon probability. We are having cluster of lines having different slopes and positions. Where positions refer to intersecting point of line with Y-axis. Each line can be uniquely distinguished by its two property: Slope(m) & Intersection point with Y-axis(c). Next step is to arrange these properties in array that how analysis can be made on it in easy way. Line separating the image vertically is selected is Y-axis. Bottom line of the image is selected as X-axis. Lines are stores in this Line Map. Following steps are taken for making Line Map: 1. Get End points of the Line (x1, y1) & (x2, y2). 2. Find Slope m and intersection of the line with Y-axis c. 3. Find angle of the Line with X-axis θ from tan inverse function i.e. atan(m). 4. Find length of the line L with distance formula. 5. Add value resulting from multiplying L by factor f into [θ, c]

PAGE 5

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

c : 0 to ImageHeight + Δ

θ : +90˚ to -90˚

Line Map Structure X – Theta Y – c (Intersection of Line with Y-axis)

Steps for making Line Map Array from Line Map: 1. Initialize 2D array LMA of dimensions [ ImageHeight + Δ ][ 180 ] 2. Find c, θ & L values of line add L * f into LMA[ ImageHeight + Δ ][ 90 - θ ]

1.1.1.5 Thresholding After these sequential procedures, we will get LMA. Now, the Lane we are trying to find will be analyzed as line and saved somewhere in LMA. It is obvious that if there is Lane in image, it will represent long line(s) than other. Thus, we can identify lane by calculating length of the line and number of lines detected for same value of [θ, c]. We can also take care that only those lines will be added to LMA whose length is more than one threshold value. Same procedures can be performed with after Line Map Compaction. It gives better results for finding Lane.

1.1.1.6 Line Map Compaction This way is chosen for optimization in memory and processing. It is obvious that sample Image may be of high resolution in that will make LMA with greater size. Lines within image may point distinct cs but those cs may lied in particular ranges. Thing is that only those range of array will be

PAGE 6

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

modified and rest will be unchanged. This will result in wastage of memory and extra useless processing. To overcome it, I have introduced one solution that will give useful results based upon probability. Solution is to Merge blocks in one group. This can be done in two ways: 1.

Merging LMA Vertically. This will represent replica of actual LMA representing probability of having lines being in particular range of cs. Actually, this is useful to do without suffering loss. 2. Further, if necessary, LMA can be merged Horizontally too. It will do same as 1 but for θ. Introducing constant cFactor, that represents number of rows those are merged. For doing this, following steps are taken: 1.

Initialize 2D array LMA of dimensions [( ImageHeight + Δ ) / cFactor ][ 180 ] 2. Find c, θ & L values of line add L * f into LMA[ ( ImageHeight + Δ ) / cFactor ][ 180 ]

1.1.2 Simple Lane Detection In Simple Line detection, these steps are performed in below sequence: 1.1.1.1 1.1.1.2 1.1.1.3 1.1.1.4 1.1.1.5

Conversion to Gray (Colorless detection) Blurring Extracting Lines Understanding Lines in Images or Thresholding

1.1.1.6 Line Map

Further & Ongoing processing is explained below: Find the [θ, c] location in which value is stored above threshold value in which θ is positive and store [θ, c] into variables maxTPos and maxCPos. Store value at [θ, c] into variable maxLPos. Find the [θ, c] location in which value is stored above threshold value in which θ is negative and store [θ, c] into variables maxTNeg and maxCNeg. Store value at [θ, c] into variable maxLNeg. Now, if maxLNeg and maxLPos is above threshold value on bases of length of Line, then we intersection of both lines is calculated. The location of intersection point will help to find the angle by which Vehicle is required to turn to go on way parallel to both Lanes.

PAGE 7

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

Its calculations are performed in this way: Intersection point (interX, interY) is calculated from this equation. interX = -(maxCPos - maxCNeg)/(maxMPos - maxMNeg) interY = maxMPos*interX + maxCPos Turn angle is calculated using below formula: turnAngle = atan(interX/ interY) Sign of turnAngle will decide whether to turn Right side or Left side. Meaningful explanation is described below with use of screenshots.

Sample Outputs & Explanation 1.

Details Original Image is first converted to Gray scale for removing noise and sharpness that will degrade line detection.

After Converting to Gray scale and Blurring

The next step is applying Canny transformation. Which detects edges in images. Threshold value is changed as per requirement. By changing threshold value, in how many quantity, edges should be detected. If threshold value is high, less edges will be detected & vice a versa.

After applying Canny transformation

PAGE 8

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

It is better to make threshold value high if sample image is noisy. For example, if there is trees around way, there will be disturbance due to edges. Else if sample image is clear enough, decreasing value of threshold will make sense. There will be less disturbance to be overcome.

LMA | Lane Detected Image After detecting edges, function applied to image having edges is HoughLine(). This function gives lines in image in the form of Nx1 Matrix. Where N is number of lines detected in image. Each element of matrix has 4 elements. These are end points of line x1, y1, x2, y2. Lines detected by HoughLine() is shown by Red lines. Their length is added to their corresponding element of LMA. In Simple Lane Detection, simple approach is applied. Cluster of lines having length more than pre-defined threshold value is put into LMA. That is detected by scanning LMA and c & θ are gotten. Applying mathematical formula on these two values, Green lines are generated to show significant long lines.

PAGE 9

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

To find *point of horizon of the road, simple method is applied. That is by finding two longest lines, one having positive θ (θ > 0) and second having negative θ (θ < 0). Those two lines are shown by Green lines. Intersection of these two lines is calculated and turnAngle is calculated.

2.

After Converting to Gray scale and Blurring

After applying Canny transformation

PAGE 10

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

LMA | Lane Detected Image Here, Green Line having positive θ has greater θ value than Line having negative t. That’s why turn angle indicates to turn Left. By negligible 1˚.

3.

After Converting to Gray scale and Blurring

PAGE 11

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

After applying Canny transformation

LMA | Lane Detected Image Here, Green Line having positive θ has greater θ value than Line having negative t. That’s why turn angle indicates to turn Left. By negligible 5˚. In LMA, there is more number of dots in Negative θ direction representing Green lines with θ < 0˚.

PAGE 12

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

Advantages  

It is not affected by recognizing the Lane whether it is Day, evening or night because we are just identifying lines. Its approach is simple enough.

Disadvantages  

If there are Lanes in the image but are short in length, then it will be difficult to detect them as concept of length of similar c & θ is used. If image is perpendicular to actual lanes, then lanes in the image will look nearly Horizontal. Thing is that we are excluding nearly horizontal lines to be analyzed because it leads overall recognition into erroneous form being in Simple Lane Detection. Thus, in this case, lanes will not be recognized.

1.1.3 Color Based Lane Detection In Simple Lane Detection, only Lines and their lengths are analyzed. That method is something like blind method. There was no intelligent identification in previous method. Now, Color based Lane has extra level of identification based on color. The sub-modules functionalities can differ with how Roads are constructed, Lanes are colored, Pattern & Positions of Lanes. These differences may imply separate implementation based on Type of Road on above parameters. This module has several sequential phases through which sample image is processed.

1.1.3.1 Converting Image from RGB to HSV/HSB Definitions and Requirement of this conversion:

RGB color model It is an additive color model in which red, green and blue light are added together in various ways to reproduce a broad array of colors. The name of the model comes from the initials of the three additive primary colors, red, green and blue. Advantages  Monitors of almost all devices are based upon RGB value of pixel.

PAGE 13

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

 

That’s why, displaying RGB valued pixel, there will be no further mathematical calculations. Hardware & Software both are implemented and supported by RGB color model.

Disadvantages 



Suppose we wont to co-relate particular color with its RGB value, it is somewhat easy if color is nearby in range of any of Red, Green, Blue color and we are able to identify proportion of these three colors visually or color is tended to particular of any one of them. But if color is such that we are not able to visualize proportion of these colors, its RGB value is too hard to be estimated. In such case, identifying particular color in image becomes too hard even if we are performing identification in larger scale that’s in range.

To overcome toughness* of identifying RGB value, other color model can be used. That is HSV/HSB color model.

HSV/HSB color model HSL and HSV are the two most common cylindrical-coordinate representations of points in an RGB color model. The two representations rearrange the geometry of RGB in an attempt to be more intuitive and perceptually relevant than the cartesian (cube) representation. HSL and HSV are both cylindrical geometries, with hue, their angular dimension, starting at the red primary at 0°, passing through the green primary at 120° and the blue primary at 240°, and then wrapping back to red at 360°. In each geometry, the central vertical axis comprises the neutral, achromatic, or gray colors, ranging from black at lightness 0 or value 0, the bottom, to white at lightness 1 or value 1, the top.

Analysis Hue is the value by which we can identify which color it is. Such functionality is not given by RGB color model. Colors are ranged within 0°

PAGE 14

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

to 360°. Particular color is ranged within few degrees. It facilitates us to identify particular color just by applying Hue range on pixel. It is useful when same code is going to be handle Real-Time lane detection whether it system is running during Morning, Noon, Evening and Night Saturation tells how dense color is. Its high value simplifies detection of the color. That’s done by seeing richness of the color. There will be different value of Saturation in Morning, Noon, Evening and Night. Thus, it is useful to identify these parts of day by analyzing value of different unique parts of image. Value/Brightness tells how much fade or dark or dull color is. It is helpful when camera is having auto-brightness functionality. Such cameras increases brightness of image when there are too many things are having less Value/Brightness for balancing overall visibility of the image. It decreases brightness of image when there are too many things are having more Value/Brightness for balancing overall visibility of the image. Thus, this value is useful in determining range for detection. Advantages  Color can be identified easily just by entering range of Hue. (Problem is overcome that occurred in RGB color model)  It gives two significant parameters i.e. Saturation and Value and both can be visualize easily. Disadvantages  It requires complex calculation for converting RGB values 

pixel into its HSV equivalent. If image has high resolution, there will be lots of calculation for each pixel. For example, for full HD screen (1920 x 1080) there will be 2073600

PAGE 15

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES



pixels. For each pixel, RGB to HSV calculations will be performed. That is costlier. Obviously, Lane Detection is performed in Real-Time environment in which Video stream is Input. Video may be of 60 fps. In this case, manipulating each pixel will be too much costly. It may delay the running program. But as computer hardware performance is growing with great scale, such conversion can be implemented through hardware as a separate module. There is also a option of using GPU for parallel conversion of many pixels.

RGB to HSV Conversion [Code] public static float[] HSV(int[] rgb) { float hsv[] = new float[3]; float _r = ((float)rgb[2] / 255); float _g = ((float)rgb[1] / 255); float _b = ((float)rgb[0] / 255); float cMin = Math.min(Math.min(_r, _g), _b); float cMax = Math.max(Math.max(_r, _g), _b); float delta = (cMax - cMin);

if(cMax != 0) hsv[1] = 100 * (delta / cMax); else {

hsv[0] = -1;

hsv[1] = 0;

}

if(cMax == _r) hsv[0] = (60 * (((_g - _b) / delta) % 6));

PAGE 16

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

else if(cMax == _g) hsv[0] = (60 * (((_b - _r) / delta) + 2)); else if(cMax == _b) hsv[0] = (60 * (((_r - _g) / delta) + 4));

if(hsv[0] < 0) hsv[0] += 360; hsv[2] = 100 * cMax; return hsv; }

Applications of HSV Conversion (sub-module) In Color Based Lane Detection, lanes having White and Yellow color are detected as these two are the most commonly used on Road. Thus, detection of lines underlying these two colors play significant role. Thus, as said before, for detecting particular color, HSV color model is quite useful. I am using HSV color model for detection of White and Yellow color because color to detection is unique and its Intensity (Saturation) and Brightness may differ. Thus, range of Hue is given for detection and value of Brightness is also put with AND logic. In some cases, Saturation is also put with AND logic.

Detection of White color Its detection is a bit difficult than detecting other colors. White and Black are unique colors as per analysis below.

On RGB scale RGB = [0, 0, 0] has Black color, RGB = [255, 255, 255] has White color for 8-bit color representation. White color detection is related to detection of different shades of Gray. These values are such that value of RBG individuals are same (R = G = B) but they vary from 0 to 255.

PAGE 17

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

For example,

RGB = [0, 0, 0] | R = G = B | Black color (pure)

RGB = [128, 128, 128] | R = G = B | Gray color

RGB = [220, 220, 220] | R = G = B | Light Gray color

RGB = [225, 225, 225] | R = G = B | White color (pure) Thus, White color does not refer any color. It is independent with respect to colors regarding to its RGB values of its different shades. Analysis, Advantages & Disadvantages of using RGB scale for detecting white color are shown below.

PAGE 18

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

Analysis Thus, if we use RGB scale for detecting White color, we have to include range let’s say more than 220 for each of R, G & B equally. But, in Real-Time situation, it is not always necessary. There may value nearby to actual gray shades’ value.

For example, RGB = [218, 220, 224] | Nearer to Gray color Thus, such type of case exists in real-time video capturing that Lane is white but it is evening time. So, there are chances that Lane came in image also reflects light coming from somewhere else, probably light of Sun! So there may variations in condition R = G = B. We cannot predict whether these variations are in wider scale or smaller* scale. So, for programming implementation, combining it with AND and OR logic is difficult because it is requiring* wide range for detection that may lead to detection of colors those we have not targeted to detect.

Advantages 



If surrounding environment is clean enough, and it is noon, then there will be less reflection on road due to other objects. So, there will be less variations for detecting White color in RGB scale. It can be implemented with taking minor variation in consideration. No pre calculations are needed and saves computational time for each pixel.

Disadvantages 



We cannot use it during all day. It needs to switch its parameters if RGB scale is used. Variations may cross certain limits in Bluish or light or dark Yellowish evening. In this cases, B values may be high or R too respectively. It may work well with cameras having auto-brightness feature that adjusts gray level automatically.

PAGE 19

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

On HSV/HSB scale Analysis Using HSV scale for detection of white color is much useful and easier than RGB scale. What we have to do is only setting the range of Saturation and Brightness. Because, White color is independent of Hue and that is making its detection excluding 1 parameter and making us to worry only for 2 parameters. Hue represents the color. Saturation represents how much color is dense. If color is less dense, then it will lead to White color. Furthermore, high value of brightness confirms color being white putting in AND logic with range of Saturation. These things are shown and explained below.

Here, Hue refers to Blue color but because of Saturation is the least and Brightness is the maximum, it shows pure White.

Here, Hue refers to Cyan color but because of Saturation is the least and Brightness is the maximum, it shows pure White.

Here, Hue refers to Cyan color but because of Saturation is the least and Brightness is the not maximum, it shows Gray. It is useful when it is evening and White lane becomes dull. So, Hue will not play role while detecting White color. Only range of Saturation and Brightness is scaled. Advantage and Disadvantages are discussed below.

PAGE 20

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

Advantages 





Like RGB scale, we need not worry for 3 values and we do not have need to fall in wide variation that may occur in RGB range. In HSV, only 2 values are analyzed those are Saturation and Brightness. Saturation and Brightness can be tune easily that how, White color detection can be done when there is case of reflection of other colors’ light just by making range of Saturation and Brightness wide. It is useful when camera is having auto-brightness feature.

Disadvantages  

As usual for HSV scale, it has higher computational complexity. Using HSL color model will make much sense as it has small scale and much accurate color spectrum.

Detection of Yellow color It is too difficult to detect Yellow color in RGB scale. Whereas in HSV scale, it is much easier. How? It is explained below.

On RGB scale Yellow color is proper mixture of R, G & B. It looks simple but when it comes to shades of Yellow, those may occur in Real-Time environment, it becomes tough. Analysis, Advantages & Disadvantages are explained below.

Analysis Different shades of Yellow and their RGB values are shown below. Pure Yellow color is made up of RGB = [255, 255, 0]. Bur observing its different shades, there may be very wide variation. In second image, RGB = [195, 226, 40] represents Greenish Yellow color that may occur in Bluish environment.

RGB = [255, 255, 0] | Pure Yellow color

PAGE 21

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

RGB = [195, 226, 40] | Greenish Yellow color

Advantages  

No pre mathematical calculations are required to be performed. It is useful when environment is clean and there is less reflections and there is clear sky.

Disadvantages 

Computational calculations are required.

On HSV/HSB scale It is quite easier for Yellow color to be recognized using HSB scale. It is explained by below examples by same color those were shown in RGB scale.

HSV = [62, 100, 100] | Pure Yellow color

HSV = [70, 82, 89] | Greenish Yellow color Analysis, Advantages and Disadvantages are explained below.

PAGE 22

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

Analysis Yellow color has its own definite spectrum in Hue range. Richness is shown by Saturation value and Shades of Yellow are differentiated by Brightness value. Yellow color occurs exactly at Hue at 60. But its variance is between 50 to 70. If there is variance in shades and even in color, this scale can detect Yellow color easily using Hue value.

Advantages   

During Morning, Noon, Evening and Night detection of color is easy. Detection of yellow color can be scaled to much wider spectrum to cover all possible shades. Easy to visualize

Disadvantage 

High resolution image will eat CPU time for RGB to HSV conversion.

1.1.3.2 Solid Yellow between two Solid White Lanes Right Side - This program implementation supports detection of Color Based Lane Detection. In this section, it is performed on typical type of lane. Here, solid yellow lane is between two solid white lanes. Procedures for detecting White and Yellow color is explained below. For detecting White Lane, white color is detected after performing RGB to HSV conversion. White color is detected by Saturation and Brightness component of HSV scale. For detecting Yellow Lane, white color is detected after performing RGB to HSV conversion. Yellow color is detected by Hue, Saturation and Brightness component of HSV scale. Here, White and Yellow color is detected with sharp threshold range. So, there will these colors are extracted without influence of nearby colors’ range. So, there is no need of blurring the image before detecting edges. Next step is to detect edges in color extracted image. After this, line detection that is HoughLines() is called. It will give detected lines in image as per threshold value set earlier. If threshold is small value, then more lines will be detected. At this point threshold value is set less as there is clear image for both White and Yellow color extracted image.

PAGE 23

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

Extracting White color

Extracting Yellow color

PAGE 24

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

TurnAngle and Side indicator with respect to White and Yellow detected lane

Explanation Here, White and Yellow lanes are detected first. Now, after getting set of lines detected in image, length of line is calculated if length is above threshold length, it will be added to LMA of Yellow lines. Same procedure is performed for White lanes. Now, there is possibility that there may be more than one White Lanes in right side or left side. If vehicle is on right side, Theta of white lane as per threshold value will be negative (see above photo but here there is only one white lane on right side). Now, nearest lane will be that whose theta is more negative. So, while on going scanning lines, if above threshold value length, white length comes and if its theta is more negative, then it will be nearest lane than previous scanned by this same procedure. If vehicle is on left(wrong) side, Theta of white lane as per threshold value will be positive (see above photo but here there is only one white lane on left side). Now, nearest lane will be that whose theta is more positive. So, while on going scanning lines, if above threshold value length, white length comes and if its theta is more positive, then it will be nearest lane than previous scanned by this same procedure. These white scanned lanes’ represented is saved in LMA of White lines.

PAGE 25

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

Next thing is to make decision based on White and Yellow detected lanes. Decision is made up of two components – Current Lane and TurnAngle (necessary if vehicle is distracted). Values of White and Yellow lanes i.e. theta and c are used for finding it. As explained in above paragraphs, theta is calculated with respect to positions of White and Yellow lanes. Calculations are performed on bases of LMAWhite & LMAYellow as shown below.

Here, left side, there is LMAYellow and right side, there is LMAWhite. It points out where Yellow and White lanes are going to intersect Y-axis and by what angle had by them. Below are threshold values oh HSV/HSB color scale for detecting White and Yellow colors.

HSV Thresholds Yellow (left): H > 40 && H < 70 && S > 40 White (right): S < 10 && V > 99

Where H ∈ [0, 360] S ∈ [0, 100] V ∈ [0, 100]

LMAYellow & LMAWhite

PAGE 26

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

Wrong Side – Here is an example shown of the case when vehicle is going on Wrong side of the road. In this case, theta of White lane will be positive and theta of Yellow lane will be negative.

Extracting White and Yellow color

PAGE 27

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

TurnAngle and Side indicator with respect to White and Yellow detected lane

HSV Thresholds Yellow (left): H > 40 && H < 70 && S > 40 White (right): S < 10 && V > 99

Where H ∈ [0, 360] S ∈ [0, 100] V ∈ [0, 100]

LMAYellow & LMAWhite

PAGE 28

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

1.1.3.3 Dashed White between two Solid Yellow Lanes Right Side - It is also possible that road may have a dashed White lane between two solid Yellow lanes. In that case, I had to change the code in some way. Here, Yellow color is detected with nearly sharp threshold range because we are detecting solid Yellow lanes those may probably thin enough in the image due to being far in actual. For that Saturation range for allowing color for being detected is widen such that, more shades of Yellow color can be detected. So, there will these colors are extracted with some influence of nearby colors’ range. So, there is need of blurring the image before detecting edges. Now, White color is extracted in different way than earlier case. Here, White lane is in term of dashed line. Being dashed line in image, potion of dashed white lane that is near has much Brightness value than the portions which are far actual. So, detecting less bright dashed white line, Brightness range for detection should be widen with allowing a bit less values. Problem due to decreasing threshold value of Brightness that may occur is there will be great influence of nearby colors’ range of Brightness’s value. For example, light Blue colored sky can come in picture while detecting lanes those are on the ground. There is more probability of having electric poles and their transmitting wires which cuts the sky and comes in image as form of lines. So, applying canny transformation, edges are detected in electric wire. But good thing is that these lines in extracted colored image are very thin. Solution applied to remove these thin lines is to blur the image in some level. It will remove thin/sharp lines before applying HoughLines(). Effect of blur can be changed as per surrounding environment. Blur should not be too much that it will disappear the White lane which was actually important. Range of HSV scale for detecting white lane is much remarkable than previous case. Next step is to detect edges in color extracted image. After this, line detection that is HoughLines() is called. It will give detected lines in image as per threshold value set earlier. If threshold is small value, then more lines will be detected. At this point threshold value is set less as there is clear image for both White and Yellow color extracted image. Lane detection is further conducted as per explanation explained below.

PAGE 29

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

[Dashed] Extracting White color with widen threshold range

Extracting Yellow color with widen threshold range

PAGE 30

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

TurnAngle and Side indicator with respect to White and Yellow detected lane

Explanation Here, case is different as per road. Yellow solid lanes are at opposite side White dashed lane. Yellow lanes will be recognized but White may or may not as they are dashed and decreasing in length too. Here, white lanes those are smaller in length may not be perfectly detected. Here, as we have made range of HSV wide in scale for detecting white color of small dashed lines, there will also be lines those are larger or smaller than average of White dashed lane’s length. It is necessary to remove those larger and smaller lines to get those only those lines which are on White lane only. For that, only those lines are extracted whose length is between pre-decided length of dashed lane. As both White and Yellow lanes are detected, next step is to extract meaningful information from them. Here, there will be Two major Yellow solid lanes side by side of White lane. So, it will be easier to find intersection point from Yellow lane only. If in case, one of Yellow lane is not there in image, then intersection point can be estimated using method similar to the earlier method that is using White dashed lane and Yellow solid lane. In this example, both

PAGE 31

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

Yellow lanes are visible. So, intersection point is found from them and Turn-angle is calculated and both are shown in output image. Now, it is decided whether vehicle is on Right side or Left side of dashed White lane. This can be found from both methods.  

Comparing slopes of both Yellow lanes Slope of White dashed lane

Comparing slopes of both Yellow lanes 





If YellowLaneright has large absolute valued slope than absolute valued YellowLaneleft, then vehicle is on Right side of the White dashed lane. If YellowLaneright has smaller absolute valued slope than absolute valued YellowLaneleft, then vehicle is on Left side of the White dashed lane. Else if both have same value, then vehicle is just on the White dashed lane if vehicle is going straight forward. That is vehicle is on center of the road.

Still, this method is not trustworthy. It is useful only when vehicle is going exactly straight forward with road. But vehicle is turned in some degree, then this method is never useful. For such case, 2nd method will make sense.

Slope of White dashed lane  

Slope of white dashed lines will be positive when vehicle is on Right side of White dashed lane. Slope of white dashed lines will be negative when vehicle is on Left side of White dashed lane.

This method is much trustworthy than earlier one. If vehicle is on the right side of White dashed lane and it is turned on right side or it is on the left side of White lane and it is turned on left side, then white dashed lane may exactly vertical. In such case, slope of White dashed lane is not enough to find vehicle’s position. Location of Intersection Point of two Yellow solid lanes. If Intersection Point is on left side, then vehicle is on the right side else it is on left side of the White dashed lane.

PAGE 32

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

HSV Thresholds Yellow (left): H > 40 && H < 70 && S > 30 White (right): S < 10 && V > 95 Where H ∈ [0, 360] S ∈ [0, 100] V ∈ [0, 100]

Here, threshold values of White and Yellow lanes are made widen as per shown above. By comparing last two sample outputs, theta of YellowLaneright is greater than YellowLaneleft. By this, we can also say that vehicle is on the Right side. It can be explained well by theta of White dashed lane that is positive and indicates that vehicle is on the right side of White dashed lane

LMAYellow & LMAWhite

PAGE 33

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

Left Side & Multiple Yellow solid lanes In the case when there may be more than one Yellow solid lanes in right side or left side. If vehicle is on right side, Theta of Yellow solid lane as per threshold value will be negative. Now, nearest lane will be that whose theta is more negative. So, while on going scanning lines, if above threshold value length, Yellow length comes and if its theta is more negative, then it will be nearest lane than previous scanned by this same procedure. If vehicle is on left side, Theta of white lane as per threshold value will be positive. Now, nearest lane will be that whose theta is more positive. So, while on going scanning lines, if above threshold value length, Yellow length comes and if its theta is more positive, then it will be nearest lane than previous scanned by this same procedure. These Yellow scanned lanes’ represented is saved in LMA of Yellow lines.

[Dashed] Extracting White color with widen threshold range

PAGE 34

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

Extracting solid Yellow lane color with widen threshold range

TurnAngle and Side indicator with respect to White and multiple Yellow detected lane

PAGE 35

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

HSV Thresholds Yellow (left): H > 42 && H < 70 && S > 30 White (right): S < 10 && V > 95 Where H ∈ [0, 360] S ∈ [0, 100]

V ∈ [0, 100]

Here, threshold values for detecting Yellow solid lane is bit changed tha earlier.

LMAYellow & LMAWhite

PAGE 36

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

1.1.3.4 Dashed Yellow between two Solid White Lanes It is also possible that road may have a dashed White lane between two solid Yellow lanes. In that case, I had to change the code in some way. Remarkable change done over previous case is that range of HSV scale. It is shown in upcoming output results. Details and explanation for this case are same as earlier one. Those are not repeated only White and Yellow colors are interchanged.

HSV Thresholds Yellow (left): H > 40 && H < 70 && S > 30 White (right):

S < 15 && V > 95 Where H ∈ [0, 360] S ∈ [0, 100] V ∈ [0, 100]

Here, threshold values of White color are changed.

LMAYellow & LMAWhite

PAGE 37

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

Extracting solid White lane color after making image blurred.

[Dashed] Extracting Yellow color with widen threshold rang

PAGE 38

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

TurnAngle and Side indicator with respect to White and Yellow detected lane

1.2

Departure Warning System

In road-transport terminology, a lane departure warning system is a mechanism designed to warn the driver when the vehicle begins to move out of its lane (unless a turn signal is on in that direction) on freeways and arterial roads. These systems are designed to minimize accidents by addressing the main causes of collisions: driver error, distractions and drowsiness. In this implementation, it is done in form of whether vehicle is on Left side or Right side when it is One-way or Wrong side or Right side when it is Two-way road.

PAGE 39

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

2. Object Recognition (for predefined single object) & Depth Perception Depth perception is the visual ability to perceive the world in three dimensions (3D) and the distance of an object. Depth sensation is the corresponding term for animals, since although it is known that animals can sense the distance of an object (because of their ability to move accurately, or to respond consistently, according to that distance), it is not known whether they "perceive" it in the same subjective way that humans do.

Both modules are interconnected. So, both are implemented in Parallel way.

Depth Perception can be implemented by 



Monocular cues o Motion parallax o Depth from motion o Kinetic depth effect o Perspective o Relative size o Familiar size o Absolute size o Aerial perspective o Accommodation o Occlusion o Curvilinear perspective o Texture gradient o Lighting and shading o Defocus blur o Elevation Binocular cues o Stereopsis, or retinal (binocular) disparity, or binocular parallax o Convergence o Shadow Stereopsis

Here, Depth perception is implemented using Binocular Cues (Stereopsis).

PAGE 40

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

Stereopsis Animals that have their eyes placed frontally can also use information derived from the different projection of objects onto each retina to judge depth. By using two images of the same scene obtained from slightly different angles, it is possible to triangulate the distance to an object with a high degree of accuracy. Each eye views a slightly different angle of an object seen by the left and right eyes. This happens because of the horizontal separation parallax of the eyes. If an object is far away, the disparity of that image falling on both retinas will be small. If the object is close or near, the disparity will be large. It is stereopsis that tricks people into thinking they perceive depth when viewing Magic Eyes, Autostereograms, 3-D movies, and stereoscopic photos.

The word "stereo" comes from the Greek word "stereos" which means firm or solid. With stereo vision you see an object as solid in three spatial dimensions--width, height and depth--or x, y and z. It is the added perception of the depth dimension that makes stereo vision so rich and special. Stereo vision or stereoscopic vision probably evolved as a means of survival. With stereo vision, we can see WHERE objects are in relation to our own bodies with much greater precision--especially when those objects are moving toward or away from us in the depth dimension. We can see a little bit around solid objects without moving our heads and we can even perceive and measure "empty" space with our eyes and brains

Source of exact Definitions: Wikipedia

PAGE 41

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

Binocular Disparity refers to the difference in image location of an object seen by the left and right eyes, resulting from the eyes’ horizontal separation (parallax). The brain uses binocular disparity to extract depth information from the two-dimensional retinal images in stereopsis. In computer vision, binocular disparity refers to the difference in coordinates of similar features within two stereo images.

PAGE 42

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

Derivation that is made for calculating Depth of object using Stereopsis Field of View of camera (Horizontal) = 40 ˚ Field of View of camera (Vertical) = 30 ˚

PAGE 43

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

Output of Stereopsis

PAGE 44

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

Outputs from different position and angles in space.

PAGE 45

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

Explanation Here, in front of Two stereo cameras, object (Red ball) is given motion. With respect to derivation mentioned in above photo, its Depth, Height and Movements are detected. 3D Graphical Interface used in above output photos is 3D Graphics Library designed by me in Q1’16. Positions of ball in real-time is calculated and its (x, y, z) are put into 3D Graphics Library and shown ball’s position. Thera are many balls (spheres) in output because upcoming video frames will generate their new positioned sphere in 3D Graphics Interface.

More details about my 3D Graphics Library: youtube.com/DhavalKadia

Object Recognition Here, object recognition is implemented using single object detection. That is done by sampling a Red ball every time and finding its coordinates in cameras. It is recognized by the OpenCV function HoughCircles(). HSV color scheme is used in it. Threshold values are changed as per requirements. In next level, it is done using OpenCV function StereoSGBM. In pre calculations, it also performs Object matching of surrounding environment in front of camera. After doing that, it performs Stereopsis concept to find depth and gives results thorough doing Object Recognition.

PAGE 46

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

Depth Map In 3D computer graphics a depth map is an image or image channel that contains information relating to the distance of the surfaces of scene objects from a viewpoint. The term is related to and may be analogous to depth buffer, Z-buffer, Z-buffering and Z-depth. The "Z" in these latter terms relates to a convention that the central axis of view of a camera is in the direction of the camera's Z axis, and not to the absolute Z axis of a scene. As a part of Depth perception, it is put in use by using StereoSBGM function of OpenCV. That stands for Semi Global Matching.

StereoSGBM (Stereo Global Matching) This algorithm is designed by H. Hirschmuller. It is put in use in OpenCV as a function. I used it with setting parameters with respect to stereo images. Constructor of the function is explained below. White colored pixels are much nearer than Black in output results.

StereoSGBM StereoSGBM stereoSG = new StereoSGBM( # Minimum DIsparities value. Normally, it is zero but sometimes rectification algorithms can shift images, so this parameter needs to be adjusted accordingly. # 16xn Maximum disparity minus minimum disparity. The value is always greater than zero. In the current implementation, this parameter must be divisible by 16. # Matched block size. It must be an odd number >=1. Normally, it should be somewhere in the 3..11 range. # p1 The first parameter controlling the disparity smoothness. See below.

PAGE 47

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

# p2 The second parameter controlling the disparity smoothness. The larger the values are, the smoother the disparity is. P1 is the penalty on the disparity change by plus or minus 1 between neighbor pixels. P2 is the penalty on the disparity change by more than 1 between neighbor pixels. The algorithm requires P2 > P1. See stereo_match.cpp sample where some reasonably good P1 and P2 values are shown # disp12MaxDiff :Maximum allowed difference (in integer pixel units) in the left-right disparity check. Set it to a non-positive value to disable the check. # prefilterCap :Truncation value for the prefiltered image pixels. The algorithm first computes x-derivative at each pixel and clips its value by [preFilterCap, preFilterCap] interval. The result values are passed to the Birchfield-Tomasi pixel cost function. # uniqueness ratio :Margin in percentage by which the best (minimum) computed cost function value should "win" the second best value to consider the found match correct. Normally, a value within the 5-15 range is good enough. spreckle Range :Maximum disparity variation within each connected component. If you do speckle filtering, set the parameter to a positive value, it will be implicitly multiplied by 16. Normally, 1 or 2 is good enough. full DP "Set it to true to run the full-scale two-pass dynamic programming algorithm. It will consume O(W*H*numDisparities) bytes, which is large for 640x480 stereo and huge for HD-size pictures. By default, it is set to false.

Source: OpenCV 2.4.8 Documentation Here. Depth is scaled into [0, 255] valued distance range that how, it can be visualized in Gray scale using those converted value. White color represents less depth than those are Black. Theoretically, this distance for measuring depth is called Z-depth.

PAGE 48

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

Depth Map Output

Left

Right

Depth Map with particular parameters set for image

PAGE 49

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

Left

Right

The Standard Stereo image pair for Depth Perception

PAGE 50

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

Left

Right

The Standard Stereo image pair for Depth Perception

PAGE 51

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

Explanation Here, main thing is parameters of the function. These parameters depend upon how cameras are placed with what distance. Somehow, it also depends upon at how much depth, average number of objects are that is what to focus with much accuracy. That depends upon convergence of camera. If there is much noise in image, then it is better to blur the image before applying it to this function. It will reduce inconsistency of Depth Map. It is quite useful when there are many sharp small objects in image. It will increase sharpness of the image and which can effect result of StereoSGBM function. To compensate it, Blurring is applied. Further improvement to result is to apply Mean-Shift. It smooths the results and removes the surface which are not available in both cameras’ each other’s view.

Parameters for above outputs

StereoSGBM Parameters Blurring is also applied when it is necessary. Output 1: StereoSGBM(0, 80, 15, 450, 1125, -1, 63, 10, 20, 32, false) Output 2: StereoSGBM(0, 80, 11, 242, 605, -1, 63, 10, 50, 32, false) Output 3: StereoSGBM(0, 48, 11, 242, 605, -1, 63, 10, 0, 32, false)

PAGE 52

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

Find More Implementations Dhaval Kadia | [email protected]

Professional Profiles Website LinkedIn ResearchGate | My Research Paper Academia | My Research Paper YouTube | Output Videos of Projects SourceForge | Executable Files of Projects GitHub

Social Profiles Google+ Facebook Twitter Blogger

PAGE 53

IMAGE PROCESSING: APPLICATIONS IN AUTONOMOUS VEHICLES

Further Milestones 1. 2. 3. 4. 5. 6. 7. 8. 9.

Curved Lane Detection Property based Object Recognition Object/Image Segmentation Focal Length based Depth Perception Use of Contours & Haar-Cascades HUD for Car & Fighter-Jet integrated LiDAR based Virtual Assistant for Driver LiDAR based Depth Perception & Object Recognition Automatic Carrier Landing System (ACLS) for Aircraft Carrier

THE END

PAGE 54