Application of Elastic Image Registration and ...

Application of Elastic Image Registration and Refraction Correction for Non-Contact Underwater Strain Measurement M. A. Haile* and P. G. Ifju† *US Army Research Laboratory, Vehicle Technology Directorate, Aberdeen Proving Ground, MD 21005, USA † Mechanical and Aerospace Engineering Department, University of Florida, Gainesville, FL 32611, USA

ABSTRACT: The refraction-induced image distortion introduces large errors in the deformation measurement of fluid submerged specimens using digital image correlation (DIC). This study provides a review of the nature of the refraction-induced image distortion, assesses experimental conditions that interact with refraction and proposes an elastic image registration technique to correct the refraction distortion of underwater images. In the elastic image registration technique, control points are selected on reference and refracted images of a template object and locally sensitive transformation functions that overlay the two images are obtained. The transformation functions so obtained are then used to reconstruct undistorted images from underwater images and the former are used as input to a DIC system. The proposed approach has shown to improve the refraction error in the order of 5–8% for typical material test samples undergoing deformation inside a water-filled glass chamber. KEY WORDS: image reconstruction, image registration, local weighted mean transform

Introduction Non-contact deformation and strain-measuring techniques, such as the digital image correlation (DIC), rely on digital images of a speckled specimen. In a typical noncontact technique, the image recording system and the test specimen are kept in the same medium, i.e. images are acquired while the specimen is being deformed in air. There is a growing need, however, to extend the use of DIC in areas where objects undergo deformation in fluid medium [1–4]. In biomechanics, for instance, mechanical tests are conducted inside an organ culture system or in a fluid nutrient medium that mimics the native physiological environment of a tissue. Image distortion resulting from refraction, however, prohibits the use of DIC in such applications and has been a challenge for experimenters. A number of researchers have attempted to solve the refraction problem mainly by developing a camera calibration algorithm using ray optics analysis. Sutton and McFadden [2] quantified the strain errors induced by an oblique glass window by exclusively focusing on the relative orientation of the interface rather than the medium itself. Barta and Horvath [3] produced a caustic curve as a function of the refractive index of water to quantify the refraction distortion on aerial image viewed from under water, the study has not mentioned ways to reconstruct the complete aerial image. Tetlow and Spours [4] adopted an image-processing scheme to create a virtual world representation of an underwater worksite. Yet their technique is limited to representation of large structures and cannot be readily adaptable for strain and deformation measurements using DIC. Lavest et al. [5] used a camera calibrated in air to infer calibration parameters for underwater application. The optical laws that Lavest used are limited to isotropic fluids and does not account for optical properties of the interface. Gennery [6] approximated the radial lens distortion using curve fitting of known points such as those measured from a calibration fixture. Both the intrinsic and

2011 Blackwell Publishing Ltd j Strain (2011) doi: 10.1111/j.1475-1305.2011.00805.x

extrinsic calibration parameters are obtained in a single least squares adjustment. Kwon and Casebolt [1] extends the direct linear transformation technique of Abdel-Aziz and Karara [7] through grouping of control points and control volumes and based on overlapping and non-overlapping strategies; however, both approaches are based on direct linear transformation hence can only be used for uniform perspective type distortions. The most complete work on the refraction problem comes from Sutton et al. [8–11]. In his work, Sutton developed a non-linear analytical solution to perform a stereo calibration for underwater imaging. The developed method incorporates a parametric representation for the orientation and position of interfaces and accounts for the effects of refraction at such interfaces. The approach presented in Sutton’s paper is a calibration approach where the camera parameters are obtained in a two-stage calibration process. The technique proposed herein, however, uses simple image-processing technique where an elastic image registration scheme that uses a local weighted mean (LWM) transform is used to reconstruct undistorted images of underwater specimens using control point matching. A non-rigid transformation matrix is derived by comparing images of a template object acquired in-water and in-air. The in-water specimen images are then swept with the transformation matrix and reconstructed before being submitted to a commercially available DIC algorithm. Detailed presentation of the approach follows in the subsequent sections.

Refraction-Induced Image Distortion Refraction occurs when light passes from one medium to another. The extent of refraction depends on the optical property of the media and the angle of incidence of the light ray. For a given test setup defined by the relative location of the camera, the interface and an object point P shown in Figure 1, Snell’s law provides [1]:

1

Application of Elastic Image Registration and Refraction Correction : M. A. Haile and P. G. Ifju

δ β

α

oP

ϕ kδ

Using the quantities defined in Figure 2, the Lagrangian normal strain ey introduced by the apparent refraction distortion is:

P P′

ϕ

o P′

Liquid

R

R

ey ¼

Plexiglass R

ϕ′ c

Air

ϕ′

co Figure 1: A simplified ray optics analysis establishing relationship between experimental parameters and angle of refraction

y1 y0 y0

(3)

The pseudo-strain given in Equation (3) is the result of refraction of light rays and has nothing to do with mechanical loading of the specimen. Also note that the pseudo-strain is not uniform as points far from the centre exhibit a lager apparent displacement than points closer to the centre of the image. A similar expression can be written for strain ex.

Image Registration

pffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi kd k2 d2 þ a2 sin u q ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ffi ¼ sin u ð1 kÞd ð1 kÞ2 d2 þ b2

where / and /¢ are the angle of incidence and angle of refraction, a is the camera to interface distance, b is the interface to object distance, d is the object to normal axis distance, k is the distance from the interface point to the normal axis normalised by d, and c is the relative refractive index of the first medium to the second medium. As shown in Figure 1, having an interface does merely shift the ray’s point of exit and has very little effect on the angle of refraction. For a given experiment with setup constants a, b, d and c, the ratio k is obtained by solving [1]:

Image registration is a process of aligning two images of the same scene through control point matching and spatial transformation [18, 20, 21]. A spatial transformation modifies the spatial relationship between pixels in a given input image to new locations in an output image. The refraction distortion can be considered as a shift in the spatial location of control points as shown in Figure 2. For a given experimental setup, control points on the refracted image can be brought into alignment with their unrefracted counterparts using LWM transformation. The later is a non-rigid mapping that allows local elastic deformation of the refracted image to match with the reference image. The LWM transform is briefly discussed in the following section.

ðc2 1Þk2 ð1 kÞ2 d2 þ c2 ð1 kÞ2 a2 k2 b2 ¼ 0

Local weighted mean transform

0

(1)

(2)

Parameter k determines the degree of image distortion induced by refraction in underwater imaging and a k value closer to the ratio a/(a + b) implies a less-distorted image and vice versa [7, 12, 13]. Figure 2 shows the refracted and unrefracted points of an arbitrary rectangular grid submerged underwater. The dark squares are the actual location of grid points on unrefracted image, and the white squares indicate the apparent position of the same grid points on the refracted image. As shown in the figure, the refracted image is larger than the unrefracted image and the degree of non-linearity increases with the angle of refraction / and the relative index of refraction of the two media.

Given N non-collinear set of control points (xi, yi) on the unrefracted (or reference image) and (ui, vi) on the refracted image as shown in Figure 3, it is possible to construct an approximate mapping between these points using polynomial functions f and g such that: ui ¼ f ðxi ; yi Þ vi ¼ g ðxi ; yi Þ i ¼ 1; . . . ; N

(4)

Equation (4) can be conveniently re-written as a loci of 3D points in the Euclidian space as (xi, yi, ui) and (xi, yi, vi) and accordingly polynomials f and g can be described by: f ðx; y Þ ¼

j n X X

ajk xk yjk

j¼0 k¼0

Refracted

g ðx; y Þ ¼

j n X X

(5) k jk

bjk x y

j¼0 k¼0

■ Non-refracted y1

y0

(u1,v1)

y z

(x1,y1)

x

Figure 2: Refracted and non-refracted points of a rectangular grid [1]

2

(u2,v2)

(ui,vi) (xi,yi)

(x2,y2) Figure 3: Control point pairs on the refracted and reference images


M. A. Haile and P. G. Ifju : Application of Elastic Image Registration and Refraction Correction

The coefficients ajk and bjk polynomials f and g can be calculated by substituting the N measurements into Equation (5) and solving the resulting system of linear equations as: j n X X

jk

ajk xki yi

(6) jk bjk xki yi

¼ vi

(9)

i¼1

1 W ½ðx xi Þ2 þðy yi Þ2 2 =ln Qi ðx; yÞ vi ¼ i¼1 N 1 P W ½ðx xi Þ2 þðy yi Þ2 2 =ln N P

¼ ui

j¼0 k¼0 j n X X

1 W ½ðx xi Þ2 þðy yi Þ2 2 =ln Pi ðx; yÞ ui ¼ i¼1 N 1 P W ½ðx xi Þ2 þðy yi Þ2 2 =ln N P

for i ¼ 1; . . . ; N

i¼1

j¼0 k¼0

If a point (x, y) in the reference image is near control point i, then the point corresponding to (x, y) in the refracted image is also near control point i. Therefore, the u value of point (x, y) can be determined from u values of the nearest control points with appropriate weights as: N P

Wi ðx; yÞui f ðx; y Þ ¼ i¼1N P Wi ðx; y Þ

where Pi(x, y) are polynomials passing through points (xi, yi, ui) and Qi(x, y) are polynomials passing through points (xi, yi, vi) for all (N ) 1) control points nearest to point i. Polynomials Pi and Qi are often chosen to be of order two however higher degrees can also be applied as the weighted sum obtained will always be continuous and smooth all over the image [17, 19].

Experimental Validation

i¼1

(7)

N P

Wi ðx; yÞvi g ðx; y Þ ¼ i¼1N P Wi ðx; y Þ i¼1

As stated earlier, we attempt to overlay the refracted and reference images using a set of polynomials that perform transformation in a certain local region of the image. This implies that the zone of applicability of polynomials f and g is limited to a small local region around control point i. The condition of locality is enforced by defining the weight function as [14–16]: Wi ðLÞ ¼ 1 3L2 þ 2L3 Wi ð LÞ ¼ 0

(10)

0L1 L>1

(8)

where L = [(x ) xi)2 + (y ) yi)2]1/2/ln and ln is the distance of point (xi, yi) from (N ) 1)th nearest control point on the reference image. The local weight function ensures that polynomial i will have no influence on points in the reference image whose distance from control point is greater than ln. The weighted mean of all polynomials passing through an arbitrary point (x, y) and having non-zero weight are [14]:

(A)

To verify the validity of the elastic image registration scheme presented earlier, a series of underwater tests were performed on a thin rubber specimen with a nominal thickness of 0.75 mm. The specimen is densely speckled (manually marked) with a felt-point water proof marker with a nominal diameter of 1 mm (or about 5 pixels). The template shown inside the hydration chamber in Figure 4 is a uniformly gridded rectangular panel of 250 mm long and 120 mm wide with 120 · 50 control-points (CPs) with 0.85 mm in diameter uniformly spaced apart. The CPs are printed in black on a white background similar to the calibration board of a commercially available DIC grids. The DIC system used in this test is shown in Figures 4 and 5.

Acquiring DIC images of the template As a first step, the template was positioned at the middle of the empty chamber and is tightly clamped with its front face parallel to the viewing side of the chamber and its edges aligned with the axis of the testing machine as shown in Figure 4A. A pair of digital images is acquired and stored in the default data format. The chamber is then filled with running water as shown in Figure 4B without disturbing the template, and a second pair of digital images is acquired and stored in the same default data format.

Empty chamber

(B)

Water filled chamber

Camera-1

Camera-2 Template

Material testing frame Figure 4: Digital image correlation setup for acquiring: (A) reference, and (B) refracted images of the template


3

Application of Elastic Image Registration and Refraction Correction : M. A. Haile and P. G. Ifju Water filled chamber

(A)

Empty chamber

(B) Specimen

Camera-2

Camera-1

Material testing frame

Figure 5: Uniaxial test setup inside (A) water-filled, and (B) an empty hydration chamber

Acquiring DIC images of the test specimen

image are shifted to the left and the image is larger in size than its unrefracted counterpart. The non-uniformity of the distortion is evident from the variation of distances between the larger control points marked on the images. The transformation matrix is recovered by registering the two images of the template and comparing the coordinates of the control points. In this work, we used MATLAB software (The MathWorks Inc., Natrick, MA, USA) to automate the basic image-processing task. The registration process involves the following three steps:

Second, the speckled specimen is securely clamped inside the empty chamber and a uniaxial stretch test was performed. Reference and deformed images of the specimen are acquired at four different strain levels (13, 20, 27, and 35%) using a pre-calibrated DIC setup. The cross head displacement of the test frame is automatically controlled by Testworks software (MTS Systems Corporation, Eden Prairie, MN, USA) installed on separate computer. As these images were captured while the chamber is empty as shown in Figure 5B, they can be directly input into the VIC-3D correlation algorithm to calculate the displacement or strain fields on the surface of the specimen. In a repeated test shown in Figure 5A, the chamber is filled with water and the same uniaxial test was performed on the same specimen. Reference and deformed images of the test specimen are acquired at the four strain level as before. As these set of images were acquired while the specimen is deforming underwater, they contain refraction distortion and need to be corrected before submitting into the DIC algorithm. The refraction correction process involving the elastic image registration scheme is presented in Transforming images of the test specimen.

Figure 7 compares the transformed and the reference (unrefracted) images of the template. As shown in the figures, the reconstructed image registers well with the reference image.

Results and Discussions

Transforming images of the test specimen

Figure 6 shows the reference and the underwater images of the template panel captured by camera-2. Only the elastic image registration scheme that determines the mapping function for camera-2 will be presented. The mapping function for camera-1 can be obtained in the same manner. As shown in the figure, control points on the underwater

For a given experimental set up, the ui and vi mapping functions that enable reconstruction of the underwater (refracted) images are generated via control point matching of the template image as stated earlier. Underwater images of the test specimen are swept by the parameters of the mapping functions to register the refracted image space

(B)

Reference image

100

100

200

200

300

300

400

(u, v)

500

Y-pixels

Y-pixels

(A)

1 Load the two images into MATLAB workspace and perform image segmentation and control point matching, 2 Extract coordinates of all matching control points (x, y) and (u, v) from both the underwater and reference images respectively, 3 Find local mapping functions ui and vi per the procedure discussed in Local weighted mean transform using Equations (9) and (10).

600

500 600 700

800

800

900

900

1000

1000 400

600

X-pixels

800

1000

1200

(x, y)

400

700

200

Underwater image

200

400

600

800

1000

1200

X-pixels

Figure 6: Template images acquired inside (A) an empty, and (B) water filled chamber

4


M. A. Haile and P. G. Ifju : Application of Elastic Image Registration and Refraction Correction

Reference image

(A)

Underwater image

(B) 100

100

200

200

200

300

300

300

400

400

400

500

500

500

600

600

600

700

700

700

800

800

800

900

900

900

1000

1000

1000

Y-pixel

100

200

400

600

800

1000

1200

200

400

X-pixel

600

800

Reconstructed image

(C)

1000

1200

200

X-pixel

400

600

800

1000

1200

X-pixel

Figure 7: Specimen at 35% strain (A) reference, (B) underwater and, (C) reconstructed images

with the reference image space. Figure 7 shows the reference image, the underwater image and the reconstructed image of the test specimen at 35% strain level. As shown in the figures, the reconstructed image registers well with the reference image. As a final step, these images were directly input into VIC3D correlation algorithm to calculate the in-plane strain field in x and y directions. Figures 8–10 shows the full-field

Figure 8: In-plane strains on the reference images at 35% strain

strains on the surface of the deformed specimen for the three cases at 35% strain level. The step size for this DIC correlation is fixed at 5 pixels with a subset size of 19 pixels. Strain calculations are done with a curvilinear 90% decay filter using a filter size of 15 pixels for data averaging and smoothing. Figure 12 shows the variation of strain along a vertical line drawn at the mid section of the uniaxial test specimen shown in Figure 11. The plots show that the strain field in the reconstructed image closely follows the reference image with the maximum error in the order of 2%. As the transformation functions f and g are derived from a finite number of discrete control points, some mismatch error is inevitable and the registration cannot be exact. In general, the mismatch error depends on the number of control point pairs used on the template matching process. The larger the number of control points used the smaller the mismatch error would be; however, use of large number of control points may increase the computational complexity of the method. Figure 13 shows the measured strain error from underwater and reconstructed images in comparison with the reference image at the four strain levels along the mid section of the specimen. Owing to the large refraction angle, points far from the centre of the specimen exhibit a larger pseudo-displacement than those closer to the centre.

Figure 9: In-plane strains on the underwater images at 35% strain

L1

Figure 10: In-plane strains on the reconstructed images at 35% strain


Figure 11: Strain values are sought along line L1 of a typical uniaxial specimen

5

Application of Elastic Image Registration and Refraction Correction : M. A. Haile and P. G. Ifju ε yy = 27%

ε yy = 35%

0.4

0.29 Reference Reconstructed Underwater

0.28

Strain ε yy

Strain ε yy

0.38 0.36 0.34

0.26 0.25

0.32 0.3 300

0.27

400

500

600 700 Y-Pixels

800

0.24 300

900 1000

400

500

900 1000

800

900 1000

0.145 Reference Reconstructed Underwater

0.14

0.2

Strain ε yy

Strain ε yy

800

ε yy = 13%

ε yy = 20%

0.21

0.19

0.135 0.13 0.125 0.12

0.18 300

600 700 Y-Pixels

400

500

600

700

800

0.115 300

900 1000

400

500

600

700

Y-Pixels

Y-Pixels

Figure 12: Comparison of longitudinal strain along a line drawn at the mid section of the uniaxial specimen

ε yy = 35%

ε yy = 27%

0.05

0.04 Underwater Reconstructed

Underwater Reconstructed 0.03

Error, ε yy

Error, ε yy

0.04 0.03 0.02

0.01

0.01 0 300

0.02

400

500

600 700 Y-Pixel

800

900

0 300

1000

ε yy = 20%

0.03

400

500

600 700 Y-Pixel

800

900

1000

800

900

1000

ε yy = 13%

0.02

0.025

Error, ε yy

Error, ε yy

0.015 0.02 0.015 0.01

0.01

0.005 0.005 0 300

400

500

600 700 Y-Pixel

800

900

1000

0 300

400

500

600 700 Y-Pixel

Figure 13: Strain errors on underwater and reconstructed images relative to reference images measured along the mid section of the uniaxial specimen

The error magnitude of the reconstructed image is in the order of 1–2%, whereas the underwater image exhibit errors in excess of 7% at the 35% strain level. The fact that the error is a function of the distance from the centre of the specimen indicates that large errors would result from the DIC strain of large underwater specimens. It should also be noted that for tests performed in fluids with higher refractive index such as sodium chloride irrigation solution

6

(a common sterile tissue cleanser), the DIC strain error will be higher than the same test performed in distilled water. In general, the presence of solid particulates will not affect the performance of the correction scheme unless the particulates are large enough to block the views of the DIC camera. Moreover, in quasistatic material tests, the motion of the fluid surrounding the test specimen is not significant to warrant consideration.


M. A. Haile and P. G. Ifju : Application of Elastic Image Registration and Refraction Correction Table 1: Per cent strain errors of an underwater and reconstructed images 3. Strain level (%)

Underwater image (%)

Reconstructed image (%)

35

7

2

27

5

1

20

4

1

13

4

0.75

4.

5.

6. 7.

Table 1 summarises the percentage strain errors obtained from direct DIC correlation of underwater and reconstructed images for multiple strain levels. 8.

Conclusions This study has proposed an elastic image registration technique to undo the refraction-induced distortion through control point matching and spatial transformation. The image registration technique uses a LWM transform that relies on local polynomial functions at regions of the image. To obtain the transformation functions, refracted and unrefracted images of a template panel were compared and the mapping functions that achieved registration are retrieved. The transformation is then applied to the underwater images of a test specimen to correct the refraction distortion. Once specimen images are corrected, they are input into a DIC algorithm to obtain the displacement and strain field using the usual correlation technique. Strain and deformation measurements of a case study have shown a good agreement between the reconstructed and the reference DIC patterns of an underwater test sample. The elastic image registration technique presented here is an approximate method and as such the registered images will have certain mismatch error, nonetheless the errors incurred because of the elastic image registration process is often contained within the uncertainty bounds of most commercially available DIC techniques. The approach used in this study is different from previous works, such Ke et al. [11], in that it uses an elastic image registration scheme to undo the refraction distortion of a submerged image rather than developing a whole new calibration algorithm.

9.

10.

11.

12.

13.

14. 15.

16. 17. 18. 19.

REFERENCES 1. Kwon, Y.-H. and Casebolt, J. B. (2006) Effect of light refraction on the accuracy of camera calibration and reconstrcution in underwater motion analysis. Sports Biomech. 5, 315–340. 2. Sutton, M. A. and McFadden, C. (1999) Development of a methodology for non-contacting strain measurements in fluid


20. 21.

environments using computer vision. Opt. Lasers Eng. 32, 367– 377. Barta, A. and Horvath, G. (2003) Underwater binocular imaging of aerial objects versus the position of eyes relative to the flat water surface. J. Opt. Soc. Am. 20, 2370–2377. Tetlow, S. and Spours, J. (1999) Three-dimensional measurement of underwater work sites using structural laser light. Meas. Sci. Technol. 10, 1162. Lavest, J. M., Rives, G. and Lapreste, J. T. (2003) Dry camera calibration for underwater applications. Mach. Vis. Appl. 13, 245–253. Gennery, D. B. (2006) Generalized camera calibration including fish-eye lenses. Int. J. Comput. Vis. 68, 239–266. Abdel-Aziz, Y. I. and Karara, H. M. (1971) Direct linear transformation from comparator coordinates into object space coordinates in closed-range photogrammetry. Proceedings of the symposium on close-range photogrametry, 1–8. Schreier, H. W., Garcia, D. and Sutton, M. A. (2004) Advances in light microscope stereo vision. Exp. Mech. 44, 278–288. Sutton, M. A., McNeill, S. R., Jang, J. and Babai, M. (1988) Effects of Subpixel image restoration on digital correlation error estimation. Opt. Eng. 27, 870–877. Sutton, M. A., McNeill, S. R., Helm, J. D. and Chao, Y. J. (2000) Advances in two-dimensional and three-dimensional computer vision, Photomechanics. Topics Appl. Phys. 77, 323–373. Ke, X., Sutton, M. A., Lessner, S. M. and Yost, M. (2008) Robust stereo vision and calibration methodology for accurate three-dimensional digital image correlation measurement on submerged objects. J. Strain Anal. Eng. Design 43, 689–704. Kwon, Y.-H. (1999a) Object plane deformation due to refraction in two dimenstional underwater motion analysis. J. Appl. Biomech. 15, 396–403. Kwon, Y.-H. (1999b) A camera calibration algorithm for underwater motion analysis. Proceedings of the XVIIIth International symposium on biomechanics in sports, 257– 260. Goshtasby, A. (1988) Image registration by local approximation methods. Image Vis. Comput. 6, 255–261. Sdika, M. (2008) A fast nonrigid image registration with constraints on the Jacobian using large scale constrained optimization. IEEE Trans. Med. Imaging 27, 271–281. Davis, H. F. (1963) Fourier Series and Orthogonal Functions. Allyn and Bacon, Boston: 403 pp. Goshtasby, A. (1987) Piecewise cubic mapping functions for image registration. Pattern Recogn. 20, 525–533. Goshtasby, A. (1985) Template matching in rotated images. IEEE Trans. Pattern Anal. Mach. Intell. 7, 338–344. Haile, M. A., Yin, W. and Ifju, P. G. MATLAB based Image PreProcessing and Digital Image Correlation of Objects in Liquid. Proceedings of the SEM Annual Conference June 1–4, 2009. Albuquerque, NM, USA. Zitova, B. and Flusser, J. (2003) Image registration methods: a survey. Image Vis. Comput. 21, 977–1000. Sorzano, C. O. S., Thevenaz, P. and Unser, M. (2005) Elastic registration of biological images using vector-spline regularization. IEEE Trans. Biomed. Eng. 52, 652–663.

7