Improvement of Accuracy of the Membrane Shape

Improvement of Accuracy of the Membrane Shape Mapping of the Artificial Ventricle by Eliminating Optical Distortion Wojciech SULEJ

Krzysztof MURAWSKI

Tadeusz PUSTELNY

Institute of Teleinformatics and Automatics Military University of Technology Warsaw, Poland [email protected]

Institute of Teleinformatics and Automatics Military University of Technology Warsaw, Poland [email protected]

Department of Optoelectronics Silesian University of Technology Gliwice, Poland [email protected]

IEEE Member # 92707852 II.

Abstract—The paper presents new technique for correcting the desired shape of the flaccid membrane used in the artificial heart chamber. The original shape of membrane was obtained using specially developed type of method - Depth from Defocus. Accurately determining the shape of the diaphragm is very important. The shape of the membrane affects the final accuracy of determining the stroke volume of extracorporeal pneumatic heart assist pump. Three rigid membranes were used in the study. Each of them was developed on the basis of the original shape of the flaccid membrane and was examined.

The Depth From Defocus (DFD) method type is presented in [1 - 5] has been developed for visual distance measuring. It can also successfully be used for the construction of the sensor defining the shape of the flaccid membrane of the extracorporeal heart support pulse pump. It can take two forms of implementation. Until now the visualization of the shape of the membrane was performed only in the virtual world (augmented reality). Currently, works are carried out to lead to the determination of the shape of the membrane in the real world. Such an implemented measurement requires overcoming significant problems arising from image distortion of the membrane, which are formed in the optical path of the camera. Overcoming them will simplify the sensor calibration, it will eliminate errors of determining the shape of the membrane resulting from the process of its natural aging and long-term operation

Keywords—membrane shape mapping; optical distortion; artificial ventricle

I.

INTRODUCTION

The article presents the use of a new technique, the Depth From Defocus (DFD) presented in [1 - 5] to determine the shape of the flaccid membrane of the extracorporeal pneumatic heart assist pump model (Fig. 1). The work carried out in this area provides an opportunity to develop a visual method for determining the stroke volume of the extracorporeal pneumatic heart assist pump blood chamber. Studies of this type are carried out in the framework of the Polish Artificial Heart (PSS) [6 - 8]. Currently, despite the lack of satisfactory solutions these works are no longer continued excluding [9, 10]. The developed method also creates the possibility of a practical use of opaque biologically inert layers, which significantly reduces the risk of the formation of clots while maintaining the safe operation of the heart support pump.

III.

MESUREMENT METHOD

In this paper, the DFD method was used to determine the points of the measurement grid that describes the shape of the observed flaccid membrane. In the adopted implementation the position of the grid points is defined in the millimetric scale. As a function of assessing the correctness of the determined membrane shape, a numerical integration method was chosen. As a result, it was possible to compare the volume below the surface of the marked grid with the volume of the membrane shape, calculated in a formal method. The described volume in a wider sense is the same as the basic operation parameter of the heart support pump – the stroke volume. a)

Convex

b)

Flat

c)

Concave

Fig. 2. Models of membranes with the well-known math formulas.

Fig. 1. The cross-section of model of artificial ventricle.

978-1-5386-2402-9/17/$31.00 ©2017 IEEE

MOTIVATION

820

Fig. 5. Flat membrane shape mapping with optical distortion. Fig. 3. The laboratory stand for measuring the stroke volume of the artificial ventricle by membrane shape mapping.

with a fixed focal length f = 1.8 mm and a viewing angle of 126°. The camera is connected to a computer using a USB 2.0 port interface. The image resolution obtained from the cameras was reduced through the hardware (ROI) to the size of 900 px x 900 px. Sharpness is set for a flat membrane focusing on the center marker. This is completed with proprietary software, which enables measurement and enables real-time imaging.

Because it is assumed that the determination of stroke volume of the chamber should be with a measurement error of no more than 10% therefore it is extremely important to accurately copy the shape of the membrane. Due to the lack of a mathematical apparatus describing the shape of the flaccid membrane it was decided to replace the examined membrane shape with a rigid model (the equivalent of one) of a known mathematical description. As a result of this, it was possible to obtain the shape mapping of the membrane in the form of a measuring grid and determining for each point of the grid a default value. With these two values the measurement errors could be determined and their causes analyzed. Three reference membrane shapes were selected for testing (Fig. 2): convex, concave and flat. Membranes were designed in CAD software. Then they were 3D printed with an accuracy of 0.001 mm on the X and Y axes with a layer thickness on the Z-axis of 0.09 mm. On the surfaces of the membranes 49 round, white, markers having a diameter of 3 mm were arranged. The study assumed an even distribution of markers forming squares. The distance between the centers of the neighboring markers was 7.7 mm. Markers were placed starting from the central marker. For the study, a laboratory station was designed and built. Different parts of the station were printed on a 3D printer. After assembly a stable structure was created eliminating random movements and changes in camera viewing angles with respect to the membrane in each of the spatial dimensions (Fig. 3). The station allows for quick and easy replacement of the tested membranes. It is equipped with a miniature monochrome XIMEA camera model MU9PM-MH with a lens

IV.

RESULTS WITH OPTICAL DISTORTION

While using the DFD method, without introducing any amendments, for determining the shape, an inaccuracy of the shape mapping of the tested reference membranes was obtained, Fig. 4 - Fig. 6. In the case of the convex membrane (Fig. 4), the shape of the projection is subject to the smallest error in the vicinity of the central marker. The biggest distortion was obtained for markers which position was close to the edge of the membrane. It is noted here that markers are located below the outline of a circular membrane. For the considered membrane shape the determined stroke volume should equal 0 ml. In practice, it is much higher. In the case of flat membranes (Fig. 5) it can be recognized that the diaphragm shape has been modeled correctly, though, the values are arranged below the real level of the membrane outline. Hence the membrane gives the impression of a concave shape. For such a membrane shape, the determined stroke volume should equal 35 ml. The value obtained from the measurement is greater by 8 ml. With a concave membrane (Fig. 6), the membrane assumes a shape resembling a more rectangular than spherical cap. As a result, the determined stroke volume significantly exceeds the expected value of 70 ml.

Fig. 4. Convex membrane shape mapping with optical distortion.

Fig. 6. Concave membrane shape mapping with optical distortion.

821

a) Stand with chess pattern

b) Detected and reprojected points

Fig. 7. Process of eliminating optical distortion and camera calibration.

V.

CLASSICAL APPROACH

Fig. 9. Convex membrane shape mapping after elimination optical distortion.

During analysis of the data it was noticed that the major problems of mapping the membrane result from the phenomenon of optical distortion of the image. The distortion phenomena could not be avoided due to the need to use the camera lens of a small focal length and a large viewing angle. It is necessary due to the short distance from the front of the camera lens to the front surface of the membrane, which is approx. 20 mm and a membrane diameter of 70 mm. The classic approach used to eliminate optical distortion uses the Brown-Conrady model [11 - 13]. According to this model, for each pixel of the image with the x and y coordinates have new values calculated xu and yu on the basis of equations: xu = x(1 + k1r2 + k2r4 + k3r6) + [2p1xy + p2(r2 + 2x2)] 2

4

6

2

2

yu = y(1 + k1r + k2r + k3r ) + [ p1(r + 2y ) + 2p2xy]

a) With optical distortion

Fig. 10. View from camera of flat membrane.

(1) (2)

As can be seen the calculation of the value of xu and yu requires knowledge of the values of coefficients k1, k2, k3, as well as p1 and p2. They are determined by placing them in the field of view of the camera a chess pattern with known area sizes (Fig. 7a). Afterwards, a few to several images have to be acquired, changing the position of the pattern between recorded images. For the acquired images characteristic points are determined and new mapping is calculated (Fig. 7b). As a result, it is possible to calculate the desirable parameters of equations (1) and (2). These parameters for the given optical system are determined only once. The described mechanism of eliminating optical distortion is available in the library of OpenCV and in Matlab. The chess pattern was used in the study with a width and height of the field equal to 10 mm. During calibration, it was placed in the position of the membrane. In the study, six different images of the pattern were acquired. On their basis, the following coefficients were determined: k1 = -4.5644, k2 = 8.8746, k3 = -5.7745, p1 = 0.0015 and p2 = 0.0018. The calculated values of the coefficients are saved in the software. Eliminating distortion was done with a function from the OpenCV library. a) With optical distortion

b) Without optical distortion

Fig. 11. Flat membrane shape mapping after elimination optical distortion. a) With optical distortion


Fig. 12. View from camera of concave membrane.


Fig. 13. Concave membrane shape mapping after elimination optical distortion.

Fig. 8. View from camera of convex membrane.

822

VI.

RESULTS AFTER ELIMINATION OPTICAL DISTORTION

Removal of the optical distortion improved the shape of the generated grids describing the membrane (Fig. 8 – Fig. 13). For the convex membrane (Fig. 8 and Fig. 9) the elimination of optical distortion resulted in a slightly improved diaphragm shape. The number of markers located below the level of the circular outline fell to eight. The stroke volume determined earlier also decreased by approx. 3 ml. During the testing of the flat membrane (Fig. 10 and Fig. 11) the elimination of the optical distortion significantly improved the resulting grid. The calculated stroke volume also improved significantly. The difference between the reference value (35 ml) was only 2.5 ml. The best results were obtained for a concave membrane (Fig. 12 and Fig. 13). This is apparent in both the image from the camera and on a dedicated membrane view, which takes on the correct shape – the spherical cap. The determined stroke volume has improved, which only differed from the correct value (70 ml) by approx. 8 ml.

Fig. 15. Chart of compensation coefficients.

Elimination of measurement error over the entire image area required the determination of the compensation coefficients for each possible position of the marker. The studies takes into account the movement of the marker in the Z-axis and the coordinates X (width) and Y (height) of the image. Having determined 49 markers, 49 compensation coefficients were obtained. Other coefficients were determined using interpolation. For this purpose a function of triangulationbased cubic interpolation of 2D scattered data was implemented in Matlab. Data of all markers were transmitted to the functions. This resulted in the formation of a measuring grid, which determines the correction result for each possible position of the marker (Fig. 15). The values of the coefficients were written in the form of a matrix with a dimension of 900 x 900. The shape of the resulting measurement grid resembles an elliptic paraboloid, which was confirmed by using the curve fitting mechanism available in Matlab. Then, using the data stored in the matrix of correction factors the equations describing the surface were determined. For this purpose, data placed in the middle row and middle column was selected from the matrix. As a result, a curve representing the values of correction factors was obtained for coordinates X

VII. INVENTED APPROACH Using the classical approach to eliminate distortion improved the obtained results, but not to the extent expected by the authors. So an attempt to develop another solution was put forth. The new approach to compensate for optical distortion of the image of the membrane is based on the properties of the used method of measurement. By observing the back of the silicon membrane with the recognizable markers it was noted that, in the absence of any observed distortion the markers should have the same surface as the central marker, indicated by number 25 on Fig. 14. The measurement shows that the surface area equals 1827 px, while the other markers had a smaller area. A marker bearing the number 6 stands out among the rest that cover an area of only 941 pixels. This is due to the optical distortion that causes objects away from the camera’s optical axis to tend to be smaller. In the case of marker number 6, this difference is as high as 886 px. It was therefore concluded that if the surface marker is almost half, then its value should be corrected with the obtained measurement result.

Fig. 14. View from camera of upper left quarter of membrane with recognized position and surface of markers.

Fig. 16. Parabola compensation for X.

823

Fig. 19. Flat membrane shape mapping after invented compensation.

Fig. 17. Parabola compensation for Y.

(Fig. 16) and Y (Fig. 17) of the marker position. On their basis, approximation was made and the following equation was determined: fx(x) = 0.0063x2 - 5.7101x + 1334.8284

(3)

fy(x) = 0.0067x2 - 6.0701x + 1409.4164

(4)

Knowing equations (3) and (4) the values of correction factors are calculated automatically for each marker position. This approach increases the speed of measurement providing, in the considered case, the calculation in real time (with a frequency f ≥ 10 Hz). The size of the surface marker after compensation is calculated from expression (5). Zc = Z + fx(X) + fy(Y)

Fig. 20. Concave membrane shape mapping after invented compensation.

For the flat membrane (Fig. 19) using the invented compensation equated the reference membrane shape with the reference shape. The calculated volume in relation to the reference value of 35 ml is practically devoid of error.

(5)

VIII. RESULTS AFTER INVENTED COMPENSATION

For the concave membrane (Fig. 20), the results are comparable to the approach described in point V. The mapping of the membrane assumes the correct shape of a spherical cap and the determined volume differs from the assumed 70 ml by about 11 ml.

Introducing invented compensation yielded even better mapping results of the membrane shape. Improved determined volumes were also achieved. The result of the calculations is shown in Fig. 18 – Fig. 20. In the case with a convex membrane (Fig. 18) invented compensation considerably improved the mapping of the membrane shape. The achieved shape, almost on the entire surface, was comparable to that of the standard membrane. The determined volume, which should equal 0 ml, is greater only by about 10 ml.

IX. SUMMARY Results of the volume measurements together with the calculated errors have been collected in tables I – III. Undoubtedly, the best results were achieved using an invented approach. Only for the concave membrane a better accuracy was obtained when compared with using a classical approach. It is also worth noting that the absolute error of volume determination at the upper and lower extremities of the membrane using the invented approach is almost identical. TABLE I.

SUMMARY FOR CONVEX MEMBRANE

Specification

Fig. 18. Convex membrane shape mapping after invented compensation.

824

Ve = 0 ml V [ml]

ΔV = Ve - V [ml]

ΔV [%]

With optical distortion

21.165

-21.165

∞

After elimination of optical distortion

18.330

-18.331

∞

After invented compensation

10.032

-10.032

∞

TABLE II.

SUMMARY FOR FLAT MEMBRANE

REFERENCES [1]

Ve = 35 ml

Specification

V [ml]

ΔV = Ve - V [ml]

ΔV [%]


43.376

-8.376

23.93


37.663

-2.663

7.61


34.483

0.516

1.47

[2]

[3]

[4]

X.

CONCLUSIONS

The paper presents a method of improving the accuracy of defining the shape of a flaccid membrane by eliminating optical distortion. The study used a flaccid membrane produced for the pneumatic extracorporeal pump model of the heart pump (Figure 1). In order to verify the results, rigid equivalents, models, flattened membranes were used. Two cases when using a flaccid membrane have been selected for the study, which are well-defined mathematically: convex and concave. Other states of the membrane, as opposed to the elastic membrane, have no mathematical description. The study was also conducted for a flat membrane, which shape cannot be obtained using a flaccid membrane. However, the use of a flat membrane is advantageous because of the evaluation of the results obtained. The flat membrane has also contributed to the development of the original technique of eliminating optical distortion. Evaluating the results was done using the numerical integration method. As a result, the volumes of the analyzed membranes were determined. This approach provided the opportunity to compare the results of the tests with the volumes determined in an analytical way. The analysis of the results in tables I – III shows that better results have been obtained after using the developed method of eliminating distortion. It follows that the elimination of distortion has allowed the determination of the grid, which shape is closer to the shape of the actual flaccid membrane. TABLE III.

[5]

[6]

[7]

[8]

[9]

[10]

[11] [12]

SUMMARY FOR CONCAVE MEMBRANE

Specification

[13]

Ve = 70 ml V [ml]

ΔV = Ve - V [ml]

ΔV [%]


80.272

-10.273

14.68


62.012

7.987

11.41


58.907

11.092

15.85

825

K. Murawski, “Method of measuring the distance using one camera”, Patent Application: P.408076, 2014. (in Polish). K. Murawski, M. Murawska, T. Pustelny, “The system and method of determining the shape of the membrane pneumatic pump of extracorporeal heart assist device”, Patent Application: nr P.414104, 2015. (in Polish). K. Murawski, “Measurement of membrane displacement using a motionless camera”, Acta Phys. Pol. A, 128, 1, 2015, 10 – 14. DOI: 10.12693/APhysPolA.128.10 K. Murawski, “Measurement of membrane displacement with a motionless camera equipped with a fixed focus lens”, Metrology and Measurement Systems, 22, 1, 2015, 69 – 78. DOI: 10.1515/mms-20150011 K. Murawski, A. Arciuch, T. Pustelny, “Studying the influence of object size on the range of distance measurement in the new Depth From Defocus method”, 2016 Federated Conference on Computer Science and Information Systems (FedCSIS), Gdansk, 2016, pp. 817822. DOI: 10.15439/2016F136 J. Sarna, R. Kustosz, E. Woźniewska, M. Gonsior, A. Jarosz, K. Szymańska, D. Hansel, E. Krzak, Program „Polskie Sztuczne Serce” Sojusz Medycyny, Nauki i Techniki, ISBN 978-83-63310-16-5, 2013, (in Polish). P. Gibinski, G. Konieczny, E. Maciak, Z. Opilski, T. Pustelny, “Acoustic device for measuring instantaneous blood volume in cardiac support chamber i.e. pneumatic heart assist driving chamber, has sensor supporting heart in openings, and audio amplifier connected with volume unit of blood-cell support”, Patent Number(s): PL394074A1, 2011. G. Konieczny, T. Pustelny, P. Marczyński, “Optical sensor for measurements of the blood chamber volume in the POLVAD Prosthesis - static measurements”, Acta Phys. Pol. A, 124, 3, 2013, 479 – 482. DOI: 10.12693/APhysPolA.124.479 L. Grad, K. Murawski, T. Pustelny, “Measuring the stroke volume of the pneumatic heart prosthesis using an artificial neural network”, Proc. SPIE 10034, 11th Conference on Integrated Optics: Sensors, Sensing Structures, and Methods, (2016; DOI: 10.1117/12.2243952 K. Murawski, T. Pustelny, L. Grad, M. Murawska, “Estimation of the blood volume in pneumatically controlled ventricular assist device by vision sensor and image processing technique”, Proc. 21st International Conference on Methods and Models in Automation and Robotics (MMAR), 2016; DOI: 10.1109/MMAR.2016.7575115 D. C. Brown, “Decentering distortion of lenses”, Photogrammetric Engineering, 32 (3), 1966, 444–462. A. E. Conrady, “Decentred Lens-Systems”, Monthly notices of the Royal Astronomical Society, 79, 1919, 384–390. D. C. Brown, “Close-Range Camera Calibration”, Photogrammetric Engineering, 37 (8), 1971, 855-866.