Camera calibration is required to correct the image due to the lens distortion and to map the 3D positions and orientations in the world corresponding to the 2D image.
The 3D point of an object in the world coordinate is specified as $(X_w, Y_w, Z_w)$. Its corresponding projection in a 2D image is specified as $(u, v)$. The equation that relates these two coordinates is:
where P is a 3x4 projection matrix consisting of two parts: a 3x3 Intrinsic matrix $K$ and a 3x4 Extrinsic matrix $E$.
The action of the camera calibration is to determine a camera's intrinsic and extrinsic parameters.
The intrinsic parameters include the focal length in the x and y directions and the principal point which is the center of an image.
The intrinsic matrix:
\begin{equation*}
K=
\left [
\begin{matrix}
f_x & \gamma & c_x \\
0 & f_y & c_y \\
0 & 0 & 1
\end{matrix}
\right ]
\end{equation*}
where $f_x$ and $f_y$ are the focal length in the x and y direstions. They are usually the same.
$c_x$ and $c_y$ are the coordinate of the optical center. They are usually the same as the image center.
$\gamma$ is the skew between the x and y axes. The skew is usually $0$.
The extrinsic matrix consistes of a rotation matrix $R$ and a translation matrix $T$:
These extrinsic paramaters describe the camera's position and orientation in the world, including:
Rotation $R$: The angle at which the camera is tilted with respect to the world cooridnate.
Translation $T$: The distance the camera is from the origin of the world coordinate.
(1) Chessboard pattern:
In this project, the chessboard with 5 rows and 8 columns of black and white squares is chosen. The chessboard sheet will be attached to a stationary wall.
(2) Camera:
This project calibrates the 5MP USB camera with with OV5640 Sensor ELP-USB500W02M-L21. The camera software on a laptop can directly connect with the USB camera.
(3) Camera Tripod:
The camera is fastened to a camera tripod. The camera can be adjusted to have 24 different orientations and positions.
(1) The chessboard pattern does not need to have dimensions in high precision. However, make sure the corners joining black and white squares are orthogonal to each other with relatively good contrast and the square size stays the same.
(2) The orientation of the chessboard sheet does not need to be precisely positioned. However, it is typical to set it to align with the numbers of rows and columns, defined in the calibration program. For example, if the calibration program is set up to have inner corners arranged in (4 rows, 7 columns), then the chessboard pattern shall have the black and white squares arranged in the orientation of 5 rows and 8 columns.
Arrange the tripod in front of the chessboard sheet. Adjust the USB camera focus by turning the ring holding the camera lens until the image has relatively good contrast.
The image below shows that the chessboard pattern in the image was distorted by the camera's features.
It is required to take at least 10 images for the camera calibration program to analyze. In this project, 24 images were taken. Please note that
It is preferred to have the chessboard pattern occupies at least 33 % of the image because only the chessboard pattern area will be calibrated. When the chessboard pattern is unwarped, the area surrounding the chessboard will become warped and unusable for the image processing applications.
Make sure there is soft and uniform lighting in the room such that the chessboard pattern has approximately equal brightness across the entire pattern.
(a) cv2.imread(image): Read the test image.
(b) cv2.cvtColor(img,cv2.COLOR_BGR2GRAY): Convert the image to grayscale.
(c) cv2.threshold(gray, thresh, maxValue, cv2.THRESH_BINARY):
Update the image to maximize the intensity of the white square; maxValue=255. The pixel intensity greater than the threshold value will be set to the maxValue. This will expedite the process of finding the square junction corners. It relaxes the criteria for test images.
(d) cv2.findChessboardCorners(dst, (innerCornerRows, innerCornerColumns),
cv2.CALIB_CB_ADAPTIVE_THRESH + cv2.CALIB_CB_FAST_CHECK + cv2.CALIB_CB_NORMALIZE_IMAGE):
Only if this function can successfully find the inner corners of the chessboard, then these identified image points are qualified to be collected for the camera calibration.
(e) cv2.cornerSubPix(image, corners, winSize, zeroZone, criteria)
Refine pixel coordinates for given 2d points. Take the coners data from the function findChessboardCorners and output the refined corners as the image points to work with the corresponding object points. The pair of image points and the object points will be the input for the camera calbration.
The winSize is one half of the side length of the search window. In this project, it is set to be (11,11). That means the search window size is (23, 23).
The zeroZone is one half of the size of the dead region in the middle of the search zone. In this project, it is set to be (-1,-1) to indicate that there is no need for such a zeroZone.
The criteria=(cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001) where sets 30 analysis iterations and 0.001 pixel tolerance.
The object points are simply the coordinate arrays in the world coordinate. For example, the (4,7) inner corner matrix will have the object points:
[0., 0., 0.], [1., 0., 0.], [2., 0., 0.], [3., 0., 0.], [0., 1., 0.], [1., 1., 0.], [2., 1., 0.], [3., 1., 0.],..., etc.
Reference:
Use cv2.drawChessboardCorners(img, (innerCornerRows, innerCornerColumns), refinedCorners, ret=True) for the visualization of each test sample.
Use cv2.calibrateCamera(objpoints, imgpoints, gray.shape[::-1], None, None) to obtain the key matrix of the camera.
The objpoints and imgpoints were collected during the procedure of finding the inner corners of each test image.
For example:
Camera matrix:
[[1.49588176e+03 0.00000000e+00 8.49679636e+02]
[0.00000000e+00 1.51614226e+03 5.78382106e+02]
[0.00000000e+00 0.00000000e+00 1.00000000e+00]]
DistortionCoefficient:
[[-0.40135481 0.21923169 0.01450131 0.00866517 -0.14121383]]
Reference:
https://docs.opencv.org/4.x/dc/dbb/tutorial_py_calibration.html
(a) Use Camera matrix, distortion coefficient, and image size as the inputs to
cv2.getOptimalNewCameraMatrix(CameraMatrix, DistortionCoefficient, (width,height), 1, (width,height)).
The outputs are NewCameraMatrix and RegionofInterest. The region of interest indicates the center of the chessboard pattern and its size.
(b) Use cv2.undistort(OriginalImage, CameraMatrix, DistortionCoefficient, None, NewCameraMatrix) to obtain the unwarped image.
Apply the RegionofInterest to crop the unwarped image to keep only the portion which corresponds to the original image.
x, y, w, h = roi
croppedImg =unwarpedImg[y:y+h, x:x+w]
Reference:
