Skip to content

face_tracker

face_tracker

LucasKanadeTracker

The LucasKanadeTracker is a Python class that implements the Lucas-Kanade optical flow algorithm to track a face in a video stream. It provides methods to track a face given an initial bounding box, and to transform the coordinates of a point in the previous frame to the current frame.

Class Members

transformation: a 2D affine transformation matrix that represents the motion of the face from the previous frame to the current frame MAX_CORNERS: the maximum number of corners to track in the face MIN_CORNERS: the minimum number of corners required to compute a 2D affine transformation QUALITY_LEVEL: the quality level used to select the corners to track MIN_DISTANCE: the minimum distance between the corners to track

Methods

transform_coords(x, y): takes in two integers x and y as input and returns their transformed coordinates after applying the current transformation matrix.

track_face(image, last_image, box): takes in two images (image and last_image) and a bounding box represented by a tuple (x, y, w, h) and returns a new bounding box that represents the face location in the current frame. If the face cannot be tracked, the method returns None. The steps performed by the method are:

* Convert both images to grayscale and equalize their histograms
* Create a binary mask to track the face within the bounding box
* Track the corners of the face using the Lucas-Kanade optical flow algorithm
* Filter the tracked corners to keep only those that are consistent with the optical flow computed in the opposite direction
* Compute the 2D affine transformation matrix that represents the motion of the face
* Transform the bounding box to the new location
* Update the transformation matrix with the computed affine transformation
* Return the transformed bounding box
Source code in redesign_pipeline/face_detectors/face_tracker.py 
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
class LucasKanadeTracker():
    """
    The LucasKanadeTracker is a Python class that implements the 
    Lucas-Kanade optical flow algorithm to track a face in a video stream. 
    It provides methods to track a face given an initial bounding box, and to transform 
    the coordinates of a point in the previous frame to the current frame.

    Class Members
    -------------

    transformation: a 2D affine transformation matrix that represents the motion of the face from the previous frame to the current frame
    MAX_CORNERS: the maximum number of corners to track in the face
    MIN_CORNERS: the minimum number of corners required to compute a 2D affine transformation
    QUALITY_LEVEL: the quality level used to select the corners to track
    MIN_DISTANCE: the minimum distance between the corners to track

    Methods
    -------

    transform_coords(x, y): takes in two integers x and y as input and returns their 
    transformed coordinates after applying the current transformation matrix.

    track_face(image, last_image, box): takes in two images (image and last_image) 
    and a bounding box represented by a tuple (x, y, w, h) and returns a new bounding 
    box that represents the face location in the current frame. If the face cannot be 
    tracked, the method returns None. The steps performed by the method are:

        * Convert both images to grayscale and equalize their histograms
        * Create a binary mask to track the face within the bounding box
        * Track the corners of the face using the Lucas-Kanade optical flow algorithm
        * Filter the tracked corners to keep only those that are consistent with the optical flow computed in the opposite direction
        * Compute the 2D affine transformation matrix that represents the motion of the face
        * Transform the bounding box to the new location
        * Update the transformation matrix with the computed affine transformation
        * Return the transformed bounding box
    """

    def __init__(self):
        """
        Initializes the LucasKanadeTracker object with a transformation attribute set to None.
        """
        self.transformation = None

    def transform_coords(self,x,y):
        """
        Transforms the given x and y coordinates using the transformation matrix.

        Parameters
        ----------
        x : float
            The x coordinate to be transformed.
        y : float
            The y coordinate to be transformed.

        Returns
        -------
        tuple of floats
            The transformed (x,y) coordinates.
        """

        if self.transformation is not None:
            x, y = cv2.transform(np.array([[[x, y]]]), self.transformation).squeeze()

        return x,y

    def track_face(self,image,last_image,box):
        """
        Tracks a face from the given image using the Lucas Kanade tracker algorithm and returns the transformed bounding box coordinates.

        Parameters
        ----------
        image : numpy.ndarray
            The current frame of the video containing the face to be tracked.
        last_image : numpy.ndarray
            The previous frame of the video containing the face to be tracked.
        box : tuple of ints
            A tuple representing the bounding box (x,y,w,h) of the face to be tracked.

        Returns
        -------
        tuple of floats or None
            If successful, the transformed bounding box coordinates in the form of (x,y,w,h). If tracking fails, returns None.
        """

        frame_gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
        frame_gray = cv2.equalizeHist(frame_gray)

        last_frame_gray = cv2.cvtColor(last_image, cv2.COLOR_BGR2GRAY)
        last_frame_gray = cv2.equalizeHist(last_frame_gray)

        trackingRegion = np.zeros((frame_gray.shape[0], frame_gray.shape[1]), np.uint8)

        points = np.array([[box[0] + 0.22 * box[2], box[1] + 0.21 * box[3]], [box[0] + 0.78 * box[2], box[1] + 0.21 * box[3]], 
                        [box[0] + 0.70 * box[2], box[1] + 0.65 * box[3]], [box[0] + 0.30 * box[2], box[1] + 0.65 * box[3]]], dtype=np.int32)

        trackingRegion = cv2.fillPoly(trackingRegion, [points], (255, 255, 255))
        corners = cv2.goodFeaturesToTrack(image=frame_gray, maxCorners=MAX_CORNERS, qualityLevel=QUALITY_LEVEL,
                                                minDistance=MIN_DISTANCE, mask=trackingRegion, blockSize=3, gradientSize=3, useHarrisDetector=False, k=0.04)
        if corners is None:
            corners = []

        corners_1, cornersFound_1, err = cv2.calcOpticalFlowPyrLK(
            last_frame_gray, frame_gray, np.float32(corners), None)
        corners_0, cornersFound_0, err = cv2.calcOpticalFlowPyrLK(
            frame_gray, last_frame_gray, corners_1, None)

        corners_1v = []
        corners_0v = []

        for j in range(len(corners)):
            if cornersFound_1[j] and cornersFound_0[j] and cv2.norm(corners[j]-corners_0[j]) < 2:
                corners_0v.append(corners_0[j])
                corners_1v.append(corners_1[j])
        if len(corners_1v) >= MIN_CORNERS:
            corners = corners_1v
            transformation, inliers = cv2.estimateAffinePartial2D(
                np.float32(corners_0v), np.float32(corners_1v))

            if transformation.size > 0:
                box_coords = np.array([[[box[0], box[1]]], [[box[0] + box[2], box[1] + box[3]]]], dtype=np.float32)
                transformedBoxCoords = cv2.transform(box_coords, transformation)
                transformedBox = (transformedBoxCoords[0][0][0], transformedBoxCoords[0][0][1], transformedBoxCoords[1][0]
                                  [0] - transformedBoxCoords[0][0][0], transformedBoxCoords[1][0][1] - transformedBoxCoords[0][0][1])
                self.transformation = transformation
                return transformedBox

            self.transformation = None

        return None

__init__()

Initializes the LucasKanadeTracker object with a transformation attribute set to None.

Source code in redesign_pipeline/face_detectors/face_tracker.py 
46
47
48
49
50
def __init__(self):
    """
    Initializes the LucasKanadeTracker object with a transformation attribute set to None.
    """
    self.transformation = None

track_face(image, last_image, box)

Tracks a face from the given image using the Lucas Kanade tracker algorithm and returns the transformed bounding box coordinates.

Parameters
numpy.ndarray

The current frame of the video containing the face to be tracked.

numpy.ndarray

The previous frame of the video containing the face to be tracked.

tuple of ints

A tuple representing the bounding box (x,y,w,h) of the face to be tracked.

Returns

tuple of floats or None If successful, the transformed bounding box coordinates in the form of (x,y,w,h). If tracking fails, returns None.

Source code in redesign_pipeline/face_detectors/face_tracker.py 
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
def track_face(self,image,last_image,box):
    """
    Tracks a face from the given image using the Lucas Kanade tracker algorithm and returns the transformed bounding box coordinates.

    Parameters
    ----------
    image : numpy.ndarray
        The current frame of the video containing the face to be tracked.
    last_image : numpy.ndarray
        The previous frame of the video containing the face to be tracked.
    box : tuple of ints
        A tuple representing the bounding box (x,y,w,h) of the face to be tracked.

    Returns
    -------
    tuple of floats or None
        If successful, the transformed bounding box coordinates in the form of (x,y,w,h). If tracking fails, returns None.
    """

    frame_gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
    frame_gray = cv2.equalizeHist(frame_gray)

    last_frame_gray = cv2.cvtColor(last_image, cv2.COLOR_BGR2GRAY)
    last_frame_gray = cv2.equalizeHist(last_frame_gray)

    trackingRegion = np.zeros((frame_gray.shape[0], frame_gray.shape[1]), np.uint8)

    points = np.array([[box[0] + 0.22 * box[2], box[1] + 0.21 * box[3]], [box[0] + 0.78 * box[2], box[1] + 0.21 * box[3]], 
                    [box[0] + 0.70 * box[2], box[1] + 0.65 * box[3]], [box[0] + 0.30 * box[2], box[1] + 0.65 * box[3]]], dtype=np.int32)

    trackingRegion = cv2.fillPoly(trackingRegion, [points], (255, 255, 255))
    corners = cv2.goodFeaturesToTrack(image=frame_gray, maxCorners=MAX_CORNERS, qualityLevel=QUALITY_LEVEL,
                                            minDistance=MIN_DISTANCE, mask=trackingRegion, blockSize=3, gradientSize=3, useHarrisDetector=False, k=0.04)
    if corners is None:
        corners = []

    corners_1, cornersFound_1, err = cv2.calcOpticalFlowPyrLK(
        last_frame_gray, frame_gray, np.float32(corners), None)
    corners_0, cornersFound_0, err = cv2.calcOpticalFlowPyrLK(
        frame_gray, last_frame_gray, corners_1, None)

    corners_1v = []
    corners_0v = []

    for j in range(len(corners)):
        if cornersFound_1[j] and cornersFound_0[j] and cv2.norm(corners[j]-corners_0[j]) < 2:
            corners_0v.append(corners_0[j])
            corners_1v.append(corners_1[j])
    if len(corners_1v) >= MIN_CORNERS:
        corners = corners_1v
        transformation, inliers = cv2.estimateAffinePartial2D(
            np.float32(corners_0v), np.float32(corners_1v))

        if transformation.size > 0:
            box_coords = np.array([[[box[0], box[1]]], [[box[0] + box[2], box[1] + box[3]]]], dtype=np.float32)
            transformedBoxCoords = cv2.transform(box_coords, transformation)
            transformedBox = (transformedBoxCoords[0][0][0], transformedBoxCoords[0][0][1], transformedBoxCoords[1][0]
                              [0] - transformedBoxCoords[0][0][0], transformedBoxCoords[1][0][1] - transformedBoxCoords[0][0][1])
            self.transformation = transformation
            return transformedBox

        self.transformation = None

    return None

transform_coords(x, y)

Transforms the given x and y coordinates using the transformation matrix.

Parameters
float

The x coordinate to be transformed.

float

The y coordinate to be transformed.

Returns

tuple of floats The transformed (x,y) coordinates.

Source code in redesign_pipeline/face_detectors/face_tracker.py 
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
def transform_coords(self,x,y):
    """
    Transforms the given x and y coordinates using the transformation matrix.

    Parameters
    ----------
    x : float
        The x coordinate to be transformed.
    y : float
        The y coordinate to be transformed.

    Returns
    -------
    tuple of floats
        The transformed (x,y) coordinates.
    """

    if self.transformation is not None:
        x, y = cv2.transform(np.array([[[x, y]]]), self.transformation).squeeze()

    return x,y