Skip to content

blazeface_facemesh

blazeface_facemesh

BlazefaceFacemesh

Bases: FaceDetector

This class implements a face detection and landmark extraction algorithm based on the BlazeFace model. It extends the FaceDetector class and provides the functionality to detect and draw facial landmarks in images.

Methods

init

The constructor of the class takes an optional boolean argument tracking. It initializes the parent class FaceDetector and sets up the face mesh classifier using the face_mesh solution from the mp module. If tracking is set to True, the classifier is set up to track faces in video frames. If tracking is False, the classifier is set up to detect faces in still images. The class also sets up several lists of facial landmarks and two lists to store detection and tracking times.

fs_to_landmark_list

This method takes a list of facial landmarks fs and returns a list of landmarks with duplicates removed.

get_facemesh_coords

This method takes a set of face detection results results, a list of facial landmarks landmark_list, and an image img. It returns the x and y coordinates of the landmarks as a list of tuples. The coordinates are scaled to the size of the image.

detect_faces

This method takes an image as input and returns a set of face detection results. It first converts the image from BGR to RGB and then runs the face mesh classifier on the image. The detection time is recorded and stored in the det_times list.

coords_for_landmark

This method takes a set of face detection results results, a list of facial landmarks lm, and an image image. It returns the x and y coordinates of the landmarks sorted in a counterclockwise order.

rgb_means

This method takes a set of facial landmark coordinates coords and an image image. It returns the mean RGB values of the pixels within the region defined by the landmark coordinates.

draw_landmarks

This method takes a set of face detection results results, a list of facial landmarks lm_list, and an image image. It draws the facial landmarks on the image and returns the result.

Source code in redesign_pipeline/face_detectors/blazeface_facemesh.py 
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
class BlazefaceFacemesh(FaceDetector):
    """
    This class implements a face detection and landmark extraction algorithm based on the BlazeFace model. It extends the FaceDetector class and provides the functionality to detect and draw facial landmarks in images.

    Methods
    -------

    __init__

    The constructor of the class takes an optional boolean argument tracking. It initializes the parent class FaceDetector and sets up the face mesh classifier using the face_mesh solution from the mp module. If tracking is set to True, the classifier is set up to track faces in video frames. If tracking is False, the classifier is set up to detect faces in still images. The class also sets up several lists of facial landmarks and two lists to store detection and tracking times.


    fs_to_landmark_list

    This method takes a list of facial landmarks fs and returns a list of landmarks with duplicates removed.


    get_facemesh_coords

    This method takes a set of face detection results results, a list of facial landmarks landmark_list, and an image img. It returns the x and y coordinates of the landmarks as a list of tuples. The coordinates are scaled to the size of the image.


    detect_faces

    This method takes an image as input and returns a set of face detection results. It first converts the image from BGR to RGB and then runs the face mesh classifier on the image. The detection time is recorded and stored in the det_times list.


    coords_for_landmark

    This method takes a set of face detection results results, a list of facial landmarks lm, and an image image. It returns the x and y coordinates of the landmarks sorted in a counterclockwise order.


    rgb_means

    This method takes a set of facial landmark coordinates coords and an image image. It returns the mean RGB values of the pixels within the region defined by the landmark coordinates.


    draw_landmarks

    This method takes a set of face detection results results, a list of facial landmarks lm_list, and an image image. It draws the facial landmarks on the image and returns the result.

    """

    def __init__(self,tracking=False):
        """
        Initializes a new instance of the BlazefaceFacemesh class.

        Parameters:
        -----------
        tracking: bool
            A boolean value indicating whether to enable tracking (default: False).
        """
        super().__init__()
        self.mpFaceMesh = mp.solutions.face_mesh

        if tracking:
            self._classifier = self.mpFaceMesh.FaceMesh(max_num_faces=1,
                                            refine_landmarks=True,
                                            min_detection_confidence=0.5,
                                            min_tracking_confidence=0.5)
        else:
            self._classifier = self.mpFaceMesh.FaceMesh(max_num_faces=1,
                                            refine_landmarks=True,
                                            min_detection_confidence=0.5,
                                            static_image_mode=True
                                            )

        self.lm_big_forehead = self.fs_to_landmark_list(FACEMESH_BIG_FOREHEAD)
        self.lm_small_forehead = self.fs_to_landmark_list(FACEMESH_SMALL_FOREHEAD)
        self.lm_cheeks_and_nose = self.fs_to_landmark_list(FACEMESH_CHEEKS_AND_NOSE)
        self.lm_left_cheek = self.fs_to_landmark_list(FACEMESH_LEFT_CHEEK)
        self.lm_right_cheek = self.fs_to_landmark_list(FACEMESH_RIGHT_CHEEK)
        self.lm_crop_face = self.fs_to_landmark_list(FACEMESH_CROP_FACE)
        self.det_times = []
        self.track_times = []

    def fs_to_landmark_list(self, fs: List[List[int]]) -> List[int]:
        """
        Converts a facemesh landmark list from a list of pairs to a list of integers.

        Args
        ----
        fs: (List[List[int]]): A facemesh landmark list.

        Returns
        -------
        A list of integers representing the facemesh landmarks.
        """
        lst = []
        for pair in fs:
            for n in pair:
                if n not in lst:
                    lst.append(n)
        return lst

    def get_facemesh_coords(self, results, landmark_list: List[int], img: np.ndarray) -> np.ndarray:
        """
        Returns the x, y coordinates of the facemesh landmarks.

        Args:
        - results (any): The results of the facemesh detection.
        - landmark_list (List[int]): A list of integers representing the facemesh landmarks.
        - img (np.ndarray): An image.

        Returns:
        - An array of x, y coordinates of the facemesh landmarks.
        """
        h,w = img.shape[:2]
        xyz = []
        for num in landmark_list:
            lm = results[0].landmark[num]
            xyz.append((lm.x, lm.y))
        return np.multiply(xyz, [w,h]).astype(int)


    # def check_visibility_lm(self, results, landmark_list):
    #     good_lm_list = []
    #     for num in landmark_list:
    #         lm = results[0].landmark[num]
    #         print(lm.HasField('visibility'),lm.visibility,lm.HasField('presence'),lm.presence)
    #         if lm.HasField('visibility') and lm.visibility > VISIBILITY_THRESHOLD:
    #             if lm.HasField('presence') and lm.presence > PRESENCE_THRESHOLD:
    #                 good_lm_list.append(num)

    #     return len(good_lm_list)/len(landmark_list)
    # https://github.com/google/mediapipe/issues/3159
    # https://github.com/google/mediapipe/issues/3200

    def detect_faces(self, image: np.ndarray) -> any:
        """
        Detects faces in an image using BlazefaceFacemesh.

        Args:
        - image (np.ndarray): An image.

        Returns:
        - The results of the facemesh detection.
        """
        time = tt.time()
        image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
        results = self._classifier.process(image)
        self.det_times.append(tt.time() - time)

        return results.multi_face_landmarks

    def coords_for_landmark(self, results: Any, lm: List[int], image: np.ndarray) -> np.ndarray:
        """
        Extracts the coordinates of the specified landmark and returns them as a numpy array.

        Parameters
        ----------
        results : Any
            The results of the BlazeFace Mesh detection.
        lm : List[int]
            The indices of the landmarks whose coordinates are to be extracted.
        image : np.ndarray
            The input image containing the face.

        Returns
        -------
        np.ndarray
            A numpy array containing the coordinates of the specified landmark.
        """
        coords = self.get_facemesh_coords(results, lm, image)            
        coords = sort_counterclockwise(coords)
        coords = np.array(coords).reshape(-1,1,2).astype(np.int32)

        return coords

    def rgb_means(self,coords: np.ndarray, image: np.ndarray, masks: List[Union[None, np.ndarray]]) -> Tuple[float, float, float]:
        """Calculates the RGB means of a given region of interest in an image.

        Args:
            coords (numpy.ndarray): The coordinates of the region of interest in the image.
            image (numpy.ndarray): The input image.
            masks (List[Union[None, numpy.ndarray]]): A list of masks to apply to the image.

        Returns:
            Tuple[float, float, float]: The RGB means of the region of interest in the image.

        """
        roi_mask = np.zeros(image.shape[:2], np.uint8)
        cv2.drawContours(roi_mask, [coords], -1, 255, -1)

        final_mask = roi_mask

        for mask in masks:
            if mask is not None:
                final_mask=cv2.bitwise_and(final_mask,mask)

        mean = cv2.mean(image, mask=final_mask)
        return mean[2], mean[1], mean[0] #Returns RGB means\

    def draw_landmarks(self, results: dict, lm_list: List[int], image: np.ndarray, masks: List[Union[None, np.ndarray]]) -> np.ndarray:
        """Draws the face mesh landmarks on the given image.

        Args:
            results (dict): The dictionary containing the results of the face mesh detection.
            lm_list (List[int]): The list of landmarks to draw.
            image (numpy.ndarray): The input image.
            masks (List[Union[None, numpy.ndarray]]): A list of masks to apply to the image.

        Returns:
            numpy.ndarray: The image with the face mesh landmarks drawn on it.

        """
        for mask in masks:
            if mask is not None:
                image = cv2.bitwise_and(image,image, mask = mask)

        # Draw the face mesh annotations on the image.
        for lm in lm_list:
            coords = self.get_facemesh_coords(results, lm, image)            
            coords = sort_counterclockwise(coords)
            coords = np.array(coords).reshape(-1,1,2).astype(np.int32)

            cv2.drawContours(image, [coords], -1, (255,255,255), 1, 4)

        return image

__init__(tracking=False)

Initializes a new instance of the BlazefaceFacemesh class.

bool

A boolean value indicating whether to enable tracking (default: False).

Source code in redesign_pipeline/face_detectors/blazeface_facemesh.py 
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
def __init__(self,tracking=False):
    """
    Initializes a new instance of the BlazefaceFacemesh class.

    Parameters:
    -----------
    tracking: bool
        A boolean value indicating whether to enable tracking (default: False).
    """
    super().__init__()
    self.mpFaceMesh = mp.solutions.face_mesh

    if tracking:
        self._classifier = self.mpFaceMesh.FaceMesh(max_num_faces=1,
                                        refine_landmarks=True,
                                        min_detection_confidence=0.5,
                                        min_tracking_confidence=0.5)
    else:
        self._classifier = self.mpFaceMesh.FaceMesh(max_num_faces=1,
                                        refine_landmarks=True,
                                        min_detection_confidence=0.5,
                                        static_image_mode=True
                                        )

    self.lm_big_forehead = self.fs_to_landmark_list(FACEMESH_BIG_FOREHEAD)
    self.lm_small_forehead = self.fs_to_landmark_list(FACEMESH_SMALL_FOREHEAD)
    self.lm_cheeks_and_nose = self.fs_to_landmark_list(FACEMESH_CHEEKS_AND_NOSE)
    self.lm_left_cheek = self.fs_to_landmark_list(FACEMESH_LEFT_CHEEK)
    self.lm_right_cheek = self.fs_to_landmark_list(FACEMESH_RIGHT_CHEEK)
    self.lm_crop_face = self.fs_to_landmark_list(FACEMESH_CROP_FACE)
    self.det_times = []
    self.track_times = []

coords_for_landmark(results, lm, image)

Extracts the coordinates of the specified landmark and returns them as a numpy array.

Parameters
Any

The results of the BlazeFace Mesh detection.

List[int]

The indices of the landmarks whose coordinates are to be extracted.

np.ndarray

The input image containing the face.

Returns

np.ndarray A numpy array containing the coordinates of the specified landmark.

Source code in redesign_pipeline/face_detectors/blazeface_facemesh.py 
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
def coords_for_landmark(self, results: Any, lm: List[int], image: np.ndarray) -> np.ndarray:
    """
    Extracts the coordinates of the specified landmark and returns them as a numpy array.

    Parameters
    ----------
    results : Any
        The results of the BlazeFace Mesh detection.
    lm : List[int]
        The indices of the landmarks whose coordinates are to be extracted.
    image : np.ndarray
        The input image containing the face.

    Returns
    -------
    np.ndarray
        A numpy array containing the coordinates of the specified landmark.
    """
    coords = self.get_facemesh_coords(results, lm, image)            
    coords = sort_counterclockwise(coords)
    coords = np.array(coords).reshape(-1,1,2).astype(np.int32)

    return coords

detect_faces(image)

Detects faces in an image using BlazefaceFacemesh.

  • image (np.ndarray): An image.
  • The results of the facemesh detection.
Source code in redesign_pipeline/face_detectors/blazeface_facemesh.py 
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
def detect_faces(self, image: np.ndarray) -> any:
    """
    Detects faces in an image using BlazefaceFacemesh.

    Args:
    - image (np.ndarray): An image.

    Returns:
    - The results of the facemesh detection.
    """
    time = tt.time()
    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
    results = self._classifier.process(image)
    self.det_times.append(tt.time() - time)

    return results.multi_face_landmarks

draw_landmarks(results, lm_list, image, masks)

Draws the face mesh landmarks on the given image.

Parameters:

Name Type Description Default
results dict

The dictionary containing the results of the face mesh detection.

 required
lm_list List[int]

The list of landmarks to draw.

 required
image numpy.ndarray

The input image.

 required
masks List[Union[None, numpy.ndarray]]

A list of masks to apply to the image.

 required

Returns:

Type Description
np.ndarray

numpy.ndarray: The image with the face mesh landmarks drawn on it.
Source code in redesign_pipeline/face_detectors/blazeface_facemesh.py 
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
def draw_landmarks(self, results: dict, lm_list: List[int], image: np.ndarray, masks: List[Union[None, np.ndarray]]) -> np.ndarray:
    """Draws the face mesh landmarks on the given image.

    Args:
        results (dict): The dictionary containing the results of the face mesh detection.
        lm_list (List[int]): The list of landmarks to draw.
        image (numpy.ndarray): The input image.
        masks (List[Union[None, numpy.ndarray]]): A list of masks to apply to the image.

    Returns:
        numpy.ndarray: The image with the face mesh landmarks drawn on it.

    """
    for mask in masks:
        if mask is not None:
            image = cv2.bitwise_and(image,image, mask = mask)

    # Draw the face mesh annotations on the image.
    for lm in lm_list:
        coords = self.get_facemesh_coords(results, lm, image)            
        coords = sort_counterclockwise(coords)
        coords = np.array(coords).reshape(-1,1,2).astype(np.int32)

        cv2.drawContours(image, [coords], -1, (255,255,255), 1, 4)

    return image

fs_to_landmark_list(fs)

Converts a facemesh landmark list from a list of pairs to a list of integers.

Args

fs: (List[List[int]]): A facemesh landmark list.

Returns

A list of integers representing the facemesh landmarks.

Source code in redesign_pipeline/face_detectors/blazeface_facemesh.py 
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
def fs_to_landmark_list(self, fs: List[List[int]]) -> List[int]:
    """
    Converts a facemesh landmark list from a list of pairs to a list of integers.

    Args
    ----
    fs: (List[List[int]]): A facemesh landmark list.

    Returns
    -------
    A list of integers representing the facemesh landmarks.
    """
    lst = []
    for pair in fs:
        for n in pair:
            if n not in lst:
                lst.append(n)
    return lst

get_facemesh_coords(results, landmark_list, img)

Returns the x, y coordinates of the facemesh landmarks.

  • results (any): The results of the facemesh detection.
  • landmark_list (List[int]): A list of integers representing the facemesh landmarks.
  • img (np.ndarray): An image.
  • An array of x, y coordinates of the facemesh landmarks.
Source code in redesign_pipeline/face_detectors/blazeface_facemesh.py 
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
def get_facemesh_coords(self, results, landmark_list: List[int], img: np.ndarray) -> np.ndarray:
    """
    Returns the x, y coordinates of the facemesh landmarks.

    Args:
    - results (any): The results of the facemesh detection.
    - landmark_list (List[int]): A list of integers representing the facemesh landmarks.
    - img (np.ndarray): An image.

    Returns:
    - An array of x, y coordinates of the facemesh landmarks.
    """
    h,w = img.shape[:2]
    xyz = []
    for num in landmark_list:
        lm = results[0].landmark[num]
        xyz.append((lm.x, lm.y))
    return np.multiply(xyz, [w,h]).astype(int)

rgb_means(coords, image, masks)

Calculates the RGB means of a given region of interest in an image.

Parameters:

Name Type Description Default
coords numpy.ndarray

The coordinates of the region of interest in the image.

 required
image numpy.ndarray

The input image.

 required
masks List[Union[None, numpy.ndarray]]

A list of masks to apply to the image.

 required

Returns:

Type Description
Tuple[float, float, float]

Tuple[float, float, float]: The RGB means of the region of interest in the image.
Source code in redesign_pipeline/face_detectors/blazeface_facemesh.py 
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
def rgb_means(self,coords: np.ndarray, image: np.ndarray, masks: List[Union[None, np.ndarray]]) -> Tuple[float, float, float]:
    """Calculates the RGB means of a given region of interest in an image.

    Args:
        coords (numpy.ndarray): The coordinates of the region of interest in the image.
        image (numpy.ndarray): The input image.
        masks (List[Union[None, numpy.ndarray]]): A list of masks to apply to the image.

    Returns:
        Tuple[float, float, float]: The RGB means of the region of interest in the image.

    """
    roi_mask = np.zeros(image.shape[:2], np.uint8)
    cv2.drawContours(roi_mask, [coords], -1, 255, -1)

    final_mask = roi_mask

    for mask in masks:
        if mask is not None:
            final_mask=cv2.bitwise_and(final_mask,mask)

    mean = cv2.mean(image, mask=final_mask)
    return mean[2], mean[1], mean[0] #Returns RGB means\