FitMonkey is a product designed to track exercise performance using computer vision. This edge (Nvidia Jetston) enabled app counts repetitions and monitors for good/bad form using models trained on pose estimation and optical flow.
Check out the website fitMonkey
Performance
The proof of concept demo has been trained on pushups, pullups and squats, with the following performance results
Pushups
Validation
Test
F1
Accuracy
F1
Accuracy
Pushup Detector
0.95
0.95
0.89
0.81
Bad Back Form
0.97
0.97
0.79
0.76
Squats
Validation
Test
F1
Accuracy
F1
Accuracy
Squats Detector
0.98
0.98
0.93
0.93
Bad Form
0.99
0.99
0.81
0.75
Pullups
Validation
Test
F1
Accuracy
F1
Accuracy
Pullup Detector
0.95
0.95
0.97
0.99
Keypoint Tracking with Human2D Pose Estimation
Human Pose Estimation is used to generate input into the form tracking models.
Human Pose Estimation localizes and captures human body parts, such as shoulders, elbows and ankles from the input video.
First we identify the confidence maps to identify body parts (fig 1a)
Then we identify the affinity fields (fig 1b) which represent the degree of association between the body parts
The confidence maps and affinity fields are used to infer 2d keypoints (fig 1c)
These keyopints map to CoCo bodyparts (fig 1d)
An aditional step is to calculate the angles between body part pairs, such as wrist and elbow
The keypoints, and angle coordinates are used to train exercise detection and posture models
Counting Reps with Optical Flow
Optical flow is the pattern of apparent motion of image objects between two consecutive frames in a sequence (e.g. frames in a video).
It shows the movement of points fetween frames using a 2D vector field, where each vector represents the displacement (difference) of frames.
We use Open CV's implementation of Dense Optical Flow using Gunner Farneback’s algorithm to track the movement of the user performing the exercise.
The algorithm calculates the optical flow for every pixel in the image.
Because anarobic exercises are Home consistent and repetitive motion, we can track the change in direction of movement as an example of a repetition
And because optical flow tracks movement - we can use the same algorithm for all exercises, negating the need to train multiple models.
