Intensity Change

The Intensity Change algorithm detects movement onset with a frame-differencing approach.

Given a current_frame and a previous_frame, the algorithm subtracts the brightness (0–255) of each pixel in the current_frame from the brightness of the corresponding pixel in the previous_frame. If the difference is greater than the set threshold, the pixel is considered to have changed.

The number of changed pixels is summed to create a timeseries of size 1xN, where N is the number of frames in the video.

Parameters

The threshold parameter adjusts the sensitivity of movement detection.

Range: 0–255 (default: 20)

The lower the threshold, the more sensitive the algorithm is to pixel changes, and therefore movement.

Preprocessing

Frames are converted to grayscale and a light boxblur filter (luma radius = 1) is applied to smooth ambient pixel noise.

Postprocessing

After the frame-differencing is applied, three optional postprocessing steps are applied to the current_frame:

Opening

Opening is a morphological operation (erosion followed by dilation) that can help remove “salt and pepper” noise from the frame.

Morphological opening

Range: 0–10000 (default: 0)

Closing

Closing is a morphological operation (dilation followed by erosion) that can help “glue” small, non-contiguous pixel detections together.

If multiple pieces/edges of a body part are detected, closing can help meld them into a single object, which has a higher signal-to-noise ratio.

Morphological closing

Range: 0–10000 (default: 0)

Small object removal

Maximum size of objects to discard from the frame. Like opening, this can help remove small bits of noise.

This is particularly effective when combined with closing, and disparate “pieces” of a single body part or object can be combined together, surviving small object removal.

Small object removal

Range: 0–10000 (default: 0)

With all postprocessing parameters combined, you get something like this:

Opening: 12, Closing: 15, Small object removal: 225

(This video required somewhat aggressive postprocessing to boost the signal-to-noise ratio.)

Event Detection

After the frames are processed and the timeseries is created, the algorithm makes a second pass along the timeseries to detect movement onset.

Baseline Estimation

Simple baseline estimation approaches, such as a mean or median filtering, tend to overestimate (in the case of mean, due to outliers) or underestimate (in the case of median, due to long quiet periods) baseline “noise”, compared with a human labeler.

Here, we use a custom baseline estimation approach that is more robust to outliers and long quiet periods:

1. Split the timeseries into short (i.e., 0.1 s) bins
2. For each bin, measure the 95th percentile of the intensity values (ignores outliers)
3. Find the most common value across all bins (via Gaussian kernel density estimation)

Time bins with 95%ile

We set our detection threshold to 2x this baseline.

Peak detection

Next, we compute the peak prominences of every detected peak. Events with a peak prominence under the detection threshold are discarded.

Rate limit

A 250 ms rate limit is applied to all detected events. This enforces a minimum “quiet period” before an event starts.

All events within 250 ms of the first detected event in a burst are discarded.

Output

The Intensity Change algorithm creates:

1. A new “overlay” video with changing pixels highlighted in blue
2. A timeseries of size 1xN, representing the number of changed pixels in each frame
3. An array of event onset times, in seconds

The next step after detecting events with this algorithm is to verify the detected events in Quick Score.