PDL Abstract

Picking Interesting Frames in Streaming Video

SysML’18, February 15–16, 2018, Stanford, CA.

Christopher Canel, Thomas Kim, Giulio Zhou, Conglong Li, Hyeontaek Lim, David G. Andersen, Michael Kaminsky†, Subramanya R. Dulloor†

Carnegie Mellon University
† Intel Labs


As video camera deployments proliferate in the smart cities of the future [2], software systems are faced with the increasing challenge of determining which segments of data are relevant. For resource-constrained edge nodes, limited network bandwidth back to the datacenter prevents sending entire video streams.

This paper presents a new application-independent interesting frame (IF) detection algorithm for identifying relevant frames in streaming video. We envision this IF detector as a preprocessing step in a larger video analytics pipeline where the expensive computation occurs later (similar to the way Bloom filters can guard expensive data structures). Given a target frame rate (or, equivalently, a target bandwidth), the algorithm decides which frames are the most generally interesting and therefore should be processed by downstream applications or forwarded to the datacenter.We decide how “interesting” a frame is based on its semantic difference from other frames. The IF detection algorithm uses a hierarchy of filters to trade off between end-to-end latency and aggressive decimation. The algorithm strives to maximize the semantic diversity of the selected frames. Compared to simply choosing frames at a fixed interval, the IF detector better handles bursty events in the stream.