Surveillance and Social Situational Awareness

Thursday, May 11, 2017 - 4:00pm to 5:30pm
Schapiro Hall (CEPSR) 750 (Costa Engineering Commons)
Columbia University
New York, NY 10027
United States

Columbia Data Science Institute Industry Innovation Seminars

(Talk 1 of 2)


Peter Tu,Senior Principal Scientist, GE Global Research

Title: Surveillance and Social Situational Awareness

Abstract: This talk will describe a variety of methods that have been developed for the purposes of understanding group level social behaviors using stand-off video surveillance methods. Three main topics are considered: 1) the GE Sherlock System: a comprehensive approach to capturing and analyzing non-verbal cues of persons in crowd/group level interactions, 2) One Shot Learning: a new approach to crowd level behavior recognition based on the concept that a new behavior can be recognized with as little as a single example and 3) Agent Based Inference: a novel approach to the analysis of individual cognitive states of person?s interacting in a group or crowd level social interactions. The talk starts with a description of the GE Sherlock system which encompasses methods such as person tracking in crowds, dynamic PTZ camera control, facial analytics from a distance such as gaze estimation and expression recognition, upper body affective pose analysis and the inference of social states such as rapport and hostility. The talk then discusses how cues derived from the Sherlock system can be used to construct semantically meaningful behavior descriptors or affects allowing for signature matching between behaviors which can be viewed as a form of one shot learning. Going beyond affects based on direct observation, we argue that more meaningful affects can be constructed via the inference of the cognitive states of each individual. To this end we introduce the Agent Based Inference framework. The talk concludes with a discussion of how such methods are making their way into commercial use via efforts such as the intelligent city, the intelligent airport and the intelligent hospital.

Biography: In 1990 Dr. Tu joined Sony Research in Tokyo Japan, where he develeped a number of computer vision algorithms for man-machine interfaces. While at Oxford University, his research was devoted to the development of computer vision methods for the autumatic analysis of seismic imagery. In 1997 Dr. Tu became a senior research scientist working at General Electric?s Global Research center. In partnership with Lockheed Martin, he has developed a set of latent fingerprint matching algorithms for the FBI Automatic Fingerprint Identification System (AFIS). Dr. Tu has also developed optical methods for the precise measurement of 3D parts in a manufacturing setting. Dr. Tu was the principal investigator for the FBI ReFace project, which is tasked with developing an automatic system for face reconstruction from skeletal remains. In 2006, he was the principal investigator for the National Institute of Justice?s 3D Face Enhancer Program. This work was focused on improving face recogntion from poor quality surveillance video. In 2008, Dr Tu led the GE video analytics team that participated in the DHS STIDP demonstration program - the goal of STIDP is to establish an effective defence against suicide bomber attack. Dr Tu is the prinicipal investigator for the DARPA sponsored effort associated with group level behavior recognition at a distance. Currently Dr. Tu is the Senior Prinicipal Scientist for a group of 15 researchers in the field of multi-view video analysis with the aim of acheiving reliable behavior recognition in complex environments. He has helped to develop a large number analytic capabilities including: person detection from fixed and moving platforms, crowd segementation, multi-view tracking, person reacquistion, face modelling, face expression analysis, face recognition at a distance, face verification from photo IDs and articulated motion analysis. Dr Tu has over 50 peer reviewed publications and has filed more than 25 U.S. patents.

Back to Top