Video processing systems and methods

2024-07-12 06:00:10   224人阅读

An exemplary calibration sheet having a plurality of dots can be used. In this embodiment, the dots can be circular dots and square dots which are interleaved among each others. The dots should be placed relatively close to each other and each dot size should not be too large, so we can have as many dots as possible in one snapshot. However, the dots should not be placed too close to each other and the dot size should not be too small, so they are not identifiable.

The camera can monitor patient activity and generates a warning if the patient has fallen. In one implementation, the system detects the speed of center of mass movement. If the center of mass movement is zero for a predetermined period, the patient is either sleeping or unconscious. The system then attempts to signal the patient and receive confirmatory signals indicating that the patient is conscious. If patient does not confirm, then the system generates an alarm. For example, if the patient has fallen, the system would generate an alarm signal that can be sent to friends, relatives or neighbors of the patient. Alternatively, a third party such as a call center can monitor the alarm signal. Besides monitoring for falls, the system performs video analysis of the patient. For example, during a particular day, the system can determine the amount of time for exercise, sleep, and entertainment, among others. The network of sensors in a patient‘s home can recognize ordinary patterns—such as eating, sleeping, and greeting visitors—and to alert caretakers to out-of-the-ordinary ones—such as prolonged inactivity or absence. For instance, if the patient goes into the bathroom then disappears off the sensor for 13 minutes and don‘t show up anywhere else in the house, the system infers that patient had taken a bath or a shower. However, if a person falls and remains motionless for a predetermined period, the system would record the event and notify a designated person to get assistance.

A fall detection process performs the following operations:

• Find floor space area
• Define camera view background 3D scene
• Calculate patient‘s key features
• Detect fall

In one implementation, pseudo-code for determining the floor space area is as follows:

• 1. Sample the camera view space by M by N, e.g. M=1000, N=500.
• 2. Calculate all sample points the 3D coordinates in room coordinate system; where Z axis is pointing up. Refer to the 3D detection for how to calculate 3D positions.
• 3. Find the lowest Z value point (Zmin)
• 4. Find all points whose Z values are less than Zmin+Ztol; where Ztol is a user adjustable value, e.g. 2 inches.
• 5. If rooms have different elevation levels, then excluding the lowest Z floor points, repeat step 2, 3 and 4 while keeping the lowest Z is within Ztol2 of previous Z. In this example Ztol2=2 feet, which means the floor level difference should be within 2 feet.
• 6. Detect stairs by finding approximate same flat area but within equal Z differences between them.
• 7. Optionally, additional information from the user can be used to define floor space more accurately, especially in single camera system where the coordinates are less accurate, e.g.:
  • a. Import the CAD file from constructors‘ blue prints.
  • b. Pick regions from the camera space to define the floor, then use software to calculate its room coordinates.
  • c. User software to find all flat surfaces, e.g. floors, counter tops, then user pick the ones, which are actually floors and/or stairs.

In the implementation, pseudo-code for determining the camera view background 3D scene is as follows:

• 1. With the same sample points, calculate x, y coordinates and the Z depth and calculate 3D positions.
• 2. Determine background scene using one the following methods, among others:
  • a. When there is nobody in the room.
  • b. Retrieve and update the previous calculated background scene.
  • c. Continuous updating every sample point when the furthest Z value was found, that is the background value.

In the implementation, pseudo-code for determining key features of the patient is as follows:

• 1. Foreground objects can be extracted by comparing each sample point‘s Z value to the background scene point‘s Z value, if it is smaller, then it is on the foreground.
• 2. In normal condition, the feet/shoe can be detected by finding the lowest Z point clouds close the floor in room space, its color will be extracted.
• 3. In normal condition, the hair/hat can be detected by finding the highest Z point clouds close the floor in room space, its color will be extracted.
• 4. The rest of the features can be determined by searching from either head or toe. E.g, hat, hair, face, eye, mouth, ear, earring, neck, lipstick, moustache, jacket, limbs, belt, ring, hand, etc.
• 5. The key dimension of features will be determined by retrieving the historic stored data or recalculated, e.g., head size, mouth width, arm length, leg length, waist, etc.
• 6. In abnormal conditions, features can be detected by detect individual features then correlated them to different body parts. E.g, if patient‘s skin is black, we can hardly get a yellow or white face, by detecting eye and nose, we know which part is the face, then we can detect other characteristics.

To detect fall, the pseudo-code for the embodiment is as follows:

• 1. The fall has to be detected in almost real time by tracking movements of key features very quickly. E.g. if patient has black hair/face, track the center of the black blob will know roughly where his head move to.
• 2. Then the center of mass will be tracked, center of mass is usually around belly button area, so the belt or borderline between upper and lower body closed will be good indications.
• 3. Patient‘s fall always coupled with rapid deceleration of center of mass. Software can adjust this threshold based on patient age, height and physical conditions.
• 4. Then if the fall is accidental and patient has difficult to get up, one or more of following will happen:
  • a. Patient will move very slowly to find support object to get up.
  • b. Patient will wave hand to camera ask for help. To detect this condition, the patient hand has to be detected first by finding a blob of points with his skin color. Hand motion can be tracked by calculate the motion of the center of the points, if it swings left and right, it means patient is waving to camera.
  • c. Patient is unconscious, motionless. To detect this condition, extract the foreground object, calculate its motion vectors, if it is within certain tolerance, it means patient is not moving. In the mean time, test how long it last, if it past a user defined time threshold, it means patient is in great danger.

In one embodiment for fall detection, the system determines a patient fall-down as when the patient‘s knee, butt or hand is on the floor. The fall action is defined a quick deceleration of center of mass, which is around belly button area. An accidental fall action is defined when the patient falls down with limited movement for a predetermined period.

The system monitors the patients‘ fall relative to a floor. In one embodiment, the plan of the floor is specified in advance by the patient. Alternatively, the system can automatically determine the floor layout by examining the movement of the patient‘s feet and estimated the surfaces touched by the feet as the floor.

The system detects a patient fall by detecting a center of mass of an exemplary feature. Thus, the software can monitor the center of one or more objects, for example the head and toe, the patient‘s belt, the bottom line of the shirt, or the top line of the pants.

The detection of the fall can be adjusted based on two thresholds:

• a. Speed of deceleration of the center of mass.
• b. The amount of time that the patient lies motionless on the floor after the fall.

In one example, once a stroke occurs, the system detects a slow motion of patient as the patient rests or a quick motion as the patient collapses. By adjust the sensitivity threshold, the system detects whether a patient is uncomfortable and ready to rest or collapse.

If the center of mass movement ceases to move for a predetermined period, the system can generate the warning. In another embodiment, before generating the warning, the system can request the patient to confirm that he or she does not need assistance. The confirmation can be in the form of a button that the user can press to override the warning. Alternatively, the confirmation can be in the form of a single utterance that is then detected by a speech recognizer.

In another embodiment, the confirmatory signal is a patient gesture. The patient can nod his or her head to request help and can shake the head to cancel the help request. Alternatively, the patient can use a plurality of hand gestures to signal to the server?20?the actions that the patient desires.

By adding other detecting mechanism such as sweat detection, the system can know whether patient is uncomfortable or not. Other items that can be monitored include chest movement (frequency and amplitude) and rest length when the patient sits still in one area, among others.

Besides monitoring for falls, the system performs video analysis of the patient. For example, during a particular day, the system can determine the amount of time for exercise, sleep, entertainment, among others. The network of sensors in a patient‘s home can recognize ordinary patterns—such as eating, sleeping, and greeting visitors—and to alert caretakers to out-of-the-ordinary ones—such as prolonged inactivity or absence. For instance, if the patient goes into the bathroom then disappears off the camera?10?view for a predetermined period and does not show up anywhere else in the house, the system infers that patient had taken a bath or a shower. However, if a person falls and remains motionless for a predetermined period, the system would record the event and notify a designated person to get assistance.

In one embodiment, changes in the patient‘s skin color can be detected by measuring the current light environment, properly calibrating color space between two photos, and then determining global color change between two states. Thus, when the patient‘s face turn red, based on the redness, a severity level warning is generated.

In another embodiment, changes in the patient‘s face are detected by analyzing a texture distortion in the images. If the patient perspires heavily, the texture will show small glisters, make-up smudges, or sweat/tear drippings. Another example is, when long stretched face will be detected as texture distortion. Agony will show certain wrinkle texture patterns, among others.

The system can also utilize high light changes. Thus, when the patient sweats or changes facial appearance, different high light areas are shown, glisters reflect light and pop up geometry generates more high light areas.

A module?62?analyzes facial changes such as facial asymmetries. The change will be detected by superimpose a newly acquired 3D anatomy structure to a historical (normal) 3D anatomy structure to detect face/eye sagging or excess stretch of facial muscles.

In one embodiment, the system determines a set of base 3D shapes, which are a set of shapes which can represent extremes of certain facial effects, e.g. frown, open mouth, smiling, among others. The rest of the 3D face shape can be generated by blending/interpolating these base shapes by applied different weight to each base shapes.

The base 3D shape can be captured using 1) a 3D camera such as cameras from Steinbichler, Genex Technology, Minolta 3D, Olympus 3D or 2) one or more 2D camera with preset camera field of view (FOV) parameters. To make it more accurate, one or more special markers can be placed on patient‘s face. For example, a known dimension square stick can be placed on the forehead for camera calibration purposes.

Using the above 3D detection method, facial shapes are then extracted. The proper features (e.g. a wrinkle) will be detected and attached to each base shape. These features can be animated or blended by changing the weight of different shape(s). The proper features change can be detected and determine what type of facial shape it will be.

Next, the system super-imposes two 3D facial shapes (historical or normal facial shapes and current facial shapes). By matching features and geometry of changing areas on the face, closely blended shapes can be matched and facial shape change detection can be performed. By overlaying the two shapes, the abnormal facial change such as sagging eyes or mouth can be detected.

The above processes are used to determine paralysis of specific regions of the face or disorders in the peripheral or central nervous system (trigeminal paralysis; CVA, among others). The software also detects eyelid positions for evidence of ptosis (incomplete opening of one or both eyelids) as a sign of innervation problems (CVA; Horner syndrome, for example). The software also checks eye movements for pathological conditions, mainly of neurological origin are reflected in aberrations in eye movement. Pupil reaction is also checked for abnormal reaction of the pupil to light (pupil gets smaller the stronger the light) may indicate various pathological conditions mainly of the nervous system. In patients treated for glaucoma pupillary status and motion pattern may be important to the follow-up of adequate treatment. The software also checks for asymmetry in tongue movement, which is usually indicative of neurological problems. Another check is neck veins: Engorgement of the neck veins may be an indication of heart failure or obstruction of normal blood flow from the head and upper extremities to the heart. The software also analyzes the face, which is usually a mirror of the emotional state of the observed subject. Fear, joy, anger, apathy are only some of the emotions that can be readily detected, facial expressions of emotions are relatively uniform regardless of age, sex, race, etc. This relative uniformity allows for the creation of computer programs attempting to automatically diagnose people‘s emotional states.

Speech recognition is performed to determine a change in the form of speech (slurred speech, difficulties in the formation of words, for example) may indicated neurological problems, such an observation can also indicate some outward effects of various drugs or toxic agents.

In one embodiment shown in?FIG. 4, a facial expression analysis process performs the following operations:

• Find floor space area
• Define camera view background 3D scene
• Calculate patient‘s key features
• Extract facial objects
• Detect facial orientation
• Detect facial expression

The first three steps are already discussed above. The patient‘s key features provide information on the location of the face, and once the face area has been determined, other features can be detected by detecting relative position to each other and special characteristics of the features:

• Eye: pupil can be detected by applying Chamfer matching algorithm, by using stock pupil objects.
• Hair: located on the top of the head, using previous stored hair color to locate the hair point clouds.
• Birthmarks, wrinkles and tattoos: pre store all these features then use Chamfer matching to locate them.
• Nose: nose bridge and nose holes usually show special characteristics for detection, sometime depend on the view angle, is side view, special silhouette will be shown.
• Eye browse, Lips and Moustache: All these features have special colors, e.g. red lipstick; and base shape, e.g. patient has no expression with mouth closed. Software will locate these features by color matching, then try to deform the base shape based on expression, and match shape with expression, we can detect objects and expression at the same time.
• Teeth, earring, necklace: All these features can be detected by color and style, which will give extra information.

In one implementation, pseudo-code for detecting facial orientation is as follows:

• Detect forehead area
• Use the previously determined features and superimpose them on the base face model to detect a patient face orientation.

Depends on where patient is facing, for a side facing view, silhouette edges will provide unique view information because there is a one to one correspondent between the view and silhouette shape.

Once the patient‘s face has been aligned to the right view, exemplary pseudo code to detect facial expression is as follows:

• 1. Detect shape change. The shape can be match by superimpose different expression shapes to current shape, and judge by minimum discrepancy. E.g. wide mouth open.
• 2. Detect occlusion. Sometime the expression can be detected by occlusal of another objects, e.g., teeth show up means mouth is open.
• 3. Detect texture map change. The expression can relate to certain texture changes, if patient smile, certain wrinkles patents will show up.
• 4. Detect highlight change. The expression can relate to certain high light changes, if patient sweats or cries, different highlight area will show up.

The cameras in?FIGS. 6-7?can communicate over PCIe bus, or can communicate over USB 3.0 protocol (SuperSpeed USB). The theoretical throughput improvement offered by USB 3.0 is a 10× jump over USB 2.0 hardware: USB 2.0 peaks at a theoretical 480 Mbps, while USB 3.0 can theoretically handle up to 5 Gbps. This is done using transmission over differential transceivers and shielded differential pair cables, the details of which can be found at http://www.usb.org/developers/ssusb/. Two differential pairs are used (dual/simplex), and the differential transmission is similar to those used in SATA and PCIe differential transmission techniques.

In one embodiment, the GPUs are used to expedite the motion estimation function, which is the most computationally intensive section of H.264 operations by performing motion estimation in parallel. The term "parallel" to refer to processes that generally run concurrently in a coordinated fashion, but does not require a strict step by step, or clock by clock parallelism.

FIG. 8?shows an exemplary stream computing programming model where programmable stream cores execute application specific programs called stream kernels such as video processing kernels. The stream cores operate with a virtualized SIMD programming model operating on streams of data. In stream computing, arrays of input video data are mapped onto a number of SIMD engines which execute kernels to generate video outputs that are written to external memory or to the cloud. Each instance of a kernel is called a thread. A specified region of the output buffer to which threads are mapped is the domain of execution. The stream processor schedules the array of threads onto a group of processors until all threads have been processed. Subsequent kernels can be executed until the application completes.

FIG. 9?shows exemplary stream processor which includes groups of SIMD engines. Each SIMD engine contains numerous thread processors, which are responsible for executing kernels, each operating on an independent data stream. Thread processors in turn contain numerous stream cores, which are programmable computation units that can perform integer, single or double precision floating point and transcendental operations. All thread processors within an SIMD engine execute the same instruction, and different SIMD engines can execute different instructions.

Advanced Video Coding (AVC)

In one embodiment, a video frame is broken down into macroblocks (each macroblock typically covers 16×16 pixels), each macroblock‘s movement from a previous frame (reference frame) is tracked and represented as a vector, called motion vector. Storing this vector and residual information instead of the complete pixel information greatly reduces the amount of data used to store the video. The pyramid (or hierarchical) motion vector prediction performs motion estimation on a significant downsampled version of the image. The vectors found in this iteration are used as estimates for the motion vector predictions in motion estimation of a slightly less-downsampled image. This process is repeated until the motion estimation is performed on the full-resolution image. One implementation started at a level of sixteen times downsampling and doubled the resolution to eight times. It continued doubling until the motion estimation is done for the full resolution image. One kernel is executed per level of hierarchy. After the kernel was done executing the motion vectors found are left on the device for the next kernel call to minimize the number of host-device memory transfers needed.

In another implementation, the GPUs operates in parallel on "slices" of video data for H.264 encoding, each containing a set of blocks that can be decoded without any other neighboring block information (from outside the slice). At each slice, the predictors are reset, trading off compression efficiency for error resilience. Thus, one slice can be used per line of blocks. If an error is introduced in any given block, the system can recover on the next line of blocks.

The video frames are first placed in to the memory from a capture device such as a camera. The GPU then executes various pixel processes of an encoder resulting in coefficients. These processes include Intra and Inter prediction, mode selection, motion estimation, motion compensation, DCT and IDCT, Quantization and inverse Quantization. The resulting coefficients and metadata is then processed by GPU. The GPU then takes the coefficient and meta data and encodes using a variable length coding process (VLC) resulting in a video stream. If there are multiple slices in the picture, the GPU can process each slice in parallel resulting in higher overall performance.

Each slice in a video stream can be decoded independently of other slices. Slices also contain blocks that are dependent on other blocks in the slice and are best decoded sequentially; therefore, in a preferred embodiment, each slice is decoded using a sequential processor in the GPU, but more than one slice can be decoded in parallel using a group of sequential processors in the GPU. Each sequential processor decodes an assigned slice, and outputs the independent coefficients and metadata into another array for subsequent use. If there are not enough sequential processors for all slices of a frame, slices may be assigned, for example in a round-robin fashion, until all slices are decoded.

Variable sized slices are packed in a buffer that contains the encoded bits from the video stream with all slices packed together. The data is pre-processed by finding the point in the buffer where each slice begins and the pointers for each slice are stored in an index array which is read by each processor in the GPU to find the location of the slice that each processor is responsible for decoding. Once the set of macroblocks in each GPU processor array has been VLC decoded to coefficients and meta data, the resulting (RLE compressed) coefficients and metadata for each block in a slice is stored in an array. Another index table is used to indicate where each macroblock is located in the coefficient buffer. Each processor in the GPU array then reads the address offset for the macroblock data for its decoding assignment. Once all the slices have been decoded, the decompressed slice data is sent for H.264 NAC assembly and decoding of the next frame of slices can be started on the GPU array. Since each macroblock is independent of other macroblocks, the GPU‘s parallel processors can be applied to decompressing all of the blocks in parallel.

One embodiment also performs scalable video coding using the GPU.

Other operations include:

Video Feature Tracking and Matching

In one embodiment, a KLT tracking process and a SIFT feature extraction process to enable real-time processing of high resolution video. The KLT tracking process computes displacement of features or interest points between consecutive video frames when image motion is fairly small. Feature selection is done by finding maximas of a saliency measure (minimum eigen-values of the 2×2 structure matrix obtained from gradient vectors. It is evaluated over the complete image and a subsequent non-maximal suppression is performed. Assuming a local translational model between subsequent video frames, the feature displacements are computed using Newton‘s method to minimize the sum of squared distances (SSD) within a tracking window around the feature position in the two images.

A multi-resolution KLT tracker allows handling larger image motion while multiple tracking iterations at each scale increases its accuracy. Features tracks are often lost after a few frames of tracking; hence new features are selected in a particular video frame only after tracking features in a few successive frames. This maintains a roughly fixed number of features in the tracker.

GPU-KLT maps these various steps to sets of different fragment programs. The multi-resolution pyramid of the image and its gradients are computed by a series of two-pass separable convolutions performed in fragment programs. The KLT cornerness map is computed in two render passes. The first pass computes the minimum eigen value of the 2×2 gradient matrix at each pixel and the two passes together accumulate the cornerness value within a 7×7 window centered at each pixel. During feature re-selection, the neighborhood of existing features is invalidated; early Z-culling avoids computations in these image regions. The cornerness map is transferred back to the CPU where non-maximal suppression is done to build the final feature-list. KLT tracking performs a fixed number of tracking iterations at each image resolution starting with the coarsest pyramid level. Each tracking iteration constructs a linear system of equations in two unknowns for each interest point, AX=B and directly solves them to update the estimated displacement. All steps are performed on the GPU. A SSD residual is computed between the two image patches of a particular KLT feature in order to reject features tracked inaccurately. Conditional statements are avoided in fragment programs by tracking a constant number of features and rejecting inaccurate tracks after the final tracking iteration on the GPU and before reading back the feature-list.

The Scale Invariant Feature Transform (SIFT) process performs extraction of interest points invariant to translation, rotation, scaling and illumination changes in images. It first constructs a Gaussian scale-space pyramid from the input image while also calculating the gradients and difference-of-gaussian (DOG) images at these scales. Interest points are detected at the local extremas within the DOG scale space. Once multiple keypoints have been detected at different scales, the image gradients in the local region around each feature point are encoded using orientation histograms and represented in the form of a rotationally invariant feature descriptor. The construction of the Gaussian scale space pyramid is accelerated on the GPU using fragment programs for separable convolution. The intensity image, gradients and the DOG values are stored in a RGBA texture and computed in the same pass. Blending operations in graphics hardware are used to find local extremas in the DOG pyramid in parallel at all pixel locations. The Depth test and the Alpha test is used to threshold these keypoints; The local principal curvatures of the image intensity around the keypoint is inspected; this involves computing the ratio of eigenvalues of the 2x2 Hessian matrix of the image intensity at that point. The keypoint locations are implicitly computed in image-sized, binary buffers, one for each scale in the pyramid. A fragment program compresses (a factor of 32) the binary bitmap into RGBA data, which is readback to the CPU and decoded there. At this stage, a list of keypoints and their scales have been retrieved. Since reading back the gradient pyramid (stored in texture memory) to the CPU is expensive, the subsequent steps in SIFT are also performed on the GPU. Gradient vectors near the keypoint location are Gaussian weighted and accumulated inside an orientation histogram by another fragment program. The orientation histogram is read back to the CPU, where its peaks are detected. Computing histograms on the GPU is expensive and doing it on the CPU along with a small readback is a little faster. The final step involves computing 128 element SIFT descriptors. These consist of a set of orientation histograms built from 16x16 image patches in invariant local coordinates determined by the associated keypoint scale, location and orientation. SIFT descriptors cannot be efficiently computed completely on the GPU, as histogram bins must be blended to remove quantization noise. This step is partitioned between the CPU and the GPU. Each feature‘s gradient vector patch is resampled, weighted using a Gaussian mask using blending support on the GPU. The resampled and weighted gradient vectors are collected into a tiled texture block which is subsequently transferred back to the CPU and then used to compute the descriptors. This CPU-GPU partition was done to minimize data readback from the GPU since transferring the whole gradient pyramid back to the CPU is impractical. Moreover texture re-sampling and blending are efficient operations on the GPU and are performed there. This also produces a compact tiled texture block which can be transferred to the CPU in a single read back. GPU-SIFT gains a large speed-up in the Gaussian scale-space pyramid construction and keypoint localization steps. The compressed readback of binary images containing feature positions reduces the readback data-size by a factor of 32. The feature orientation and descriptors computation is partitioned between the CPU and GPU in a way that minimizes data transfer from GPU to CPU.?

Video Conferencing Compression

In one embodiment, the video feature tracking and matching described above is used to compress conferencing sessions. Typically, in videoconferencing, the background remains the same, but the facial expression can change. The operation is as follows:

1) send the first few minutes of video using conventional or compressed video and simultaneously determine predetermine facial and body features;?
2) after the start up period, for each frame determine whether the current frame only has facial/body changes and if so

• look for an updated position of the features and transmit a vector indicating facial and body feature changes to the remote computer
• the remote computer converts the vector of changed facial features to an image of the user‘s face and body position?

3) otherwise, there are significant changes to the frame and so loop back to (1) to do a fresh compression cycle.

The process achieves a very high compression ratio since only a vector of feature position changes are sent as a vector and the vector is converted back into frame image by the remote computer. Moreover, if significant scene changes occur (such as new participants entering the conference, or participant picks up a book and show book to the camera), then the system reverts back to H.264 compression of full image.

Face Recognition

Face detection can be performed on board the camera for autofocus of the camera. Additionally, the face detection can be used to identify regions in the video that should be encoded at high resolution for certain applications.

A parallelized implementation of convolutional neural networks (CNNs) is done with parallelizing the detection process using the GPU. The convolutional network consists of a set of layers each of which contains one or more planes. Approximately centered and normalized images enter at the input layer. Each unit in a plane receives input from a small neighborhood in the planes of the previous layer. The idea of connecting units to local receptive fields dates back to the 1960s with the perceptron and Hubel and Wiesel‘s discovery of locally sensitive, orientation-selective neurons in the cat‘s visual system. The general strategy of a convolutional network is to extract simple features at a higher resolution, and then convert them into more complex features at a coarser resolution. The simplest was to generate coarser resolution is to sub-sample a layer by a factor of 2. This, in turn, is a clue to the convolutions kernel‘s size.

The weights forming the receptive field for a plane are forced to be equal at all points in the plane. Each plane can be considered as a feature map which has a fixed feature detector that is convolved with a local window which is scanned over the planes in the previous layer. Multiple planes are usually used in each layer so that multiple features can be detected. These layers are called convolutional layers.

The GPU supports a fast, automatic system for face recognition which is a combination of a local image sample representation, a self-organizing map network, and a convolutional network for face recognition. For the images in the training set, a fixed size window is stepped over the entire image and local image samples are extracted at each step. At each step the window is moved by 4 pixels. Next, a self-organizing map (e.g. with three dimensions and five nodes per dimension) is trained on the vectors from the previous stage. The SOM quantizes the 25-dimensional input vectors into 125 topologically ordered values. The three dimensions of the SOM can be thought of as three features. The SOM can be replaced with the Karhunen-Loeve transform. The KL transform projects the vectors in the 25-dimensional space into a 3-dimensional space. Next, the same window as in the first step is stepped over all of the images in the training and test sets. The local image samples are passed through the SOM at each step, thereby creating new training and test sets in the output space created by the self-organizing map. (Each input image is now represented by 3 maps, each of which corresponds to a dimension in the SOM. The size of these maps is equal to the size of the input image divided by the step size. A convolutional neural network, or alternatively a multilayer perceptron neural network, is trained on the newly created training set.

The self-organizing map provides a quantization of the image samples into a topological space where inputs that are nearby in the original space are also nearby in the output space, which results in invariance to minor changes in the image samples, and the convolutional neural network provides for partial invariance to translation, rotation, scale, and deformation. Substitution of the Karhunen-Lo‘eve transform for the self organizing map produced similar but slightly worse results. The method is capable of rapid classification, requires only fast, approximate normalization and preprocessing, and consistently exhibits better classification performance than the eigenfaces approach on the database considered as the number of images per person in the training database is varied from 1 to 5.

Face Detection/Gesture Detection

As discussed above, a parallelized implementation of convolutional neural networks (CNNs) is done with parallelizing the detection process using the GPU. This can be used for autofocus of the camera. Once the face is detected, the GPUs can also be used to detect gestures as commands. Motion features are first computed on the input image sequence (stationary camera assumed). The face detector is then employed to obtain a user-centric representation, and again a classifier to discriminate between gestures is learned using a variant of AdaBoost. A real-time version of this classifier is deployed using the GPU.

To calculate the motion features, the optical flow for each frame is determined. The optical flow vector field F is then split into horizontal and vertical components of the flow, Fx and Fy, each of which is then half-wave rectified into four non-negative channels Fx+, Fx?, Fy+, Fy?. A channel corresponding to motion magnitude F0 is obtained by computing the L2 norm of the four basic channels. These five non-negative channels are then normalized to facilitate gesture recognition in soft-real time where frame rates can be variable, and to account for different speed of motion by different users.

Given a vector v that represents the optical flow for a given pixel, the system computes v=v/(∥v∥+e), where e is used to squash optical flow vectors with very small magnitude introduced by noise. Next, each of the five channels is box-filtered to reduce sensitivity to small translations by the user performing the gesture. This final set of five channels: ?Fx+, ?Fx?, ?Fy+, ?Fy?, ?F0 will be used as the motion features for each frame.

A gesture is represented as a collection of movements required to complete a single phase of the gesture, rather than just capture a subset of the gesture phase. Hence, the system aggregates the motion features over a temporal history of the last k frames, for some k which is large enough to capture all frames from a gesture phase.

Face detection is used to create a normalized, user centric view of the user. The image is scaled based on the radius of the detected face, and is then cropped and centered based on the position of the face. The frame is cropped and resized to a 50×50 pixel region centered around the user. All five motion feature channels described above are flattened into a single vector which will be used to determine the gesture being performed.

A multi-class boosting process AdaBoost is used such as the one at http://multiboost.sourceforge.net. AdaBoost takes the motion features as input. The supervised training is based on a set of labeled gestures. A set of weak learners is generated based on thresholding value from a particular component of the motion feature vector. The output of the final strong learner on motion feature v for class label is determined using weights chosen by Ada Boost.

Panaroma Stitching

Referring back to the telepresence video conferencing system of?FIG. 2, the GPUs of?FIGS. 4-7?can perform panaroma stitching so that 3 inexpensive cameras can be used to provide a 180 degree immersive view. The GPU operations are done pipeline fashion as follows: Radial Distortion Correction. Next, the GPUs perform Keypoint Detection & Extraction (Shi-Tomasi/SIFT). Keypoint Matching is done, and the GPUs recover Homography (RANSAC). Next, the GPUs create a Laplacian Pyramid. A Projective Transform is done, and a Multi-Band Blend is the last stage of the pipeline.

Object Recognition

The CPU/GPU can identify video objects and then generate metadata for video objects contained within the captured video frames. For each of the captured video frames, the CPU/GPUs may execute program instructions to: (i) detect whether the video frame contains a video object, (ii) generate meta data for each detected video object, and (iii) cause data storage to store the relevant portions of the video and the metadata describing objects in the video for search purposes.

Generating the metadata may be carried out in various ways. For instance, generating the meta data may be carried out by segmenting video objects within a video frame and then representing features of each segmented video object. As an example, the feature representation may be color appearance information, that is, the analytical data may comprise color data based on the color or colors of a video object. The analytical data based on the color or colors of a video object may include any of a variety of color data. For example, for any given video object, the color data may include Red Green Blue (RGB) color space data, Hue Saturation Value (HSV) color space data, YCrCb color space data, and/or YUV color space data.

As another example, the metadata may include data based on pixel intensity, data indicating which pixels are part of the video object, a unique identifier of the video object, and/or structural information associated with the video object. The structural information may include information pertaining to edges, curvature, and/or texture of the video object, for example. The structural information may include information that indicates how close a structure of the video object matches a circle, rectangle, star, or some other arbitrary shape. The metadata may also include confidence measures of the other types of data in the metadata. The confidence measures may indicate a determination of how likely a video object is a given type of object, such as a vehicle, person, animal, bag, or some other type of object.

In one embodiment, the GPU detects an object using color and then tracks the object by:

• 1. Create a masking image by comparing each pixel with a target color value. Convert pixels that fall within the range to white, and convert those that fall outside the range to black.
• 2. Find the centroid of the target color. The centroid of the tracked color defines the center position for the overlay image. A multipass pixel-processing kernel is used to compute a location. The output of this phase is a 1×1-pixel image, containing the coordinate of the centroid in the first two components (pixel.rg) and the area in the third component (pixel.b). The area is used to estimate the distance of the object from the camera.
• 3. Composite an image over the detected object. Assuming the shape of the object does not change with respect to the frame, then the change in area of the tracked color is proportional to the square of the distance of the object from the viewer. This information is used to scale the overlay image, so that the overlay image increases or decreases in size appropriately.

In another embodiment, Shape-Based Matching (SBM) is used for recognizing partially occluded objects which is very robust against illumination changes. It is closely related to correlation. The main difference is that SBM uses the inner product of normalized edge gradient vectors as similarity measure instead of raw image intensity. To recognize an object, the search image is correlated with the model using the similarity measure of SBM. The model image is fitted in all possible translations within the search image and a score is assigned to each position. As correlation based methods do not cope with any other object transformation those must be handled by evaluating the results of multiple correlations.

In one embodiment, a Hough transform (HT) can be used to recognize any analytically describable shape]. A generalization of the Hough transform (GHT) who manages any discrete shape is also available [1]. Ulrich presents a modification of the generalized Hough transform (MGHT), gaining real-time performance by taking advantage of resolution hierarchies, which is not as straight forward as it might first seem. The principles of it are described below. Initially, a model of the object to be recognized is generated. A gradient map is generated from the model image and edges are extracted. Then a table, mapping edge gradient direction (represented as an angle) to offset vectors to the center of the model, is created.

In the search procedure a gradient map of the search image is used. For each gradient direction (represented as an angle) a number of displacement vectors are looked up using the table. A vote is placed on each position pointed at by a displacement vector. If the point belongs to an edge of the object, at least one of the vectors point at the center of the object that is to be found, and a vote will be placed for that position. After this is done, a peak has appeared in the voting space at the center of the object as this is the single point that the most dislocation vectors points at (at least one for each gradient belonging to the edge of the object in the search image).

To utilize a resolution pyramid the model image is divided into tiles for all levels but the coarsest. A separate lookup table is generated for each tile. Searching then begins with a regular search at the coarsest pyramid level which yields a coarse position of the object which will be refined for each level in the pyramid.

In the refinement, the entire gradient map is considered as before, but only the mapping table connected to the tiles that have dislocation vectors pointing to a neighborhood of the known position are evaluated for each gradient. This way the amount of votes to place, and the voting space itself is much smaller than before.

In another embodiment, a chamfer distance transform of an edge image is used as a model. Another edge image is extracted from the search image. To find the position of the object a similarity measure is evaluated for each potential position (as for cross correlation). The similarity measure is the mean distance from each edge in the search image to the closest edge in the model for the current translation. This is obtained by accumulating samples from the distance map of the model for each edge in the search image. If there is a perfect match, the mean distance is 0.

Other object recognition techniques can be used to tap into the GPU‘s parallel stream processing.

Cloud Based Parallel Video Search Engine

Preferably the video data can be operated in parallel using clusters of cloud based processors. Video analysis applications from anywhere in the world can, after authentication and security clearance, access and analyze video data that represents video streams in parallel, and annotate portions of the video data (e.g., frames and groups of frames), based on the analyses performed, with information that describes the portion of the video data. From an event stored in the database, the associated frames and/or groups of frames can be replayed for further human-based or application-based analyses. Applications plugged into the pipeline, via application program interfaces (APIs), can perform complex analyses and update the search engine in parallel.

The system identifies video objects and then generates metadata for video objects contained within the captured video frames. For each of the captured video frames, the CPU/GPUs may execute program instructions to: (i) detect whether the video frame contains a video object, (ii) generate meta data for each detected video object, and (iii) cause data storage to store the relevant portions of the video and the metadata describing objects in the video for search purposes.

The system can generate metadata live at the camera end, or alternatively can generate data using processors in the EC2 compute cloud operating on data stored in the S3 cloud data storage, in one embodiment. Generating the metadata may be carried out in various ways. For instance, generating the meta data may be carried out by segmenting video objects within a video frame and then representing features of each segmented video object. As an example, the feature representation may be color appearance information, that is, the analytical data may comprise color data based on the color or colors of a video object. The analytical data based on the color or colors of a video object may include any of a variety of color data. For example, for any given video object, the color data may include Red Green Blue (RGB) color space data, Hue Saturation Value (HSV) color space data, YCrCb color space data, and/or YUV color space data.

As another example, the metadata may include data based on pixel intensity, data indicating which pixels are part of the video object, a unique identifier of the video object, and/or structural information associated with the video object. The structural information may include information pertaining to edges, curvature, and/or texture of the video object, for example. The structural information may include information that indicates how close a structure of the video object matches a circle, rectangle, star, or some other arbitrary shape. The metadata may also include confidence measures of the other types of data in the metadata. The confidence measures may indicate a determination of how likely a video object is a given type of object, such as a vehicle, person, animal, bag, or some other type of object.

A user can search a video database by providing examples of the people and/or objects for whom they are looking. This is sometimes referred to as query by example. One use of the system is for determining actions taken before an event. For example, by examining the video data recorded by a security system in a place of business over the course of a week, one can determine the routes taken through a building by a particular person, or develop a list of all people someone interacted with while in a building that particular week.

In one approach discussed in US Pat. 20080204569, the content of which is incorporated by reference, a covariance matrix based appearance model is used. Specifically, this covariance matrix appearance model is used to query both within a single camera and with multiple cameras. There are several advantages of the covariance appearance model. First, it can efficiently fuse many heterogeneous features. Second, it does not require motion information of objects and third can handle rigid and non-rigid objects observed by non-stationary cameras. Therefore it is robust to objects‘ pose and illumination changes which can occur when tracking across different cameras. This is particularly advantageous when the video repository contains video data from non-overlapping cameras, where the views of the different cameras can be different and the appearance of the objects might vary greatly in scale, pose, and shape. In one particular embodiment, the system performs its query of the video data based on a blob appearance model. In this embodiment, an appearance model for a blob is first defined. Since a goal of such a system is to support un-calibrated non-overlapping cameras, the appearance models have to be robust to changes in color, scale, pose, and other similar appearance factors.

In one embodiment, the elastic cloud compute (EC2) provides a plurality of processor to search for objects in the captured videos. Each video object is included in only one frame. In this regard, each video object is unique. However, multiple video frames may include a video object that represents a particular object captured on the video frames, and the multiple video frames may be captured by one or more video cameras. A particular object captured on a video frame may include a person, a group of people, an item carried by a person or the group of people, an animal, a vehicle, or some other arbitrary object that may be captured on a video frame (e.g., any object within a video frame). Analytical data may be determined for video objects that are detected as being in motion at the time the video objects are captured. Any video object of interest to a user may be selected by the user while a video frame comprising that video object is being displayed.

One embodiment uses Amazon‘s SimpleDB which stores items in domains. Items contain one or more attribute-value pairs. Each attribute can have multiple values. Queries let users perform range & equality operators (including ‘starts-with‘) comparing attribute values to constant values. Queries can have multiple conditions on a given attribute, and can union and intersect sets of results across different attributes. Queries return paged result sets of item names. Item names can then be pulled in parallel from SimpleDB to get the full set of attribute-value pairs for each item. An Inverted Index on SimpleDB is done that supports ‘sharding‘. Ferret‘s DefaultAnalyzer tokenizes the input and filter terms. The overall goal is to build an inverted index of terms that will let the system quickly looks up which video sequences contain which features. The system has a textual index running in the cloud with shards, fields, suffix matching, and exact phrase matching, with boolean operator.

Amazon SimpleDB is a web service for running queries on structured data in real time. This service works in close conjunction with Amazon Simple Storage Service (Amazon S3) and Amazon Elastic Compute Cloud (Amazon EC2), collectively providing the ability to store, process and query data sets in the cloud. These services are designed to make web-scale computing easier and more cost-effective for developers.

Amazon SimpleDB service was built to support massively parallel operations. To improve the put performance, the system can distribute the data among multiple threads, or processes, that execute BatchPutAttributes calls in parallel. The improvement is proportional to the number of threads. Parallelizing queries by splitting the search range can also be done. The pattern for executing parallel queries that are independent from each other is very similar to executing any other request: the application spawns multiple threads or processes, and each one is responsible for processing Select calls against Amazon SimpleDB independently. In a use case for parallelizing queries that involves retrieving a large number of pages for a single query expression, the system splits the search range into multiple segments to parallelize across these segments.

Building applications that work well in a networked environment with unpredictable latencies opens a new set of potential error conditions when compared to typical desktop or client-server applications. Invoking requests to a web service such as Amazon SimpleDB, whether over the Internet or from an EC2 instance within the Amazon cloud, requires application designers to consider issues such as network connectivity, request timeouts and remote errors due to overload or throttling. There are numerous components on the network, such as DNS servers, switches, load-balancers, and others that could generate errors anywhere in the life of a given request.

Once a request arrives at the service, Amazon SimpleDB uses standard HTTP error codes to communicate server and overload errors through the REST interface. One technique for dealing with such error responses in a networked environment is to implement retries in the client application layer, regardless of whether those errors come from the service itself or any component on the network between Amazon SimpleDB and client application. This technique increases the reliability of the applications and reduces operational costs for the developer. In addition to simple retries, we an exponential backoff algorithm can be used for better flow control. Exponential backoff uses progressively longer waits between retries for consecutive error responses: up to 400 milliseconds before the first retry, up to 1600 milliseconds before the second, up to 6400 milliseconds before third, and so on. The exponential back off algorithm is as follows:

A user can perform a "seed search" (i.e., a search of a subset of the analytical data) to identify video objects that most closely match the selected video object. The identified video objects that most closely match the selected video object may include the selected video object. The subset of analytical data may comprise analytical data for which there is a high probability that the data will correspond to video objects that most closely match the selected video object.

A "complete search" (i.e., a search of the entire set of analytical data) may be carried out for each of the video objects identified during the seed search so as to identify video objects that most closely match the video objects identified during the seed search. Thereafter, the video frames comprising the video objects identified during the complete searches may be displayed at a display device.

After performance of the seed search and the complete searches, a user may view video frames in which the selected video object appears and/or video frames in which a video object identified as one of a plurality of video objects most closely matching the selected video object appear, without having to view video frames in which the selected video object or video objects identified as most closely matching the selected video object do not appear. In this way, the user does not have to waste time viewing video frames that are not of interest to the user.

Additionally, while viewing the video frames comprising video objects detected during a seed search and/or a complete search, the user may select another video object displayed in a viewed video frame. The other video object may comprise an object that interacts with the previously selected video object. Thereafter, the system may perform a seed search for the other video object so as to detect video objects that most closely match the other video object, and then perform one or more complete searches for video objects that most closely match the detected video object that most closely matches the other video object. Video frames comprising the other video object and/or video objects that most closely match the other video object may be displayed at the display device.

Although parallel video searching is described above, alternatively, the cloud based processors can be configured as a sequential processing systems where video analysis applications can access and analyze video data that represents video streams flowing through the pipeline, and annotate portions of the video data (e.g., frames and groups of frames), based on the analyses performed, with information that describes the portion of the video data. These annotations flow through the pipeline, possibly along with corresponding frames or groups of frames, to subsequent stages of processing, at which increasingly complex analyses can be performed. Analyses performed at the various stages of the pipeline can take advantage of the analyses performed at prior stages of the pipeline through use of the information embodied in the annotations. At each stage of the pipeline, portions of the video streams determined to be of no interest to subsequent stages are removed from the video data, which reduces the processing requirements of the subsequent stages.

From an event stored in the database, the associated frames and/or groups of frames can be replayed for further human-based or application-based analyses. Generally, as the videos flow down the pipeline, (1) portions of the videos or frames that are considered uninteresting to all the applications at a given stage are removed, thereby reducing the amount of data that flows further down the pipeline; (2) portions of the videos or frames that are considered interesting to an application at a given stage are analyzed, with a goal of identifying features, activities, objects, etc. of interest; and (3) analyzed portions of the videos or frames are annotated by the applications with information that describes what the applications identified as interesting in that portion of the video. In one embodiment, the pipeline comprises four different successive stages of processing: (1) quick frame processing; (2) deep frame processing; (3) cluster processing; and (4) database processing. Due to the nature of the pipeline, applications plugged into the pipeline, via application program interfaces (APIs) associated with each respective stage, can perform increasingly more complex analyses at each successive stage of processing.

GUI for Parallel Search

In?FIG. 10, video objects contained in a video frame are identified by a rectangle, and a selected video object in a video frame is identified by an ellipse, such as ellipse?450?shown in?FIG. 3. In particular, a video object identified by a solid-line rectangle indicates that video object has been identified as a video object that most closely matches the selected video object, whereas a video object identified by a dashed-line rectangle indicates that video object has not been identified as a video object that most closely matches the selected video object.

The rectangles and ellipses are not part of the video frame captured by a video camera but may be overlaid on the video frame when the video frame is displayed via GUI?400. Other means for indicating the presence of a video object in a video frame may also be used, such as an outline of another shape (e.g., a circle, a polygon, and/or an outline of the video object) and other means for indicating the presence of a selected video object may also be used. The rectangle or other shape may indicate an outer boundary of all the pixels of the video object such that the video object includes all of the pixels within the shape.

In?FIG. 10, video frame 1, 009 is displayed in video frame window?402, video frame 2, 000 is displayed in video frame window?404, video frame 3, 006 is displayed in video frame window?406, and video frame 4, 009 is displayed in video frame window?408. Video frame 1, 009 includes video objects 15, 16, 26, 27. Video frame 2, 000 includes video objects 111, 137, 138. Video frame 3, 006 includes video objects 231, 232, 233, 247, 248. Video frame 4, 009 includes video objects 335, 336. Video object 335 is a selected video object as indicated by ellipse?450. In?FIG. 3, video objects 15, 111, 231, and 335 have been identified as video objects most closely matching the selected video object 335 in video frames 1, 009, 2, 000, 3, 006, and 4, 009, respectively. The user can select each video object in?FIG. 10?to be searched by drag-and-drop a rectangular region and the GUI shows a dashed line rectangle in response. After performing the search for video objects that most closely match the selected video object, each video frame displayed at GUI may include a video object identified by a solid line rectangle indicating that video object most closely matches the selected video object. Alternatively, if no video objects in video frames captured by a given video camera are identified as most closely matching the selected video object, then the video frame window?402?for the given video camera may provide an indication that no video frames were captured by the given video camera. The indication may be a blank video display window, a video frame test pattern, or some other indication.

A given object may appear in a non-contiguous series of video frames captured by a single video camera. Thus, a search for video frames that include video objects matching the given object may result in identifying video objects from a non-contiguous series of video frames captured by one or more video cameras.

SRC=http://www.google.com/patents/US20140085501