METHOD AND DEVICE FOR THE SELECTION OF A STABILIZATION METHOD

FIELD OF THE INVENTION

The invention relates to the evaluation and the selection of the stabilization quality of a video sequence stabilization method applied on a video sequence, and to a corresponding stabilization device incorporating said evaluation and selection method.

BACKGROUND OF THE INVENTION

Video footage from hand-held camcorders is typically jerky due to small, unwanted camera movements. Removal of those undesired movements requires video stabilization techniques. As its name suggests, video stabilization is the process of generating a compensated video sequence where image motion by the camera's undesirable shake or jitter is removed. Digital image processing techniques are often used to perform such a task and are preferred over mechanical or optical video stabilization approaches since modern VLSI techniques allow a more compact camera design.

To estimate the quality of different stabilization methods, stabilized versions obtained from the application of different stabilization methods to the same original sequence, are watched by viewers who decide which one they prefer. The decision criterion is the visual comfort. This method allows a fine ranking of the stabilization methods. However, subjective assessment of viewer is highly time consuming and expensive.

Evaluation of the perceptual quality of stabilization method is also performed with the help of computers. The Interframe Transformation Fidelity (ITF) method can, for example, be used. This method computes the Pick Signal to Noise Ratio (PSNR) between consecutive images of the stabilized sequence. Another possible method is the Global Transformation Fidelity (GTF) method. This method calculates the PSNR between the current image and a reference image, for example the first image when the stabilization starts. These Interframe Transformation Fidelity (ITF) and the Global Transformation Fidelity (GTF) methods are simple to implement and to understand. However, they are biased when local motion exists in the image or when the images differ with lighting conditions. They are also biased when jitters are simultaneous with an intentional global motion during, for example, a zoom movement of the camera. In addition, they cannot differentiate the quality of a motion extraction and a motion compensation. For example, the use of different interpolation filters to generate the

stabilized sequence would lead to different PSNR, which are not necessarily linked to the quality of the stabilization method. Further, the ITF and the GTF methods do not give an indication about the smoothness of the global motion of the sequence from image to image. Therefore, it is desirable to develop a new evaluation method to estimate and select a stabilization method among the existing stabilization methods and to provide an estimation quality factor as close as possible to human perception that is not influenced by the global motion of the image.

SUMMARY OF THE INVENTION

Accordingly, it is an object of the invention to provide an improved method to evaluate and select the quality of a stabilization method.

To this end, the invention relates to a method to evaluate and select the stabilization quality of at least a stabilization method applied to an original video sequence composed of several images to obtain a stabilized video sequence, the evaluation method being adapted to assign a quality factor (Q) to the stabilization method according to the quality of the obtained stabilized video sequence, wherein the evaluation method comprising the following steps:

- extracting a set of initial global motion vectors (GMV°2, GMV ⁰ *) from the original video sequence ;

- determining a set of initial energy quantities (W°(a), W°(b), W°(c)) per predefined frequency band (a, b, c), from the energy spectrum of the set of initial global motion vectors (GMV° ₂ , GMV° ₄ ) ;

- stabilizing the original video sequence using the stabilization method, to obtain the stabilized video sequence ;

- extracting a set of final global motion vectors (GMV ^f io, GMV ^f i2) from the stabilized video sequence ;

- determining a set of final energy quantities (W ^f (a), W ^f (b), W ^f (c)) per predefined frequency band (a, b, c), from the energy spectrum of the set of final global motion vectors (GMV ^f i ₀ , GMV ^f ₁₂ );

- determining a set of energy differences (W (a), W (b), W (c)) per predefined frequency band (a, b, c), each energy difference (W ^d (a), W ^d (b), W ^d (c)) resulting from the subtraction for each predefined frequency band (a, b, c) of a final energy quantity (W ^f (a), W ^f (b), W ^f (c)) to the initial energy quantity (W°(a), W°(b), W°(c)); and

- determining the quality factor (Q) featuring the stabilization method, wherein the step of determining the quality factor (Q) includes a step of determining a motion component, the motion component being proportional to a weighted sum of the energy differences (W ^d (a), W ^d (b), W ^d (c)) of the set of energy differences (W ^d (a), W ^d (b), W ^d (c)). Since the evaluation and selection method computes the energy difference of the energy spectrums of the original video sequence and of the stabilized video sequence, the quality factor is not influenced by the global motion of the sequence.

Other features and advantages of the method are recited in the dependent claims. In addition, the invention relates to a device to evaluate and select the stabilization quality of at least a stabilization method applied to an original video sequence composed of several images to obtain a stabilized video sequence, the evaluation method being adapted to assign a quality factor (Q) to the stabilization method according to the quality of the obtained stabilized video sequence, wherein the device comprises:

- motion estimation means adapted to extract a set of initial global motion vectors (GMV° ₂ , GMV° ₄ ) from the original video sequence and a set of final global motion vectors (GMV o, GMV ₁₂ ) from the stabilized video sequence ;

- energy analysis means adapted to determine a set of initial energy quantities (W°(a), W°(b), W°(c)) by predefined frequency band (a, b, c), of the energy spectrum of the set of initial global motion vectors (GMV°2, GMV°4) and a set of final energy quantities (W ^f (a), W ^f (b), W ^f (c)) by predefined frequency band (a, b, c) of the energy spectrum of the set of final global motion vectors (GMV ^f io, GMV ^f i ₂ );

- computing means adapted to determine a set of energy differences (W ^d (a), W ^d (b), W ^d (c)) by frequency band (a, b, c), each energy difference (W ^d (a), W ^d (b), W ^d (c)) resulting from the subtraction of a final energy quantity (W (a), W (b), W (c)) of a defined frequency band (a, b, c) to the initial energy quantity (W°(a), W°(b), W°(c)) of the same frequency band (a, b, c); and

- the computing means being adapted to determine a quality factor (Q) characterizing the evaluated stabilization method, wherein the step of determining the quality factor (Q) includes a step of determining a motion component, the motion component being proportional to a weighted sum of the energy differences (W ^d (a), W ^d (b), W ^d (c)) of the set of energy differences (W ^d (a), W ^d (b), W ^d (c)).

The invention also relates to a device for digital image stabilization for removing unwanted camera movements, called jitter, from an original sequence of images generated by said camera and obtaining a stabilized sequence of images, said device incorporating a stabilization quality metric based on a previous evaluation and selection of stabilization quality which are performed in a device according to claim 12.

These and other aspects of the invention will be apparent from the following description, drawings and from the claims.

BRIEF DESCRIPTION OF THE DRAWINGS - Figure 1 is a schematic block diagram of a stabilization method to be tested;

- Figure 2 is a block diagram of a device according to the invention, which allows to estimate the stabilization quality of a stabilization method;

- Figure 3 is a flow chart of a method according to the invention, to estimate the stabilization quality of a stabilization method; - Figure 4 is a block diagram of an example of a set of motion vectors associated to macroblocks of an image of an original video sequence; and

- Figure 5 is a graph representing the energy spectrum of the set of initial global motion vectors of the original video sequence.

DETAILED DESCRIPTION

Referring to Figure 1, a method and device for the evaluation and the selection of a stabilization method is illustrated. This stabilization method also referenced 1 is applied to a set of consecutive images 2, 4, 6, etc, of an original video sequence 8 (obtained at the output of a camera) to obtain a set of consecutive images 10, 12, 14, etc, of a stabilized video sequence 16.

Referring to Figure 2, a device 15 according to the invention is adapted to evaluate the stabilization method 1. It comprises a motion estimation block 17 connected to an energy analysis block 19, and an image analysis block 21, and the energy analysis block 19 and to the image analysis block 21 are themselves connected to a computing block 23. The device 15 is adapted to receive both the original sequence 8 and the stabilized sequence 16 in order to evaluate the quality of the stabilization method 1 and to deliver a quality factor Q featuring this quality. The highest quality factor is assigned to the stabilization method having an image quality similar to that obtained when the camera is stable or unshaken.

To this end, the device 15 performs an evaluation method 18 (see Fig.3) according to the invention. To obtain efficient result, the evaluation method 18 is applied to video sequences 8, 16 comprising between 500 and 700 images.

As illustrated in Figures 2 and 3, the evaluation method 18 comprises a first step 20 of extracting a set of initial global motion vectors GMV° ₂ , GMV° ₄ , GMV° ₆ , GMW° ₈ , etc, from the original video sequence 8. This step is performed in Fig.2 by the motion estimation block 17. To obtain an initial motion vector of the two first images 2, 4 of the original sequence 8, different methods can be used.

One of them consists in considering the image 2 and its subsequent image 4 during a sub-step 22 and determining a set of motion vectors V _k of macroblocks 24 of these images. As illustrated in Figure 4, each motion vector Vk represents the movement of the scene from one image 2 to the subsequent image 4, in each macroblock 24. Typically, each macroblock 24 comprises 16x16 pixels of the image.

In a sub-step 26, the median GMV°2 of the set of motion vectors Vk is determined. At the first iteration, this median motion vector is the initial global motion vector GMV° ₂ and represents the global movement of the scene realized between images 2 and 4.

Once the initial global motion vector GMV°2 of the considered images 2, 4 is determined, the sub-steps 22 and 26 are repeated for all images of the original sequence 8 to obtain a set of initial global motion vectors GMV°2, GMV° ₄ , etc. They are repeated for the 700 images of the original sequence to obtain a set of 699 global motion vectors, and more generally for all images of the original sequence, where the sequence have more or less frame than 700.

For reason of simplification, in the following the method is described only in connection to the global motion vectors GMV° ₂ and GMV° ₄ but it is applied to the 699 global motion vectors GMW°i with 1=2, 4, 6, 8, etc, determined in the sequence.

Each initial global motion vector comprises an horizontal u° ₂ and a vertical v° ₂ components which are the projection on the horizontal and respectively vertical axis of the global motion vector GMV°2. GMV°4 = (u°4, v° ₄ )

The evaluation method 18 affects to each tested stabilization method 1 a quality factor Q that comprises a motion component and an image component.

During steps 28 to 42, the motion component of the quality factor is calculated by performing an analysis of the energy spectrum of the original 8 and the stabilized 16 sequences.

In the step 28, an energy analysis is performed by the energy analysis block 19, on both horizontal u° ₂ and vertical v° ₂ components of the set of initial global motion vectors

The transformation of each component of the initial global motion vectors GMV° _2, GMV° ₄ , from the time domain to the frequency domain is made for example with a Cooley-Tukey Fast Fourier Transform algorithm (FFT) using the Danielson-Lanczos Lemma.

From the FFT transform of the horizontal u°2 and the vertical v°2 components of the initial global motion vectors, the energy spectrum is obtained using the following relation:

W°(F) = Re (U(F)) ² + Im (U(F)) ² + Re ((V(F)) ² + Im (V(F)) ² where: - U(F) is the FFT transform of the horizontal component u ⁰ ;

- V(F) is the FFT transform of the vertical component v° ; and

- "Re" is the real part and "Im" the imaginary part of the complex numbers resulting of the FFT transform.

Thereafter, W°(F) (as well as all other energy values) is expressed in decibels, and V is a non-negative value, its value in decibels Vdb is given by: Vdb = 10 log V.

Figure 5 shows the graph representing the evolution of the energy in function of the frequency, namely the spectrum W°(F), obtained from the FFT transform of the set of the initial global motion vectors.

During step 30, a set of initial energy quantities W°(a), W°(b), W°(c) are computed from the energy spectrum W°(F). Each initial energy quantity W°(i) corresponds to the integral of the energy of the energy spectrum W°(F) over a predefined frequency band i= a, b, c.

Besides, during step 32, the stabilization method 1 to be evaluated, is applied to the original sequence 8 to obtain the stabilized sequence 16. At step 34, a set of final global motion vectors GMV ^f io, GMV ^f i ₂ are extracted from the stabilised sequence 16 by the motion estimation block 17 and using the same method as the one used at step 20.

At step 36, a set of final energy quantities W (a), W (b), W (c) is determined from the energy spectrum W ^f (F) of the set of final global motion vectors GMV ^f io, GMV ^f i2 etc, using the same method as the one explained at steps 28 and 30.

At step 38, a set of energy differences W ^d (i) is computed. Each energy difference results from the subtraction of a final energy quantity W (i) of a defined frequency band i to the initial energy quantity W°(i) of the same frequency band i, as shown for example with the following expressions : W ^d (a) = W°(a) - W ^f (a) W ^d (b) = W°(b) - W ^f (b) As explained later in the description, the motion component of the quality factor Q is proportional to a weighted sum of the set of energy differences W ^d (a), W ^d (b) , W ^d (c) over each predefined frequency band a, b, c.

During steps 40 and 42, a set of frequency-based weighting coefficients of the motion components are determined by the computing block 23. At step 40, a couple comprising an inside band energy coefficient "Energy_in_i" and an outside band energy coefficient "Energy_out_i" is computed for each frequency band i.

For example, the frequency band a is considered. The inside band energy coefficient "Energy_in_a" of the band a, is equal to the final energy quantity W°(a) of the frequency band a. The outside band energy coefficient "Energy_out_a" of the band a, is equal to the sum of the initial energy quantity W°(i) of all frequency bands i of the energy spectrum W°(F), at the exception of the band a. In the above mentioned example, Energy_out_a = W°(b) + W°(c).

The inside band energy coefficient and the outside band energy coefficient are calculated for all predefined frequency bands a, b, c to obtain a couple of coefficients per predefined frequency band a, b, c. Each couple is associated to a frequency band and is one of the frequency -based weighting coefficients.

At step 42, a set of relative inside "AJnJ" and the set of relative outside "A_out_F band energy coefficients are determined by the computing block 23, for each frequency band i, using the following equation :

Energyjnj _ Energy_ouji

- ^{iπJ ~} Total_enegy - ^{0UtJ ~} TotaTenegy

where the "Total_energy" is the integral over all frequency band i of the energy spectrum W(F) of the set of initial global motion vectors and where "Energy_in_i" and "Energy_out_i" are the inside band and respectively the outside band energy coefficients.

During steps 44 to 50, the image component of the quality factor Q is calculated by the image analysis block 21, performing image analysis on the original sequence 8.

At step 44, a global inner motion coefficient IM is computed. This coefficient IM constitutes an image-based weighting coefficient and represents the global motion within the scene. This coefficient is computed, for example, by calculating the disparity of motion vectors V _k inside the image using the following equation : m 1 ⁿ

= — σ -∑ abs(V k - GMV .° ;

where:

- GMV° _j is the global motion vector of the image j computed at step 26, by the motion estimation block 17;

- V _k is the motion vector of one macro block 24 of the image j of the original sequence 8 computed at sub-step 22, by the motion estimation block 17 ;

- "abs" is the absolute value;

- "n" is a number of macro blocks 24 in an image j which corresponds to the number of motion vectors in this image; and

- "m" is a number of images.

At step 46, a mean texture rate of each image 2, 4, 6 is determined using a classical texture analysis method. An average TI of the mean of the texture rates determined for all images is computed. The average texture rate TI constitutes an image-based weighting coefficient.

At step 48, an Interframe Transformation Fidelity (ITF) is computed. This index is a standard Pick Signal to Noise Ratio (PSNR). It is determined on the luminance data between two consecutive images.

2

255

PSNR(lk,lk + i)=10. |og _{1 0}

#pixels 4λ ^a 'i ^b 'i where

- a _y and b _y are the luminance data of each pixel of the current image and of the corresponding pixel of the consecutive image,

- "# pixels" is the number of pixels in the image. The ITF is given by :

.. nb_image-1

ITF = y (PSNR(l _K ,l _K+1 ))

(nb_image - 1 ) ^ v κ ^> κ.+ ⁱ "

where "nb_image" is the number of images in the sequence 8.

The ITF is computed for each image of the original sequence 8 and the median value of the computed ITF is used for the determination of the quality factor Q. At step 50, the quality factor Q is computed from the following equation:

Q = V UxO ₁ + (Xl ₁ * A + α2 _j * A _{out l} + α3 _j * IM + α4 ₁ * TI U lTF ie frequency_ bands V /

TpO ₁ + βl _x * Energy_in_ i H- β2 ₁ * Energy_out _i ^j ⁺ σ + PS ^ A _{111 1} + β4 ₁ * A _{out l} + β5 ₁ * IM + β6 ₁ * TI * ^{w d} ( ⁱ )

IG frequency_ bands I J where:

- the factors βθ _1; βli, β2 _b β3 _b β4 _1; βS^re constant function of the frequency band i;

- the factors CtO ₁ , CtI ₁ , α2 _b α3 _1; ct^ are constant function of the frequency band i; - "Energy_in_i" is the inside band energy coefficient for the frequency band i;

- "Energy_out_i" is the outside band energy coefficient for the frequency band i;

- "AJnJ" is the relative inside band energy coefficient (= Energy_in_z / Total energy);

- ^u A_outJ" is the relative outside band energy coefficient (= Energy_out_z / Total energy); - ITF is the median value of the computed Interframe Transformation Fidelity index;

- IM is the global inner motion coefficient; and

- IT is the average for all images of the mean of the texture rates determined.

The factors βθ _1; βli, β2 _b β3 _b β4 _1; β5j and CtO ₁ , CtI ₁ , a2 _u α3 _1; ct^ are determined empirically, during a training phase, by viewing several stabilized sequences 16 obtained from performing a stabilization method 1 on original sequences 8 showing different types of scene. Once the factors determined, they are fixed and used to determine the quality factor Q of other stabilization method to evaluate.

The method 18 according to the invention can be used in a particular case where the energy spectrum W°(F) is only divided into two energy bands i, one for high frequency, for example above 1 Hz, and one for low frequency, for example below 1 Hz. In such cases, the above equation is given by:

Q = (αO + αl * A _Below + α2 * A _Above + α3 * IM + α4 * Tl) * ITF + (βO + βl * A _Below + βl * A _Above + β2 * IM + β3 * Tl ) * redHF with ⁺ ^° ^{+ γl} * ^{ABelow + βl} * ^{A Above + γ2} * ^{IM + γ3} * ^TI ^ * ^redLF

_ HF_Energy _ LF_Energy

^{Above "} HF_Energy ₊ LF_Energy ^{Below "} HF_Energy ₊ LF_Energy ^{" " Above}

and "HF_energy", "LF_energy" are absolute levels of high and low frequency energy, and redHF is the energy reduction in the high frequency band and redLF is the energy reduction in the low frequency band.

This method can be implemented in hardware on a micro-controller or in software on a logiciel support executed by a microprocessor from a non volatile memory of the ROM (Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory) kind or an equivalent. The method of the invention can also be carried out by a computer program for a processing unit comprising a set of instructions, which, when loaded into said processing unit, causes the processing unit to carry out the method described above. There are indeed numerous ways of implementing functions by means of items of hardware or software, or both. In this respect, the drawings are very diagrammatic, each representing only one possible embodiment of the invention. Thus, although a drawing shows different functions as different blocks, this by no means excludes that a single item of hardware or software carries out several functions. Nor does it exclude that an assembly of items of hardware or software or both carry out a function.

The remarks made herein before demonstrate that the detailed description, with reference to the drawings, illustrates rather than limits the invention, and that numerous alternatives, which fall within the scope of the appended claims, are possible. Any reference sign in a claim should not be construed as limiting the claim. The word "comprising" does not exclude the presence of other elements or steps than those listed in

a claim. The word "a" or "an" preceding an element or step does not exclude the presence of a plurality of such elements or steps.