BILATERAL HIGH FREQUENCY BOOST
Methods and apparatuses for video processing. In one aspect, filtering is applied after applying a set of offset values to one or more coefficients. The application of an offset value may be based on a determination as to whether the coefficient meets a threshold value.
Latest Telefonaktiebolaget LM Ericsson (publ) Patents:
 TECHNIQUE FOR CONFIGURING A RANDOM ACCESS PROCEDURE
 FAIRNESS AND COMPLEXITY CONSTRAINED UPLINK TRANSMISSION USING COORDINATED MULTIPOINT (COMP) NONORTHOGONAL MULTIPLE ACCESS (NOMA)
 NETWORK DEVICE, TERMINAL DEVICE, AND METHODS THEREIN
 A NETWORK NODE AND METHOD IN A WIRELESS COMMUNICATIONS NETWORK
 METHODS, APPARATUSES, COMPUTER PROGRAMS AND COMPUTER PROGRAM PRODUCTS FOR DATA STORAGE
This disclosure relates to apparatuses and methods for the processing of media objects, such as videos. Some aspects of this disclosure relate to apparatuses and methods for encoding, decoding, and/or filtering one or more components of a video object.
BACKGROUNDBilateral filtering of image data directly after forming the reconstructed image block can be beneficial for video compression. As described by Wennersten et al., in “Bilateral Filtering for Video Coding”, IEEE VCIP '17 , Dec. 1013, 2017, it is possible to reach a bit rate reduction of 0.5% with maintained visual quality for a complexity increase of 3% (encode) and 0% (decode) for random access. Other forms of filtering are used with respect to video images as well.
However, there remains a need for effective processing to offset certain downsides of filtering. This includes, for example, offsets for bilateral filtering.
SUMMARYAccording to some embodiments, a method for processing a media object is provided. The method may include applying a set of offset values to a set of coefficients associated with the media object, wherein each offset value of the set of offset values corresponds to a separate coefficient of the set of coefficients. In certain aspects, applying the set of offset values comprises: (1) determining whether a coefficient value is larger than a positive threshold or smaller than a negative threshold, and (2) based on the determination, adding a corresponding offset value to the coefficient value when the coefficient value is larger than the positive threshold or subtracting the corresponding offset value from the coefficient value when the coefficient value is smaller than the negative threshold. After applying the set of offset values, method further includes performing a filtering operation on the media object.
According to some embodiments, a method for processing a media object is provided. The method may include determining an initial offset value for at least one coefficient of a set of coefficients associated with the media object, and determining an adjusted offset value for the coefficient. In certain embodiments, the adjusted offset value is determined based at least in part on the initial offset value and a modification factor. In certain embodiments, determining the initial offset value and determining the adjusted offset value is based on determining that a value of the coefficient meets or exceeds a threshold. In some instances, the method may include further steps such as: generating the set of coefficients by performing an inverse quantization process on a plurality of quantized values associated with the media object; applying the adjusted offset value to the coefficient to generate a modified coefficient; performing an inverse transform using the modified coefficient; and performing a filtering operation. The filtering operation may comprise, for example, one or more of bilateral filtering, deblocking filtering, adaptive loop filtering, and sample adaptive offset filtering.
According to embodiments, there is provided a method for applying filtering to a media object comprising a plurality of samples. The method includes applying a set of offset values to a set of coefficients, wherein each offset value of the set of offset values corresponds to a separate coefficient of the set of coefficients. In some embodiments, the step of applying the set of offset values includes determining whether a coefficient value is larger than a positive threshold or smaller than a negative threshold, and applying a corresponding offset value to the coefficient value based on the determination. If the coefficient value is larger than the positive threshold, then the method adds the corresponding value to the coefficient value, and if the coefficient value is smaller than the negative threshold, the method subtracts the corresponding value from the coefficient value. In some embodiments, the method includes a further step in which the number of coefficients in the set of coefficients having a value (i) larger than the positive threshold or (ii) smaller than the negative threshold is determined. A factor value can be determined based on the determined number of coefficients, which can affect the amount of offset. The method can also include applying filtering after applying the set of offset values.
Also, and in certain aspects, the step of adding the corresponding offset value to the coefficient value includes multiplying the corresponding offset value, a determined factor value, and/or a quantization factor to obtain a multiplied offset value and adding the multiplied offset value to the coefficient value. Similarly, the step of subtracting the corresponding offset value from the coefficient value can include multiplying the corresponding offset value, the determined factor value, and/or a quantization factor to obtain a multiplied offset value and subtracting the multiplied offset value from the coefficient value.
In some embodiments, after dequantization, coefficients are modified by constants to be further closer to or further away from zero. The size of the modification can be based, for example, on the number of nonzero coefficients.
According to some embodiments, an encoder or decoder is provided that is configured to perform one or more of the methods of offset and filtering methods.
According to some embodiments, a computer program product is provided, which comprises a nontransitory computer readable medium storing instructions that when performed by processing circuitry of a video processing device, causes the device to perform one or more the foregoing methods.
According to some embodiments, a video processing device (e.g., an encoder or decoder) is provided. The device may include a memory and a processor, wherein the processor is configured to perform one or more of the foregoing methods.
The accompanying drawings, which are incorporated herein and form part of the specification, illustrate various embodiments.
Although a bilateral filter is an adaptive filter, it still has much in common with a regular lowpass filter. In particular, a bilateral filter on average acts as a lowpass filter by reducing high frequencies. Because frequencies are correct—on average—before filtering, this reduction in high frequencies can be a problem. According to embodiments, this and other issues may be addressed by boosting high frequencies before filtering, such that the frequencies will be correct after the filtering is applied. In certain aspects, by boosting the high frequencies before filtering, the benefits of the filtering are obtained while maintaining correct average values for all frequencies. Additionally, and in accordance with some embodiments, by boosting the high frequencies before rather than after applying the filtering, the boost to the high frequencies may be applied in the frequency domain while the filtering is applied in the pixel domain after the inverse transform. In some instances, compression efficiency may be improved by approximately 0.1%. Although illustrated with bilateral filtering as an example, embodiments may implement other forms of filtering.
Referring now to
The values of
Referring now to
According to embodiments, only nonzero coefficients in a block should be changed. If secondary transforms are used, however, coefficients between 100*QF and −100*QF may be processed as if zero. A version of the Versatile Video Coding (VVC) codec includes what is referred to a secondary transform. Without the secondary transform, the decoder performs the following: decode coefficients>inverse quantize coefficients using QP>inverse transform rows (using a primary transform)>inverse transform columns (also using a primary transform)>add prediction. In this instance, different primary transforms can be used for the rows and the columns. As an example, the rows could have been transformed with DCT2, and the columns with DST7.
However, with a secondary transform, part of the block can be transformed a second time in the encoder. Instead of transforming rows and columns individually, the transform can act on the 2D coefficients simultaneously, i.e., it is nonseparable. Since the decoder does everything in reverse, the inverse secondary transform actually comes before the primary transforms in the decoder. Hence, a codec that uses secondary transforms can operate according to: decode coefficients>inverse quantize coefficients using QP>inverse transform 2D block (using a secondary transform)>inverse transform rows (using a primary transform)>inverse transform columns (also using a primary transform)>add prediction. Since a nonseparable 2D transform is often more expensive than a separable one (that acts on rows and columns separately), only part of the block is typically transformed using the secondary transform. As an example, if the block consists of 16×16 coefficients, perhaps only the top left 4×4 coefficients (corresponding to the lowest frequencies) are treated with the secondary transform. In embodiments, coefficients with values between 100*QF and −100*QF may be processed as if zero.
According to some embodiments, the amount of offset used may be based, at least in part, on the number of nonzero coefficients. Specifically, it is observed that the effect of the filtering on each coefficient is larger if a block has fewer nonzero coefficients. Accordingly, a new factor (NF) can be use in some embodiments. For instance, an NF of 8 can be used one or two coefficients of a block are nonzero. In some embodiments, the NF is 4 if less than five coefficients of the block are nonzero. In some embodiments, the NF is 2 if less than nine coefficients of the block are nonzero. Otherwise, the NF is 1 according to embodiments.
In some embodiments, the NF depends on the maximum frequency of a nonzero coefficient. In such embodiments, the NF would be 2, otherwise 1, for the coefficients in an 8×8 block in the topleft corner. For example, NF may be 2 if there are no nonzero coefficients outside of the topleft part of the block. If there is at least one nonzero coefficient outside of the topleft area, NF may be 1. According to embodiments, topleft can be defined as a specific region or dimensions, such as x+y<3.
In some embodiments, the operation performed for each coefficient (x,y) in a block can be stated as:

 If(coefficient(x,y)>100*QF)
 Add table(x,y)*QF*NF to coefficient(x,y)
 Else if(coefficient(x,y)<−100*QF)
 Subtract table(x,y)*QF*NF from coefficient(x,y)
FIG. 3 shows code 300 implementing certain embodiments described herein.
 Subtract table(x,y)*QF*NF from coefficient(x,y)
 If(coefficient(x,y)>100*QF)
The proposed offset process can be expanded in a number of ways, including: (1) enabling the processing of larger blocks, either with a larger table or only applied to the topleft coefficients of the larger block; (2) having parameters depend on the block size, such as, unique tables per block size or multiplying the coefficients by a unique constant per block size; and (3) further optimization of the table, as described above.
In some embodiments, process 400 includes a further step in which the number of coefficients in the set of coefficients having a value (i) larger than the positive threshold (e.g., 100*Quantization Factor) or (ii) smaller than the negative threshold (e.g., −100*QF) is determined.
In some embodiments, process 400 includes a further step in which a factor value (e.g., New Factor) is determined based on the determined number of coefficients.
In some embodiments, the step of adding the corresponding offset value to the coefficient value includes multiplying the corresponding offset value, the determined factor value, and a quantization factor to obtain a multiplied offset value and adding the multiplied offset value to the coefficient value.
In some embodiments, the step of subtracting the corresponding offset value from the coefficient value includes multiplying the corresponding offset value, the determined factor value, and a quantization factor to obtain a multiplied offset value and subtracting the multiplied offset value from the coefficient value.
Referring now to
According to embodiments, applying (s502) may comprise determining (s504) whether a coefficient value is larger than a positive threshold or smaller than a negative threshold. Based on this determination, a corresponding offset value can be added (s506) to the coefficient value when the coefficient value is larger than the positive threshold or subtracting the corresponding offset value from the coefficient value when the coefficient value is smaller than the negative threshold. After the offset is applied, the process 500 may then perform a filtering operation (s508) on the media object. This may include, for instance, applying a filter to the result of an inverse transform that uses the modified coefficients after offset. According to embodiments, the media object comprises a plurality of video images and the filtering operation is one or more of bilateral filtering, deblocking filtering, adaptive loop filtering, and sample adaptive offset filtering
In some embodiments, the process 500 also includes determining the number of coefficients in the set of coefficients having a value that is either larger than the positive threshold or smaller than the negative threshold. In this instance, the corresponding offset value that is used is based at least in part on the determined number. Also, the process 500 can include determining a factor value based on the number of coefficients in the set of coefficients having a value that is either larger than the positive threshold or smaller than the negative threshold. In this instance, the corresponding offset value is a product of at least an initial offset value and the determined factor value. In some embodiments, the offset value is based at least in part on a quantization factor or quantization parameter value.
According to embodiments, the process 500 may implement a frequency multiplier that is based on determining the maximum frequency of any nonzero coefficients in the set of coefficients. In this instance, the corresponding offset value can be a product of at least an initial offset value and the determined frequency multiplier. For example, the frequency multiplier might be 2.
In some embodiments, the corresponding offset value is a product of at least an initial offset value and the multiplier based on a block size for samples of the media object.
Referring now to
According to some embodiments, the modification factor used in process 550 is based at least in part on: (i) the number of coefficients in the set of coefficients having a magnitude that is greater than the threshold, (ii) quantization factor or quantization parameter value, and (3) the maximum frequency of any nonzero coefficients of the set of coefficients.
In some embodiments, the set of coefficients corresponds to an image block and at least one of determining an initial offset value or adjusted offset value is repeated for each coefficient of the set. At least of the modification factor and the initial offset value can be based on a size of the block.
In some embodiments, the set of coefficients corresponds to an image block and the modification factor is further based on the number of nonzero coefficients for said block. In some embodiments of process 550, the modification factor may defined as:
QF*NF,
where NF is determined according to the number of nonzero coefficients in the block and QF is based on one or more of the quantization factor and quantization parameter. The value of NF may be, for example, 8 if one or two coefficients of the block are nonzero, 4 if less than five coefficients of the block are nonzero, 2 if less than nine coefficients of the block are nonzero, and otherwise 1.
According to some embodiments, device 700 may be adapted to perform one or more the methods of
While various embodiments of the present disclosure are described herein, it should be understood that they have been presented by way of example only, and not limitation. Thus, the breadth and scope of the present disclosure should not be limited by any of the abovedescribed exemplary embodiments. Moreover, any combination of the abovedescribed elements in all possible variations thereof is encompassed by the disclosure unless otherwise indicated herein or otherwise clearly contradicted by context.
Additionally, while the processes described above and illustrated in the drawings are shown as a sequence of steps, this was done solely for the sake of illustration. Accordingly, it is contemplated that some steps may be added, some steps may be omitted, the order of the steps may be rearranged, and some steps may be performed in parallel.
Claims
1. A method for processing a media object, the method comprising:
 applying a set of offset values to a set of coefficients associated with the media object, wherein each offset value of the set of offset values corresponds to a separate coefficient of the set of coefficients, and wherein applying the set of offset values comprises: determining whether a coefficient value is larger than a positive threshold or smaller than a negative threshold, and based on the determination, adding a corresponding offset value to the coefficient value when the coefficient value is larger than the positive threshold or subtracting the corresponding offset value from the coefficient value when the coefficient value is smaller than the negative threshold; and
 after applying the set of offset values, performing a filtering operation on the media object.
2. The method of claim 1, further comprising:
 determining the number of coefficients in the set of coefficients having a value that is either larger than the positive threshold or smaller than the negative threshold,
 wherein the corresponding offset value is based at least in part on the determined number.
3. The method of claim 1, further comprising:
 determining a factor value based at least in part on the number of coefficients in the set of coefficients having a value that is either larger than the positive threshold or smaller than the negative threshold, and
 wherein the corresponding offset value is a product of at least an initial offset value and the determined factor value.
4. The method of claim 1, wherein the offset value is based at least in part on a quantization factor or quantization parameter value.
5. The method of claim 1, further comprising:
 determining a frequency multiplier based on the maximum frequency of any nonzero coefficients of the set of coefficients, and
 wherein the corresponding offset value is a product of at least an initial offset value and the determined frequency multiplier.
6. The method of claim 5, wherein the frequency multiplier is 2.
7. The method of claim 1, wherein the media object comprises a plurality of video images and the filtering operation is one or more of bilateral filtering, deblocking filtering, adaptive loop filtering, and sample adaptive offset filtering.
8. The method of claim 1, wherein the set of coefficients corresponds to an entire block of the media object.
9. The method of claim 1, wherein the set of coefficients corresponds to portion of a block of the media object.
10. The method of claim 1, wherein the set of coefficients is a subset of a larger set of coefficients corresponding to the media object, and wherein the set of coefficients are the topleft coefficients of the larger set.
11. The method of claim 1,
 wherein the corresponding offset value is a product of at least an initial offset value and the multiplier based on a block size for samples of the media object.
12. A decoder for processing a media object, the decoder being adapted to:
 apply a set of offset values to a set of coefficients associated with the media object, wherein each offset value of the set of offset values corresponds to a separate coefficient of the set of coefficients, and wherein applying the set of offset values comprises: determining whether a coefficient value is larger than a positive threshold or smaller than a negative threshold, and based on the determination, adding a corresponding offset value to the coefficient value when the coefficient value is larger than the positive threshold or subtracting the corresponding offset value from the coefficient value when the coefficient value is smaller than the negative threshold; and after applying the set of offset values, perform a filtering operation on the media object.
13. The decoder of claim 12, where the decoder is further configured to:
 determine the number of coefficients in the set of coefficients having a value that is either larger than the positive threshold or smaller than the negative threshold,
 wherein the corresponding offset value is based at least in part on the determined number.
14. An encoder for processing a media object, the encoder being adapted to:
 apply a set of offset values to a set of coefficients associated with the media object, wherein each offset value of the set of offset values corresponds to a separate coefficient of the set of coefficients, and wherein applying the set of offset values comprises: determining whether a coefficient value is larger than a positive threshold or smaller than a negative threshold, and based on the determination, adding a corresponding offset value to the coefficient value when the coefficient value is larger than the positive threshold or subtracting the corresponding offset value from the coefficient value when the coefficient value is smaller than the negative threshold; and
 after applying the set of offset values, perform a filtering operation on the media object.
15. The encoder of claim 14, where the encoder is further configured to:
 determine the number of coefficients in the set of coefficients having a value that is either larger than the positive threshold or smaller than the negative threshold,
 wherein the corresponding offset value is based at least in part on the determined number.
16. A method for processing a media object, the method comprising:
 determining an initial offset value for at least one coefficient of a set of coefficients associated with the media object; and
 determining an adjusted offset value for the coefficient,
 wherein the adjusted offset value is determined based at least in part on the initial offset value and a modification factor.
17. The method of claim 16, wherein one or more of determining the initial offset value and determining the adjusted offset value is based on determining that a value of the coefficient meets or exceeds a threshold.
18. The method of claim 16, further comprising:
 generating the set of coefficients by performing an inverse quantization process on a plurality of quantized values associated with the media object.
19. The method of claim 16, further comprising:
 applying the adjusted offset value to the coefficient to generate a modified coefficient.
20. The method of claim 19, further comprising:
 performing an inverse transform using the modified coefficient;
 performing a filtering operation comprising one or more of bilateral filtering, deblocking filtering, adaptive loop filtering, and sample adaptive offset filtering.
21. The method of claim 16, wherein determining the initial offset value comprises selecting the value from a table.
22. The method of claim 16, wherein the modification factor is based at least in part on: (i) the number of coefficients in the set of coefficients having a magnitude that is greater than the threshold, (ii) a quantization factor or quantization parameter value, and (iii) the maximum frequency of any nonzero coefficients of the set of coefficients.
23. The method of claim 17, wherein the threshold is zero.
24. The method of claim 16, wherein the set of coefficients corresponds to an image block and at least one of determining an initial offset value or adjusted offset value is repeated for each coefficient of the set.
25. The method of claim 24, wherein at least one of the modification factor and the initial offset value is based on a size of the block.
26. The method of claim 16, a wherein the set of coefficients corresponds to an image block and the modification factor is further based on the number of nonzero coefficients for said block.
27. The method of claim 26, wherein the modification factor is defined as:
 QF*NF,
 where NF is determined according to the number of nonzero coefficients in the block and QF is based on one or more of the quantization factor and quantization parameter.
28. The method of claim 27, wherein the value of NF is 8 if one or two coefficients of the block are nonzero, is 4 if less than five coefficients of the block are nonzero, is 2 if less than nine coefficients of the block are nonzero, and is otherwise 1.
29. A decoder for processing a media object, the decoder being adapted to:
 determine an initial offset value for at least one coefficient of a set of coefficients associated with the media object; and
 determine an adjusted offset value for the coefficient,
 wherein the adjusted offset value is determined based at least in part on the initial offset value and a modification factor.
30. (canceled)
31. An encoder for processing a media object, the encoder being adapted to:
 determine an initial offset value for at least one coefficient of a set of coefficients associated with the media object; and
 determine an adjusted offset value for the coefficient,
 wherein the adjusted offset value is determined based at least in part on the initial offset value and a modification factor.
32. (canceled)
33. A method for applying filtering to a media object comprising a plurality of samples, the method comprising:
 applying a set of offset values to a set of coefficients, wherein each offset value of the set of offset values corresponds to a separate coefficient of the set of coefficients, and wherein applying the set of offset values comprises: determining whether a coefficient value is larger than a positive threshold or smaller than a negative threshold, and applying a corresponding offset value to the coefficient value based on the determination whether the coefficient value is larger than the positive threshold or smaller than the negative threshold, such that, if the coefficient value is larger than the positive threshold, then adding the corresponding value to the coefficient value, and if the coefficient value is smaller than the negative threshold, then subtracting the corresponding value from the coefficient value; and
 after applying the set of offset values, applying the filtering.
34. The method of claim 33, further comprising:
 determining the number of coefficients in the set of coefficients having a value (i) larger than the positive threshold or (ii) smaller than the negative threshold.
35. The method of claim 34, further comprising:
 determining a factor value based on the determined number of coefficients.
36. The method of claim 35, wherein adding the corresponding offset value to the coefficient value comprises:
 multiplying the corresponding offset value, the determined factor value, and a quantization factor to obtain a multiplied offset value; and
 adding the multiplied offset value to the coefficient value.
37. The method of claim 35, wherein subtracting the corresponding offset value from the coefficient value comprises:
 multiplying the corresponding offset value, the determined factor value, and a quantization factor to obtain a multiplied offset value; and
 subtracting the multiplied offset value from the coefficient value.
38. A node for applying filtering to a media object comprising a plurality of samples, the network node configured to:
 apply a set of offset values to a set of coefficients, wherein each offset value of the set of offset values corresponds to a separate coefficient of the set of coefficients, and wherein applying the set of offset values comprises: determining whether a coefficient value is larger than a positive threshold or smaller than a negative threshold, and applying a corresponding offset value to the coefficient value based on the determination whether the coefficient value is larger than the positive threshold or smaller than the negative threshold, such that, if the coefficient value is larger than the positive threshold, then adding the corresponding value to the coefficient value, and if the coefficient value is smaller than the negative threshold, then subtracting the corresponding value from the coefficient value; and
 after applying the set of offset values, apply the filtering.
39. The node of claim 38, further configured to:
 determine the number of coefficients in the set of coefficients having a value (i) larger than the positive threshold or (ii) smaller than the negative threshold.
40. The node of claim 39, further configured to:
 determine a factor value based on the determined number of coefficients.
41. The node of claim 40, wherein adding the corresponding offset value to the coefficient value comprises:
 multiplying the corresponding offset value, the determined factor value, and a quantization factor to obtain a multiplied offset value; and
 adding the multiplied offset value to the coefficient value.
42. The node of claim 40, wherein subtracting the corresponding offset value from the coefficient value comprises:
 multiplying the corresponding offset value, the determined factor value, and a quantization factor to obtain a multiplied offset value; and
 subtracting the multiplied offset value from the coefficient value.
43. A computer program product comprising a nontransitory computer readable medium storing a computer program comprising instructions which, when executed by processing circuitry, causes the processing circuitry to perform the method of claim 1.
44. (canceled)
Type: Application
Filed: Jul 2, 2019
Publication Date: Oct 14, 2021
Applicant: Telefonaktiebolaget LM Ericsson (publ) (Stockholm)
Inventors: Per WENNERSTEN (Årsta), Jacob STRÖM (Stockholm), Jack ENHORN (Kista), Du LIU (Solna)
Application Number: 17/255,240