summaryrefslogtreecommitdiff
path: root/vp9/encoder/vp9_quantize.c
AgeCommit message (Collapse)Author
2013-04-15Fix width/height switch-up in U/V SB quantize code.Ronald S. Bultje
Change-Id: I697514efd6024e1b4153bbde58ae5e323b030981
2013-04-11Remove unused macroblock versions of reconstruction functions.Ronald S. Bultje
More specifically, remove vp9_quantize_mb*, vp9_optimize_mb*, vp9_inverse_transform_mb* and vp9_transform_mb*. Instead, use the generic _sb* functions that take a size argument, and call them with BLOCK_SIZE_MB16X16. Change-Id: I33024afea95d3a23ffbc1df7da426e4645110f29
2013-04-10Merge "Fixing upper case names." into experimentalDmitry Kovalev
2013-04-09Make SB coding size-independent.Ronald S. Bultje
Merge sb32x32 and sb64x64 functions; allow for rectangular sizes. Code gives identical encoder results before and after. There are a few macros for rectangular block sizes under the sbsegment experiment; this experiment is not yet functional and should not yet be used. Change-Id: I71f93b5d2a1596e99a6f01f29c3f0a456694d728
2013-04-09Fixing upper case names.Dmitry Kovalev
Renaming Y1dequant to y_dequant, UVdequant to uv_dequant, QIndex to qindex. Change-Id: I1c356e5f886deb3f8807dc212de9799b55b09d58
2013-04-04Move EOB to per-plane dataJohn Koleszar
Continue migrating data from BLOCKD/MACROBLOCKD to the per-plane structures. Change-Id: Ibbfa68d6da438d32dcbe8df68245ee28b0a2fa2c
2013-04-04Move qcoeff, dqcoeff from BLOCKD to per-plane dataJohn Koleszar
Start grouping data per-plane, as part of refactoring to support additional planes, and chroma planes with other-than 4:2:0 subsampling. Change-Id: Idb76a0e23ab239180c818025bae1f36f1608bb23
2013-03-26Add col/row-based coefficient scanning patterns for 1D 8x8/16x16 ADSTs.Ronald S. Bultje
These are mostly just for experimental purposes. I saw small gains (in the 0.1% range) when playing with this on derf. Change-Id: Ib21eed477bbb46bddcd73b21c5c708a5b46abedc
2013-03-25Modeling default coef probs with distributionDeb Mukherjee
Replaces the default tables for single coefficient magnitudes with those obtained from an appropriate distribution. The EOB node is left unchanged. The model is represeted as a 256-size codebook where the index corresponds to the probability of the Zero or the One node. Two variations are implemented corresponding to whether the Zero node or the One-node is used as the peg. The main advantage is that the default prob tables will become considerably smaller and manageable. Besides there is substantially less risk of over-fitting for a training set. Various distributions are tried and the one that gives the best results is the family of Generalized Gaussian distributions with shape parameter 0.75. The results are within about 0.2% of fully trained tables for the Zero peg variant, and within 0.1% of the One peg variant. The forward updates are optionally (controlled by a macro) model-based, i.e. restricted to only convey probabilities from the codebook. Backward updates can also be optionally (controlled by another macro) model-based, but is turned off by default. Currently model-based forward updates work about the same as unconstrained updates, but there is a drop in performance with backward-updates being model based. The model based approach also allows the probabilities for the key frames to be adjusted from the defaults based on the base_qindex of the frame. Currently the adjustment function is a placeholder that adjusts the prob of EOB and Zero node from the nominal one at higher quality (lower qindex) or lower quality (higher qindex) ends of the range. The rest of the probabilities are then derived based on the model from the adjusted prob of zero. Change-Id: Iae050f3cbcc6d8b3f204e8dc395ae47b3b2192c9
2013-03-08Merge "Consistent usage of ROUND_POWER_OF_TWO macro." into experimentalDmitry Kovalev
2013-03-07Consistent usage of ROUND_POWER_OF_TWO macro.Dmitry Kovalev
Change-Id: I44660975e9985310d8c654c158ee7a61291b5a08
2013-03-07Re-add support for ADST in superblocks.Ronald S. Bultje
This also changes the RD search to take account of the correct block index when searching (this is required for ADST positioning to work correctly in combination with tx_select). Change-Id: Ie50d05b3a024a64ecd0b376887aa38ac5f7b6af6
2013-03-07Coding con-zero count rather than EOB for coeffsDeb Mukherjee
This patch revamps the entropy coding of coefficients to code first a non-zero count per coded block and correspondingly remove the EOB token from the token set. STATUS: Main encode/decode code achieving encode/decode sync - done. Forward and backward probability updates to the nzcs - done. Rd costing updates for nzcs - done. Note: The dynamic progrmaming apporach used in trellis quantization is not exactly compatible with nzcs. A suboptimal approach has been used instead where branch costs are updated to account for changes in the nzcs. TODO: Training the default probs/counts for nzcs Change-Id: I951bc1e22f47885077a7453a09b0493daa77883d
2013-03-04Make superblocks independent of macroblock code and data.Ronald S. Bultje
Split macroblock and superblock tokenization and detokenization functions and coefficient-related data structs so that the bitstream layout and related code of superblock coefficients looks less like it's a hack to fit macroblocks in superblocks. In addition, unify chroma transform size selection from luma transform size (i.e. always use the same size, as long as it fits the predictor); in practice, this means 32x32 and 64x64 superblocks using the 16x16 luma transform will now use the 16x16 (instead of the 8x8) chroma transform, and 64x64 superblocks using the 32x32 luma transform will now use the 32x32 (instead of the 16x16) chroma transform. Lastly, add a trellis optimize function for 32x32 transform blocks. HD gains about 0.3%, STDHD about 0.15% and derf about 0.1%. There's a few negative points here and there that I might want to analyze a little closer. Change-Id: Ibad7c3ddfe1acfc52771dfc27c03e9783e054430
2013-02-27Move eob from BLOCKD to MACROBLOCKD.Ronald S. Bultje
Consistent with VP8. Change-Id: I8c316ee49f072e15abbb033a80e9c36617891f07
2013-02-22Experimental removal of over quant codePaul Wilkins
The over quant code was added in VP8 post bitstream freeze to allow compression to lower data rates In VP9 the real qualtizer range has been greatly extended anyway. Change-Id: I5d384fa5e9a83ef75a3df34ee30627bd21901526
2013-02-19Use lossless for Q0Yaowu Xu
The commit changes the coding mode to lossless whenever the lowest quantizer is choosen. As expected, test results showed no difference for cif and std-hd set where Q0 is rarely used. For yt and yt-hd set, Q0 is used for a number of clips, where this commit helped a lot in the high end. Average over all clips in the sets: yt: 2.391% 1.017% 1.066% hd: 1.937% .764% .787% Change-Id: I9fa9df8646fd70cb09ffe9e4202b86b67da16765
2013-02-15Remove y2dc/ac Q delta values from the bitstream.Ronald S. Bultje
Since there is no Y2, these values are always zero. This changes the bitstream results slightly, hence a separate commit. Change-Id: I2f838f184341868f35113ec77ca89da53c4644e0
2013-02-15Remove some Y2-related code.Ronald S. Bultje
Change-Id: I4f46d142c2a8d1e8a880cfac63702dcbfb999b78
2013-02-13Fixed encoder decoder mismatch.Paul Wilkins
Reverted part of change I19981d1ef0b33e4e5732739574f367fe82771a84 That gives rise to an enc/dec mismatch. As things stand the memsets are still needed. Change-Id: I9fa076a703909aa0c4da0059ac6ae19aa530db30
2013-02-13Merge "enable bitstream lossless support" into experimentalYaowu Xu
2013-02-13enable bitstream lossless supportYaowu Xu
1. Added a bit in frame header to to indicate if a frame is encoded in lossless mode, so decoder does not make the decision based on Q0 2. Minor changes to make sure that lossy coding works same as when the lossless experiment is not enabled. 3. Renamed function pointers for transforms to be consistent, using prefix fwd_txm and inv_txm for forward and inverse respectively To encode in lossless mode, using "--lossless=1 --min-q=0 --max-q=0" with vpxenc. Change-Id: Ifae53b26d2ffbe378d707e29d96817b8a5e6c068
2013-02-12Faster vp9_regular_quantize_b_8x8.Christian Duvivier
A couple of scalar optimizations speeding up quantization by about 1.6x. Overall encoder speedup is around 3%. Change-Id: I19981d1ef0b33e4e5732739574f367fe82771a84
2013-01-29Remove eob_max_offset markers.Paul Wilkins
Remove eob_max_offset markers and replace with the generic skip_block flag to indicate to the quantizer that all coeffs to be set to 0 and eob position set to 0; Change-Id: Id477e8f8d4ec1a5562758904071013c24b76bfd7
2013-01-28Segment Skip FlagPaul Wilkins
First step in simplifying the segment mode and segment EOB flags into a simpler segment skip flag that implies 0,0 mv and EOB at position 0. Change-Id: Ib750cac31a7a02dc21082580498efd9f7d8d72a5
2013-01-28Simplify Zero bin and zero bin run code.Paul Wilkins
Simplification to eliminate a number of very large data data structures. All zero run, zbin boosts for different transform sizes are now limited to a maximum run length of 15 before they max out the boost. Some further work still needs be done to refactor, rationalize and optimize the multiple quantizer functions. The simplification coupled with tweaks to the 16 element array now used for all transform sizes, has minimal effect on quality. Change-Id: I6f3948b8ca0418b60d4db9030ff19026a34ed423
2013-01-10Merge tx32x32 experiment.Ronald S. Bultje
Change-Id: I615651e4c7b09e576a341ad425cf80c393637833
2013-01-08Merge superblocks (32x32) experiment.Ronald S. Bultje
Change-Id: I0df99742029834a85c4933652b0587cf5b6b2587
2012-12-18Use standard integer types for pixel values and coefficients.Ronald S. Bultje
For coefficients, use int16_t (instead of short); for pixel values in 16-bit intermediates, use uint16_t (instead of unsigned short); for all others, use uint8_t (instead of unsigned char). Change-Id: I3619cd9abf106c3742eccc2e2f5e89a62774f7da
2012-12-18Give 4x4 scan and coef_band tables a _4x4 suffix.Ronald S. Bultje
This matches the names of tables for all other transform sizes. Change-Id: Ia7681b7f8d34c97c27b0eb0e34d490cd0f8d02c6
2012-12-0732x32 transform for superblocks.Ronald S. Bultje
This adds Debargha's DCT/DWT hybrid and a regular 32x32 DCT, and adds code all over the place to wrap that in the bitstream/encoder/decoder/RD. Some implementation notes (these probably need careful review): - token range is extended by 1 bit, since the value range out of this transform is [-16384,16383]. - the coefficients coming out of the FDCT are manually scaled back by 1 bit, or else they won't fit in int16_t (they are 17 bits). Because of this, the RD error scoring does not right-shift the MSE score by two (unlike for 4x4/8x8/16x16). - to compensate for this loss in precision, the quantizer is halved also. This is currently a little hacky. - FDCT and IDCT is double-only right now. Needs a fixed-point impl. - There are no default probabilities for the 32x32 transform yet; I'm simply using the 16x16 luma ones. A future commit will add newly generated probabilities for all transforms. - No ADST version. I don't think we'll add one for this level; if an ADST is desired, transform-size selection can scale back to 16x16 or lower, and use an ADST at that level. Additional notes specific to Debargha's DWT/DCT hybrid: - coefficient scale is different for the top/left 16x16 (DCT-over-DWT) block than for the rest (DWT pixel differences) of the block. Therefore, RD error scoring isn't easily scalable between coefficient and pixel domain. Thus, unfortunately, we need to compute the RD distortion in the pixel domain until we figure out how to scale these appropriately. Change-Id: I00386f20f35d7fabb19aba94c8162f8aee64ef2b
2012-11-28Fixing 8x8/4x4 ADST for intra modes with tx selectDeb Mukherjee
This patch allows use of 8x8 and 4x4 ADST correctly for Intra 16x16 modes and Intra 8x8 modes when the block size selected is smaller than the prediction mode. Also includes some cleanups and refactoring. Rebase. Change-Id: Ie3257bdf07bdb9c6e9476915e3a80183c8fa005a
2012-11-28fixed includes to be fully specifiedJim Bankoski
Change-Id: Ia1cce221f8511561b9cbd8edb7726fbc286ff243
2012-11-27Add vp9_ prefix to all vp9 filesJohn Koleszar
Support for gyp which doesn't support multiple objects in the same static library having the same basename. Change-Id: Ib947eefbaf68f8b177a796d23f875ccdfa6bc9dc