summaryrefslogtreecommitdiff
path: root/vp8/decoder/detokenize.c
AgeCommit message (Collapse)Author
2012-10-22Change entropy context assignment style from *(x + y) = z to x[y] = z.Ronald S. Bultje
Change-Id: Ibc4425bb6e01606751021e57c93106c365a2da21
2012-10-22Merge changes I02e7f64a,Ide954b00,Idc8b5977 into experimentalRonald S. Bultje
* changes: Fix another typo in 4x4-transform-for-i8x8-intra-pred coeff contexts. 8x8 transform support in splitmv. Use SPLITMV_PARTITIONING instead of a plain integer type.
2012-10-22Merging the hybrid transform experimentsDeb Mukherjee
Change-Id: I99f1982b30a630a9a070a8326d83b34a33cba14c
2012-10-228x8 transform support in splitmv.Ronald S. Bultje
For splitmv, where partitioning is 8x16, 16x8 or 8x8, this patch uses the 8x8 transform (instead of the 4x4) if txfm_mode is ALLOW_8X8 or ALLOW_16X16. For TX_MODE_SELECT, splitmv can indicate which of the 2 transform sizes (4x4 or 8x8) it wants to use. Gains (with hybridtx4x4/8x8/16x16 and tx_select experiments enabled) on derf: +0.9%, HD: +0.4%, STD/HD: +0.8% (SSIM or overall PSNR, both metrics show similar improvements). Change-Id: Ide954b000b415548ed92a7ac78e24f36e60fcf06
2012-10-19Remove bc, bc2 from pbi,cpi,xdJohn Koleszar
Pass the bool coder to be used explicitly. This avoids cases where two different bool coders can be addressed from the same function. Also be more consistent with bool coder variable naming, start to standardize on 'bc'. Change-Id: I1c95e2fdbe24ebe8c0f84924daa1728e3b054a31
2012-10-19Some cleanups and fixes.Deb Mukherjee
Separates the logic on transform type selection previously spread out over a number of files into a separate function. Currently the tx_type field in b_mode_info is not used, but still left in there to eventually use for signaling the transform type in the bitstream. Also, now for tx_type = DCT_DCT, the regular integer DCT is used, as opposed to the floating point DCT used in conjuction with hybrid transform. Results change somewhat due to the transform change, but are within reasonable limits. The hd/std-hd sets are slightly up, while derf/yt are slightly down. Change-Id: I5776840c2239ca2da31ca6cfd7fd1148dc5f9e0f
2012-10-15Add and consistently use PLANE_TYPE.Ronald S. Bultje
Change the macros PLANE_TYPE_{Y_NO_DC,Y2,UV,Y_WITH_DC} to a typed enum, and use this typed enum consistently across all places where relevant. In places where the type is implied (e.g. in functions that only handle second order planes or chroma planes), remove it as a function argument and instead hardcode the proper enum in the code directly. Change-Id: I93652b4a36aa43163d49c732b0bf5c4442738c47
2012-10-10Merge of the TX_16X16 experimentDeb Mukherjee
Change-Id: I22aa803ffff330622cdb77277e7b196a9766f882
2012-10-05Mismatch bug-fix when transform expts are offDeb Mukherjee
A variety of issues related to tangling of various hybrid transform experiments. Change-Id: I84dac65ff513f8b024407760df55fffa95e198de
2012-09-27Entropy coding for hybrid transformDeb Mukherjee
Separates the entropy coding context models for 4x4, 8x8 and 16x16 ADST variants. There is a small improvement for HD (hd/std-hd) by about 0.1-0.2%. Results on derf/yt are about the same, probably because there is not enough statistics. Results may improve somewhat once the initial probability tables are updated for the hybrid transforms which is coming soon. Change-Id: Ic7c0c62dacc68ef551054fdb575be8b8507d32a8
2012-08-30hybrid transform of 16x16 dimensionJingning Han
Enable ADST/DCT of dimension 16x16 for I16X16 modes. This change provides benefits mostly for hd sequences. Set up the framework for selectable transform dimension. Also allowing quantization parameter threshold to control the use of hybrid transform (This is currently disabled by setting threshold always above the quantization parameter. Adaptive thresholding can be built upon this, which will further improve the coding performance.) The coding performance gains (with respect to the codec that has all other configuration settings turned on) are derf: 0.013 yt: 0.086 hd: 0.198 std-hd: 0.501 Change-Id: Ibb4263a61fc74e0b3c345f54d73e8c73552bf926
2012-08-21Fix compiler warningsDaniel Kang
Change-Id: I887cdcf90b87736f2a3805420aa41eed93324bee
2012-08-14Some build fixesDeb Mukherjee
Fixes two typos that breaks the build introduced in recent check-ins. Change-Id: I3f15f3c088c48a63fede8941c9a4ba05dd0403c2
2012-08-14Code clean up.Paul Wilkins
References to MACROBLOCKD that use "x" changed to "xd" to comply with convention elsewhere that x = MACROBLOCK and xd = MACROBLOCKD. Simplify some repeat references using local variables. Change-Id: I0ba2e79536add08140a6c8b19698fcf5077246bc
2012-08-03Replacing the 8x8 DCT with 8x8 ADST/DCT for I8x8Jingning Han
Fixed the code review comments. Under the htrans8x8 experiment the 8X8 DCT in the I8X8 mode is replaced with a combination of 8X8 ADST and DCT. Overall coding gains with the htrans8x8 experiment are: derf: 0.486 std-hd: 1.040 hd: 1.063 yt: 0.506 Note that part of the gain comes from bigger transforms (8x8 instead of 4x4) and part comes from replacing the DCT wth the ADST. Change-Id: I92ca6bbfce11b4165d612b81d9adfad4d010c775
2012-08-0216x16 DCT blocks.Daniel Kang
Set on all 16x16 intra/inter modes Features: - Butterfly fDCT/iDCT - Loop filter does not filter internal edges with 16x16 - Optimize coefficient function - Update coefficient probability function - RD - Entropy stats - 16x16 is a config option Have not tested with experiments. hd: 2.60% std-hd: 2.43% yt: 1.32% derf: 0.60% Change-Id: I96fb090517c30c5da84bad4fae602c3ec0c58b1c
2012-08-02Use 8x8 DCT transform for I8X8 prediction modeJingning Han
Apply 2D-DCT transform of dimension 8x8 to encode prediction residuals of I8X8 mode. Brought back block type 3 probability context model for 8x8 tokens, which is used for the coefficients of Y blocks in I8x8 modes. The coefficient costs estimate of I8X8 mode in rate-distortion is also changed appropriately. Performance results: derf: 0.246 yt: 0.114 std-hd: 0.730 hd: 0.670 Change-Id: If1d970eeb4e1827c9f0d2c5b27d33089b347ea27
2012-07-27Merges several experimentsDeb Mukherjee
The following five experiments are merged: newentropy newupdate adaptive_entropy (also includes a couple of parameter changes that improves results a little in common/entropymode.c and encoder/modecosts.c that were not merged from the internal branch) newintramodes expanded_coef_context Change-Id: I8a142a831786ee9dc936f22be1d42a8bced7d270
2012-07-19Adds hybrid transformJingning Han
Adds ADST/DCT hybrid transform coding for Intra4x4 mode. The ADST is applied to directions in which the boundary pixels are used for prediction, while DCT applied to directions without corresponding boundary prediction. Adds enum TX_TYPE in b_mode_infor to indicate the transform type used. Make coding style consistent with google style. Fixed the commented issues. Experimental results in terms of bit-rate reduction: derf: 0.731% yt: 0.982% std-hd: 0.459% hd: 0.725% Will be looking at 8x8 transforms next. Change-Id: I46dbd7b80dbb3e8856e9c34fbc58cb3764a12fcf
2012-07-17Restyle codeJohn Koleszar
Approximate the Google style guide[1] so that that there's a written document to follow and tools to check compliance[2]. [1]: http://google-styleguide.googlecode.com/svn/trunk/cppguide.xml [2]: http://google-styleguide.googlecode.com/svn/trunk/cpplint/cpplint.py Change-Id: Idf40e3d8dddcc72150f6af127b13e5dab838685f
2012-07-16Simplify vp8_decode_mb_tokens_8x8.Daniel Kang
Change-Id: Icac5b18e5c832681cea866679796a1dd4278cfb8
2012-07-16Simplify vp8_decode_mb_tokens.Daniel Kang
Change-Id: Ib6866e491994f6c7eeb6c2e5ecd1ff54cc187610
2012-06-15Adaptive entropy coding of coefficients, modes, mv.Deb Mukherjee
This patch incorporates adaptive entropy coding of coefficient tokens, and mode/mv information based on distributions encountered in a frame. Specifically, there is an initial forward update to the probabilities in the bitstream as before for coding the symbols in the frame, however at the end of decoding each frame, the forward update to the probabilities is reverted and instead the probabilities are updated towards the actual distributions encountered within the frame. The amount of update is weighted by the number of hits within each context. Results on derf/hd/std-hd are all up by 1.6%. On derf, the most of the gains come from coefficients, however for the hd and std-hd sets, the most of the gains come from the mode/mv information updates. Change-Id: I708c0e11fdacafee04940fe7ae159ba6844005fd
2012-05-04Expanding the coefficient encoding contextsDeb Mukherjee
This patch expands the set of prev contexts used for video coding from 3 to 4. There is a small improvement of the order of 0.08% for derf and 0.15% on the HD set. The tests were rerun after the various merges last week. There are two columns in each test - the first are the results with the mbskip change, and the second with expanded contexts added on top of that. Derf: http://www.corp.google.com/~debargha/vp8_results/explibvpx_newentropy_expcontext.html HD: http://www.corp.google.com/~debargha/vp8_results/explibvpx_hd_newentropy_expcontext.html Rebased. Broke up 80 char lines. Change-Id: I82d2e72d054e530cbf5ce9aa0e6d85c582965675
2012-05-02chagned the decoder band to match the encoderYaowu Xu
missed the decoder side in last commit Change-Id: Ie97f35189e93f78783e3d8072a36eea768beed27
2012-03-21Adding contextual coding of mb_skip_coeff flag.Deb Mukherjee
Using contextual coding of the mkb_skip_coeff flag using the values of this flag from the left and above. There is a small improvement of about 0.15% on Derf: http://www.corp.google.com/~debargha/vp8_results/mbskipcontext.html Refactored to use pred_common.c by adding a new context type. Results on HD set (about 0.66% improvement): http://www.corp.google.com/~debargha/vp8_results/mbskipcontext_hd.html Incliding missing refactoring to use the pred_common utilities. Change-Id: I95373382d429b5a59610d77f69a0fea2be628278
2012-03-05Changed how UV r/d estimates are done for Intra ModesYaowu Xu
The commit changed to compute UV intra RD estimates for 4x4 and 8x8 separately to be used in mode decision for MB modes associated with the appropriate transform size respectively. Now finally after many other changes related 8x8 quantizer zbin boost and zbin_mode_boost, this change overall helps the HD(with 8x8) by around ~.13%. (avg .13% glb .13% ssim .17%) The commit also has a few changes for eliminating compiler warnings. Change-Id: Ibab35dad44820c87e6b44799c66f8d519cc37344
2012-03-01Merge t8x8 experimentsYaowu Xu
Change-Id: I8e9b6b154e1a0d0cb42d596366380d69c00ac15f
2012-02-16Reworked context conversion between 8x8 and 4x4Yaowu Xu
The commit rationized and simplified the entropy context conversion betwen MB using 8x8 transform and MB using 4x4 transform. The old version had a number of weirdness in how 4x4 transform MB's context is used for 8x8 blocks other than the first 8x8 within a MB. Test showed the change has a gain ~.1% for avg psnr, glb psnr and ssim on the limited HD set. Change-Id: I774536c416baa6845aa741f956d8a69fa40e5d47
2012-02-13Fixed typo on #define namePaul Wilkins
SE_LVL_EOB => SEG_LVL_EOB Change-Id: I6d10169878a709bc9b82f03e5d5903c629fa7679
2012-02-10fixed an issue related to 2nd order size due to merge artifacts.Yaowu Xu
For 8x8 transformed macroblock, the 2nd order transform is a 2x2 haar transform, here there is only 4 coefficients total. A previous merge changed these to 64, causing crashes when encoding with 8x8 transform enabled. (i.e. when input video image size > 640x360 ) This commit reverts them back to 4 and fixes the crashes. Change-Id: I3290b81f8c0d32c7efec03093a61ea57736c0550
2012-02-10Improved coding using 8x8 transformRonald S. Bultje
In summary, this commit encompasses a series of changes in attempt to improve the 8x8 transform based coding to help overall compression quality, please refer to the detailed commit history below for what are the rationale underly the series of changes: a. A frame level flag to indicate if 8x8 transform is used at all. b. 8x8 transform is not used for key frames and small image size. c. On inter coded frame, macroblocks using modes B_PRED, SPLIT_MV and I8X8_PRED are forced to using 4x4 transform based coding, the rest uses 8x8 transform based coding. d. Encoder and decoder has the same assumption on the relationship between prediction modes and transform size, therefore no signaling is encoded in bitstream. e. Mode decision process now calculate the rate and distortion scores using their respective transforms. Overall test results: 1. HD set http://www.corp.google.com/~yaowu/no_crawl/t8x8/HD_t8x8_20120206.html (avg psnr: 3.09% glb psnr: 3.22%, ssim: 3.90%) 2. Cif set: http://www.corp.google.com/~yaowu/no_crawl/t8x8/cif_t8x8_20120206.html (avg psnr: -0.03%, glb psnr: -0.02%, ssim: -0.04%) It should be noted here, as 8x8 transform coding itself is disabled for cif size clips, the 0.03% loss is purely from the 1 bit/frame flag overhead on if 8x8 transform is used or not for the frame. ---patch history for future reference--- Patch 1: this commit tries to select transform size based on macroblock prediction mode. If the size of a prediction mode is 16x16, then the macroblock is forced to use 8x8 transform. If the prediction mode is B_PRED, SPLITMV or I8X8_PRED, then the macroblock is forced to use 4x4 transform. Tests on the following HD clips showed mixed results: (all hd clips only used first 100 frames in the test) http://www.corp.google.com/~yaowu/no_crawl/t8x8/hdmodebased8x8.html http://www.corp.google.com/~yaowu/no_crawl/t8x8/hdmodebased8x8_log.html while the results are mixed and overall negative, it is interesting to see 8x8 helped a few of the clips. Patch 2: this patch tries to hard-wire selection of transform size based on prediction modes without using segmentation to signal the transform size. encoder and decoder both takes the same assumption that all macroblocks use 8x8 transform except when prediciton mode is B_PRED, I8X8_PRED or SPLITMV. Test results are as follows: http://www.corp.google.com/~yaowu/no_crawl/t8x8/cifmodebase8x8_0125.html http://www.corp.google.com/~yaowu/no_crawl/t8x8/hdmodebased8x8_0125log.html Interestingly, by removing the overhead or coding the segmentation, the results on this limited HD set have turn positive on average. Patch 3: this patch disabled the usage of 8x8 transform on key frames, and kept the logic from patch 2 for inter frames only. test results on HD set turned decidedly positive with 8x8 transform enabled on inter frame with 16x16 prediction modes: (avg psnr: .81% glb psnr: .82 ssim: .55%) http://www.corp.google.com/~yaowu/no_crawl/t8x8/hdintermode8x8_0125.html results on cif set still negative overall Patch 4: continued from last patch, but now in mode decision process, the rate and distortion estimates are computed based on 8x8 transform results for MBs with modes associated with 8x8 transform. This patch also fixed a problem related to segment based eob coding when 8x8 transform is used. The patch significantly improved the results on HD clips: http://www.corp.google.com/~yaowu/no_crawl/t8x8/hd8x8RDintermode.html (avg psnr: 2.70% glb psnr: 2.76% ssim: 3.34%) results on cif also improved, though they are still negative compared to baseline that uses 4x4 transform only: http://www.corp.google.com/~yaowu/no_crawl/t8x8/cif8x8RDintermode.html (avg psnr: -.78% glb psnr: -.86% ssim: -.19%) Patch 5: This patch does 3 things: a. a bunch of decoder bug fixes, encodings and decodings were verified to have matched recon buffer on a number of encodes on cif size mobile and hd version of _pedestrian. b. the patch further improved the rate distortion calculation of MBS that use 8x8 transform. This provided some further gain on compression. c. the patch also got the experimental work SEG_LVL_EOB to work with 8x8 transformed macroblock, test results indicates it improves the cif set but hurt the HD set slightly. Tests results on HD clips: http://www.corp.google.com/~yaowu/no_crawl/t8x8/HD_t8x8_20120201.html (avg psnr: 3.19% glb psnr: 3.30% ssim: 3.93%) Test results on cif clips: http://www.corp.google.com/~yaowu/no_crawl/t8x8/cif_t8x8_20120201.html (avg psnr: -.47% glb psnr: -.51% ssim: +.28%) Patch 6: Added a frame level flag to indicate if 8x8 transform is allowed at all. temporarily the decision is based on frame size, can be optimized later one. This get the cif results to basically unchanged, with one bit per frame overhead on both cif and hd clips. Patch 8: Rebase and Merge to head by PGW. Fixed some suspect 4s that look like hey should be 64s in regard to segmented EOB. Perhaps #defines would be bette. Bulit and tested without T8x8 enabled and produces unchanged output. Patch 9: Corrected misalligned code/decode of "txfm_mode" bit. Limited testing for correct encode and decode with T8x8 configured on derf clips. Change-Id: I156e1405d25f81579d579dff8ab9af53944ec49c
2012-02-09Removal of SEGFEATURES placeholder commentsPaul Wilkins
This commit only involves the removal of placeholder comments //#if CONFIG_SEGFEATURES. Change-Id: I94b350daaf998ee0cfdde5aa25b1d3b0522ab816
2012-02-09Merge Extended Q experiment.Paul Wilkins
Merge the extended Q experiment as indicated by the Change-Id: I02d9e654fff9998cc7e9e2f1f5cd838dad8fb431
2011-12-07Removed #if CONFIG_I8X8Yaowu Xu
This commit removed the macro CONFIG_I8X8, which was used to indicate the 8x8 intra prediction experiment, made the change fully merged in. Change-Id: Iafa4443781ce6e83f5591c12ba615a0e92ce0ea0
2011-12-02enabled 8x8 intra prediction modes on inter framesYaowu Xu
This commit enabled the usage of 8x8 intra prediction modes on inter frames. There are a few TODO items related to this: 1)baseline entropy need be calibrated; 2)cost of UV need to be done more properly rather than using decision only relying on Y; 3)Threshold for allowing picking 8x8 intra prediction should be lowered to lower than the B_PRED. Even with all the TODOs, tests showed consistent gain on derf set ~0.1% (PSNR:0.08% and SSIM:0.14%). It is assumed that 8x8 intra prediction will help more on large resolution clips, especially with above TODOs addressed. Change-Id: I398ada49dfc32575cfab962a569c2885111ae3ba
2011-11-04Segment Feature Data AccessPaul Wilkins
No change to functionality or output. Updates to the segment feature data structure now all done through functions such as set_segdata() and get_segdata() in seg_common.c. The reason for this is to make changing the structures (if needed) and debug easier. In addition it provides a single location for subsequent addition of range and validity checks. For example valid combination of mode and reference frame. Change-Id: I2e866505562db4e4cb6f17a472b25b4465f01add
2011-11-03Segment Features. Removal of #ifdefsPaul Wilkins
Removal of configure #ifdefs so that segment features always available. Removal of code supporting old segment feature method. Still a good deal of tidying up to do. Change-Id: I397855f086f8c09ab1fae0a5f65d9e06d2e3e39f
2011-10-24Further segment feature extensions.Paul Wilkins
This quite large check in includes the following: Merge in some code from Ronald (mbgraph.c) that scans a Gf/arf group. This is used as a basis for a simple segmentation for the normal frames in a gf/arf group. This code also uses satd functions from Yaowu. Adds functionality for coding the latest possible position of an EOB for blocks in the segment. (Currently 0-15 only, hence just for 4x4 dct). Where the EOB position is 0 this acts like "skip" and the normal coding of skip at the per mb level is disabled. Added functions (seg_common.c) for setting and reading segment feature elements. These may want to be optimized away at some point but while the mecahnism is in a state of flux they provide a single location for making changes and keep things a bit cleaner. This is still proof of concept code. Currently the tested feature set:- Quantizer, Loop Filter level, Reference frame, Prediction Mode, EOB end stop. TBD:- Add functions for setting and reading the feature data with range and validity checking. Handling of signed and unsigned feature data. At the moment all is assumed to be signed and a sign bit is coded but many cannot be negative. Correct handling of EOB feature with intra coded blocks. Testing/trapping of legal/illegal ref frame and mode combinations. Transform size switch plus merge and test with 8c8 DCT work Merge and test with Sumans Segmenation coding optimizations Change-Id: Iee12e83661c7abbd1e0ce6810915eb4ec35e2d8e
2011-09-30Experimental: segfeature added.Paul Wilkins
New setting added to configure script
2011-09-16add 8x8 intra prediction modesYaowu Xu
Patch 1 to Patch 3 is an initial implementation of 8x8 intra prediction modes, here are with the following assumptions: a. 8x8 has 4 prediction modes DC, H, V and TM b. UV 4x4 block use the same mode as corresponding 8x8 area c. i8x8 modes are enabled for key frame only for now Patch 4: d. removed debug code from previous patches Patch 5: e. added stats code to collect entropy stats and further cleaned up Patch 6: f. changed mode stats code to collect finer stats of modes Patch 7: g. normalized i8x8 modes distribution to total at 256 (8bits). Patch 8: h. fixed a bug in decoder and removed debug printf output. Patch 9: i. more cleanups to address paul's comment Patch 10: j. messy rebase/merges to bring the commit up to date. Tests on HD clips encoded with all key frame showing consistent gain on all clips and all metrics:~0.5%(psnr) and 0.6%(ssim): http://www.corp.google.com/~yaowu/no_crawl/i8x8hd_allkey_fixedq.html To build and test, configure with: --enable-experimental --enable-i8x8 Change-Id: I9813fe07ae48cab5fdb5d904bca022514ad01e7f
2011-09-03fixed a decoder bugYaowu Xu
the bug appears to be introduced from a merge at Commit:62400028.. Change-Id: I332d78b673f12e5ef2b0cdd6bab57dc2c8af7a72
2011-08-11Merge remote branch 'internal/upstream-experimental' into HEADJohn Koleszar
Conflicts: vp8/decoder/detokenize.c vp8/decoder/onyxd_if.c vp8/vp8_common.mk Change-Id: Ifca1108186a8bc715da86a44021ee2fa5550b5b8
2011-07-26Make cat6 probs properly dependent on CONFIG_EXTEND_QRANGEJohn Koleszar
Change-Id: I2ac5d8818acb50f9db38de8cb562f337e51006b2
2011-07-26Merge remote branch 'internal/upstream' into HEADJohn Koleszar
Conflicts: vp8/decoder/detokenize.c vp8/decoder/onyxd_int.h Change-Id: Ib9b516b939358ac8bf694200a8425fdd62c8d149
2011-07-26Merge remote branch 'origin/master' into experimentalJohn Koleszar
Conflicts: vp8/decoder/detokenize.c vp8/decoder/onyxd_int.h Change-Id: Idc301ae630dc1aedeb85674ecfdcf1eb28420f81
2011-07-25Eliminated TOKENEXTRABITSScott LaVarnway
Noticed small performance gains, depending on material. Change-Id: I334369f6312bc19aa73481fc3f790ab181e11867
2011-07-21fix more merge issuesYaowu Xu
With this fix, the experimental branch now builds and encodes correctly with the following two configure options respectively: --enable-experimental --enable-t8x8 --enable-experimental Change-Id: I3147c33c503fe713a85fd371e4f1a974805778bf
2011-07-20Add 8x8 transform to experimental branchDeb Mukherjee
Please refer to previous commit messages for detailed info: https://on2-git.corp.google.com/g/#change,5940 https://on2-git.corp.google.com/g/#change,6045 Change-Id: I8b16992f2f69c5a808ad40a3e32ef589cce7c59d
2011-07-20Merge remote branch 'origin/master' into experimentalJohn Koleszar
Change-Id: I956822324c046c254806dd712a2d3be4dcf8564b