summaryrefslogtreecommitdiff
path: root/vp8/encoder/rdopt.c
AgeCommit message (Collapse)Author
2011-12-07Removed #if CONFIG_I8X8Yaowu Xu
This commit removed the macro CONFIG_I8X8, which was used to indicate the 8x8 intra prediction experiment, made the change fully merged in. Change-Id: Iafa4443781ce6e83f5591c12ba615a0e92ce0ea0
2011-12-07made vp8_mode_context adaptiveYaowu Xu
vp8_mode_contexts[] is an entropy table used to code inter mode choices. It was a fixed constant table. This commit made the entropy context adaptive. Tests on derf set showed very good consistent gains on all metrics: avg psnr .47%, overall psnr .46% and ssim .40%. http://www.corp.google.com/~yaowu/no_crawl/newModeContext.html Change-Id: Ia62b14485c948e2b74586118619c5eb2068b43b2
2011-12-07Minor fixes:Paul Wilkins
fixed issues caused by conflicts between two experiments. Change-Id: I56a9bd69493e4850c121ea057a6233c55777c2a5
2011-12-06Dual 16x16 inter prediction.Ronald S. Bultje
This patch introduces the concept of dual inter16x16 prediction. A 16x16 inter-predicted macroblock can use 2 references instead of 1, where both references use the same mvmode (new, near/est, zero). In the case of newmv, this means that two MVs are coded instead of one. The frame can be encoded in 3 ways: all MBs single-prediction, all MBs dual prediction, or per-MB single/dual prediction selection ("hybrid"), in which case a single bit is coded per-MB to indicate whether the MB uses single or dual inter prediction. In the future, we can (maybe?) get further gains by mixing this with Adrian's 32x32 work, per-segment dual prediction settings, or adding support for dual splitmv/8x8mv inter prediction. Gain (on derf-set, CQ mode) is ~2.8% (SSIM) or ~3.6% (glb PSNR). Most gain is at medium/high bitrates, but there's minor gains at low bitrates also. Output was confirmed to match between encoder and decoder. Note for optimization people: this patch introduces a 2nd version of 16x16/8x8 sixtap/bilin functions, which does an avg instead of a store. They may want to look and make sure this is implemented to their satisfaction so we can optimize it best in the future. Change-ID: I59dc84b07cbb3ccf073ac0f756d03d294cb19281
2011-12-06Some further QIndex issues with extended QPaul Wilkins
Resolved or factored out some further issues with Q index. Put in a 3rd order polynomial instead of less accurate power function as the best fit on gf and kf boost adjustment. Added avg_q value to use instead of ni_av_qi. Compute segment delta Q values based on avg_q. Fixed bug in adjust_maxq_qrange(). The extended range Q on the derf set, using standard data rates (which do not extend high enough to get big benefits) still show a shortfall of between 0.5 and 1% though so there would appear to be further issues that need to be tracked down. Change-Id: Icfd49b9f401906ba487ef1bef7d397048295d959
2011-12-02enabled 8x8 intra prediction modes on inter framesYaowu Xu
This commit enabled the usage of 8x8 intra prediction modes on inter frames. There are a few TODO items related to this: 1)baseline entropy need be calibrated; 2)cost of UV need to be done more properly rather than using decision only relying on Y; 3)Threshold for allowing picking 8x8 intra prediction should be lowered to lower than the B_PRED. Even with all the TODOs, tests showed consistent gain on derf set ~0.1% (PSNR:0.08% and SSIM:0.14%). It is assumed that 8x8 intra prediction will help more on large resolution clips, especially with above TODOs addressed. Change-Id: I398ada49dfc32575cfab962a569c2885111ae3ba
2011-12-02Further work on extended Q range.Paul Wilkins
Fixed some further QIndex related issues and replaced some tables (eg zbin and rounding) Also Added function (currently disabled by default) to populate the main AC and DC quantizer tables. Using the original AC range the resulting computed DC values give behavior broadly comparable on the DERF set. That is not to say that the equations will hold good over a more extended range. The purpose of this code is to make it easier to experiment with further alterations to the Q range and distribution of Q values plus the relative weights given to AC and DC. The function find_fp_qindex() ensures that changes to the Q tables are reflected in the value passed in to the first pass code. Slight experimental adjustment to static segment Q offset. Change-Id: I36186267d55dfc2a3d565d0cff7218ef300d1cd5
2011-11-29Further work on extended Q range.Paul Wilkins
Fixed bug in firspass.c call to vp8_initialize_rd_consts() This was passing in vp8_dc_quant(cm->base_qindex, cm->y1dc_delta_q) instead of (cm->base_qindex + cm->y1dc_delta_q). It just so happens that for the value 26 used for cm->base_qindex in the unextended Q case, the two give similar results. However, when using the extended Q range the two are very different. Also added more stats output and partly disabled another broken feature. Change-Id: Iddf6cf5ea8467c44b7c133f38e629f6ba6f2581e
2011-11-28changed find_near_mvs search to include a mb from last frameYaowu Xu
This is an experiment to include a mv contribution from last frame to nearest and near mv definition. Initial test showed some small though consistent gain. latest patch slightly better result ~.13%-~.18%. TODO: the entropy used to encode the mode choice, i.e. the mv counts based conditional distribution of modes should be re-collected to reflect this change, it is expected that there is some further gain from that. Change-Id: Ief1e284a36d8aa56b49ae5b360c91419ec494fa4
2011-11-23Further resolution of QIndex LUTS;Paul Wilkins
This commit resolves further QIndex look up tables to facilitate experimentation with the quantizer range. In some cases rather than remove the look up tables completely I have created functions that are called once to populate them using a formulaic approach base on the actual quantizer. The use of these functions based on best fit of data from the original tables does affect the results on some clips but across the derf test set the effect was broadly neutral. Change-Id: I8baa61c97ce87dc09a6340d56fdeb681b9345793
2011-11-22Remove redundant code (lf_or_gf and frame_lf_or_gf)Adrian Grange
Removed unused variables lf_or_gf and frame_lf_or_gf. Change-Id: I88692cd7d53e532d303c4525ee4667c1ecea3026
2011-11-22Removal of Qindex LUTS.Paul Wilkins
One of the problems arising when tweaking or adjusting the quantizer tables is that there are a lot of look up tables that depend on the QINDEX. Any adjustment to the link between QINDEX and real quantizer therefore tends to break aspects of for example the rate control. In this check in I have replaced several of the look up tables with functions that approximate the same results as the old Q luts but use a formulaic approach based on real Q values rather than QIndex. This should hopefully make it easier to experiment with changes to the Q tables without always having to go through and hand optimize a set of look up tables. Once things stabilize we may choose to re-instate luts for the sake of performance. Patch 2: Addressed Ronald's comments. vp8_init_me_luts() Added so luts only initialized once. Change-Id: Ic80db2212d2fd01e08e8cb5c7dca1fda1102be57
2011-11-21CONFIG_T8X8 experiment.:Paul Wilkins
Block the selection of 4x4 modes in key frames if 8x8 is selected. Change-Id: Ie5729ec22a999d9a1996f020bd4b941e29514992
2011-11-09T8x8 experiment merge.Paul Wilkins
For ease of testing and merging experiments I have removed in line code in encode_frame() that assigns MBs to be t8x8 or t4x4 coded segments and have moved the decision point and segment setup to the init_seg_features0 test function. Keeping everything in one place helps make sure for now that experiments using segmentation are not fighting each other. Also made sure mode selection code can't choose 4x4 modes if t8x8 is selected. Patch2: In init_seg_features() add checks for SEG_LVL_TRANSFORM active. Change-Id: Ia1767edd99b78510011d4251539f9bc325842e3a
2011-11-09Segmentation feature logic fix.Paul Wilkins
Bug fix to logic in vp8_pick_inter_mode() and vp8_rd_pick_inter_mode(). The block on the use of segment features for the cm->refresh_alt_ref_frame case was just for testing and is not correct. The special case code for alt ref can be re-enabled as an else clause. Change-Id: Ic9b57cdb5f04ea7737032b8fb953d84d7717b3ce
2011-11-08Segment signaling of TX sizePaul Wilkins
Initial attempt at using new segment feature signaling to indicate 4x4 or 8x8 transform. needs --enable-experimental --enable-t8x8 Note this is work in progress. Change-Id: Ib160d46a5d810307bfcbc79853ce1a65b5b870b7
2011-11-04Segment Feature Data AccessPaul Wilkins
No change to functionality or output. Updates to the segment feature data structure now all done through functions such as set_segdata() and get_segdata() in seg_common.c. The reason for this is to make changing the structures (if needed) and debug easier. In addition it provides a single location for subsequent addition of range and validity checks. For example valid combination of mode and reference frame. Change-Id: I2e866505562db4e4cb6f17a472b25b4465f01add
2011-11-03Segment Features. Removal of #ifdefsPaul Wilkins
Removal of configure #ifdefs so that segment features always available. Removal of code supporting old segment feature method. Still a good deal of tidying up to do. Change-Id: I397855f086f8c09ab1fae0a5f65d9e06d2e3e39f
2011-11-02Segmentation: Reference framesPaul Wilkins
Modify reference frame segmentation so that ONE or MORE reference frames may be marked as a available for a given segment. Fixed bugs relating to segment coding of INTRA and some INTER modes at the segment level. Modified Q boost for static areas based on ambient average Q. Strong results now on clips with significant static areas. (some data points in derf set as high as 9% and some static & slide show type content in YT set > 20%) Change-Id: Ia79f912efa84b977f35a23683ae3643251e24f0c
2011-10-24Further segment feature extensions.Paul Wilkins
This quite large check in includes the following: Merge in some code from Ronald (mbgraph.c) that scans a Gf/arf group. This is used as a basis for a simple segmentation for the normal frames in a gf/arf group. This code also uses satd functions from Yaowu. Adds functionality for coding the latest possible position of an EOB for blocks in the segment. (Currently 0-15 only, hence just for 4x4 dct). Where the EOB position is 0 this acts like "skip" and the normal coding of skip at the per mb level is disabled. Added functions (seg_common.c) for setting and reading segment feature elements. These may want to be optimized away at some point but while the mecahnism is in a state of flux they provide a single location for making changes and keep things a bit cleaner. This is still proof of concept code. Currently the tested feature set:- Quantizer, Loop Filter level, Reference frame, Prediction Mode, EOB end stop. TBD:- Add functions for setting and reading the feature data with range and validity checking. Handling of signed and unsigned feature data. At the moment all is assumed to be signed and a sign bit is coded but many cannot be negative. Correct handling of EOB feature with intra coded blocks. Testing/trapping of legal/illegal ref frame and mode combinations. Transform size switch plus merge and test with 8c8 DCT work Merge and test with Sumans Segmenation coding optimizations Change-Id: Iee12e83661c7abbd1e0ce6810915eb4ec35e2d8e
2011-09-30Segment coding of mode and reference frame.Paul Wilkins
Proof of concept test code that encodes mode and reference frame data at the segment level. Decode-able bit stream but some issues not yet resolved. As it this helps a little on a couple of clips but hurts on most as the basis for segmentation is unsound. To build and test, configure with --enable-experimental --enable-segfeatures Change-Id: I22a60774f69273523fb152db8c31f4b10b07c7f4
2011-09-16enable selecting&transmitting to for intra mode entropyYaowu Xu
This commit added a 3 bit index to the bitstream, the index is used to look into the intra mode coding entropy context table. The commit uses the mode stats to calculate the cost of transmitting modes using 8 possible entropy distributions, and selects the distribution that provides the lowest cost to do the actual mode coding. Initial test show this provides additional .2%~.3% gain over quantizer adaptive intra mode coding. So the adaptive intra mode coding provides a total of .5%(psnr) to .6% gain(ssim) combined for all-key-encoding To build and test, configure with --enable-experimental --enable-qimode Change-Id: I7c41cd8bfb352bc1fe7c5da1848a58faea5ed74a
2011-09-16add 8x8 intra prediction modesYaowu Xu
Patch 1 to Patch 3 is an initial implementation of 8x8 intra prediction modes, here are with the following assumptions: a. 8x8 has 4 prediction modes DC, H, V and TM b. UV 4x4 block use the same mode as corresponding 8x8 area c. i8x8 modes are enabled for key frame only for now Patch 4: d. removed debug code from previous patches Patch 5: e. added stats code to collect entropy stats and further cleaned up Patch 6: f. changed mode stats code to collect finer stats of modes Patch 7: g. normalized i8x8 modes distribution to total at 256 (8bits). Patch 8: h. fixed a bug in decoder and removed debug printf output. Patch 9: i. more cleanups to address paul's comment Patch 10: j. messy rebase/merges to bring the commit up to date. Tests on HD clips encoded with all key frame showing consistent gain on all clips and all metrics:~0.5%(psnr) and 0.6%(ssim): http://www.corp.google.com/~yaowu/no_crawl/i8x8hd_allkey_fixedq.html To build and test, configure with: --enable-experimental --enable-i8x8 Change-Id: I9813fe07ae48cab5fdb5d904bca022514ad01e7f
2011-09-16Merge remote branch 'internal/upstream' into HEADJohn Koleszar
Conflicts: vp8/decoder/decodframe.c vp8/encoder/encodeframe.c vp8/encoder/encodemb.c Change-Id: I6e0d1669e4409a2dfd73ba2c7038d730842d3953
2011-08-24Removed bmi copy to/from BLOCKDScott LaVarnway
for SPLITMV and B_PRED modes. Modified code to use the bmi found in mode_info_context instead of BLOCKD. On the decode side, the uvmvs are calculated only when required, instead of every macroblock. This is WIP. (bmi should eventually be removed from BLOCKD) Small performance gains noticed for RT encodes and decodes.(VGA) Change-Id: I2ed7f0fd5ca733655df684aa82da575c77a973e7
2011-08-11Merge remote branch 'internal/upstream-experimental' into HEADJohn Koleszar
Conflicts: vp8/decoder/detokenize.c vp8/decoder/onyxd_if.c vp8/vp8_common.mk Change-Id: Ifca1108186a8bc715da86a44021ee2fa5550b5b8
2011-07-28Merge remote branch 'origin/master' into experimentalJohn Koleszar
Change-Id: Iaca87acc9726b5173d638528684d154538ec01e6
2011-07-27Preload reference area in sub-pixel motion search (real-time mode)Yunqing Wang
This change implemented same idea in change "Preload reference area to an intermediate buffer in sub-pixel motion search." The changes were made to vp8_find_best_sub_pixel_step() and vp8_find_best_half _pixel_step() functions which are called when speed >= 5. Test result (using tulip clip): 1. On Core2 Quad machine(Linux) rt mode, speed (-5 ~ -8), encoding speed gain: 2% ~ 3% rt mode, speed (-9 ~ -11), encoding speed gain: 1% ~ 2% rt mode, speed (-12 ~ -14), no noticeable encoding speed gain 2. On Xeon machine(Linux) Test on speed (-5 ~ -14) didn't show noticeable speed change. Change-Id: I21bec2d6e7fbe541fcc0f4c0366bbdf3e2076aa2
2011-07-27Merge "Fix range checks in motion search"Yunqing Wang
2011-07-27Fix range checks in motion searchYunqing Wang
There were some situations that the start motion vectors were out of range. This fix adjusted range checks to make sure they are checked and clamped. Change-Id: Ife83b7fed0882bba6d1fa559b6e63c054fd5065d
2011-07-27Merge remote branch 'origin/master' into experimentalJohn Koleszar
Change-Id: I1ae82458536ba2f0969e1bea78f41cd16fe96b79
2011-07-26cosmetics: consistently use [u]int64_tJames Zern
Removes mixed usage of (unsigned) long long and INT64. Fixes Issue #208. Change-Id: I220d3ed5ce4bb1280cd38bb3715f208ce23cf83a
2011-07-21fix more merge issuesYaowu Xu
With this fix, the experimental branch now builds and encodes correctly with the following two configure options respectively: --enable-experimental --enable-t8x8 --enable-experimental Change-Id: I3147c33c503fe713a85fd371e4f1a974805778bf
2011-07-20Add 8x8 transform to experimental branchDeb Mukherjee
Please refer to previous commit messages for detailed info: https://on2-git.corp.google.com/g/#change,5940 https://on2-git.corp.google.com/g/#change,6045 Change-Id: I8b16992f2f69c5a808ad40a3e32ef589cce7c59d
2011-07-13Merge remote branch 'origin/master' into experimentalJohn Koleszar
Change-Id: I9dab62c24d71f71cdc36732ed8ed469bee67d7e1
2011-07-13Merge remote branch 'internal/upstream' into HEADJohn Koleszar
2011-07-09Merge remote branch 'origin/master' into experimentalJohn Koleszar
Change-Id: Ica63d16cb39e2d65a3414f0b9f86c8a64112dfa3
2011-07-09Merge remote branch 'internal/upstream' into HEADJohn Koleszar
2011-07-08Minor change in pick_inter_mode()Yunqing Wang
Scott suggested to move vp8_mv_pred() under "case NEWMV" to save extra checks. Change-Id: I09e69892f34a08dd425a4d81cfcc83674e344a20
2011-07-08Adjust full-pixel clamping and motion vector limit calculationYunqing Wang
Do mvp clamping in full-pixel precision instead of 1/8-pixel precision to avoid error caused by right shifting operation. Also, further fixed the motion vector limit calculation in change: b7480454706a6b15bf091e659cd6227ab373c1a6 Change-Id: Ied88a4f7ddfb0476eb9f7afc6ceeddbf209fffd7
2011-07-01Merge remote branch 'origin/master' into experimentalJohn Koleszar
Change-Id: Iaf6e9e14d0cfe5cef3895cfb68524d51139a6d23
2011-07-01Merge remote branch 'internal/upstream' into HEADJohn Koleszar
2011-06-30Merge "Copy macroblock data to a buffer before encoding it"Yunqing Wang
2011-06-30Bug fix in motion vector limit calculationYunqing Wang
Motion vector limits are calculated using right shifts, which could give wrong results for negative numbers. James Berry's test on one clip showed encoder produced some artifacts. This change fixed that. Change-Id: I035fc02280b10455b7f6eb388f7c2e33b796b018
2011-06-30Merge remote branch 'internal/upstream-experimental' into HEADJohn Koleszar
Conflicts: vp8/encoder/bitstream.c Change-Id: I44c00f98dcb99eb728ce4f5256aefb135a711a74
2011-06-30Merge remote branch 'origin/master' into experimentalJohn Koleszar
Change-Id: I35c9ca116aecd0d03e762942d9cf1289edb4f23d
2011-06-28Use MAX_ENTROPY_TOKENS and ENTROPY_NODES more consistentlyJohn Koleszar
There were many instances in the code of vp8_coef_tokens and vp8_coef_tokens-1, which was a preprocessor macro despite the naming convention. Replace these with MAX_ENTROPY_TOKENS and ENTROPY_NODES, respectively. Change-Id: I72c4f6c7634c94e1fa066cd511471e5592c748da
2011-06-28Merge remote branch 'internal/upstream-experimental' into HEADJohn Koleszar
2011-06-23Copy macroblock data to a buffer before encoding itYunqing Wang
I got this idea from Pascal (Thanks). Before encoding a macroblock, copy it to a 16x16 buffer, and then read source data from there instead. This will help keep the source data in cache, and help with the performance. Change-Id: Id05f4cb601299150511d59dcba0ae62c49b5b757
2011-06-23fix build issues for experimental branchYaowu Xu
experimental branch build was broken from some merge artifacts, this commit fixes those issues to enable the experimental branch to build. Change-Id: Ic52b2d2f1d1b80abb7ecaa4c0927bcf887ac0c2a