diff options
| author | James Zern <jzern@google.com> | 2012-11-05 17:14:32 -0800 |
|---|---|---|
| committer | James Zern <jzern@google.com> | 2012-11-05 17:14:32 -0800 |
| commit | e47d9f1d072e99c8c952e8ffa278ff7aef2ef1b7 (patch) | |
| tree | 254d3d94bcb602d6207f5e703731cd5a48c5c8a7 | |
| parent | f2541f8a4a1a6503c43de0ea8954f4b7ea2d2499 (diff) | |
| download | libvpx-e47d9f1d072e99c8c952e8ffa278ff7aef2ef1b7.tar libvpx-e47d9f1d072e99c8c952e8ffa278ff7aef2ef1b7.tar.gz libvpx-e47d9f1d072e99c8c952e8ffa278ff7aef2ef1b7.tar.bz2 libvpx-e47d9f1d072e99c8c952e8ffa278ff7aef2ef1b7.zip | |
rd_pick_inter_mode: prevent signed integer overflow
calculate the txfm_cache difference first as both values may be
INT64_MAX with the intent that they cancel each other out.
Change-Id: I214d072458e1b24f60289974e6302af1aff7b66c
| -rw-r--r-- | vp9/encoder/rdopt.c | 4 |
1 files changed, 3 insertions, 1 deletions
diff --git a/vp9/encoder/rdopt.c b/vp9/encoder/rdopt.c index ffff248ac..d0f905b1d 100644 --- a/vp9/encoder/rdopt.c +++ b/vp9/encoder/rdopt.c @@ -4090,7 +4090,9 @@ static void rd_pick_inter_mode(VP9_COMP *cpi, MACROBLOCK *x, for (i = 0; i < NB_TXFM_MODES; i++) { int64_t adj_rd; if (this_mode != B_PRED) { - adj_rd = this_rd + txfm_cache[i] - txfm_cache[cm->txfm_mode]; + const int64_t txfm_mode_diff = + txfm_cache[i] - txfm_cache[cm->txfm_mode]; + adj_rd = this_rd + txfm_mode_diff; } else { adj_rd = this_rd; } |