diff options
| author | Jingning Han <jingning@google.com> | 2018-06-28 12:06:12 -0700 |
|---|---|---|
| committer | Jingning Han <jingning@google.com> | 2018-06-28 12:06:44 -0700 |
| commit | 3e3ca7faa6c9e68be3058e175187f382f050f8da (patch) | |
| tree | 691b642bb0ae54bc7bdade25de7ecf8e41cbeb98 | |
| parent | 4a42d0918e199233723195ea0c9561a2343df1bd (diff) | |
| download | libvpx-3e3ca7faa6c9e68be3058e175187f382f050f8da.tar libvpx-3e3ca7faa6c9e68be3058e175187f382f050f8da.tar.gz libvpx-3e3ca7faa6c9e68be3058e175187f382f050f8da.tar.bz2 libvpx-3e3ca7faa6c9e68be3058e175187f382f050f8da.zip | |
Avoid operation on INT64_MAX value
If the rate cost returns as INT_MAX, directly set the rdcost as
INT64_MAX.
Change-Id: I3ea1963aff10040dd9cef805beed9aebeedb93bc
| -rw-r--r-- | vp9/encoder/vp9_encodeframe.c | 7 |
1 files changed, 4 insertions, 3 deletions
diff --git a/vp9/encoder/vp9_encodeframe.c b/vp9/encoder/vp9_encodeframe.c index f6fcd9d33..8b426451c 100644 --- a/vp9/encoder/vp9_encodeframe.c +++ b/vp9/encoder/vp9_encodeframe.c @@ -1967,13 +1967,14 @@ static void rd_pick_sb_modes(VP9_COMP *cpi, TileDataEnc *tile_data, vp9_caq_select_segment(cpi, x, bsize, mi_row, mi_col, rd_cost->rate); } - rd_cost->rdcost = RDCOST(x->rdmult, x->rddiv, rd_cost->rate, rd_cost->dist); - x->rdmult = orig_rdmult; // TODO(jingning) The rate-distortion optimization flow needs to be // refactored to provide proper exit/return handle. - if (rd_cost->rate == INT_MAX) rd_cost->rdcost = INT64_MAX; + if (rd_cost->rate == INT_MAX) + rd_cost->rdcost = INT64_MAX; + else + rd_cost->rdcost = RDCOST(x->rdmult, x->rddiv, rd_cost->rate, rd_cost->dist); ctx->rate = rd_cost->rate; ctx->dist = rd_cost->dist; |