diff options
| author | Alex Converse <aconverse@google.com> | 2015-11-20 13:48:51 -0800 |
|---|---|---|
| committer | Alex Converse <aconverse@google.com> | 2015-11-24 16:32:01 -0800 |
| commit | 022c848b4d04a2c1f56a54748eee503767269023 (patch) | |
| tree | 3a27a7582fb2719361e679c5ad738c69db8193cb /test/variance_test.cc | |
| parent | 64a60ce3ba78c33a69e0adf23f68638d644015e3 (diff) | |
| download | libvpx-022c848b4d04a2c1f56a54748eee503767269023.tar libvpx-022c848b4d04a2c1f56a54748eee503767269023.tar.gz libvpx-022c848b4d04a2c1f56a54748eee503767269023.tar.bz2 libvpx-022c848b4d04a2c1f56a54748eee503767269023.zip | |
Change highbd variance rounding to prevent negative variance.
Always round sum error and sum square error toward zero in variance
calculations. This prevents variance from becoming negative.
Avoiding rounding variance at all might be better but would be far
more invasive.
Change-Id: Icf24e0e75ff94952fc026ba6a4d26adf8d373f1c
Diffstat (limited to 'test/variance_test.cc')
| -rw-r--r-- | test/variance_test.cc | 4 |
1 files changed, 2 insertions, 2 deletions
diff --git a/test/variance_test.cc b/test/variance_test.cc index 6f50f78f2..1e10cdee1 100644 --- a/test/variance_test.cc +++ b/test/variance_test.cc @@ -54,11 +54,11 @@ static void RoundHighBitDepth(int bit_depth, int64_t *se, uint64_t *sse) { switch (bit_depth) { case VPX_BITS_12: *sse = (*sse + 128) >> 8; - *se = (*se + 8) >> 4; + *se = *se / (1 << 4); break; case VPX_BITS_10: *sse = (*sse + 8) >> 4; - *se = (*se + 2) >> 2; + *se = *se / (1 << 2); break; case VPX_BITS_8: default: |