summaryrefslogtreecommitdiff
path: root/test/variance_test.cc
AgeCommit message (Collapse)Author
2012-11-27Add vp9_ prefix to all vp9 filesJohn Koleszar
Support for gyp which doesn't support multiple objects in the same static library having the same basename. Change-Id: Ib947eefbaf68f8b177a796d23f875ccdfa6bc9dc
2012-11-15support building vp8 and vp9 into a single libJohn Koleszar
Change-Id: Ib8f8a66c9fd31e508cdc9caa662192f38433aa3d
2012-11-06Fix variance (signed integer) overflowJames Zern
In the variance calculations the difference is summed and later squared. When the sum exceeds sqrt(2^31) the value is treated as a negative when it is shifted which gives incorrect results. To fix this we force the multiplication to be unsigned. The alternative fix is to shift sum down by 4 before multiplying. However that will reduce precision. For 16x16 blocks the maximum sum is 65280 and sqrt(2^31) is 46340 (and change). This change is based on: 1698234 Missed some variance casts fea3556 Fix variance overflow Change-Id: I2c61856cca9db54b9b81de83b4505ea81a050a0f