shithub: libvpx

Info  •  Files  •  Log  •  Branches

Prevent negative cost for highbitdepth

Adds proper scaling for highbitdepth in a rdopt cost.

Change-Id: I066694799a7f491b830945ef1c66eb202071c355

--- a/vp9/encoder/vp9_rdopt.c

+++ b/vp9/encoder/vp9_rdopt.c

@@ -491,7 +491,9 @@

       if (x->plane[plane].eobs[block]) {

         int64_t dc_correct = coeff[0] * coeff[0] -

             (coeff[0] - dqcoeff[0]) * (coeff[0] - dqcoeff[0]);

-

+#if CONFIG_VP9_HIGHBITDEPTH

+        dc_correct >>= ((xd->bd - 8) * 2);

+#endif

         if (tx_size != TX_32X32)

           dc_correct >>= 2;