shithub: libvpx

Info  •  Files  •  Log  •  Branches

Fix error in get_smooth_intra_threshold() for 10/12 bit.

This function seems to scale the threshold for testing an
SSE value in the wrong direction for 10 and 12 bit inputs.

Also for a true SSE the scalings should probably be << 4 and 8

Change-Id: Iba8047b3f70d04aa46d9688a824f3d49c1c58e90

--- a/vp9/encoder/vp9_firstpass.c

+++ b/vp9/encoder/vp9_firstpass.c

@@ -532,10 +532,10 @@

         ret_val = SMOOTH_INTRA_THRESH;

         break;

       case VPX_BITS_10:

-        ret_val = SMOOTH_INTRA_THRESH >> 2;

+        ret_val = SMOOTH_INTRA_THRESH << 4;

         break;

       case VPX_BITS_12:

-        ret_val = SMOOTH_INTRA_THRESH >> 4;

+        ret_val = SMOOTH_INTRA_THRESH << 8;

         break;

       default:

         assert(0 && "cm->bit_depth should be VPX_BITS_8, "