shithub: openh264

Info  •  Files  •  Log  •  Branches

Avoid trying to downscale layers to dimensions below 2

When calculating what resolution to actually downscale to,
it can end up smaller than what the caller set. When scaling
down to resolutions close to the limit of allowed values,
this can end up setting values lower than the limit.

Previously, e.g. a downscale from 2266x8 to 566x2 will end
up as 566x1 after this calculation. When scaling to a
566x1, the chroma plane gets a height of 0, which doesn't
make sense, and which breaks e.g. the SSE2 scaler. Therefore,
make sure none of the dimensions end up set below 2.

--- a/codec/encoder/core/src/wels_preprocess.cpp

+++ b/codec/encoder/core/src/wels_preprocess.cpp

@@ -430,9 +430,9 @@

     if (iInputWidthXDstHeight > iInputHeightXDstWidth) {

       pScaledPicture->iScaledWidth[iSpatialIdx] = iCurDstWidth;

-      pScaledPicture->iScaledHeight[iSpatialIdx] = iInputHeightXDstWidth / kiInputPicWidth;

+      pScaledPicture->iScaledHeight[iSpatialIdx] = WELS_MAX (iInputHeightXDstWidth / kiInputPicWidth, 2);

     } else {

-      pScaledPicture->iScaledWidth[iSpatialIdx] = iInputWidthXDstHeight / kiInputPicHeight;

+      pScaledPicture->iScaledWidth[iSpatialIdx] = WELS_MAX (iInputWidthXDstHeight / kiInputPicHeight, 2);

       pScaledPicture->iScaledHeight[iSpatialIdx] = iCurDstHeight;

     }

   }