Limit bn_mul_mont() usage to sizes less than or equal to 8192 bits.
authorjsing <jsing@openbsd.org>
Tue, 7 Mar 2023 06:28:36 +0000 (06:28 +0000)
committerjsing <jsing@openbsd.org>
Tue, 7 Mar 2023 06:28:36 +0000 (06:28 +0000)
commit40a54edf7638fa6c300cdbd7c6ad81180124d16b
tree4c18c814bc8045c852d25ec5d380abddec9bc71b
parent6a253d70dbd70da4fa8ca67a9e99b52569eddb28
Limit bn_mul_mont() usage to sizes less than or equal to 8192 bits.

The assembly bn_mul_mont() implementations effectively use alloca() to
allocate space for computation (at up to 8x the input size), without
any limitation. This means that sufficiently large inputs lead to the
stack being blown. Prevent this by using the C based implementation
instead.

Thanks to Jiayi Lin <jlin139 at asu dot edu> for reporting this to us.

ok beck@ tb@
lib/libcrypto/bn/bn_mont.c