Cap the size of numbers we check for primality

We refuse to generate RSA keys larger than 16k and DH keys larger than 10k.
Primality checking with adversarial input is a DoS vector, so simply don't
do this. Introduce a cap of 32k for numbers we try to test for primality,
which should be more than large enough for use withing a non-toolkit crypto
library. This is one way of mitigating the DH_check()/EVP_PKEY_param_check()
issue.

ok jsing miod

diff --git a/lib/libcrypto/bn/bn_prime.c b/lib/libcrypto/bn/bn_prime.c
index a09bac4..5a4aa50 100644 (file)
--- a/lib/libcrypto/bn/bn_prime.c
+++ b/lib/libcrypto/bn/bn_prime.c
@@ -1,4 +1,4 @@
-/* $OpenBSD: bn_prime.c,v 1.33 2023/07/08 12:21:58 beck Exp $ */
+/* $OpenBSD: bn_prime.c,v 1.34 2023/07/20 06:26:27 tb Exp $ */
 /* Copyright (C) 1995-1998 Eric Young (eay@cryptsoft.com)
  * All rights reserved.
  *
@@ -240,6 +240,8 @@ BN_is_prime_ex(const BIGNUM *a, int checks, BN_CTX *ctx_passed, BN_GENCB *cb)
 }
 LCRYPTO_ALIAS(BN_is_prime_ex);
 
+#define BN_PRIME_MAXIMUM_BITS (32 * 1024)
+
 int
 BN_is_prime_fasttest_ex(const BIGNUM *a, int checks, BN_CTX *ctx_passed,
     int do_trial_division, BN_GENCB *cb)
@@ -249,6 +251,15 @@ BN_is_prime_fasttest_ex(const BIGNUM *a, int checks, BN_CTX *ctx_passed,
        if (checks < 0)
                return -1;
 
+       /*
+        * Prime numbers this large do not appear in everyday cryptography
+        * and checking such numbers for primality is very expensive.
+        */
+       if (BN_num_bits(a) > BN_PRIME_MAXIMUM_BITS) {
+               BNerror(BN_R_BIGNUM_TOO_LONG);
+               return -1;
+       }
+
        if (checks == BN_prime_checks)
                checks = BN_prime_checks_for_size(BN_num_bits(a));