Bound cofactor in EC_GROUP_set_generator()

Instead of bounding only bounding the group order, also bound the
cofactor using Hasse's theorem. This could probably be made a lot
tighter since all curves of cryptographic interest have small
cofactors,  but for now this is good enough.

A timeout found by oss-fuzz creates a "group" with insane parameters
over a 40-bit field: the order is 14464, and the cofactor has 4196223
bits (which is obviously impossible by Hasse's theorem). These led to
running an expensive loop in ec_GFp_simple_mul_ct() millions of times.

Fixes oss-fuzz #46056

Diagnosed and fix joint with jsing

ok inoguchi jsing (previous version)

diff --git a/lib/libcrypto/ec/ec_lib.c b/lib/libcrypto/ec/ec_lib.c
index 455d44a..888f1ed 100644 (file)
--- a/lib/libcrypto/ec/ec_lib.c
+++ b/lib/libcrypto/ec/ec_lib.c
@@ -1,4 +1,4 @@
-/* $OpenBSD: ec_lib.c,v 1.43 2022/03/29 13:48:40 tb Exp $ */
+/* $OpenBSD: ec_lib.c,v 1.44 2022/03/29 14:03:12 tb Exp $ */
 /*
  * Originally written by Bodo Moeller for the OpenSSL project.
  */
@@ -385,6 +385,12 @@ EC_GROUP_set_generator(EC_GROUP *group, const EC_POINT *generator,
        } else if (!ec_guess_cofactor(group))
                return 0;
 
+       /* Use Hasse's theorem to bound the cofactor. */
+       if (BN_num_bits(&group->cofactor) > BN_num_bits(&group->field) + 1) {
+               ECerror(EC_R_INVALID_GROUP_ORDER);
+               return 0;
+       }
+
        return 1;
 }