The RSA_FLAG_SIGN_VER is not yet supported and the current code uses

the rsa_priv_enc() and rsa_pub_dec() callbacks for sign and verify
operations.

A tale from OpenSSL's rsa.h:

  New sign and verify functions: some libraries don't allow arbitrary
  data to be signed/verified: this allows them to be used. Note: for
  this to work the RSA_public_decrypt() and RSA_private_encrypt() should
  *NOT* be used RSA_sign(), RSA_verify() should be used instead. Note:
  for backwards compatibility this functionality is only enabled if the
  RSA_FLAG_SIGN_VER option is set in 'flags'.

In OpenSSL, RSA engines should provide the rsa_sign() and rsa_verify()
callbacks and this should be the default.  By the "default" is
disabled by default and RSA engines that provide extra sign and verify
callbacks have to set the non-default RSA_FLAG_SIGN_VER flag.  This is
not used by OpenSSL's own RSA code and was only set by two non-default
RSA engines: IBM 4758 and Windows CAPI - both of them got removed from
our library.  And btw., this comment about the new non-default default
was added in 1999.

Thanks to Piotr Sikora, who pointed out that I didn't handle the
sign/verify case.

diff --git a/usr.sbin/relayd/ca.c b/usr.sbin/relayd/ca.c
index c08a64d..4804d8e 100644 (file)
--- a/usr.sbin/relayd/ca.c
+++ b/usr.sbin/relayd/ca.c
@@ -1,4 +1,4 @@
-/*     $OpenBSD: ca.c,v 1.1 2014/04/18 13:55:26 reyk Exp $     */
+/*     $OpenBSD: ca.c,v 1.2 2014/04/18 14:32:22 reyk Exp $     */
 
 /*
  * Copyright (c) 2014 Reyk Floeter <reyk@openbsd.org>
@@ -411,6 +411,9 @@ ca_engine_init(struct relayd *x_env)
            (rsa_default = ENGINE_get_RSA(e)) == NULL)
                return (-1);
 
+       if (rsa_default->flags & RSA_FLAG_SIGN_VER)
+               fatalx("unsupported RSA engine");
+
        if (rsa_default->rsa_mod_exp == NULL)
                rsae_method.rsa_mod_exp = NULL;
        if (rsa_default->rsa_mod_exp == NULL)