The RSA_FLAG_SIGN_VER is not yet supported and the current code uses
the rsa_priv_enc() and rsa_pub_dec() callbacks for sign and verify
operations.
A tale from OpenSSL's rsa.h:
New sign and verify functions: some libraries don't allow arbitrary
data to be signed/verified: this allows them to be used. Note: for
this to work the RSA_public_decrypt() and RSA_private_encrypt() should
*NOT* be used RSA_sign(), RSA_verify() should be used instead. Note:
for backwards compatibility this functionality is only enabled if the
RSA_FLAG_SIGN_VER option is set in 'flags'.
In OpenSSL, RSA engines should provide the rsa_sign() and rsa_verify()
callbacks and this should be the default. By the "default" is
disabled by default and RSA engines that provide extra sign and verify
callbacks have to set the non-default RSA_FLAG_SIGN_VER flag. This is
not used by OpenSSL's own RSA code and was only set by two non-default
RSA engines: IBM 4758 and Windows CAPI - both of them got removed from
our library. And btw., this comment about the new non-default default
was added in 1999.
Thanks to Piotr Sikora, who pointed out that I didn't handle the
sign/verify case.