Rework the MD setting in the RSA ASN.1 method

Rework the MD setting in the RSA ASN.1 method

This streamlines the code to use safer idioms, do proper error checking
and be slightly less convoluted. Sprinkle a few references to RFC 8017
and explain better what we are doing and why. Clarify ownership and use
more consistent style.

This removes the last internal use of X509_ALGOR_set_md().

ok jsing