Fix incorrect RSA_public_decrypt() return check

Fix incorrect RSA_public_decrypt() return check

RSA_public_decrypt() returns <= 0 on error. Assigning to a size_t and
checking for == 0 is not the right thing to do here. Neither is blindly
turning the check into <= 0...

Found by Niels Dossche

ok jsing