Align hex_to_string with OpenSSL 1.1 behavior

Align hex_to_string with OpenSSL 1.1 behavior

This is a bit of a strange one. When this function was moved and renamed,
somehow checks for NULL pointers and 0 length were lost. Whether that was
due to great review missing it or great review suggesting it, is unclear.

Now the function can actually legitimately be called with a length of 0
(as ASN.1 OCTET STRINGS can have length 0) and "" is the appropriate
representation for that, so the fix is to allocate a 0 octet. That much
was correct. What was completely missed is that a long can be negative
which will then still lead to an out-of-bounds access. So fix that as
well.

Finally, don't malloc 3 * len + 1 without overflow checking. Rather
use calloc's internal checks. The + 1 isn't really needed anyway.

All this is still really gross and can be done much more cleanly and
safely with CBB/CBS. This will done later once we have better regress
coverage.

ok jsing