Add size_t to int checks for SSL functions.

Add size_t to int checks for SSL functions.

libtls accepts size_t for lengths but libssl accepts int.  This verifies
that the input does not exceed INT_MAX.  It also avoids truncating size_t
when comparing with int and adds printf-style attributes for
tls_set_error().

with input from deraadt@ and tedu@
ok tedu@