List:       openssl-users
Subject:    64 bit problem in RSA_generate_key in 0.9.5a
From:       Karsten Spang <ks () bellesystems ! com>
Date:       2000-05-29 13:46:24

I have found a problem in the generation of an RSA key. Under 64 bit HP-UX,
the problem manifests itself as RSA_F4 becoming 0x1000100010001. The bug
is that 1 is an int, and according to the ANSI C standard, shifting more
than the size of the an expression is undefined. On this architechture,
only the 5 least significant bits of the shift is used, thus 1<<32==1 !
The patch below changes 1 to 1UL, and 1UL<<32!=1UL when a long is 64 bits.
--
Karsten Spang
Senior Software Developer, Ph.D.
Belle Systems A/S
Tel.:   +45 59 44 25 00
Fax.:   +45 59 44 25 88
E-mail: kspang@bellesystems.com
Web:    http://www.bellesystems.com/
Defining the Future of IP Services

*** rsa_gen.c.dist	Sat Feb  5 15:17:30 2000
--- rsa_gen.c	Mon May 29 15:19:31 2000
***************
*** 95,101 ****
  	 * unsigned long can be larger */
  	for (i=0; i<sizeof(unsigned long)*8; i++)
  		{
! 		if (e_value & (1<<i))
  			BN_set_bit(rsa->e,i);
  		}
  #else
--- 95,101 ----
  	 * unsigned long can be larger */
  	for (i=0; i<sizeof(unsigned long)*8; i++)
  		{
! 		if (e_value & (1UL<<i))
  			BN_set_bit(rsa->e,i);
  		}
  #else
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
User Support Mailing List                    openssl-users@openssl.org
Automated List Manager                           majordomo@openssl.org