Monday, June 25, 2012

Multiple Rackspace Security Vulnerabilities Discovered

Several Rackspace cloud security vulnerabilities have been discovered. Problems with a Rackspace-supplied agent running on cloud servers have been documented, along with a much more severe issue with the method Rackspace has used to generate default root passwords for cloud servers. In short, root password hashes were generated using a legacy hashing function (resulting in cryptographically weaker hashes to start with) and used the system hostname as the first portion of the password.

Thus, cloud servers deployed in this manner would only consider the first eight characters of the root password significant, potentially allowing an attacker with simple knowledge of this weakness and the system's hostname to remotely log in via SSH as root. As hostnames are easily determined by a number of means, the potential for damage is significant. Additionally, evidences exists that Rackspace is storing customer root passwords internally in a recoverable format.

These issues were reported to the company, as described in the previously published Rackspace cloud security pre-advisory. To date, Rackspace has apparently mitigated some of the issues for newly deployed instances, but serious questions remain regarding the integrity of servers in the wild which were deployed using the flawed methods. As the company is a large hosting provider with well known IP space, and the time at which these problems were first manifested is unknown, the number of vulnerable servers could be significant.

During the late afternoon and early evening of April 3, 2004, this supercell thunderstorm dropped 2 inch-diameter hail over Chaparral, N.M. causing widespread damage. [NWS/NOAA/Wikipedia]

nova-agent root password hashing weakness

Rackspace has recently begun installing nova-agent on customer servers to aid with various tasks, such as file injection during deployment and recovery of root passwords. The nova-agent communicates with Rackspace's infrastructure via the Xen bus. Although the agent is largely written in Python (with some C for flavor), the source code is not included on customer instances. It is, however, available at

(Before you ask, killing it outright is a bad idea. The Rackspace Cloud control panel will hang for a considerable length of time waiting for a reply from nova-agent when it needs to talk to it, and won't let you do anything until it times out. We invite the community to create a dummy nova-agent that allows us to disable this rootkit without breaking our servers.)

Prior to commit ab52bd9 (2012-02-13), passwords were set using the legacy DES-based crypt scheme. This resulted in only the first eight characters of root passwords set externally (via the control panel, API, etc) being significant. In other words, root passwords set by Rackspace were truncated to eight characters.

This significantly limited the possible set of passwords (to 56 bits), allowing for easier-than-expected remote bruteforcing. Also, should an attacker gain access to /etc/shadow, the root password could be recovered with ease.

Workaround: Reset the root password using the normal Linux tool (passwd).
Fix: Better password hashes are being generated with new installs of the following images, as of at least June 17th: 
  • Arch 2011.10 
  • CentOS 6.2 
  • Debian 6.0 
  • Gentoo 11.0 
  • Fedora 17 
  • OpenSUSE 12.1 
  • Red Hat Enterprise Linux 6.1 
  • Ubuntu 10.04 LTS 
  • Ubuntu 12.04 LTS 
CentOS 6.2 and OpenSUSE 12.1 are using the SHA2-based $6$ scheme, while the rest are using the MD5-based $1$ scheme. Other images have not been tested. It is recommended that you visually verify the password hashing scheme used in your /etc/shadow.

Predictable default root password generation

Prior to (approximately) May 14th, Rackspace Cloud Server generated default passwords using an algorithm similar to:
hostname = ''
alphabet = string.letters[0:52] + string.digits
randombits = ''.join(random.choice(alphabet) for _ in range(9))
initial_password = hostname + randombits
This would result in a root password of, as an example,
 Combined with the root password hashing weakness, mentioned above, this means that the root password on any server deployed prior to May 14th with a hostname of more than 8 characters is, effectively, the server's hostname. In other words, the default root password for the above server would be
This significantly reduces the set of possible passwords to check, particularly if the hostname of the machine can be easily recovered (perhaps from a HTTP error or SMTP server response or mail headers, etc).

Workaround: Reset the root password using the normal Linux tool (passwd).
Fix: The order of the initial password has been reversed, such that it is randombits + hostname. Or, for the above, Forcing the user to set their own root password would be a more suitable fix, however.

On-the-fly silent root password resets

Prior to the week of May 14th, the "Reset Root Password" option on the Rackspace Cloud control panel would generate a new password (using the algorithm described in #2) and, if the instance were running nova-agent, set the root password without requiring a server reboot, and without notifying any of the contacts on the account. The only indication that the root password had changed was a log entry similar to the following in /var/log/nova-agent.log:
2012-05-13 19:24:30,090 [INFO] Received command 'password' with argument: 'eKWjeHx/2qk+1fCFNurF8/YFfm7N7DVTjKC0hDGrcNhDWNjoY/4mGWsWb5rYpXN0'
(Note that nova-agent does NOT use the standard facilities for setting passwords and instead edits /etc/shadow directly, so this would not be announced via syslog, and it would not appear in the default reports of log analysis/summarization tools.)

This could be triggered by anyone/anything with access to the Cloud control panel or the API.

Fix: As of sometime in mid-May, e-mails are now sent to account contacts when the password is changed via the Cloud control panel. API calls have not yet been tested. (And no, the plaintext e-mail will no longer include the new root password.)

Insecure password storage for generated root passwords

When a server is initially deployed, or the Reset Root Password button is pressed, a root password is generated (per #2 above). On any subsequent rebuilds of this server, the same root password is used as the initial password. This suggests one of two possible problems: 1) There is a deterministic algorithm to generate the apparently-random part of the password, perhaps including a counter to permit new passwords to be generated, or 2) The Rackspace Cloud control panel saves the password in plaintext, or recoverable cyphertext, for future use.

We believe the first option is not what is happening, as rebuilds of servers created before the change in #2 above will re-use the existing hostname + random password. So, it seems likely that Rackspace is storing -- for reasons unfathomable -- randomly-generated initial root passwords. It is probable that someone with privileged access to the Rackspace Cloud platform could access these passwords.
It is not known if Rackspace stores new passwords which are set via the API.