Where has that cloud server been?

Recently I've wanted to beef up my malware analysis capabilities for the malicious hosts list. I wanted some "burner servers" spread across various providers and countries to grab samples. Being, as always, the-researcher-on-a-budget, I've become intricately familiar with the folks over at lowendbox.com and the deals they post. Now, as one can expect, with cheap providers, you find some diamonds in the rough... and you also find some providers that are worth exactly what you paid for them. Sometimes this means slow bandwidth, high latency, and oversold processors, but other times it can burn you on the security side as well.

Let's talk about a provider I started to use called FlyByNightVPS (names and other details obviously changed). I signed up for FlyByNight, forked over my payment details, and a few minutes later, the e-mail with my server's details arrived. I fired up my ssh client and was jazzed up for some good ol' configurin'

ben@workstation:~$ ssh $MY_IP
The authenticity of host '$MY_IP ($MY_IP)' can't be established.
RSA key fingerprint is d3:ad:b3:3f:ea:5b:52:f5:ab:ad:1d:3a:e7:f0:8d:99.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '$MY_IP' (RSA) to the list of known hosts.
ben@$MY_IP's password:
Last login: Sat Apr 11 08:49:58 2012 from $SOME_UK_HOSTNAME
[ben@cloudaloudadingdong ~]$

Hmmph. That's odd. I don't remember being in the UK in May, let along logging into this server that shouldn't have existed prior to a couple of minutes ago. This smelled bad. Putting on my investigator hat, I started poking around to determine if there were any other suprises on the system: The logs beyond the lastlog was cleared, the .bash_history files were empty, most of the logging was disabled, and the MD5s of most of the files matched with "known good" references. So far, so good.

While pondering what else to check, a light went off...

[ben@cloudaloudadingdong ~]$ ls /etc/ssh/ssh_host_* -las
4 -rw------- 1 root root  668 Apr 11  2009 /etc/ssh/ssh_host_dsa_key
4 -rw-r--r-- 1 root root  604 Apr 11  2009 /etc/ssh/ssh_host_dsa_key.pub
4 -rw------- 1 root root 1679 Apr 11  2009 /etc/ssh/ssh_host_rsa_key
4 -rw-r--r-- 1 root root  396 Apr 11  2009 /etc/ssh/ssh_host_rsa_key.pub
[ben@cloudaloudadingdong ~]$

Oh crap... That doesn't look good... That might mean...

[ben@cloudaloudadingdong ~]$ ssh $MY_IP
The authenticity of host '$MY_IP ($MY_IP)' can't be established.
RSA key fingerprint is d3:ad:b3:3f:ea:5b:52:f5:ab:ad:1d:3a:e7:f0:8d:99.
Are you sure you want to continue connecting (yes/no)? ^C
[ben@cloudaloudadingdong ~]$ ssh $ADJACENT_IP
The authenticity of host '$ADJACENT_IP ($ADJACENT_IP)' can't be established.
RSA key fingerprint is d3:ad:b3:3f:ea:5b:52:f5:ab:ad:1d:3a:e7:f0:8d:99.
Are you sure you want to continue connecting (yes/no)?

Facepalm

For those of you not familiar with the intricacies of SSH, this means that the public and private keys are identical across my and the server on the adjacent IP. This is a situation I've run into before. Back in 2002, I particpated in a capture the flag competition that was run by my unversity. The admins cloned all the contestant servers from a single Norton Ghost image. One of the teams (not mine) figured out that because they cloned the machines post-installation, all the SSH keys were identical across all the contestant servers and then proceeded to sniff all the traffic from the contestants, grabbing all their passwords, and then ripping off all their flags. This means that using "my" private key, I could intercept and monitor all the SSH traffic for the other server (and, presumably, every other Linux server at FlyByNightVPS) as if it were unencrypted.

Oops...

Well, I'm currently working with FlyByNightVPS in an attempt to fix this giant oozing and gaping security hole problem across their images and hopefully across all their existing servers. But, when doing some research for this writeup, searching for the hostname some posts came up in an Asian VPS forum discussing another (US based) VPS provider, meaning this key may be in other places as well. In the new age of servers in the cloud, one needs to worry about where that server image you're using has been and what "bonus features" the provider may have included either accidently or on purpose.