Security risks of ssh in legacy applications: Suggested mitigations (2024)

Businesses are moving to cloud and adopting REST API based architectures, but lot of legacy applications which are fork lifted on to the cloud stack, still have to be in production. Often these legacy applications use ssh to secure their communication links. This use of ssh pose lots of security challenges in the current context. In this article I will share my thoughts on this problem and discuss possible solutions.

Why traditional ssh implementation is a security risk?

Secure ssh implementation use ssh keys (public-private key pairs) for authentication instead of passwords/passphrases. This use of ssh keys makes it resistant to brute force password cracking attack. However in most legacy implementation, private keys are kept in the ssh client's local storage. This becomes a bigger problem when private keys are kept unencrypted to allow client applications function without a human user in the loop (to enter passphrase to decrypt the private key). This pattern of ssh implementations has following kinds of risks:

• Theft of private key is possible due to presence of other vulnerabilities: In past, local storage locations were considered reasonably secure. But with time, software stacks and applications have become more complex and have increased the attack surface. The number of known vulnerabilities have also increased in each component. As a result, now hackers have more ways to break into the ssh clients to steal the private keys. Thus today this legacy pattern of ssh implementation have become vulnerable.
• Key theft by insiders, or ex-employees: People who have access to client systems, have access to keys.
• Keys can be exposed through the memory dump, or can be read out from memory by different attack techniques.
• Keys can be exposed during key rotation: This can happen because many businesses do not have a secure way of rotating keys.
• Public key may be tampered during exchange between client and server side maintenance personnel, or when the public key is at rest in the server. This is especially a problem when this exchange takes place with human interventions and without use of a secure system or procedure in place, or when server side compensatory security controls are missing on the server.

How can we mitigate these risks?

There are several methods to mitigate these risks, some of these are discussed below:

Method 1: Use HTTPS (TLS) based data transfer instead of ssh: This is by far the best option, however the pros and cons are presented below.

Advantages: TLS use certificates to implement a trust mechanism based authentication. This has advantages over non-PKI public-private key pair based authentication used in ssh. TLS based technology stack comes prepackaged with periodic certificate expiration, revocation and public key authentication capabilities. This improves security. Whereas ssh-key based technology stack generally do not come with these features (however take a look at this article that talks about ssh key expiry). Expiring and revoking ssh keys will actually need additional software and mechanisms.

Disadvantages: Swapping TLS with ssh needs lot of work. As of today, use of TLS means using HTTP REST APIs on top of TLS. This means that the application has to be modified to use REST API based data transfer. Therefore, substantial changes need to be made in the application, at both server and client side, in application code as well as, in the platform infrastructure. When the communications link crosses business boundaries, and especially when multiple clients are involved, coordinating such change becomes too unwieldy. It is better to schedule this change with other changes or complete overhaul/replacement of the application.

Method 2: Use certificate based ssh: Now a days there are ssh implementations that use either ssh-certificates (non x.509) or x.509 certificates for authentication. See the links here (link1, link2) for x.509 certificates based ssh, and ssh-certificate based ssh. This will also need to changes in the server and client side platform infrastructure but not in the application code. This is comparatively less work than method #1, as there is no need to implement HTTP REST API based data transfer which needed application code change in method 1.

Advantages: Less work than method # 1. But still has the advantages of PKI (which are: certificate expiration, revocation and public key authentication capabilities), just like TLS.

Disadvantages: Not having other security benefits of HTTP REST API based technology stack. This still requires lot of co-ordination to implement when client and servers belong to multiple parties.

Method 3: Store ssh private key in key vault and let the vault manage the key life cycle: The weakness of using keys for ssh authentication primarily arise because private key is stored in the local storage and handled in the unsecured memory in the client application. This weakness can be removed, if the client application can be made to work with keys stored in a vault.

This is how it will work - In ssh authentication process (refer for details here), the server sends a message signed by client's public key to the client, then the client decrypts that message with its private key and provide proof to the server that it was able to successfully decrypt the cipher text. Once the server get assurance that the client has successfully decrypted the message, server knows that the client is indeed in possession of the right private key, therefore client's identity is now established.

In regular ssh key based authentication implementation, the client does the decryption using the locally stored private key. But in this vault based private key storage method, the ssh client will forwards the cipher text to the vault and request it to be decrypted and returned to the client. The ssh client will use the decrypted message to complete the ssh authentication handshake with the ssh server. I will later share some links on details / sample code/implementation on ssh handshake.

Therefore this vault need these following features:

• APIs to encrypt/decrypt messages.
• Features to manage the entire life cycle of ssh key in a secure manner- creation, storage, rotation, and discarding.

Advantages: This involves only changes in the ssh client side infrastructure (at the side that needs protection) and may not need code changes in the client application layer. The server and other clients that need not be protected does not need to change. This is a great advantage over method 1 and 2.

Disadvantages: Ssh client code that will implement ssh handshake might have vulnerabilities, and as this code may not be standard implementation. So this code needs to be thoroughly reviewed to eliminate implementation vulnerabilities. This is because many vulnerabilities creep in the implementation of strong crypto algorithms to make them effectively useless in the practical world. In addition this has the same weaknesses as ssh keys, which PKI based approach mitigates by offering: certificate expiration, revocation and public key authentication capabilities.

Method 4: Manage and rotate ssh keys in the local storage, manually or using orchestration/automation tools: Here are some orchestration tools that others have used. Rotate keys periodically, and rotate when key personnel who had access to the keys, leave the organization.

Advantages: No change in client-server components needed at all, all changes are out of band for ssh components or application layer.

Disadvantages: Lot of expensive features are needed to manage the ssh keys, that PKI based methods avoided.

Method 5: Adopt an end-to-end secure system to deliver public key to the ssh server. This method is needed in conjunction with one of the other method # 3, or 4. This method is not needed with method # 1 or 2, because those two certificate based methods have their own mechanism to establish authenticity of public keys. Signed/certified ssh public key, which can be authenticated against a root of trust can be one alternative to implement this secure system. In this alternative, the authentication is not based on certificates, only the public key is authenticated using certificates. So this is a half way towards full use of certificates like method #2. There can be other ways to get assurance about public keys.

Method 6: Implement compensatory controls in the client and server sides that needs protection, to reduce private key theft and public key tampering.

Which option to choose and why ?

Choice of method depends on situation, and cost- benefit analysis. Here are some case studies to illustrate this:

Case 1: Multi-party legacy ssh situation: Here the the legacy application used ssh client, the ssh server belonged to a different business. The ssh server had many clients, all of them belong to different businesses. Therefore it was a real messy situation ! The business that needed to protect the ssh key on the client, chose method # 3, because it had an existing vault infrastructure with needed features. It did not choose method # 4, even though had been using orchestration tools for other work.

Case 2: Legacy eco-system of multiple applications using ssh keys: A large number of legacy systems were using ssh keys. All server and clients belonged to the same business, which used orchestration tool extensively. It is obvious that method # 4 is a better choice here for short and medium term. Other alternatives could be adopt more formal ssh key management systems of zero trust products.

Case 3: New application that need to transfer files/data: The right choice for a new applications would be method # 1, as this is a green field situation where use of HTTPS can be afforded to avoid use of ssh to create a technical debt.

Conclusion

Use of ssh pose several security risks and manageability problem. Fortunately we have multiple options to mitigate the risks. The choice of mitigation method will need a cost-benefit analysis. Legacy applications may be allowed to continue using ssh (but with compensatory controls). But new applications should avoid use of ssh and all he associated technical debt.

#ssh

#security

#keymanagement