When it comes to developing and managing workloads on AWS EC2 instances, the ability to seamlessly integrate your development environment is crucial. While Visual Studio Code (VS Code) offers excellent extensions for remote development via SSH, connecting Visual Studio Code to EC2 instances securely and efficiently via AWS SSM can be challenging. Ynpact, a leading technology company, has developed a game-changing solution to simplify this process, offering a powerful integration between VS Code Remote SSH Extension and AWS System Manager via its SSH-SSM-Proxy tool.

Many blog posts on the net already proposed solutions to connect to EC2 instances using SSH, via the VSCode remote SSH extension. And it works in a quite simple and straightforward way. But none of them are really convenient for an intensive daily use across multiple AWS accounts and regions. Here are some of the drawback of these solutions : 

• EC2 instanceId had to be hardcoded in the SSH config file. This is particularly annoying as this instance id may change over time, particularly in High Available solution that use a cluster that auto-scale
• Region and AWS credential profile had to be hardcoded into the SSH config file for each instance we want to connect to. If you have many instances spread into several regions and several AWS accounts, this makes a huge SSH config file that is not convenient and readable. In addition you must repeat the ProxyCommand in the config file and cannot refactor it. This can not scale.
• When getting AWS credential with AWS Identity Center (sso) it was complicated to use those credential in the remote SSH extension scope of execution ; or sso login had to be done into the SSH config ProxyCommand and could happen multiple time while the extension was loading a new shell or mounting a directory into VS Code (the SSO login browser could popup multiple time and it was annoying).
• SSH key pair to use and SSH user had to be configured per instanceId as well in the SSH config file. This is not a problem if you use the same user (or same type of host OS) in all EC2 instances and the same key pair for all your hosts, if not, you have to add news entries in the SSH config file.
• None of the existing solutions were capable of forwarding the AWS credential gained locally by the operator to be used into the EC2 host shell. Once into the shell, only the IAM role of the instance profile can be used. Forwarding the credentials gained locally into the EC2 instances allows us to continue to manage authentication & authorization centrally in the IAM Identity Center. This is a unique feature of our SSH-SSM-Proxy tool.

The Ynpact  SSH-SSM-Proxy tool for VS Code Remote SSH Extension is a comprehensive solution that addresses all those problems. It comes with several unique features that streamline the operation and development processes and enhance security. Let’s delve into its features, how it works and how you can set it up and use it for your own development needs.

Features

1. Enhanced Security with AWS SSM

Ynpact’s extension allows you to connect to your EC2 instances via AWS Systems Manager (SSM), significantly improving security. This means your EC2 instances can remain in a private network, eliminating the need for a bastion host. Neither you need to open the SSH port in the instance’s security group. With SSM, you can securely manage and access your instances while keeping them isolated from the public internet.

2. Direct File System Integration

With this SSH-SSM-Proxy tool, you can directly mount the EC2 instance’s file system into VS Code via SSH and SSM. This seamless integration ensures that you can work with your EC2 instance’s files as if they were local, streamlining your development workflow. With the Ynpact solution, you can use SSO credential in a manner that the remote SSH extension does not keep asking you for SSO credential by popping-up a browser window.

3. Friendly Naming Convention

The SSH-SSM-Proxy tool simplifies the connection process by allowing you to use friendly names based on EC2 instance name tags. You no longer need to remember complex instance IDs that can change over time. Instead, you can connect using the easily recognizable names you’ve assigned to your EC2 instances.

4. Local credential forwarding into target EC2 instance

Ynpact’s solution accommodates various authentication methods. You can use AWS Single Sign-On (SSO) or traditional static access keys, ensuring flexibility and compatibility with your preferred authentication approach.

The SSH-SSM-Proxy tool enables you to forward your local AWS credentials (either static or obtained through SSO) into the EC2 instance SSH session. This keeps user rights management centralized and consistent, ensuring that access and permissions are maintained per user instead of per EC2 instance. If you choose not to forward credentials, the sessions use the IAM role defined for the EC2 instance.

5. Efficient EC2 Instance Configuration Handling

The SSH-SSM-Proxy tool allows you to store and manage EC2 instance configuration details, including the AWS credentials to use, the target AWS region, the SSH user, and the SSH key to use. This centralized configuration management simplifies your setup and ensures consistency across your development environment. You don’t have to rewrite multiple times the same SSH ProxyCommand and this allows you to scale operations and development onto EC2 instances across OS, region and accounts seamlessly.

In addition, if the host is not an EC2 instance, it transparently uses a regular SSH client to connect to the host, so that it remains compatible with non AWS hosted hosts.

How it works

Here is a high level overview of connection flow on a windows host using WSL (on Linux and MacOS, the flow would be similar) :

1. Using VS Code remote SSH extension (in this diagram running into a windows host), connect to the host using it’s alias as defined while adding the new host (see configuring a new host in our github repository)
2. The “path” parameter of the Remote SSH extension is configured to call sshProxy.bat instead of an SSH client. This script, located in the windows host, will invoke the sshProxy.sh script that resides in your WSL distribution. This is handy when workspaces and tools are all running into WSL and that AWS credentials configuration also resides there.
3. From the alias name of the host, the script retrieves the configuration of the host from its own config file : AWS region, AWS credentials to use, SSH user & key, target EC2 instance name-tag value.
4. If SSO credentials are used, the script login into AWS using “aws sso login” with the defined profile. A web browser windows popup so that you can login as usual into AWS using IAM Identity Center or your corporate IdP. If static credentials are used, this step is skipped.
5. The script retrieves the EC2 instanceId from the tag name via an AWS CLI call toward the EC2 service.
6. As the Remote SSH extension needs to interact only with a regular SSH client, the script initiates an SSH connection using the local SSH client with the specified SSH key and user.
7. But the local ssh config file specify a ProxyCommand for host starting with “i-*” (aws hosts). This proxy command is the sshProxy.sh script called with different arguments from the ones specified by the remote SSH extension. This indicates the sshProxy to directly connect to the EC2 instances.
8. The sshProxy.sh uses the AWS cli to connect to the EC2 instance using AWS SSM, with the SSM document ensuring regular SSH client compatibility. It uses the specified AWS region and AWS credentials and connects using the instance ID retrieved in step 5.
9. The connection is established with the EC2 instances. If when configuring the host, the credential forwarding has been enabled, the operator gets a remote shell session that uses the locally gained credentials (by SSO or static key) instead of the EC2 instance profile role (forwarded opening the ssh session with a initial command that export AWS credential by setting environment variables).

This whole flow can repeat multiple times as the VS Code Remote SSH extension can initiate multiple SSH connections to mount the local folder and invoke multiple shells. However, this is transparent to the user as a mechanism in the script allows to cache the SSO gained credentials and to not popup a new browser window each time for SSO login. Every hour, the script might ask again for SSO login, as the SSO gained credential will expire.

Setting Up

See instruction in the Github repository : 

https://github.com/Ynpact/ynpact-ssh-ssm-proxy?tab=readme-ov-file#set-up

Configuration and Connection

See instruction in the Github repository : 

https://github.com/Ynpact/ynpact-ssh-ssm-proxy?tab=readme-ov-file#managing-target-hosts

Security concerns

Used SSH Key

The SSH key used to connect to the EC2 instances is not the main element that ensures the authentication at the remote host. While this offers the same level of security as does the key in traditional SSH connection, in this flow, the key is mainly used to ensure SSH client compatibility. Indeed, as long as the SSH port is closed and/or the instance in a private network, the main element that ensures the authentication is the AWS IAM credentials used to open the session via AWS SSM.

Considering that, it makes simpler to handle SSH private keys in your company : you could create a few private keys only (like one for prod, one for other environment and one for shared/internal services), install it on all EC2 instances of a group/env and distribute it to all the employees of a kind, as long as you connect to all your instances via SSM. Even once distributed connection can be locked by modifying the AWS role of each user or user group.

A leak of an SSH private key is thus not a critical security incident, as long as you keep the SSH port of the EC2 instance closed. In case of exposure of the key, I would still recommend to rotate the SSH key, if for some reason or incident, you need to re-open port 22 on any host (for example, an issue with the SSM agent that runs in the instance).

Forwarding credentials

This feature allows users to continue to centrally manage authorization through AWS Identity Center or IAM users, even when an operator is logged into the EC2 instances. Indeed, by default, once logged in an EC2 instance, the credentials used are the one of the EC2 instance profile roles. This might be annoying if, for example, the user must upload/download a file only from a particular directory it owns in S3 (or if the use of any other AWS service is restricted per user).

If you define minimal permission to the EC2 instance profile role (basically only the rights that let the SSM agent to interact with the SSM service, allowing to start SSH sessions), then the logged in operator will have to use it’s own credential (forwarded by the tool) to perform additional operation like accessing S3 objects, invoking lambda functions and so on.

Nothing will prevent the operator to use the EC2 instance profile role to perform the AWS operation (using curl and the EC2 instance metadata endpoint) instead of its own credential, but that EC2 instance role will not allow him to perform much.

The drawback of this approach is that, if your AWS SSM session has log enabled, the credentials used might appear in the session logs in CloudWatch, as the command used to initiate the remote bash session exports locally gained credentials. This should not be a problem if you close down access to those logs to only admins, but those last could perform action as their colleague if intercepting the key within the hour.

I would not use this feature while connecting to sensitive or prod instances. Indeed, this feature has been designated for testing and development workflows where team members need to connect to the instance and interact from it with AWS services using the AWS CLI/SDK to run investigations, experimentation, tests or to process / transform data.

Credential caching

The SSH-SSM proxy tool cache the credential gained locally via SSO in order to avoid the browser constantly popping out when mounting a directory. Those temporary credentials are valid for 1h and stored in an unencrypted file with permissions 400 in the user directory.

This should not be a concern if the work stations are properly secured (hard drive encrypted, screen locked, firewall…). This is anyway still much more secure than having long term access keys in the standard AWS credential file ~/.aws/credentials.

Wrap up

Ynpact’s SSH-SSM proxy tool for VS Code Remote SSH Extension empowers your development team to log in, execute commands, and browse files on EC2 instances effortlessly. This solution combines the convenience of VS Code, the security of AWS SSM and the power of centralized user rights management, making it an indispensable tool for AWS EC2 operation and development.

Embrace this game-changing tool from Ynpact to supercharge your development workflow and streamline your AWS EC2 instance management. Elevate your AWS development experience with enhanced security and unparalleled convenience in Visual Studio Code.

Please contact us at contact@ypnact.com for more information.