I have an application running on an EC2 instance, unfortunately I have lost private file to connect to this instance and now can not shut down this VM as the requirement is to have “ZERO DOWNTIME”

Step-by-Step Process:

Modify Security Group to Allow Session Manager Access:
- Go to the EC2 Dashboard → Instances.
- Select the locked instance.
- Navigate to Security Groups and ensure it allows outbound internet access or is connected to a VPC endpoint for SSM or add IP address of VPN network to connect directly.
Start a Session Using Session Manager:
- Go to AWS Systems Manager → Session Manager in the AWS Console.
- Click Start Session.
- Select the locked EC2 instance and click Start Session.
Create a New Key Pair:
- In a separate terminal or local environment, run the following command to create a new key pair:
```
  ssh-keygen -t rsa -b 4096 -f new_key
```
- This generates:
  - new_key (private key)
  - new_key.pub (public key)
    
    Command Breakdown →
    - ssh-keygen: Utility to create a new SSH key pair.
    - -t rsa: Specifies the type of key to generate (rsa in this case).
    - -b 4096: Sets the key length to 4096 bits for stronger encryption.
    - -f ~/.ssh/new_key: Specifies the file path and name for the new key pair (in this case, ~/.ssh/new_key).
Convert the Private Key to PEM Format:
- Use the following command to convert the private key to PEM format.
  
  Used to change the passphrase of an existing private key and optionally convert the key format to PEM format.
```
  ssh-keygen -p -m PEM -f new_key
```

Command Breakdown →

-p: This option tells ssh-keygen to change the passphrase of an existing private key.

-m PEM: This specifies the output format of the key. PEM (Privacy Enhanced Mail) format is commonly used for compatibility with older systems or third-party applications that require keys in a traditional PEM encoding.

-f new_key: This specifies the file containing the existing private key whose passphrase should be changed.

Set correct Permissions to .ssh directory
- ~/.ssh directory: The directory should have 700 permissions, which means only the owner can read, write, and execute (access) the directory.
```
  chmod 700 /.ssh
```
- Add the new public key to the authorized_keys file:
  
  Private key The private key should have 600 permissions, meaning only the owner can read and write the file.
```
  cat new_key.pub
  echo "<content-of-new_key.pub>" >> ~/.ssh/authorized_keys
  chmod 600 ~/.ssh/authorized_keys
```
Copy the Private Key to a File Locally and change it’s permission:
- On your local machine, copy the content of the new_key file into a .pem file:
  
  Private key (new_key.pem): owner can read the file
```
  cp new_key new_key.pem
  chmod 400 new_key.pem
```
Test the SSH Connection:
- Use the newly created key pair to SSH into the locked instance:
```
  ssh -i new_key.pem ec2-user@<instance-public-ip>
```

Summary:

This method ensures that you can recover access without restarting the instance.
You only need AWS Systems Manager Session Manager configured properly and a new key pair.
The key pair creation and injection process can be done in a few simple steps.

Here’s a script to automate the process of recovering access to an AWS EC2 instance using Session Manager:

Bash Script: Recover Access via Session Manager

#!/bin/bash

# Step 1: Generate a new key pair
echo "Generating a new RSA key pair..."
ssh-keygen -t rsa -b 4096 -f new_key -q -N ""
echo "Key pair generated: new_key (private), new_key.pub (public)"

# Step 2: Convert private key to PEM format
echo "Converting private key to PEM format..."
ssh-keygen -p -m PEM -f new_key -q -N ""
echo "Private key converted to PEM format."

# Step 3: Start Session Manager session and add public key to authorized_keys
echo "Starting AWS Session Manager session..."
INSTANCE_ID="<your-instance-id>"  # Replace with your EC2 instance ID

aws ssm start-session --target "$INSTANCE_ID" --document-name AWS-StartInteractiveCommand > session_script.sh <<EOF
#!/bin/bash

# Inside the session: Switch to ec2-user
sudo su - ec2-user

# Ensure .ssh directory exists
mkdir -p ~/.ssh
chmod 700 ~/.ssh

# Add new public key to authorized_keys
echo "$(cat new_key.pub)" >> ~/.ssh/authorized_keys
chmod 600 ~/.ssh/authorized_keys

echo "Public key added to authorized_keys successfully."
EOF

chmod +x session_script.sh
./session_script.sh

# Step 4: Test SSH access
echo "Testing SSH access with the new key..."
PUBLIC_IP="<your-instance-public-ip>"  # Replace with your EC2 public IP
ssh -i new_key.pem ec2-user@"$PUBLIC_IP"

echo "Access restored successfully using the new key!"

How to Use the Script:

Replace placeholders:
- Replace <your-instance-id> with your EC2 instance ID.
- Replace <your-instance-public-ip> with the EC2 public IP address.
Ensure AWS CLI is configured:
- Run aws configure and set up your credentials and region.
Run the script:
- Save the script as recover_ec2_key.sh.
- Make it executable:
```
  chmod +x recover_ec2_key.sh
```
- Run the script:
```
  ./recover_ec2_key.sh
```

Note:

This script assumes you have AWS CLI and Session Manager Plugin installed.
Ensure your IAM role and security group allow Session Manager access.

Thanks for spending your valuable time in learning to enhance your knowledge!😃🙏

𝗙𝗼𝗹𝗹𝗼𝘄 𝗺𝗲🔗╰┈➤ Hashnode LinkedIn

Easiest way to recover lost PEM key of AWS EC2 Instance | *WITHOUT RESTART*