My experience with CERN's SSH 2FA

[!NOTE] Update: (2025-09-24)

CERN now enforces 2FA for SSH on the lxplus perimeter. You cannot replace the first-hop password with key-only from off-site. The practical goal is to enter password + 2FA once per work window, and reuse that authentication for everything else (SSH, scp/rsync, Git, editor SFTP). The method below works with the built-in SSH agent on Linux and macOS, without requiring 3rd-party tools. A 1Password agent setup is included later as an optional edge case. Thanks for Patrik Adlarson for pointing out that the original post was outdated and needed an update.

Key points to remember:

First hop to lxplus requires password + 2FA.
Use SSH multiplexing (ControlMaster/ControlPersist) to pay the interactive cost once.
Use any agent you like (system ssh-agent, macOS Keychain, GNOME Keyring). 1Password/Bitwarden does support them.
Never put your CERN password in editor configs.

Universal setup (works for everyone)

Generate an SSH key (skip if you already have one)

ssh-keygen -t ed25519 -C "<cern-username>@cern.ch"

Start or use your platform agent

Linux (OpenSSH agent):

eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519

GNOME (keyring acts as agent): usually automatic if SSH_AUTH_SOCK is set by the session.

macOS (Keychain-backed OpenSSH): Add this to ~/.ssh/config:

Host *
  AddKeysToAgent yes
  UseKeychain yes

Then:

ssh-add -K ~/.ssh/id_ed25519

Install your public key on lxplus (this will prompt for password + 2FA once)

ssh-copy-id -i ~/.ssh/id_ed25519.pub <cern-username>@lxplus.cern.ch

Enable multiplexing to reuse one authenticated channel ~/.ssh/config

Host lxplus
  HostName lxplus.cern.ch
  User <cern-username>
  IdentitiesOnly yes
  IdentityFile ~/.ssh/id_ed25519

  ControlMaster auto
  ControlPersist 8h
  ControlPath ~/.ssh/ctl-%r@%h:%p

  ServerAliveInterval 30
  ServerAliveCountMax 6
  TCPKeepAlive yes
  GSSAPIAuthentication no

Optional: multi-hop examples that reuse the same master channel

Host lbgw
  HostName lbgw
  User <cern-username>
  ProxyJump lxplus
  IdentitiesOnly yes

Host gpu-farm
  HostName <gpu-farm-host>
  User <cern-username>
  ProxyJump lbgw
  IdentitiesOnly yes

Start one master connection per work block

ssh lxplus
# enter password + 2FA once

Or background the master:

ssh -fN lxplus

Verify reuse in a second terminal

ssh -v lxplus
# Expect: "Connection multiplexed."

Editors and tools

VS Code Remote SSH, Sublime SFTP, scp/rsync, and Git will reuse the multiplexed channel automatically when they honor your ~/.ssh/config. Do not store passwords in app configs.

Example Sublime SFTP (sftp-config.json, keep it .gitignored):

{
  "type": "sftp",
  "host": "lxplus",
  "user": "<cern-username>",
  "ssh_key_file": "/home/<local-user>/.ssh/id_ed25519",
  "remote_path": "/afs/cern.ch/user/<first-letter>/<cern-username>/project",
  "upload_on_save": true,
  "save_before_upload": true,
  "sync_down_on_open": true,
  "sync_same_age": true,
  "sftp_flags": ["-F", "/home/<local-user>/.ssh/config"]
}

Security hygiene

chmod 700 ~/.ssh
chmod 600 ~/.ssh/config

Remove a key by editing ~/.ssh/authorized_keys on lxplus if needed. Avoid plaintext passwords anywhere.

Edge case: 1Password SSH agent (my personal setup)

If you do use 1Password’s SSH agent locally, add:

Host *
  IdentityAgent ~/.1password/agent.sock

All other steps stay the same. Your local agent signs SSH challenges with your local auth. You still must do password + 2FA exactly once to enter lxplus from off-site.

FAQ

Q: Can I do key + 2FA without the account password on the first hop?

A: No. The perimeter requires keyboard-interactive with password + second factor.

Q: Why use a key if I must enter password + 2FA once?

A: The key plus multiplexing eliminates repeated prompts for the rest of your work window, across SSH/scp/rsync/Git/editors.

Q: Does ProxyJump bypass 2FA?

A: No. You authenticate to lxplus once. Subsequent jumps reuse the authenticated master connection.

Q: Can I auto-start the master if missing?

A: Yes:

ssh -O check lxplus || ssh -fN lxplus

Troubleshooting

Stale control socket: remove it if the master died uncleanly:
```
rm -f ~/.ssh/ctl-<cern-username>@lxplus.cern.ch:22
```
Multiple aliases for the same host: keep a single Host lxplus stanza and consistently use ssh lxplus.
Scripts that must fail fast on auth: add BatchMode yes inside a specific Host block to avoid interactive prompts.

[!WARNING] Original blog post: (2024-11-01) : Please ignore the rest of the post below, it is outdated and no longer relevant. I have left it here for historical purposes.

So, as if juggling research and coffee consumption wasn't enough, CERN has decided to spice things up. They're rolling out 2FA for SSO for all users which is great for security, of course! But here's where it gets interesting: I volunteered to be part of a test group that's trying out 2FA for SSH connections as well. Yep, I signed up for extra layers of login fun!

Now, don't get me wrong, I’m all for keeping our data safe from the clutches of cyber gremlins. But entering a 2FA code every time I SSH into a server? That gets old faster than a quark decays. Especially when you're bouncing between machines like it's a game of computational hopscotch.

So, in true nerdy fashion, I looked for a workaround to make my life a tad easier. The solution? Of course, it is obvious, SSH keys! Even with the new 2FA requirements and using lxplus.cern.ch as a proxy jump, SSH keys have been my saving grace.

Here's how I made it happen:

Generated an SSH key pair on my local machine. Good old ssh-keygen never lets me down. This can be done with the following command:
```
ssh-keygen -t rsa -b 4096 -C "username@cern.ch"
```
Added my public key to my CERN account. Now we visit our old friend ssh-copy-id. I ran the following command to copy my public key to lxplus:
```
ssh-copy-id -i ~/.ssh/id_rsa.pub lxplus
```
This command copies the public key to the ~/.ssh/authorized_keys file on lxplus, allowing me to SSH in without a password.

[!NOTE] I am using lxplus instead of lxplus.cern.ch because I have already updated my ~/.ssh/config file to include lxplus as a hostname and my username on local machine is the same as my username on lxplus.

Updated my SSH config file (~/.ssh/config) to include lxplus as a proxy jump. It looks something like this:
```
Host lxplus
    HostName lxplus.cern.ch
    User melashri
    IdentityFile ~/.ssh/id_rsa

Host lbgw
    HostName lbgw
    User melashri
    ProxyJump lxplus
    IdentityFile ~/.ssh/id_rsa
```
Just swap out lbgw with whatever server you're trying to reach. lbgw is the CERN gateway server. For me, I need that to access the restricted GPU machine that Allen Developers (LHCb HLT1 Trigger framework) have access to.
Gave it a whirl to make sure everything worked smoothly. And guess what? It did! No more constant 2FA prompts. Now you can log it using ssh lxplus or ssh lbgw and you are in.

Now, whenever I SSH into a server, it gracefully hops through lxplus using my SSH keys, sidestepping the extra 2FA step. It's like having a VIP pass to the server club. If you're also part of the 2FA SSH test group or just want to get ahead of the curve I highly recommend setting up SSH keys. Or maybe you will read that when we are already all have 2FA on SSH connections.