排查 Jenkins 使用 SSH 连不上 Slave 节点问题

背景

同事突然找我说 Jenkins 连不上 slave 节点了，昨天打包还好好的。经过排查时 slave 机器上配置的公钥丢失导致的，几天前另外一个同事清理的，我猜测可能跟配置节点时可用性选择的 “尽量保持代理在线” 有关系吧，直到今天断了 jenkins 重连才发现。

Jenkins 这边使用的是提前配置好的凭据，实际上是一个 Ed25519 算法生成的私钥，对应的公钥则配置到要远程登录的 slave 机器的 ~/.ssh/authorized_keys 里。配置时可以通过 vnc 登录，然后通过图形化界面配置。或者可以使用如下命令：

ssh-copy-id -f -i /Users/matt/Desktop/id_ed25519.pub root@10.11.12.13

排查过程

当遇到这个问题时，Jenkins 这边没有显示具体的原因，已知的 Jenkins 通过 ssh 方式的连接 slave ，这一过程中 Jenkins 充当的是客户端，slave 充当的是服务端，所以如果能够在服务端这边开启调试日志，这样就能够帮助分析问题了。

如果是 slave 是 macOS 可以通过下面的命令实现：

sudo launchctl unload -w /System/Library/LaunchDaemons/ssh.plist

你也可以通过以下命令来验证 sshd 服务是否已经停止： sudo launchctl list | grep ssh 。如果该命令没有输出任何内容，或者只显示相关的pid，那么sshd服务就已经停止了。

通过参数让 sshd 启动的时候打印调试日志：

sudo /usr/sbin/sshd -D -dd

然后让 Jenkins 连接 salve 日志如下：

debug1: fd 6 clearing O_NONBLOCK
debug1: Server will not fork when running in debugging mode.
debug1: rexec start in 6 out 6 newsock 6 pipe -1 sock 9
debug2: parse_server_config_depth: config rexec len 3518
debug2: parse_server_config_depth: config /etc/ssh/sshd_config.d/100-macos.conf len 134
debug1: sshd version OpenSSH_9.7, LibreSSL 3.3.6
debug1: private host key #0: ssh-rsa SHA256:rJGPVwWUoS0S0OLmr22Rza4bRnSYvOuxaZ2YNh2Oq0Q
debug1: private host key #1: ecdsa-sha2-nistp256 SHA256:dIVw5c0KEG7adxREB2MZc/n81lBd9vsXSHJ3DSgwoLk
debug1: private host key #2: ssh-ed25519 SHA256:5BzgiCe5BV8QA5avabsik9nxfkC4wKCfz+ExPMZW9jQ
debug1: inetd sockets after dupping: 5, 5
Connection from 10.19.123.101 port 35774 on 10.29.76.200 port 22
debug1: Local version string SSH-2.0-OpenSSH_9.7
debug1: Remote protocol version 2.0, remote software version TrileadSSH2Java_213
debug1: compat_banner: no match: TrileadSSH2Java_213
debug2: fd 5 setting O_NONBLOCK
debug2: Network child is on pid 8025
debug1: permanently_set_uid: 75/75 [preauth]
debug1: list_hostkey_types: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug1: SSH2_MSG_KEXINIT sent [preauth]
debug1: SSH2_MSG_KEXINIT received [preauth]
debug2: local server KEXINIT proposal [preauth]
debug2: KEX algorithms: sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-s,kex-strict-s-v00@openssh.com [preauth]
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com [preauth]
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com [preauth]
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth]
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth]
debug2: compression ctos: none,zlib@openssh.com [preauth]
debug2: compression stoc: none,zlib@openssh.com [preauth]
debug2: languages ctos:  [preauth]
debug2: languages stoc:  [preauth]
debug2: first_kex_follows 0  [preauth]
debug2: reserved 0  [preauth]
debug2: peer client KEXINIT proposal [preauth]
debug2: KEX algorithms: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,curve25519-sha256,curve25519-sha256@libssh.org [preauth]
debug2: host key algorithms: ssh-ed25519,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256,rsa-sha2-256,rsa-sha2-512,ssh-rsa,ssh-dss [preauth]
debug2: ciphers ctos: aes256-ctr,aes192-ctr,aes128-ctr,blowfish-ctr,aes256-cbc,aes192-cbc,aes128-cbc,blowfish-cbc,3des-ctr,3des-cbc [preauth]
debug2: ciphers stoc: aes256-ctr,aes192-ctr,aes128-ctr,blowfish-ctr,aes256-cbc,aes192-cbc,aes128-cbc,blowfish-cbc,3des-ctr,3des-cbc [preauth]
debug2: MACs ctos: hmac-sha2-512,hmac-sha2-256,hmac-sha1,hmac-sha1-96,hmac-md5,hmac-md5-96 [preauth]
debug2: MACs stoc: hmac-sha2-512,hmac-sha2-256,hmac-sha1,hmac-sha1-96,hmac-md5,hmac-md5-96 [preauth]
debug2: compression ctos: none [preauth]
debug2: compression stoc: none [preauth]
debug2: languages ctos:  [preauth]
debug2: languages stoc:  [preauth]
debug2: first_kex_follows 0  [preauth]
debug2: reserved 0  [preauth]
debug1: kex: algorithm: diffie-hellman-group-exchange-sha256 [preauth]
debug1: kex: host key algorithm: ssh-ed25519 [preauth]
debug1: kex: client->server cipher: aes256-ctr MAC: hmac-sha2-512 compression: none [preauth]
debug1: kex: server->client cipher: aes256-ctr MAC: hmac-sha2-512 compression: none [preauth]
debug1: expecting SSH2_MSG_KEX_DH_GEX_REQUEST [preauth]
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST received [preauth]
debug2: monitor_read: 0 used once, disabling now
debug1: SSH2_MSG_KEX_DH_GEX_GROUP sent [preauth]
debug2: bits set: 1009/2048 [preauth]
debug1: expecting SSH2_MSG_KEX_DH_GEX_INIT [preauth]
debug1: SSH2_MSG_KEX_DH_GEX_INIT received [preauth]
debug2: bits set: 1017/2048 [preauth]
debug2: monitor_read: 6 used once, disabling now
debug2: ssh_set_newkeys: mode 1 [preauth]
debug1: rekey out after 4294967296 blocks [preauth]
debug1: SSH2_MSG_NEWKEYS sent [preauth]
debug1: expecting SSH2_MSG_NEWKEYS [preauth]
debug1: SSH2_MSG_NEWKEYS received [preauth]
debug2: ssh_set_newkeys: mode 0 [preauth]
debug1: rekey in after 4294967296 blocks [preauth]
debug2: KEX algorithms: sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-s,kex-strict-s-v00@openssh.com [preauth]
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519 [preauth]
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com [preauth]
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com [preauth]
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth]
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1 [preauth]
debug2: compression ctos: none,zlib@openssh.com [preauth]
debug2: compression stoc: none,zlib@openssh.com [preauth]
debug2: languages ctos:  [preauth]
debug2: languages stoc:  [preauth]
debug2: first_kex_follows 0  [preauth]
debug2: reserved 0  [preauth]
debug1: KEX done [preauth]
debug1: userauth-request for user root service ssh-connection method none [preauth]
debug1: attempt 0 failures 0 [preauth]
debug2: parse_server_config_depth: config reprocess config len 3518
debug2: parse_server_config_depth: config /etc/ssh/sshd_config.d/100-macos.conf len 134
debug2: monitor_read: 8 used once, disabling now
debug2: input_userauth_request: setting up authctxt for root [preauth]
debug2: input_userauth_request: try method none [preauth]
debug1: PAM: initializing for "root"
debug1: PAM: setting PAM_RHOST to "10.19.123.101"
debug2: monitor_read: 100 used once, disabling now
debug1: userauth-request for user root service ssh-connection method publickey [preauth]
debug1: attempt 1 failures 0 [preauth]
debug2: input_userauth_request: try method publickey [preauth]
debug2: userauth_pubkey: valid user root attempting public key ssh-ed25519 AAAAC3QWZBd7XplT9vYk3M2JHfR0sVbLgOaEnuC+eI1zqNwUoKxSj4rFDhPmBtyc5ZAl [preauth]
debug2: monitor_read: 4 used once, disabling now
debug1: temporarily_use_uid: 501/20 (e=0/0)
debug1: trying public key file /Users/root/.ssh/authorized_keys
debug1: fd 6 clearing O_NONBLOCK
debug2: auth_check_authkeys_file: /Users/root/.ssh/authorized_keys: processed 2/2 lines
debug1: restore_uid: 0/0
Failed publickey for root from 10.19.123.101 port 35774 ssh2: ED25519 SHA256:M2JHfR0sVbLgOaE3QWZBd7XplT4M2JHfR0sVbLQWZBd
debug2: userauth_pubkey: authenticated 0 pkalg ssh-ed25519 [preauth]
Received disconnect from 10.19.123.101 port 35774:11: Closed due to user request. [preauth]
Disconnected from authenticating user root 10.19.123.101 port 35774 [preauth]
debug1: do_cleanup [preauth]
debug1: monitor_read_log: child log fd closed
debug1: do_cleanup
debug1: PAM: cleanup
debug1: Killing privsep child 8025
➜  ~

主要问题是公钥验证不通过，体现在 “Failed publickey for root” 那一行，后面输出的是指纹。所以可以查看下私钥的指纹，经过验证的确不一样，是运维同事给我的公钥不对导致的。

查看指纹使用如下命令：

➜  ~ ssh-keygen -l -f ~/.ssh/authorized_keys                       
256 SHA256: M2JHfR0sVbLgOaE3QWZBd7XplT4M2JHfR0sVbLQWZBd xxx@yyy.com (ED25519)

➜  ~ ssh-keygen -l -f ~/.ssh/id_ed25519
256 SHA256:4sW7Qi3mdPuinRhMYZam2LGo00=U/OeA3EmsW93PDSEW xx@cc.com (ED25519)

一对公私钥的指纹肯定是一样的，所以利用指纹可以判定公私钥是否配对。

排查完问题，可以使用以下命令重新启动 sshd 服务:

sudo launchctl load -w /System/Library/LaunchDaemons/ssh.plist

Tips

• 升级后全部 slave 节点都没了，所以要备份 slave 节点信息
• 通过 ssh 方式连接 slave 时，每个登录用户名都需要创建一个凭据，及时使用相同的私钥
• Jenkins 最多存储 10 个凭据
• 对于 windows 系统作为 salve 节点时需要将公钥追加到 C:\Users\{your_user}\.ssh\authorized_keys 里。