Permission denied (publickey,gssapi-keyex,gssapi-with-mic) while configuring passwordless SSH

I have wasted hour to figure out why i am getting issue while configuring the passwordless ssh connectivity between Linux7.2 nodes(pg01 and pg02) for PostgreSQL backup configuration with BARMAN.

Even google didn't helped me much to figure out what went wrong in configuration.

I am getting permission denied while transferring the authorized_keys file from pg01 server to pg03. I have gone through multiple blogs and tried almost all of the solutions but nothing worked for me. There will be multiple reason for the same issue. But in my case, this is something not working for me even all the permissions are correct for files and directories.

[postgres@pg01 .ssh]$ scp authorized_keys pg03:/home/postgres/.ssh
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).
lost connection

Finally, I have enabled the SCP in debug mode to see more details as below.

[postgres@pg01 .ssh]$ scp -v authorized_keys pg03:/home/postgres/.ssh
Executing: program /usr/bin/ssh host pg03, user (unspecified), command scp -v -t /home/postgres/.ssh
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 56: Applying options for *
debug1: Connecting to pg03 [192.168.0.3] port 22.
debug1: Connection established.
debug1: identity file /home/postgres/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/postgres/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/postgres/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/postgres/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/postgres/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/postgres/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/postgres/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/postgres/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: match: OpenSSH_7.4 pat OpenSSH* compat 0x04000000
debug1: Authenticating to pg03:22 as 'postgres'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:pXs/ZCX85K7tKmGk7cNfI3RQHf+3gvDcvdb207jExCY
debug1: Host 'pg03' is known and matches the ECDSA host key.
debug1: Found key in /home/postgres/.ssh/known_hosts:1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/postgres/.ssh/id_rsa
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug1: Trying private key: /home/postgres/.ssh/id_dsa
debug1: Trying private key: /home/postgres/.ssh/id_ecdsa
debug1: Trying private key: /home/postgres/.ssh/id_ed25519
debug1: No more authentication methods to try.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

Now, i can see the issue is with accepting the public key instead its trying to authenticate with other DSA keys and getting out after few more attempts even i have not generated any DSA keys and its trying with default keys available during software installation.

So what i did, i have commented out all the entries in sshd_config file and kept the minimum parameters in sshd_config file as.

SyslogFacility AUTHPRIV
StrictModes no
PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM yes
X11Forwarding yes
Subsystem sftp /usr/libexec/openssh/sftp-server

Restart the sshd service.

[root@pg01 ssh]# systemctl restart sshd_config
[root@pg01 ssh]# systemctl status sshd_config

Now, run the SCP again.. Bingo this time its worked.

[postgres@pg01 ~]$ scp -v authorized_keys pg03:/home/postgres/.ssh
Executing: program /usr/bin/ssh host pg03, user (unspecified), command scp -v -t /home/postgres/.ssh
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017
debug1: Reading configuration data /etc/ssh/ssh_config

<< Truncated msg for clarity purpose >>

debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/postgres/.ssh/id_rsa
debug1: Server accepts key: pkalg rsa-sha2-512 blen 279
debug1: Authentication succeeded (publickey).
Authenticated to pg03 ([192.168.0.3]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
debug1: Sending command: scp -v -t /home/postgres/.ssh
Sending file modes: C0664 0 test
Sink: C0664 0 test
authorized_keys                                                                                                                100%    0     0.0KB/s   00:00    
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 0 clearing O_NONBLOCK
debug1: fd 1 clearing O_NONBLOCK
Transferred: sent 2956, received 2776 bytes, in 0.5 seconds
Bytes per second: sent 5970.6, received 5607.1
debug1: Exit status 0
[postgres@pg01 ~]$ 

Now, you can see the public keys are finally accepted and its able to connect to pg03 server. So i can see these entries is causing the issue.

HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key