I found that after opening many SSH sessions using a Yubikey for authentication that gpg-agent would bork out and take increasingly long to authenticate. Eventually the time taken to provide the authentication key would take too long and opening new SSH sessions would just time out. The temporary fix just involved killing and restarting the gpg-agent, but luckily the permanent fix for the issue is very simple. All you need to do is create/edit the file ~/.gnupg/scdaemon.conf and add the following line:

disable-ccid

That’s it! You can now authenticate using your Yubikey as many times as you like without gpg-agent grinding to a halt.