I tried to set this up so I could use my yubikey as a portable authenticator following the Debian guide, and ran in to a bit of an issue.
Despite what I was doing, ssh-agent would auto start, set environment variables and give me lots of trouble.
Killing ssh-agent and manually setting the
$SSH_AUTH_SOCK to my gnupgp socket fixed the issue, however I couldn’t get ssh-agent to stop starting and setting that variable on login.
Having gone through some anguish I stumbled across xfce documentation detailing how to stop this behaviour.
Now it works.