No, nothing to install on your server. Everything is done via SSH from your phone and the server, there is nothing in-between.
When you first SSH into the server, the key is stored in your keychain and reused for these operations.
This way, you don't have to enter it everytime, but it's still safely stored.
Does this need a special agent on my server?
No, nothing to install on your server. Everything is done via SSH from your phone and the server, there is nothing in-between. When you first SSH into the server, the key is stored in your keychain and reused for these operations. This way, you don't have to enter it everytime, but it's still safely stored.