Workaround for paramiko key rotation
Created by: agraubert
Paramiko deadlocks (then times out) if key negotiation begins while an active, non sftp channel is in use (see #13 (closed)). This PR adds three workarounds for this problem:
- Before and after every command invoked through the a Remote backend instance, the backend checks if paramiko is close to its threshold for key rotation. If so, it closes and restarts the connection. This method, while disruptive and ugly, seems more reliable than
paramiko.Transport.renegotiate_keys()
. To minimize disruption, canine transports are automatically reconnected in this process (although open file handles will not be restored). - If the user can ensure that their environment is "sufficiently" protected from MITM attacks, they can call
backend.disable_paramiko_rekey()
. This attempts to completely disable paramiko's counters which trigger key rotation. This should only be used if the user trusts their connection (such as when communicating between two machines in a VPN) as the AES ciphers used will become increasingly vulnerable to observation as they are used without rotating keys. - In the event that either of these two systems is not sufficient to avoid paramiko's natural rekey process and paramiko deadlocks while executing a command, additional error handling has been added so that commands can be re-attempted after restoring the connection
These three workarounds do not represent a perfect solution, but I feel that they will greatly improve the odds of canine continuing to function over long-running pipelines