Skip to content

Workaround for paramiko key rotation

Aaron Graubert requested to merge keyrot into master

Created by: agraubert

Paramiko deadlocks (then times out) if key negotiation begins while an active, non sftp channel is in use (see #13 (closed)). This PR adds three workarounds for this problem:

  1. Before and after every command invoked through the a Remote backend instance, the backend checks if paramiko is close to its threshold for key rotation. If so, it closes and restarts the connection. This method, while disruptive and ugly, seems more reliable than paramiko.Transport.renegotiate_keys(). To minimize disruption, canine transports are automatically reconnected in this process (although open file handles will not be restored).
  2. If the user can ensure that their environment is "sufficiently" protected from MITM attacks, they can call backend.disable_paramiko_rekey(). This attempts to completely disable paramiko's counters which trigger key rotation. This should only be used if the user trusts their connection (such as when communicating between two machines in a VPN) as the AES ciphers used will become increasingly vulnerable to observation as they are used without rotating keys.
  3. In the event that either of these two systems is not sufficient to avoid paramiko's natural rekey process and paramiko deadlocks while executing a command, additional error handling has been added so that commands can be re-attempted after restoring the connection

These three workarounds do not represent a perfect solution, but I feel that they will greatly improve the odds of canine continuing to function over long-running pipelines

Merge request reports