Transport plugins

This chapter describes the generic implementation of a transport plugin. The currently implemented are the local and the ssh plugin. The local plugin makes use only of some standard python modules like os and shutil. The ssh plugin is a wrapper to the library paramiko, that you installed with AiiDA.

A generic set of tests is contained in plugin_test.py, while plugin-specific tests are written separately.

Generic transport class

class aiida.transport.FileAttribute(init=None)[source]

A class, resembling a dictionary, to describe the attributes of a file, that is returned by get_attribute(). Possible keys: st_size, st_uid, st_gid, st_mode, st_atime, st_mtime

class aiida.transport.Transport(*args, **kwargs)[source]

Abstract class for a generic transport (ssh, local, …) Contains the set of minimal methods

__enter__()[source]

For transports that require opening a connection, opens all required channels (used in ‘with’ statements)

__exit__(type, value, traceback)[source]

Closes connections, if needed (used in ‘with’ statements).

chdir(path)[source]

Change directory to ‘path’

Parameters:path (str) – path to change working directory into.
Raises:IOError, if the requested path does not exist
Return type:str
chmod(path, mode)[source]

Change permissions of a path.

Parameters:
  • path (str) – path to file
  • mode (int) – new permissions
chown(path, uid, gid)[source]

Change the owner (uid) and group (gid) of a file. As with python’s os.chown function, you must pass both arguments, so if you only want to change one, use stat first to retrieve the current owner and group.

Parameters:
  • path (str) – path to the file to change the owner and group of
  • uid (int) – new owner’s uid
  • gid (int) – new group id
close()[source]

Closes the local transport channel

copy(remotesource, remotedestination, *args, **kwargs)[source]

Copy a file or a directory from remote source to remote destination (On the same remote machine)

Parameters:
  • remotesource (str) – path of the remote source directory / file
  • remotedestination (str) – path of the remote destination directory / file
Raises:

IOError, if one of src or dst does not exist

copy_from_remote_to_remote(transportdestination, remotesource, remotedestination, **kwargs)[source]

Copy files or folders from a remote computer to another remote computer.

Parameters:
  • transportdestination – transport to be used for the destination computer
  • remotesource (str) – path to the remote source directory / file
  • remotedestination (str) – path to the remote destination directory / file
  • kwargs – keyword parameters passed to the call to transportdestination.put, except for ‘dereference’ that is passed to self.get

Note

the keyword ‘dereference’ SHOULD be set to False for the final put (onto the destination), while it can be set to the value given in kwargs for the get from the source. In that way, a symbolic link would never be followed in the final copy to the remote destination. That way we could avoid getting unknown (potentially malicious) files into the destination computer. HOWEVER, since dereference=False is currently NOT supported by all plugins, we still force it to True for the final put.

Note

the supported keys in kwargs are callback, dereference, overwrite and ignore_nonexisting.

copyfile(remotesource, remotedestination, *args, **kwargs)[source]

Copy a file from remote source to remote destination (On the same remote machine)

Parameters:
  • remotesource (str) – path of the remote source directory / file
  • remotedestination (str) – path of the remote destination directory / file
Raises:

IOError – if one of src or dst does not exist

copytree(remotesource, remotedestination, *args, **kwargs)[source]

Copy a folder from remote source to remote destination (On the same remote machine)

Parameters:
  • remotesource (str) – path of the remote source directory / file
  • remotedestination (str) – path of the remote destination directory / file
Raises:

IOError – if one of src or dst does not exist

exec_command_wait(command, **kwargs)[source]

Execute the command on the shell, waits for it to finish, and return the retcode, the stdout and the stderr.

Enforce the execution to be run from the pwd (as given by self.getcwd), if this is not None.

Parameters:command (str) – execute the command given as a string
Returns:a list: the retcode (int), stdout (str) and stderr (str).
get(remotepath, localpath, *args, **kwargs)[source]

Retrieve a file or folder from remote source to local destination dst must be an absolute path (src not necessarily)

Parameters:
  • remotepath – (str) remote_folder_path
  • localpath – (str) local_folder_path
get_attribute(path)[source]

Return an object FixedFieldsAttributeDict for file in a given path, as defined in aiida.common.extendeddicts Each attribute object consists in a dictionary with the following keys:

  • st_size: size of files, in bytes
  • st_uid: user id of owner
  • st_gid: group id of owner
  • st_mode: protection bits
  • st_atime: time of most recent access
  • st_mtime: time of most recent modification
Parameters:path (str) – path to file
Returns:object FixedFieldsAttributeDict
get_mode(path)[source]

Return the portion of the file’s mode that can be set by chmod().

Parameters:path (str) – path to file
Returns:the portion of the file’s mode that can be set by chmod()
classmethod get_short_doc()[source]

Return the first non-empty line of the class docstring, if available

classmethod get_valid_auth_params()[source]

Return the internal list of valid auth_params

classmethod get_valid_transports()[source]
Returns:a list of existing plugin names
getcwd()[source]

Get working directory

Returns:a string identifying the current working directory
getfile(remotepath, localpath, *args, **kwargs)[source]

Retrieve a file from remote source to local destination dst must be an absolute path (src not necessarily)

Parameters:
  • remotepath (str) – remote_folder_path
  • localpath (str) – local_folder_path
gettree(remotepath, localpath, *args, **kwargs)[source]

Retrieve a folder recursively from remote source to local destination dst must be an absolute path (src not necessarily)

Parameters:
  • remotepath (str) – remote_folder_path
  • localpath (str) – local_folder_path
glob(pathname)[source]

Return a list of paths matching a pathname pattern.

The pattern may contain simple shell-style wildcards a la fnmatch.

gotocomputer_command(remotedir)[source]

Return a string to be run using os.system in order to connect via the transport to the remote directory.

Expected behaviors:

  • A new bash session is opened
  • A reasonable error message is produced if the folder does not exist
Parameters:remotedir (str) – the full path of the remote directory
iglob(pathname)[source]

Return an iterator which yields the paths matching a pathname pattern.

The pattern may contain simple shell-style wildcards a la fnmatch.

isdir(path)[source]

True if path is an existing directory.

Parameters:path (str) – path to directory
Returns:boolean
isfile(path)[source]

Return True if path is an existing file.

Parameters:path (str) – path to file
Returns:boolean
listdir(path='.', pattern=None)[source]

Return a list of the names of the entries in the given path. The list is in arbitrary order. It does not include the special entries ‘.’ and ‘..’ even if they are present in the directory.

Parameters:
  • path (str) – path to list (default to ‘.’)
  • pattern (str) – if used, listdir returns a list of files matching filters in Unix style. Unix only.
Returns:

a list of strings with the file/directory names

listdir_withattributes(path='.', pattern=None)[source]

Return a list of the names of the entries in the given path. The list is in arbitrary order. It does not include the special entries ‘.’ and ‘..’ even if they are present in the directory.

Parameters:
  • path (str) – path to list (default to ‘.’)
  • pattern (str) – if used, listdir returns a list of files matching filters in Unix style. Unix only.
Returns:

a list of dictionaries, one per entry. The schema of the dictionary is the following:

{
   'name': String,
   'attributes': FileAttributeObject,
   'isdir': Bool
}

where ‘name’ is the file or folder directory, and any other information is metadata (if the file is a folder, a directory, …). ‘attributes’ behaves as the output of transport.get_attribute(); isdir is a boolean indicating if the object is a directory or not.

logger

Return the internal logger. If you have set extra parameters using _set_logger_extra(), a suitable LoggerAdapter instance is created, bringing with itself also the extras.

makedirs(path, ignore_existing=False)[source]

Super-mkdir; create a leaf directory and all intermediate ones. Works like mkdir, except that any intermediate path segment (not just the rightmost) will be created if it does not exist.

Parameters:
  • path (str) – directory to create
  • ignore_existing (bool) – if set to true, it doesn’t give any error if the leaf directory does already exist
Raises:

OSError, if directory at path already exists

mkdir(path, ignore_existing=False)[source]

Create a folder (directory) named path.

Parameters:
  • path (str) – name of the folder to create
  • ignore_existing (bool) – if True, does not give any error if the directory already exists
Raises:

OSError, if directory at path already exists

normalize(path='.')[source]

Return the normalized path (on the server) of a given path. This can be used to quickly resolve symbolic links or determine what the server is considering to be the “current folder”.

Parameters:path (str) – path to be normalized
Raises:IOError – if the path can’t be resolved on the server
open()[source]

Opens a local transport channel

path_exists(path)[source]

Returns True if path exists, False otherwise.

put(localpath, remotepath, *args, **kwargs)[source]

Put a file or a directory from local src to remote dst. src must be an absolute path (dst not necessarily)) Redirects to putfile and puttree.

Parameters:
  • localpath (str) – absolute path to local source
  • remotepath (str) – path to remote destination
putfile(localpath, remotepath, *args, **kwargs)[source]

Put a file from local src to remote dst. src must be an absolute path (dst not necessarily))

Parameters:
  • localpath (str) – absolute path to local file
  • remotepath (str) – path to remote file
puttree(localpath, remotepath, *args, **kwargs)[source]

Put a folder recursively from local src to remote dst. src must be an absolute path (dst not necessarily))

Parameters:
  • localpath (str) – absolute path to local folder
  • remotepath (str) – path to remote folder
remove(path)[source]

Remove the file at the given path. This only works on files; for removing folders (directories), use rmdir.

Parameters:path (str) – path to file to remove
Raises:IOError – if the path is a directory
rename(oldpath, newpath)[source]

Rename a file or folder from oldpath to newpath.

Parameters:
  • oldpath (str) – existing name of the file or folder
  • newpath (str) – new name for the file or folder
Raises:
  • IOError – if oldpath/newpath is not found
  • ValueError – if oldpath/newpath is not a valid string
rmdir(path)[source]

Remove the folder named path. This works only for empty folders. For recursive remove, use rmtree.

Parameters:path (str) – absolute path to the folder to remove
rmtree(path)[source]

Remove recursively the content at path

Parameters:path (str) – absolute path to remove
symlink(remotesource, remotedestination)[source]

Create a symbolic link between the remote source and the remote destination.

Parameters:
  • remotesource – remote source
  • remotedestination – remote destination
whoami()[source]

Get the remote username

Returns:list of username (str), retval (int), stderr (str)
aiida.transport.TransportFactory(module)[source]

Used to return a suitable Transport subclass.

Parameters:module (str) – name of the module containing the Transport subclass
Returns:the transport subclass located in module ‘module’
exception aiida.transport.TransportInternalError[source]

Raised if there is a transport error that is raised to an internal error (e.g. a transport method called without opening the channel first).

aiida.transport.copy_from_remote_to_remote(transportsource, transportdestination, remotesource, remotedestination, **kwargs)[source]

Copy files or folders from a remote computer to another remote computer.

Parameters:
  • transportsource – transport to be used for the source computer
  • transportdestination – transport to be used for the destination computer
  • remotesource (str) – path to the remote source directory / file
  • remotedestination (str) – path to the remote destination directory / file
  • kwargs – keyword parameters passed to the final put, except for ‘dereference’ that is passed to the initial get

Note

it uses the method transportsource.copy_from_remote_to_remote

Existing plugins

class aiida.transport.plugins.ssh.SshTransport(machine, **kwargs)[source]

Support connection, command execution and data transfer to remote computers via SSH+SFTP.

chdir(path)[source]

Change directory of the SFTP session. Emulated internally by paramiko.

Differently from paramiko, if you pass None to chdir, nothing happens and the cwd is unchanged.

chmod(path, mode)[source]

Change permissions to path

Parameters:
  • path – path to file
  • mode – new permission bits (integer)
close()[source]

Close the SFTP channel, and the SSHClient.

Todo:correctly manage exceptions
Raises:InvalidOperation – if the channel is already open
copy(remotesource, remotedestination, dereference=False)[source]

Copy a file or a directory from remote source to remote destination. Flags used: -r: recursive copy; -f: force, makes the command non interactive; -L follows symbolic links

Parameters:
  • remotesource – file to copy from
  • remotedestination – file to copy to
  • dereference – if True, copy content instead of copying the symlinks only Default = False.
Raises:

IOError – if the cp execution failed.

Note

setting dereference equal to True could cause infinite loops.

copyfile(remotesource, remotedestination, dereference=False, pattern=None)[source]

Copy a file from remote source to remote destination Redirects to copy().

Parameters:
  • remotesource
  • remotedestination
  • dereference
  • pattern
copytree(remotesource, remotedestination, dereference=False, pattern=None)[source]

copy a folder recursively from remote source to remote destination Redirects to copy()

Parameters:
  • remotesource
  • remotedestination
  • dereference
  • pattern
exec_command_wait(command, stdin=None, combine_stderr=False, bufsize=-1)[source]

Executes the specified command and waits for it to finish.

Parameters:
  • command – the command to execute
  • stdin – (optional,default=None) can be a string or a file-like object.
  • combine_stderr – (optional, default=False) see docstring of self._exec_command_internal()
  • bufsize – same meaning of paramiko.
Returns:

a tuple with (return_value, stdout, stderr) where stdout and stderr are strings.

get(remotepath, localpath, callback=None, dereference=True, overwrite=True, ignore_nonexisting=False)[source]

Get a file or folder from remote to local. Redirects to getfile or gettree.

Parameters:
  • remotepath – a remote path
  • localpath – an (absolute) local path
  • dereference – follow symbolic links. Default = True (default behaviour in paramiko). False is not implemented.
  • overwrite – if True overwrites files and folders. Default = False
Raises:
  • ValueError – if local path is invalid
  • IOError – if the remotepath is not found
get_attribute(path)[source]

Returns the object Fileattribute, specified in aiida.transport Receives in input the path of a given file.

getcwd()[source]

Return the current working directory for this SFTP session, as emulated by paramiko. If no directory has been set with chdir, this method will return None. But in __enter__ this is set explicitly, so this should never happen within this class.

getfile(remotepath, localpath, callback=None, dereference=True, overwrite=True)[source]

Get a file from remote to local.

Parameters:
  • remotepath – a remote path
  • localpath – an (absolute) local path
  • overwrite – if True overwrites files and folders. Default = False
Raises:
  • ValueError – if local path is invalid
  • OSError – if unintentionally overwriting
gettree(remotepath, localpath, callback=None, dereference=True, overwrite=True)[source]

Get a folder recursively from remote to local.

Parameters:
  • remotepath – a remote path
  • localpath – an (absolute) local path
  • dereference – follow symbolic links. Default = True (default behaviour in paramiko). False is not implemented.
  • overwrite – if True overwrites files and folders. Default = False
Raises:
  • ValueError – if local path is invalid
  • IOError – if the remotepath is not found
  • OSError – if unintentionally overwriting
gotocomputer_command(remotedir)[source]

Specific gotocomputer string to connect to a given remote computer via ssh and directly go to the calculation folder.

isdir(path)[source]

Return True if the given path is a directory, False otherwise. Return False also if the path does not exist.

isfile(path)[source]

Return True if the given path is a file, False otherwise. Return False also if the path does not exist.

listdir(path='.', pattern=None)[source]

Get the list of files at path.

Parameters:
  • path – default = ‘.’
  • pattern – returns the list of files matching pattern. Unix only. (Use to emulate ls * for example)
makedirs(path, ignore_existing=False)[source]

Super-mkdir; create a leaf directory and all intermediate ones. Works like mkdir, except that any intermediate path segment (not just the rightmost) will be created if it does not exist.

NOTE: since os.path.split uses the separators as the host system (that could be windows), I assume the remote computer is Linux-based and use ‘/’ as separators!

Parameters:
  • path – directory to create (string)
  • ignore_existing – if set to true, it doesn’t give any error if the leaf directory does already exist (bool)
Raises:

OSError – If the directory already exists.

mkdir(path, ignore_existing=False)[source]

Create a folder (directory) named path.

Parameters:
  • path – name of the folder to create
  • ignore_existing – if True, does not give any error if the directory already exists
Raises:

OSError – If the directory already exists.

normalize(path)[source]

Returns the normalized path (removing double slashes, etc…)

open()[source]

Open a SSHClient to the machine possibly using the parameters given in the __init__.

Also opens a sftp channel, ready to be used. The current working directory is set explicitly, so it is not None.

Raises:InvalidOperation – if the channel is already open
path_exists(path)[source]

Check if path exists

put(localpath, remotepath, callback=None, dereference=True, overwrite=True, ignore_nonexisting=False)[source]

Put a file or a folder from local to remote. Redirects to putfile or puttree.

Parameters:
  • localpath – an (absolute) local path
  • remotepath – a remote path
  • dereference – follow symbolic links (boolean). Default = True (default behaviour in paramiko). False is not implemented.
  • overwrite – if True overwrites files and folders (boolean). Default = False.
Raises:
  • ValueError – if local path is invalid
  • OSError – if the localpath does not exist
putfile(localpath, remotepath, callback=None, dereference=True, overwrite=True)[source]

Put a file from local to remote.

Parameters:
  • localpath – an (absolute) local path
  • remotepath – a remote path
  • overwrite – if True overwrites files and folders (boolean). Default = True.
Raises:
  • ValueError – if local path is invalid
  • OSError – if the localpath does not exist, or unintentionally overwriting
puttree(localpath, remotepath, callback=None, dereference=True, overwrite=True)[source]

Put a folder recursively from local to remote.

Parameters:
  • localpath – an (absolute) local path
  • remotepath – a remote path
  • dereference – follow symbolic links (boolean) Default = True (default behaviour in paramiko). False is not implemented.
  • overwrite – if True overwrites files and folders (boolean). Default = True
Raises:
  • ValueError – if local path is invalid
  • OSError – if the localpath does not exist, or trying to overwrite
  • IOError – if remotepath is invalid

Note

setting dereference equal to True could cause infinite loops. see os.walk() documentation

remove(path)[source]

Remove a single file at ‘path’

rename(src, dst)[source]

Rename a file or folder from src to dst.

Parameters:
  • oldpath (str) – existing name of the file or folder
  • newpath (str) – new name for the file or folder
Raises:
  • IOError – if src/dst is not found
  • ValueError – if src/dst is not a valid string
rmdir(path)[source]

Remove the folder named ‘path’ if empty.

rmtree(path)[source]

Remove a file or a directory at path, recursively Flags used: -r: recursive copy; -f: force, makes the command non interactive;

Parameters:path – remote path to delete
Raises:IOError – if the rm execution failed.
symlink(remotesource, remotedestination)[source]

Create a symbolic link between the remote source and the remote destination.

Parameters:
  • remotesource – remote source. Can contain a pattern.
  • remotedestination – remote destination
class aiida.transport.plugins.local.LocalTransport(**kwargs)[source]

Support copy and command execution on the same host on which AiiDA is running via direct file copy and execution commands.

chdir(path)[source]

Changes directory to path, emulated internally. :param path: path to cd into :raise OSError: if the directory does not have read attributes.

chmod(path, mode)[source]

Changes permission bits of object at path :param path: path to modify :param mode: permission bits

Raises:IOError – if path does not exist.
close()[source]

Closes the local transport channel

Raises:InvalidOperation – if the channel is already open
copy(source, destination, dereference=False)[source]

Copies a file or a folder from ‘remote’ source to ‘remote’ destination. Automatically redirects to copyfile or copytree.

Parameters:
  • source – path to local file
  • destination – path to remote file
  • dereference – follow symbolic links. Default = False
Raises:
  • ValueError – if ‘remote’ source or destination is not valid
  • OSError – if source does not exist
copyfile(source, destination)[source]

Copies a file from ‘remote’ source to ‘remote’ destination.

Parameters:
  • source – path to local file
  • destination – path to remote file
Raises:
  • ValueError – if ‘remote’ source or destination is not valid
  • OSError – if source does not exist
copytree(source, destination, dereference=False)[source]

Copies a folder from ‘remote’ source to ‘remote’ destination.

Parameters:
  • source – path to local file
  • destination – path to remote file
  • dereference – follow symbolic links. Default = False
Raises:
  • ValueError – if ‘remote’ source or destination is not valid
  • OSError – if source does not exist
curdir

Returns the _internal_dir, if the channel is open. If possible, use getcwd() instead!

exec_command_wait(command, stdin=None)[source]

Executes the specified command and waits for it to finish.

Parameters:command – the command to execute
Returns:a tuple with (return_value, stdout, stderr) where stdout and stderr are strings.
get(source, destination, dereference=True, overwrite=True, ignore_nonexisting=False)[source]

Copies a folder or a file recursively from ‘remote’ source to ‘local’ destination. Automatically redirects to getfile or gettree.

Parameters:
  • source – path to local file
  • destination – absolute path to remote file
  • dereference – follow symbolic links default = True
  • overwrite – if True overwrites destination default = False
Raises:
  • IOError – if ‘remote’ source is not valid
  • ValueError – if ‘local’ destination is not valid
get_attribute(path)[source]

Returns an object FileAttribute, as specified in aiida.transport. :param path: the path of the given file.

getcwd()[source]

Returns the current working directory, emulated by the transport

getfile(source, destination, overwrite=True)[source]

Copies a file recursively from ‘remote’ source to ‘local’ destination.

Parameters:
  • source – path to local file
  • destination – absolute path to remote file
  • overwrite – if True overwrites destination. Default = False

:raise IOError if ‘remote’ source is not valid or not found :raise ValueError: if ‘local’ destination is not valid :raise OSError: if unintentionally overwriting

gettree(source, destination, dereference=True, overwrite=True)[source]

Copies a folder recursively from ‘remote’ source to ‘local’ destination.

Parameters:
  • source – path to local file
  • destination – absolute path to remote file
  • dereference – follow symbolic links. Default = True
  • overwrite – if True overwrites destination. Default = False
Raises:
  • IOError – if ‘remote’ source is not valid
  • ValueError – if ‘local’ destination is not valid
  • OSError – if unintentionally overwriting
gotocomputer_command(remotedir)[source]

Return a string to be run using os.system in order to connect via the transport to the remote directory.

Expected behaviors:

  • A new bash session is opened
  • A reasonable error message is produced if the folder does not exist
Parameters:remotedir (str) – the full path of the remote directory
isdir(path)[source]

Checks if ‘path’ is a directory. :return: a boolean

isfile(path)[source]

Checks if object at path is a file. Returns a boolean.

listdir(path='.', pattern=None)[source]
Returns:

a list containing the names of the entries in the directory.

Parameters:
  • path – default =’.’
  • pattern – if set, returns the list of files matching pattern. Unix only. (Use to emulate ls * for example)
makedirs(path, ignore_existing=False)[source]

Super-mkdir; create a leaf directory and all intermediate ones. Works like mkdir, except that any intermediate path segment (not just the rightmost) will be created if it does not exist.

Parameters:
  • path – directory to create
  • ignore_existing – if set to true, it doesn’t give any error if the leaf directory does already exist
Raises:

OSError – If the directory already exists and is not ignore_existing

mkdir(path, ignore_existing=False)[source]

Create a folder (directory) named path.

Parameters:
  • path – name of the folder to create
  • ignore_existing – if True, does not give any error if the directory already exists
Raises:

OSError – If the directory already exists.

normalize(path)[source]

Normalizes path, eliminating double slashes, etc.. :param path: path to normalize

open()[source]

Opens a local transport channel

Raises:InvalidOperation – if the channel is already open
path_exists(path)[source]

Check if path exists

put(source, destination, dereference=True, overwrite=True, ignore_nonexisting=False)[source]

Copies a file or a folder from source to destination. Automatically redirects to putfile or puttree.

Parameters:
  • source – absolute path to local file
  • destination – path to remote file
  • dereference – if True follows symbolic links. Default = True
  • overwrite – if True overwrites destination. Default = False
Raises:
  • IOError – if destination is not valid
  • ValueError – if source is not valid
putfile(source, destination, overwrite=True)[source]

Copies a file from source to destination. Automatically redirects to putfile or puttree.

Parameters:
  • source – absolute path to local file
  • destination – path to remote file
  • overwrite – if True overwrites destination Default = False
Raises:
  • IOError – if destination is not valid
  • ValueError – if source is not valid
  • OSError – if source does not exist
puttree(source, destination, dereference=True, overwrite=True)[source]

Copies a folder recursively from source to destination. Automatically redirects to putfile or puttree.

Parameters:
  • source – absolute path to local file
  • destination – path to remote file
  • dereference – follow symbolic links. Default = True
  • overwrite – if True overwrites destination. Default = False
Raises:
  • IOError – if destination is not valid
  • ValueError – if source is not valid
  • OSError – if source does not exist
remove(path)[source]

Removes a file at position path.

rename(src, dst)[source]

Rename a file or folder from oldpath to newpath.

Parameters:
  • oldpath (str) – existing name of the file or folder
  • newpath (str) – new name for the file or folder
Raises:
  • IOError – if src/dst is not found
  • ValueError – if src/dst is not a valid string
rmdir(path)[source]

Removes a folder at location path. :param path: path to remove

rmtree(path)[source]

Remove tree as rm -r would do

Parameters:path – a string to path
symlink(remotesource, remotedestination)[source]

Create a symbolic link between the remote source and the remote destination

Parameters:
  • remotesource – remote source. Can contain a pattern.
  • remotedestination – remote destination

Developing a plugin

The transport class is actually almost never used directly by the user. It is mostly utilized by the ExecutionManager, that use the transport plugin to connect to the remote computer to manage the calculation. The ExecutionManager has to be able to use always the same function, or the same interface, regardless of which kind of connection is actually really using.

The generic transport class contains a set of minimal methods that an implementation must support, in order to be fully compatible with the other plugins. If not, a NotImplementedError will be raised, interrupting the managing of the calculation or whatever is using the transport plugin.

Since it is important that all plugins have the same interface, or the same response behavior, a set of generic tests has been written (alongside with set of tests that are implementation specific). After every modification, or when implementing a new plugin, it is crucial to run the tests and verify that everything is passed. The modification of tests possibly means breaking back-compatibility and/or modifications to every piece of code using a transport plugin.

If an unexpected behavior is observed during the usage, the way of fixing it is:

  1. Write a new test that shows the problem (one test for one problem when possible)
  2. Fix the bug
  3. Verify that the test is passed correctly

The importance of point 1) is often neglected, but unittesting is a useful tool that helps you avoiding the repetition of errors. Despite the appearence, it’s a time-saver! Not only, the tests help you seeing how the plugin is used.

As for the general functioning of the plugin, the __init__ method is used only to initialize the class instance, without actually opening the transport channel. The connection must be opened only by the __enter__ method, (and closed by __exit__. The __enter__ method let you use the transport class using the with statement (see Python docs), in a way similar to the following:

t = TransportPlugin()
with open(t):
    t.do_something_remotely

To ensure this, for example, the local plugin uses a hidden boolean variable _is_open that is set when the __enter__ and __exit__ methods are called. The Ssh logic is instead given by the property sftp.

The other functions that require some care are the copying functions, called using the following terminology:

  1. put: from local source to remote destination
  2. get: from remote source to local destination
  3. copy: copying files from remote source to remote destination

Note that these functions must copy files or folders regardless, internally, they will fallback to functions like putfile or puttree.

The last function requiring care is exec_command_wait, which is an analogue to the subprocess Python module. The function gives the freedom to execute a string as a remote command, thus it could produce nasty effects if not written with care. Be sure to escape any string for bash!

Currently, the implemented plugins are the Local and the Ssh transports. The Local one is simply a wrapper to some standard Python modules, like shutil or os, those functions are simply interfaced in a different way with AiiDA. The SSh instead is an interface to the Paramiko library.

Below, you can find a template to fill for a new transport plugin, with a minimal docstring that also work for the sphinx documentation.

class NewTransport(aiida.transport.Transport):

  def __init__(self, machine, **kwargs):
      """
      Initialize the Transport class.

      :param machine: the machine to connect to
      """

  def __enter__(self):
      """
      Open the connection
      """

  def __exit__(self, type, value, traceback):
      """
      Close the connection
      """

  def chdir(self,path):
      """
      Change directory to 'path'

      :param str path: path to change working directory into.
      :raises: IOError, if the requested path does not exist
      :rtype: string
      """

  def chmod(self,path,mode):
      """
      Change permissions of a path.

      :param str path: path to file
      :param int mode: new permissions
      """

  def copy(self,remotesource,remotedestination,*args,**kwargs):
      """
      Copy a file or a directory from remote source to remote destination
      (On the same remote machine)

      :param str remotesource: path of the remote source directory / file
      :param str remotedestination: path of the remote destination directory / file

      :raises: IOError, if source or destination does not exist
      """
      raise NotImplementedError

  def copyfile(self,remotesource,remotedestination,*args,**kwargs):
      """
      Copy a file from remote source to remote destination
      (On the same remote machine)

      :param str remotesource: path of the remote source directory / file
      :param str remotedestination: path of the remote destination directory / file

      :raises IOError: if one of src or dst does not exist
      """

  def copytree(self,remotesource,remotedestination,*args,**kwargs):
      """
      Copy a folder from remote source to remote destination
      (On the same remote machine)

      :param str remotesource: path of the remote source directory / file
      :param str remotedestination: path of the remote destination directory / file

      :raise IOError: if one of src or dst does not exist
      """

  def exec_command_wait(self,command, **kwargs):
      """
      Execute the command on the shell, waits for it to finish,
      and return the retcode, the stdout and the stderr.

      Enforce the execution to be run from the pwd (as given by
      self.getcwd), if this is not None.

      :param str command: execute the command given as a string
      :return: a tuple: the retcode (int), stdout (str) and stderr (str).
      """

  def get_attribute(self,path):
      """
      Return an object FixedFieldsAttributeDict for file in a given path,
      as defined in aiida.common.extendeddicts
      Each attribute object consists in a dictionary with the following keys:

      * st_size: size of files, in bytes

      * st_uid: user id of owner

      * st_gid: group id of owner

      * st_mode: protection bits

      * st_atime: time of most recent access

      * st_mtime: time of most recent modification

      :param str path: path to file
      :return: object FixedFieldsAttributeDict
      """

  def getcwd(self):
      """
      Get working directory

      :return: a string identifying the current working directory
      """

  def get(self, remotepath, localpath, *args, **kwargs):
      """
      Retrieve a file or folder from remote source to local destination
      dst must be an absolute path (src not necessarily)

      :param remotepath: (str) remote_folder_path
      :param localpath: (str) local_folder_path
      """

  def getfile(self, remotepath, localpath, *args, **kwargs):
      """
      Retrieve a file from remote source to local destination
      dst must be an absolute path (src not necessarily)

      :param str remotepath: remote_folder_path
      :param str localpath: local_folder_path
      """

  def gettree(self, remotepath, localpath, *args, **kwargs):
      """
      Retrieve a folder recursively from remote source to local destination
      dst must be an absolute path (src not necessarily)

      :param str remotepath: remote_folder_path
      :param str localpath: local_folder_path
      """

  def gotocomputer_command(self, remotedir):
      """
      Return a string to be run using os.system in order to connect
      via the transport to the remote directory.

      Expected behaviors:

      * A new bash session is opened

      * A reasonable error message is produced if the folder does not exist

      :param str remotedir: the full path of the remote directory
      """

  def isdir(self,path):
      """
      True if path is an existing directory.

      :param str path: path to directory
      :return: boolean
      """

  def isfile(self,path):
      """
      Return True if path is an existing file.

      :param str path: path to file
      :return: boolean
      """

  def listdir(self, path='.',pattern=None):
      """
      Return a list of the names of the entries in the given path.
      The list is in arbitrary order. It does not include the special
      entries '.' and '..' even if they are present in the directory.

      :param str path: path to list (default to '.')
      :param str pattern: if used, listdir returns a list of files matching
                          filters in Unix style. Unix only.
      :return: a list of strings
      """

  def makedirs(self,path,ignore_existing=False):
      """
      Super-mkdir; create a leaf directory and all intermediate ones.
      Works like mkdir, except that any intermediate path segment (not
      just the rightmost) will be created if it does not exist.

      :param str path: directory to create
      :param bool ignore_existing: if set to true, it doesn't give any error
                                   if the leaf directory does already exist

      :raises: OSError, if directory at path already exists
      """

  def mkdir(self,path,ignore_existing=False):
      """
      Create a folder (directory) named path.

      :param str path: name of the folder to create
      :param bool ignore_existing: if True, does not give any error if the
                                   directory already exists

      :raises: OSError, if directory at path already exists
      """

  def normalize(self,path='.'):
      """
      Return the normalized path (on the server) of a given path.
      This can be used to quickly resolve symbolic links or determine
      what the server is considering to be the "current folder".

      :param str path: path to be normalized

      :raise IOError: if the path can't be resolved on the server
      """

  def put(self, localpath, remotepath, *args, ** kwargs):
      """
      Put a file or a directory from local src to remote dst.
      src must be an absolute path (dst not necessarily))
      Redirects to putfile and puttree.

      :param str localpath: path to remote destination
      :param str remotepath: absolute path to local source
      """

  def putfile(self, localpath, remotepath, *args, ** kwargs):
      """
      Put a file from local src to remote dst.
      src must be an absolute path (dst not necessarily))

      :param str localpath: path to remote file
      :param str remotepath: absolute path to local file
      """

  def puttree(self, localpath, remotepath, *args, ** kwargs):
      """
      Put a folder recursively from local src to remote dst.
      src must be an absolute path (dst not necessarily))

      :param str localpath: path to remote folder
      :param str remotepath: absolute path to local folder
      """

 def rename(src,dst):
      """
      Rename a file or folder from src to dst.

      :param str oldpath: existing name of the file or folder
      :param str newpath: new name for the file or folder

      :raises IOError: if src/dst is not found
      :raises ValueError: if src/dst is not a valid string
      """

  def remove(self,path):
      """
      Remove the file at the given path. This only works on files;
      for removing folders (directories), use rmdir.

      :param str path: path to file to remove

      :raise IOError: if the path is a directory
      """

  def rmdir(self,path):
      """
      Remove the folder named path.
      This works only for empty folders. For recursive remove, use rmtree.

      :param str path: absolute path to the folder to remove
      """
      raise NotImplementedError

  def rmtree(self,path):
      """
      Remove recursively the content at path

      :param str path: absolute path to remove
      """