aiida.transports package¶
Module for classes and utilities to define transports to other machines.
-
class
aiida.transports.
Transport
(*args, **kwargs)[source]¶ Bases:
object
Abstract class for a generic transport (ssh, local, …) Contains the set of minimal methods
-
_MAGIC_CHECK
= re.compile('[*?[]')¶
-
__abstractmethods__
= frozenset({})¶
-
__dict__
= mappingproxy({'__module__': 'aiida.transports.transport', '__doc__': '\n Abstract class for a generic transport (ssh, local, ...)\n Contains the set of minimal methods\n ', '_valid_auth_params': None, '_MAGIC_CHECK': re.compile('[*?[]'), '_valid_auth_options': [], '_common_auth_options': [('safe_interval', {'type': <class 'float'>, 'prompt': 'Connection cooldown time (s)', 'help': 'Minimum time interval in seconds between consecutive connection openings', 'callback': <function validate_positive_number>})], '__init__': <function Transport.__init__>, '__enter__': <function Transport.__enter__>, '__exit__': <function Transport.__exit__>, 'is_open': <property object>, 'open': <function Transport.open>, 'close': <function Transport.close>, '__repr__': <function Transport.__repr__>, '__str__': <function Transport.__str__>, 'set_logger_extra': <function Transport.set_logger_extra>, 'get_short_doc': <classmethod object>, 'get_valid_transports': <classmethod object>, 'get_valid_auth_params': <classmethod object>, 'auth_options': <aiida.common.lang.classproperty object>, '_get_safe_interval_suggestion_string': <classmethod object>, 'logger': <property object>, 'get_safe_open_interval': <function Transport.get_safe_open_interval>, 'chdir': <function Transport.chdir>, 'chmod': <function Transport.chmod>, 'chown': <function Transport.chown>, 'copy': <function Transport.copy>, 'copyfile': <function Transport.copyfile>, 'copytree': <function Transport.copytree>, 'copy_from_remote_to_remote': <function Transport.copy_from_remote_to_remote>, '_exec_command_internal': <function Transport._exec_command_internal>, 'exec_command_wait': <function Transport.exec_command_wait>, 'get': <function Transport.get>, 'getfile': <function Transport.getfile>, 'gettree': <function Transport.gettree>, 'getcwd': <function Transport.getcwd>, 'get_attribute': <function Transport.get_attribute>, 'get_mode': <function Transport.get_mode>, 'isdir': <function Transport.isdir>, 'isfile': <function Transport.isfile>, 'listdir': <function Transport.listdir>, 'listdir_withattributes': <function Transport.listdir_withattributes>, 'makedirs': <function Transport.makedirs>, 'mkdir': <function Transport.mkdir>, 'normalize': <function Transport.normalize>, 'put': <function Transport.put>, 'putfile': <function Transport.putfile>, 'puttree': <function Transport.puttree>, 'remove': <function Transport.remove>, 'rename': <function Transport.rename>, 'rmdir': <function Transport.rmdir>, 'rmtree': <function Transport.rmtree>, 'gotocomputer_command': <function Transport.gotocomputer_command>, 'symlink': <function Transport.symlink>, 'whoami': <function Transport.whoami>, 'path_exists': <function Transport.path_exists>, 'glob': <function Transport.glob>, 'iglob': <function Transport.iglob>, 'glob1': <function Transport.glob1>, 'glob0': <function Transport.glob0>, 'has_magic': <function Transport.has_magic>, '__dict__': <attribute '__dict__' of 'Transport' objects>, '__weakref__': <attribute '__weakref__' of 'Transport' objects>, '__abstractmethods__': frozenset(), '_abc_impl': <_abc_data object>})¶
-
__enter__
()[source]¶ For transports that require opening a connection, opens all required channels (used in ‘with’ statements).
This object can be used in nested with statements and the connection will only be opened once and closed when the final with scope finishes e.g.:
t = Transport() with t: # Connection is now open.. with t: # ..still open.. pass # ..still open.. # ...closed
-
__exit__
(type_, value, traceback)[source]¶ Closes connections, if needed (used in ‘with’ statements).
-
__module__
= 'aiida.transports.transport'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
_abc_impl
= <_abc_data object>¶
-
_common_auth_options
= [('safe_interval', {'type': <class 'float'>, 'prompt': 'Connection cooldown time (s)', 'help': 'Minimum time interval in seconds between consecutive connection openings', 'callback': <function validate_positive_number>})]¶
-
_exec_command_internal
(command, **kwargs)[source]¶ Execute the command on the shell, similarly to os.system.
Enforce the execution to be run from the cwd (as given by self.getcwd), if this is not None.
If possible, use the higher-level exec_command_wait function.
- Parameters
command (str) – execute the command given as a string
- Returns
stdin, stdout, stderr and the session, when this exists (can be None).
-
classmethod
_get_safe_interval_suggestion_string
(computer)[source]¶ Default time in seconds between consecutive checks.
Set to a non-zero value to be safe e.g. in the case of transports with a connection limit, to avoid overloading the server (and being banned). Should be overriden in plugins. This is anyway just a default, as the value can be changed by the user in the Computer properties, for instance. Currently both the local and the ssh transport override this value, so this is not used, but it will be the default for possible new plugins.
-
_valid_auth_options
= []¶
-
_valid_auth_params
= None¶
-
auth_options
= {'safe_interval': {'callback': <function validate_positive_number>, 'help': 'Minimum time interval in seconds between consecutive connection openings', 'prompt': 'Connection cooldown time (s)', 'type': <class 'float'>}}¶
-
chdir
(path)[source]¶ Change directory to ‘path’
- Parameters
path (str) – path to change working directory into.
- Raises
IOError, if the requested path does not exist
- Return type
str
-
chmod
(path, mode)[source]¶ Change permissions of a path.
- Parameters
path (str) – path to file
mode (int) – new permissions
-
chown
(path, uid, gid)[source]¶ Change the owner (uid) and group (gid) of a file. As with python’s os.chown function, you must pass both arguments, so if you only want to change one, use stat first to retrieve the current owner and group.
- Parameters
path (str) – path to the file to change the owner and group of
uid (int) – new owner’s uid
gid (int) – new group id
-
copy
(remotesource, remotedestination, dereference=False, recursive=True)[source]¶ Copy a file or a directory from remote source to remote destination (On the same remote machine)
- Parameters
remotesource (str) – path of the remote source directory / file
remotedestination (str) – path of the remote destination directory / file
dereference (bool) – if True copy the contents of any symlinks found, otherwise copy the symlinks themselves
recursive (bool) – if True copy directories recursively, otherwise only copy the specified file(s)
- Raises
IOError, if one of src or dst does not exist
-
copy_from_remote_to_remote
(transportdestination, remotesource, remotedestination, **kwargs)[source]¶ Copy files or folders from a remote computer to another remote computer.
- Parameters
transportdestination – transport to be used for the destination computer
remotesource (str) – path to the remote source directory / file
remotedestination (str) – path to the remote destination directory / file
kwargs – keyword parameters passed to the call to transportdestination.put, except for ‘dereference’ that is passed to self.get
Note
the keyword ‘dereference’ SHOULD be set to False for the final put (onto the destination), while it can be set to the value given in kwargs for the get from the source. In that way, a symbolic link would never be followed in the final copy to the remote destination. That way we could avoid getting unknown (potentially malicious) files into the destination computer. HOWEVER, since dereference=False is currently NOT supported by all plugins, we still force it to True for the final put.
Note
the supported keys in kwargs are callback, dereference, overwrite and ignore_nonexisting.
-
copyfile
(remotesource, remotedestination, dereference=False)[source]¶ Copy a file from remote source to remote destination (On the same remote machine)
- Parameters
remotesource (str) – path of the remote source directory / file
remotedestination (str) – path of the remote destination directory / file
dereference (bool) – if True copy the contents of any symlinks found, otherwise copy the symlinks themselves
- Raises
IOError – if one of src or dst does not exist
-
copytree
(remotesource, remotedestination, dereference=False)[source]¶ Copy a folder from remote source to remote destination (On the same remote machine)
- Parameters
remotesource (str) – path of the remote source directory / file
remotedestination (str) – path of the remote destination directory / file
dereference (bool) – if True copy the contents of any symlinks found, otherwise copy the symlinks themselves
- Raises
IOError – if one of src or dst does not exist
-
exec_command_wait
(command, **kwargs)[source]¶ Execute the command on the shell, waits for it to finish, and return the retcode, the stdout and the stderr.
Enforce the execution to be run from the pwd (as given by self.getcwd), if this is not None.
- Parameters
command (str) – execute the command given as a string
- Returns
a list: the retcode (int), stdout (str) and stderr (str).
-
get
(remotepath, localpath, *args, **kwargs)[source]¶ Retrieve a file or folder from remote source to local destination dst must be an absolute path (src not necessarily)
- Parameters
remotepath – (str) remote_folder_path
localpath – (str) local_folder_path
-
get_attribute
(path)[source]¶ Return an object FixedFieldsAttributeDict for file in a given path, as defined in aiida.common.extendeddicts Each attribute object consists in a dictionary with the following keys:
st_size: size of files, in bytes
st_uid: user id of owner
st_gid: group id of owner
st_mode: protection bits
st_atime: time of most recent access
st_mtime: time of most recent modification
- Parameters
path (str) – path to file
- Returns
object FixedFieldsAttributeDict
-
get_mode
(path)[source]¶ Return the portion of the file’s mode that can be set by chmod().
- Parameters
path (str) – path to file
- Returns
the portion of the file’s mode that can be set by chmod()
-
get_safe_open_interval
()[source]¶ Get an interval (in seconds) that suggests how long the user should wait between consecutive calls to open the transport. This can be used as a way to get the user to not swamp a limited number of connections, etc. However it is just advisory.
If returns 0, it is taken that there are no reasons to limit the frequency of open calls.
In the main class, it returns a default value (>0 for safety), set in the _DEFAULT_SAFE_OPEN_INTERVAL attribute of the class. Plugins should override it.
- Returns
The safe interval between calling open, in seconds
- Return type
float
-
classmethod
get_short_doc
()[source]¶ Return the first non-empty line of the class docstring, if available
-
getfile
(remotepath, localpath, *args, **kwargs)[source]¶ Retrieve a file from remote source to local destination dst must be an absolute path (src not necessarily)
- Parameters
remotepath (str) – remote_folder_path
localpath (str) – local_folder_path
-
gettree
(remotepath, localpath, *args, **kwargs)[source]¶ Retrieve a folder recursively from remote source to local destination dst must be an absolute path (src not necessarily)
- Parameters
remotepath (str) – remote_folder_path
localpath (str) – local_folder_path
-
glob
(pathname)[source]¶ Return a list of paths matching a pathname pattern.
The pattern may contain simple shell-style wildcards a la fnmatch.
-
glob0
(dirname, basename)[source]¶ Wrap basename i a list if it is empty or if dirname/basename is an existing path, else return empty list.
-
gotocomputer_command
(remotedir)[source]¶ Return a string to be run using os.system in order to connect via the transport to the remote directory.
Expected behaviors:
A new bash session is opened
A reasonable error message is produced if the folder does not exist
- Parameters
remotedir (str) – the full path of the remote directory
-
iglob
(pathname)[source]¶ Return an iterator which yields the paths matching a pathname pattern.
The pattern may contain simple shell-style wildcards a la fnmatch.
-
property
is_open
¶
-
isdir
(path)[source]¶ True if path is an existing directory.
- Parameters
path (str) – path to directory
- Returns
boolean
-
isfile
(path)[source]¶ Return True if path is an existing file.
- Parameters
path (str) – path to file
- Returns
boolean
-
listdir
(path='.', pattern=None)[source]¶ Return a list of the names of the entries in the given path. The list is in arbitrary order. It does not include the special entries ‘.’ and ‘..’ even if they are present in the directory.
- Parameters
path (str) – path to list (default to ‘.’)
pattern (str) – if used, listdir returns a list of files matching filters in Unix style. Unix only.
- Returns
a list of strings
-
listdir_withattributes
(path='.', pattern=None)[source]¶ Return a list of the names of the entries in the given path. The list is in arbitrary order. It does not include the special entries ‘.’ and ‘..’ even if they are present in the directory.
- Parameters
path (str) – path to list (default to ‘.’)
pattern (str) – if used, listdir returns a list of files matching filters in Unix style. Unix only.
- Returns
a list of dictionaries, one per entry. The schema of the dictionary is the following:
{ 'name': String, 'attributes': FileAttributeObject, 'isdir': Bool }
where ‘name’ is the file or folder directory, and any other information is metadata (if the file is a folder, a directory, …). ‘attributes’ behaves as the output of transport.get_attribute(); isdir is a boolean indicating if the object is a directory or not.
-
property
logger
¶ Return the internal logger. If you have set extra parameters using set_logger_extra(), a suitable LoggerAdapter instance is created, bringing with itself also the extras.
-
makedirs
(path, ignore_existing=False)[source]¶ Super-mkdir; create a leaf directory and all intermediate ones. Works like mkdir, except that any intermediate path segment (not just the rightmost) will be created if it does not exist.
- Parameters
path (str) – directory to create
ignore_existing (bool) – if set to true, it doesn’t give any error if the leaf directory does already exist
- Raises
OSError, if directory at path already exists
-
mkdir
(path, ignore_existing=False)[source]¶ Create a folder (directory) named path.
- Parameters
path (str) – name of the folder to create
ignore_existing (bool) – if True, does not give any error if the directory already exists
- Raises
OSError, if directory at path already exists
-
normalize
(path='.')[source]¶ Return the normalized path (on the server) of a given path. This can be used to quickly resolve symbolic links or determine what the server is considering to be the “current folder”.
- Parameters
path (str) – path to be normalized
- Raises
IOError – if the path can’t be resolved on the server
-
put
(localpath, remotepath, *args, **kwargs)[source]¶ Put a file or a directory from local src to remote dst. src must be an absolute path (dst not necessarily)) Redirects to putfile and puttree.
- Parameters
localpath (str) – absolute path to local source
remotepath (str) – path to remote destination
-
putfile
(localpath, remotepath, *args, **kwargs)[source]¶ Put a file from local src to remote dst. src must be an absolute path (dst not necessarily))
- Parameters
localpath (str) – absolute path to local file
remotepath (str) – path to remote file
-
puttree
(localpath, remotepath, *args, **kwargs)[source]¶ Put a folder recursively from local src to remote dst. src must be an absolute path (dst not necessarily))
- Parameters
localpath (str) – absolute path to local folder
remotepath (str) – path to remote folder
-
remove
(path)[source]¶ Remove the file at the given path. This only works on files; for removing folders (directories), use rmdir.
- Parameters
path (str) – path to file to remove
- Raises
IOError – if the path is a directory
-
rename
(oldpath, newpath)[source]¶ Rename a file or folder from oldpath to newpath.
- Parameters
oldpath (str) – existing name of the file or folder
newpath (str) – new name for the file or folder
- Raises
IOError – if oldpath/newpath is not found
ValueError – if oldpath/newpath is not a valid string
-
rmdir
(path)[source]¶ Remove the folder named path. This works only for empty folders. For recursive remove, use rmtree.
- Parameters
path (str) – absolute path to the folder to remove
-
rmtree
(path)[source]¶ Remove recursively the content at path
- Parameters
path (str) – absolute path to remove
-
set_logger_extra
(logger_extra)[source]¶ Pass the data that should be passed automatically to self.logger as ‘extra’ keyword. This is typically useful if you pass data obtained using get_dblogger_extra in aiida.backends.djsite.utils, to automatically log also to the DbLog table.
- Parameters
logger_extra – data that you want to pass as extra to the self.logger. To write to DbLog, it should be created by the aiida.backends.djsite.utils.get_dblogger_extra function. Pass None if you do not want to have extras passed.
-
Subpackages¶
Submodules¶
Common cli utilities for transport plugins.
-
aiida.transports.cli.
common_params
(command_func)[source]¶ Decorate a command function with common click parameters for all transport plugins.
-
aiida.transports.cli.
configure_computer_main
(computer, user, **kwargs)[source]¶ Configure a computer via the CLI.
-
aiida.transports.cli.
create_configure_cmd
(transport_type)[source]¶ Create verdi computer configure subcommand for a transport type.
-
aiida.transports.cli.
create_option
(name, spec)[source]¶ Create a click option from a name and partial specs as used in transport auth_options.
-
aiida.transports.cli.
interactive_default
(transport_type, key, also_noninteractive=False)[source]¶ Create a contextual_default value callback for an auth_param key.
-
aiida.transports.cli.
match_comp_transport
(ctx, param, computer, transport_type)[source]¶ Check the computer argument against the transport type.
-
aiida.transports.cli.
transport_option_default
(name, computer)[source]¶ Determine the default value for an auth_param key.
-
aiida.transports.cli.
transport_options
(transport_type)[source]¶ Decorate a command with all options for a computer configure subcommand for transport_type.
Transport interface.
-
class
aiida.transports.transport.
Transport
(*args, **kwargs)[source]¶ Bases:
object
Abstract class for a generic transport (ssh, local, …) Contains the set of minimal methods
-
_MAGIC_CHECK
= re.compile('[*?[]')¶
-
__abstractmethods__
= frozenset({})¶
-
__dict__
= mappingproxy({'__module__': 'aiida.transports.transport', '__doc__': '\n Abstract class for a generic transport (ssh, local, ...)\n Contains the set of minimal methods\n ', '_valid_auth_params': None, '_MAGIC_CHECK': re.compile('[*?[]'), '_valid_auth_options': [], '_common_auth_options': [('safe_interval', {'type': <class 'float'>, 'prompt': 'Connection cooldown time (s)', 'help': 'Minimum time interval in seconds between consecutive connection openings', 'callback': <function validate_positive_number>})], '__init__': <function Transport.__init__>, '__enter__': <function Transport.__enter__>, '__exit__': <function Transport.__exit__>, 'is_open': <property object>, 'open': <function Transport.open>, 'close': <function Transport.close>, '__repr__': <function Transport.__repr__>, '__str__': <function Transport.__str__>, 'set_logger_extra': <function Transport.set_logger_extra>, 'get_short_doc': <classmethod object>, 'get_valid_transports': <classmethod object>, 'get_valid_auth_params': <classmethod object>, 'auth_options': <aiida.common.lang.classproperty object>, '_get_safe_interval_suggestion_string': <classmethod object>, 'logger': <property object>, 'get_safe_open_interval': <function Transport.get_safe_open_interval>, 'chdir': <function Transport.chdir>, 'chmod': <function Transport.chmod>, 'chown': <function Transport.chown>, 'copy': <function Transport.copy>, 'copyfile': <function Transport.copyfile>, 'copytree': <function Transport.copytree>, 'copy_from_remote_to_remote': <function Transport.copy_from_remote_to_remote>, '_exec_command_internal': <function Transport._exec_command_internal>, 'exec_command_wait': <function Transport.exec_command_wait>, 'get': <function Transport.get>, 'getfile': <function Transport.getfile>, 'gettree': <function Transport.gettree>, 'getcwd': <function Transport.getcwd>, 'get_attribute': <function Transport.get_attribute>, 'get_mode': <function Transport.get_mode>, 'isdir': <function Transport.isdir>, 'isfile': <function Transport.isfile>, 'listdir': <function Transport.listdir>, 'listdir_withattributes': <function Transport.listdir_withattributes>, 'makedirs': <function Transport.makedirs>, 'mkdir': <function Transport.mkdir>, 'normalize': <function Transport.normalize>, 'put': <function Transport.put>, 'putfile': <function Transport.putfile>, 'puttree': <function Transport.puttree>, 'remove': <function Transport.remove>, 'rename': <function Transport.rename>, 'rmdir': <function Transport.rmdir>, 'rmtree': <function Transport.rmtree>, 'gotocomputer_command': <function Transport.gotocomputer_command>, 'symlink': <function Transport.symlink>, 'whoami': <function Transport.whoami>, 'path_exists': <function Transport.path_exists>, 'glob': <function Transport.glob>, 'iglob': <function Transport.iglob>, 'glob1': <function Transport.glob1>, 'glob0': <function Transport.glob0>, 'has_magic': <function Transport.has_magic>, '__dict__': <attribute '__dict__' of 'Transport' objects>, '__weakref__': <attribute '__weakref__' of 'Transport' objects>, '__abstractmethods__': frozenset(), '_abc_impl': <_abc_data object>})¶
-
__enter__
()[source]¶ For transports that require opening a connection, opens all required channels (used in ‘with’ statements).
This object can be used in nested with statements and the connection will only be opened once and closed when the final with scope finishes e.g.:
t = Transport() with t: # Connection is now open.. with t: # ..still open.. pass # ..still open.. # ...closed
-
__exit__
(type_, value, traceback)[source]¶ Closes connections, if needed (used in ‘with’ statements).
-
__module__
= 'aiida.transports.transport'¶
-
__weakref__
¶ list of weak references to the object (if defined)
-
_abc_impl
= <_abc_data object>¶
-
_common_auth_options
= [('safe_interval', {'type': <class 'float'>, 'prompt': 'Connection cooldown time (s)', 'help': 'Minimum time interval in seconds between consecutive connection openings', 'callback': <function validate_positive_number>})]¶
-
_exec_command_internal
(command, **kwargs)[source]¶ Execute the command on the shell, similarly to os.system.
Enforce the execution to be run from the cwd (as given by self.getcwd), if this is not None.
If possible, use the higher-level exec_command_wait function.
- Parameters
command (str) – execute the command given as a string
- Returns
stdin, stdout, stderr and the session, when this exists (can be None).
-
classmethod
_get_safe_interval_suggestion_string
(computer)[source]¶ Default time in seconds between consecutive checks.
Set to a non-zero value to be safe e.g. in the case of transports with a connection limit, to avoid overloading the server (and being banned). Should be overriden in plugins. This is anyway just a default, as the value can be changed by the user in the Computer properties, for instance. Currently both the local and the ssh transport override this value, so this is not used, but it will be the default for possible new plugins.
-
_valid_auth_options
= []¶
-
_valid_auth_params
= None¶
-
auth_options
= {'safe_interval': {'callback': <function validate_positive_number>, 'help': 'Minimum time interval in seconds between consecutive connection openings', 'prompt': 'Connection cooldown time (s)', 'type': <class 'float'>}}¶
-
chdir
(path)[source]¶ Change directory to ‘path’
- Parameters
path (str) – path to change working directory into.
- Raises
IOError, if the requested path does not exist
- Return type
str
-
chmod
(path, mode)[source]¶ Change permissions of a path.
- Parameters
path (str) – path to file
mode (int) – new permissions
-
chown
(path, uid, gid)[source]¶ Change the owner (uid) and group (gid) of a file. As with python’s os.chown function, you must pass both arguments, so if you only want to change one, use stat first to retrieve the current owner and group.
- Parameters
path (str) – path to the file to change the owner and group of
uid (int) – new owner’s uid
gid (int) – new group id
-
copy
(remotesource, remotedestination, dereference=False, recursive=True)[source]¶ Copy a file or a directory from remote source to remote destination (On the same remote machine)
- Parameters
remotesource (str) – path of the remote source directory / file
remotedestination (str) – path of the remote destination directory / file
dereference (bool) – if True copy the contents of any symlinks found, otherwise copy the symlinks themselves
recursive (bool) – if True copy directories recursively, otherwise only copy the specified file(s)
- Raises
IOError, if one of src or dst does not exist
-
copy_from_remote_to_remote
(transportdestination, remotesource, remotedestination, **kwargs)[source]¶ Copy files or folders from a remote computer to another remote computer.
- Parameters
transportdestination – transport to be used for the destination computer
remotesource (str) – path to the remote source directory / file
remotedestination (str) – path to the remote destination directory / file
kwargs – keyword parameters passed to the call to transportdestination.put, except for ‘dereference’ that is passed to self.get
Note
the keyword ‘dereference’ SHOULD be set to False for the final put (onto the destination), while it can be set to the value given in kwargs for the get from the source. In that way, a symbolic link would never be followed in the final copy to the remote destination. That way we could avoid getting unknown (potentially malicious) files into the destination computer. HOWEVER, since dereference=False is currently NOT supported by all plugins, we still force it to True for the final put.
Note
the supported keys in kwargs are callback, dereference, overwrite and ignore_nonexisting.
-
copyfile
(remotesource, remotedestination, dereference=False)[source]¶ Copy a file from remote source to remote destination (On the same remote machine)
- Parameters
remotesource (str) – path of the remote source directory / file
remotedestination (str) – path of the remote destination directory / file
dereference (bool) – if True copy the contents of any symlinks found, otherwise copy the symlinks themselves
- Raises
IOError – if one of src or dst does not exist
-
copytree
(remotesource, remotedestination, dereference=False)[source]¶ Copy a folder from remote source to remote destination (On the same remote machine)
- Parameters
remotesource (str) – path of the remote source directory / file
remotedestination (str) – path of the remote destination directory / file
dereference (bool) – if True copy the contents of any symlinks found, otherwise copy the symlinks themselves
- Raises
IOError – if one of src or dst does not exist
-
exec_command_wait
(command, **kwargs)[source]¶ Execute the command on the shell, waits for it to finish, and return the retcode, the stdout and the stderr.
Enforce the execution to be run from the pwd (as given by self.getcwd), if this is not None.
- Parameters
command (str) – execute the command given as a string
- Returns
a list: the retcode (int), stdout (str) and stderr (str).
-
get
(remotepath, localpath, *args, **kwargs)[source]¶ Retrieve a file or folder from remote source to local destination dst must be an absolute path (src not necessarily)
- Parameters
remotepath – (str) remote_folder_path
localpath – (str) local_folder_path
-
get_attribute
(path)[source]¶ Return an object FixedFieldsAttributeDict for file in a given path, as defined in aiida.common.extendeddicts Each attribute object consists in a dictionary with the following keys:
st_size: size of files, in bytes
st_uid: user id of owner
st_gid: group id of owner
st_mode: protection bits
st_atime: time of most recent access
st_mtime: time of most recent modification
- Parameters
path (str) – path to file
- Returns
object FixedFieldsAttributeDict
-
get_mode
(path)[source]¶ Return the portion of the file’s mode that can be set by chmod().
- Parameters
path (str) – path to file
- Returns
the portion of the file’s mode that can be set by chmod()
-
get_safe_open_interval
()[source]¶ Get an interval (in seconds) that suggests how long the user should wait between consecutive calls to open the transport. This can be used as a way to get the user to not swamp a limited number of connections, etc. However it is just advisory.
If returns 0, it is taken that there are no reasons to limit the frequency of open calls.
In the main class, it returns a default value (>0 for safety), set in the _DEFAULT_SAFE_OPEN_INTERVAL attribute of the class. Plugins should override it.
- Returns
The safe interval between calling open, in seconds
- Return type
float
-
classmethod
get_short_doc
()[source]¶ Return the first non-empty line of the class docstring, if available
-
getfile
(remotepath, localpath, *args, **kwargs)[source]¶ Retrieve a file from remote source to local destination dst must be an absolute path (src not necessarily)
- Parameters
remotepath (str) – remote_folder_path
localpath (str) – local_folder_path
-
gettree
(remotepath, localpath, *args, **kwargs)[source]¶ Retrieve a folder recursively from remote source to local destination dst must be an absolute path (src not necessarily)
- Parameters
remotepath (str) – remote_folder_path
localpath (str) – local_folder_path
-
glob
(pathname)[source]¶ Return a list of paths matching a pathname pattern.
The pattern may contain simple shell-style wildcards a la fnmatch.
-
glob0
(dirname, basename)[source]¶ Wrap basename i a list if it is empty or if dirname/basename is an existing path, else return empty list.
-
gotocomputer_command
(remotedir)[source]¶ Return a string to be run using os.system in order to connect via the transport to the remote directory.
Expected behaviors:
A new bash session is opened
A reasonable error message is produced if the folder does not exist
- Parameters
remotedir (str) – the full path of the remote directory
-
iglob
(pathname)[source]¶ Return an iterator which yields the paths matching a pathname pattern.
The pattern may contain simple shell-style wildcards a la fnmatch.
-
property
is_open
¶
-
isdir
(path)[source]¶ True if path is an existing directory.
- Parameters
path (str) – path to directory
- Returns
boolean
-
isfile
(path)[source]¶ Return True if path is an existing file.
- Parameters
path (str) – path to file
- Returns
boolean
-
listdir
(path='.', pattern=None)[source]¶ Return a list of the names of the entries in the given path. The list is in arbitrary order. It does not include the special entries ‘.’ and ‘..’ even if they are present in the directory.
- Parameters
path (str) – path to list (default to ‘.’)
pattern (str) – if used, listdir returns a list of files matching filters in Unix style. Unix only.
- Returns
a list of strings
-
listdir_withattributes
(path='.', pattern=None)[source]¶ Return a list of the names of the entries in the given path. The list is in arbitrary order. It does not include the special entries ‘.’ and ‘..’ even if they are present in the directory.
- Parameters
path (str) – path to list (default to ‘.’)
pattern (str) – if used, listdir returns a list of files matching filters in Unix style. Unix only.
- Returns
a list of dictionaries, one per entry. The schema of the dictionary is the following:
{ 'name': String, 'attributes': FileAttributeObject, 'isdir': Bool }
where ‘name’ is the file or folder directory, and any other information is metadata (if the file is a folder, a directory, …). ‘attributes’ behaves as the output of transport.get_attribute(); isdir is a boolean indicating if the object is a directory or not.
-
property
logger
¶ Return the internal logger. If you have set extra parameters using set_logger_extra(), a suitable LoggerAdapter instance is created, bringing with itself also the extras.
-
makedirs
(path, ignore_existing=False)[source]¶ Super-mkdir; create a leaf directory and all intermediate ones. Works like mkdir, except that any intermediate path segment (not just the rightmost) will be created if it does not exist.
- Parameters
path (str) – directory to create
ignore_existing (bool) – if set to true, it doesn’t give any error if the leaf directory does already exist
- Raises
OSError, if directory at path already exists
-
mkdir
(path, ignore_existing=False)[source]¶ Create a folder (directory) named path.
- Parameters
path (str) – name of the folder to create
ignore_existing (bool) – if True, does not give any error if the directory already exists
- Raises
OSError, if directory at path already exists
-
normalize
(path='.')[source]¶ Return the normalized path (on the server) of a given path. This can be used to quickly resolve symbolic links or determine what the server is considering to be the “current folder”.
- Parameters
path (str) – path to be normalized
- Raises
IOError – if the path can’t be resolved on the server
-
put
(localpath, remotepath, *args, **kwargs)[source]¶ Put a file or a directory from local src to remote dst. src must be an absolute path (dst not necessarily)) Redirects to putfile and puttree.
- Parameters
localpath (str) – absolute path to local source
remotepath (str) – path to remote destination
-
putfile
(localpath, remotepath, *args, **kwargs)[source]¶ Put a file from local src to remote dst. src must be an absolute path (dst not necessarily))
- Parameters
localpath (str) – absolute path to local file
remotepath (str) – path to remote file
-
puttree
(localpath, remotepath, *args, **kwargs)[source]¶ Put a folder recursively from local src to remote dst. src must be an absolute path (dst not necessarily))
- Parameters
localpath (str) – absolute path to local folder
remotepath (str) – path to remote folder
-
remove
(path)[source]¶ Remove the file at the given path. This only works on files; for removing folders (directories), use rmdir.
- Parameters
path (str) – path to file to remove
- Raises
IOError – if the path is a directory
-
rename
(oldpath, newpath)[source]¶ Rename a file or folder from oldpath to newpath.
- Parameters
oldpath (str) – existing name of the file or folder
newpath (str) – new name for the file or folder
- Raises
IOError – if oldpath/newpath is not found
ValueError – if oldpath/newpath is not a valid string
-
rmdir
(path)[source]¶ Remove the folder named path. This works only for empty folders. For recursive remove, use rmtree.
- Parameters
path (str) – absolute path to the folder to remove
-
rmtree
(path)[source]¶ Remove recursively the content at path
- Parameters
path (str) – absolute path to remove
-
set_logger_extra
(logger_extra)[source]¶ Pass the data that should be passed automatically to self.logger as ‘extra’ keyword. This is typically useful if you pass data obtained using get_dblogger_extra in aiida.backends.djsite.utils, to automatically log also to the DbLog table.
- Parameters
logger_extra – data that you want to pass as extra to the self.logger. To write to DbLog, it should be created by the aiida.backends.djsite.utils.get_dblogger_extra function. Pass None if you do not want to have extras passed.
-
General utilities for Transport classes.
-
class
aiida.transports.util.
FileAttribute
(init=None)[source]¶ Bases:
aiida.common.extendeddicts.FixedFieldsAttributeDict
A class, resembling a dictionary, to describe the attributes of a file, that is returned by get_attribute(). Possible keys: st_size, st_uid, st_gid, st_mode, st_atime, st_mtime
-
__module__
= 'aiida.transports.util'¶
-
_valid_fields
= ('st_size', 'st_uid', 'st_gid', 'st_mode', 'st_atime', 'st_mtime')¶
-
-
class
aiida.transports.util.
_DetachedProxyCommand
(command_line)[source]¶ Bases:
paramiko.proxy.ProxyCommand
Modifies paramiko’s ProxyCommand by launching the process in a separate process group.
-
__init__
(command_line)[source]¶ Create a new CommandProxy instance. The instance created by this class can be passed as an argument to the .Transport class.
- Parameters
command_line (str) – the command that should be executed and used as the proxy.
-
__module__
= 'aiida.transports.util'¶
-
-
aiida.transports.util.
copy_from_remote_to_remote
(transportsource, transportdestination, remotesource, remotedestination, **kwargs)[source]¶ Copy files or folders from a remote computer to another remote computer.
- Parameters
transportsource – transport to be used for the source computer
transportdestination – transport to be used for the destination computer
remotesource (str) – path to the remote source directory / file
remotedestination (str) – path to the remote destination directory / file
kwargs – keyword parameters passed to the final put, except for ‘dereference’ that is passed to the initial get
Note
it uses the method transportsource.copy_from_remote_to_remote