aiida.tools.importexport.migration package¶
Migration export files from old export versions to the newest, used by verdi export migrate command.
-
aiida.tools.importexport.migration.
migrate_recursively
(metadata, data, folder)[source]¶ Recursive migration of export files from v0.1 to newest version, See specific migration functions for detailed descriptions.
- Parameters
metadata – the content of an export archive metadata.json file
data – the content of an export archive data.json file
folder – SandboxFolder in which the archive has been unpacked (workdir)
-
aiida.tools.importexport.migration.
verify_metadata_version
(metadata, version=None)[source]¶ Utility function to verify that the metadata has the correct version number. If no version number is passed, it will just extract the version number and return it.
- Parameters
metadata – the content of an export archive metadata.json file
version – string version number that the metadata is expected to have
Submodules¶
Utility functions for migration of export-files.
-
aiida.tools.importexport.migration.utils.
remove_fields
(metadata, data, entities, fields)[source]¶ Remove fields under entities from data.json and metadata.json
- Parameters
metadata – the content of an export archive metadata.json file
data – the content of an export archive data.json file
entities – list of ORM entities
fields – list of fields to be removed from the export archive files
-
aiida.tools.importexport.migration.utils.
update_metadata
(metadata, version)[source]¶ Update the metadata with a new version number and a notification of the conversion that was executed
- Parameters
metadata – the content of an export archive metadata.json file
version – string version number that the updated metadata should get
-
aiida.tools.importexport.migration.utils.
verify_metadata_version
(metadata, version=None)[source]¶ Utility function to verify that the metadata has the correct version number. If no version number is passed, it will just extract the version number and return it.
- Parameters
metadata – the content of an export archive metadata.json file
version – string version number that the metadata is expected to have
Migration from v0.1 to v0.2, used by verdi export migrate command.
-
aiida.tools.importexport.migration.v01_to_v02.
migrate_v1_to_v2
(metadata, data, *args)[source]¶ Migration of export files from v0.1 to v0.2, which means generalizing the field names with respect to the database backend
- Parameters
metadata – the content of an export archive metadata.json file
data – the content of an export archive data.json file
Migration from v0.2 to v0.3, used by verdi export migrate command.
-
aiida.tools.importexport.migration.v02_to_v03.
migrate_v2_to_v3
(metadata, data, *args)[source]¶ Migration of export files from v0.2 to v0.3, which means adding the link types to the link entries and making the entity key names backend agnostic by effectively removing the prefix ‘aiida.backends.djsite.db.models’
- Parameters
data – the content of an export archive data.json file
metadata – the content of an export archive metadata.json file
Migration from v0.3 to v0.4, used by verdi export migrate command.
The migration steps are named similarly to the database migrations for Django and SQLAlchemy. In the description of each migration, a revision number is given, which refers to the Django migrations. The individual Django database migrations may be found at:
aiida.backends.djsite.db.migrations.00XX_<migration-name>.py
Where XX are the numbers in the migrations’ documentation: REV. 1.0.XX And migration-name is the name of the particular migration. The individual SQLAlchemy database migrations may be found at:
aiida.backends.sqlalchemy.migrations.versions.<id>_<migration-name>.py
Where id is a SQLA id and migration-name is the name of the particular migration.
-
aiida.tools.importexport.migration.v03_to_v04.
add_extras
(data)[source]¶ Update data.json with the new Extras Since Extras were not available previously and usually only include hashes, the Node ids will be added, but included as empty dicts
-
aiida.tools.importexport.migration.v03_to_v04.
migrate_v3_to_v4
(metadata, data, folder, *args)[source]¶ Migration of export files from v0.3 to v0.4
Note concerning migration 0032 - REV. 1.0.32: Remove legacy workflow tables: DbWorkflow, DbWorkflowData, DbWorkflowStep These were (according to Antimo Marrazzo) never exported.
-
aiida.tools.importexport.migration.v03_to_v04.
migration_add_node_uuid_unique_constraint
(data)[source]¶ Apply migration: 0014 - REV. 1.0.14, 0018 - REV. 1.0.18 Check that no entries with the same uuid are present in the export file if yes - stop the import process
-
aiida.tools.importexport.migration.v03_to_v04.
migration_base_data_plugin_type_string
(data)[source]¶ Apply migration: 0009 - REV. 1.0.9 DbNode.type content changes: ‘data.base.Bool.’ -> ‘data.bool.Bool.’ ‘data.base.Float.’ -> ‘data.float.Float.’ ‘data.base.Int.’ -> ‘data.int.Int.’ ‘data.base.Str.’ -> ‘data.str.Str.’ ‘data.base.List.’ -> ‘data.list.List.’
-
aiida.tools.importexport.migration.v03_to_v04.
migration_calc_job_option_attribute_keys
(data)[source]¶ Apply migrations: 0023 - REV. 1.0.23 custom_environment_variables -> environment_variables jobresource_params -> resources _process_label -> process_label parser -> parser_name
-
aiida.tools.importexport.migration.v03_to_v04.
migration_code_sub_class_of_data
(data)[source]¶ Apply migrations: 0016 - REV. 1.0.16 The Code class used to be just a sub class of Node, but was changed to act like a Data node. code.Code. -> data.code.Code.
-
aiida.tools.importexport.migration.v03_to_v04.
migration_dbgroup_name_to_label_type_to_type_string
(metadata, data)[source]¶ Apply migrations: 0021 - REV. 1.0.21 Rename dbgroup fields: name -> label type -> type_string
-
aiida.tools.importexport.migration.v03_to_v04.
migration_dbgroup_type_string_change_content
(data)[source]¶ Apply migrations: 0022 - REV. 1.0.22 Change type_string according to the following rule: ‘’ -> ‘user’ ‘data.upf.family’ -> ‘data.upf’ ‘aiida.import’ -> ‘auto.import’ ‘autogroup.run’ -> ‘auto.run’
-
aiida.tools.importexport.migration.v03_to_v04.
migration_dbnode_type_to_dbnode_node_type
(metadata, data)[source]¶ Apply migration: 0030 - REV. 1.0.30 Renaming DbNode.type to DbNode.node_type
-
aiida.tools.importexport.migration.v03_to_v04.
migration_migrate_builtin_calculations
(data)[source]¶ Apply migrations: 0019 - REV. 1.0.19 Remove ‘simpleplugin’ from ArithmeticAddCalculation and TemplatereplacerCalculation type
ATTENTION:
The ‘process_type’ column did not exist before migration 0010, consequently, it could not be present in any export archive of the currently existing stable releases (0.12.*). Here, however, the migration acts on the content of the ‘process_type’ column, which could only be introduced in alpha releases of AiiDA 1.0. Assuming that ‘add’ and ‘templateplacer’ calculations are expected to have both ‘type’ and ‘process_type’ columns, they will be added based solely on the ‘type’ column content (unlike the way it is done in the DB migration, where the ‘type_string’ content was also checked).
-
aiida.tools.importexport.migration.v03_to_v04.
migration_move_data_within_node_module
(data)[source]¶ Apply migrations: 0025 - REV. 1.0.25 The type string for Data nodes changed from data.* to node.data.*.
-
aiida.tools.importexport.migration.v03_to_v04.
migration_process_type
(metadata, data)[source]¶ Apply migrations: 0010 - REV. 1.0.10 Add DbNode.process_type column
-
aiida.tools.importexport.migration.v03_to_v04.
migration_provenance_redesign
(data)[source]¶ Apply migration: 0020 - REV. 1.0.20 Provenance redesign
-
aiida.tools.importexport.migration.v03_to_v04.
migration_remove_dbcomputer_enabled
(metadata, data)[source]¶ Apply migration: 0031 - REV. 1.0.31 Remove DbComputer.enabled
-
aiida.tools.importexport.migration.v03_to_v04.
migration_remove_node_prefix
(data)[source]¶ Apply migrations: 0028 - REV. 1.0.28 Change node type strings: ‘node.data.’ -> ‘data.’ ‘node.process.’ -> ‘process.’
-
aiida.tools.importexport.migration.v03_to_v04.
migration_rename_parameter_data_to_dict
(data)[source]¶ Apply migration: 0029 - REV. 1.0.29 Update ParameterData to Dict
-
aiida.tools.importexport.migration.v03_to_v04.
migration_replace_text_field_with_json_field
(data)[source]¶ Apply migration 0033 - REV. 1.0.33 Store dict-values as JSON serializable dicts instead of strings NB! Specific for Django backend
-
aiida.tools.importexport.migration.v03_to_v04.
migration_trajectory_symbols_to_attribute
(data, folder)[source]¶ Apply migrations: 0026 - REV. 1.0.26 and 0027 - REV. 1.0.27 Create the symbols attribute from the repository array for all TrajectoryData nodes.
Migration from v0.4 to v0.5, used by verdi export migrate command.
The migration steps are named similarly to the database migrations for Django and SQLAlchemy. In the description of each migration, a revision number is given, which refers to the Django migrations. The individual Django database migrations may be found at:
aiida.backends.djsite.db.migrations.00XX_<migration-name>.py
Where XX are the numbers in the migrations’ documentation: REV. 1.0.XX And migration-name is the name of the particular migration. The individual SQLAlchemy database migrations may be found at:
aiida.backends.sqlalchemy.migrations.versions.<id>_<migration-name>.py
Where id is a SQLA id and migration-name is the name of the particular migration.
-
aiida.tools.importexport.migration.v04_to_v05.
migrate_v4_to_v5
(metadata, data, *args)[source]¶ Migration of export files from v0.4 to v0.5
This is from migration 0034 (drop_node_columns_nodeversion_public) and onwards
-
aiida.tools.importexport.migration.v04_to_v05.
migration_drop_computer_transport_params
(metadata, data)[source]¶ Apply migration 0036 - REV. 1.0.36 Drop the column transport_params from the Computer model
-
aiida.tools.importexport.migration.v04_to_v05.
migration_drop_node_columns_nodeversion_public
(metadata, data)[source]¶ Apply migration 0034 - REV. 1.0.34 Drop the columns nodeversion and public from the Node model
Migration from v0.5 to v0.6, used by verdi export migrate command.
The migration steps are named similarly to the database migrations for Django and SQLAlchemy. In the description of each migration, a revision number is given, which refers to the Django migrations. The individual Django database migrations may be found at:
aiida.backends.djsite.db.migrations.00XX_<migration-name>.py
Where XX are the numbers in the migrations’ documentation: REV. 1.0.XX And migration-name is the name of the particular migration. The individual SQLAlchemy database migrations may be found at:
aiida.backends.sqlalchemy.migrations.versions.<id>_<migration-name>.py
Where id is a SQLA id and migration-name is the name of the particular migration.
-
aiida.tools.importexport.migration.v05_to_v06.
migrate_deserialized_datetime
(data, conversion)[source]¶ Deserialize datetime strings from export archives, meaning to reattach the UTC timezone information.
-
aiida.tools.importexport.migration.v05_to_v06.
migrate_v5_to_v6
(metadata, data, *args)[source]¶ Migration of export files from v0.5 to v0.6
-
aiida.tools.importexport.migration.v05_to_v06.
migration_migrate_legacy_job_calculation_data
(data)[source]¶ Apply migration 0038 - REV. 1.0.38
Migrates legacy JobCalculation data to the new process system. Essentially old JobCalculation nodes, which have already been migrated to CalcJobNodes, are missing important attributes process_state, exit_status and process_status. These are inferred from the old state attribute, which is then discarded as its values have been deprecated.
-
aiida.tools.importexport.migration.v05_to_v06.
migration_serialize_datetime_objects
(data)[source]¶ Apply migration 0037 - REV. 1.0.37
Migrates the node attributes and extras from the EAV schema to JSONB columns. Since JSON does not support datetime objects, and the EAV did, existing datetime objects have to be serialized to strings. Just like the database migration they were serialized to the standard ISO format, except that they were first converted to UTC timezone and then the stored without a timezone reference. Since existing datetimes in the attributes and extras in the database were timezone aware and have been migrated to an ISO format string including the timezone information we should now add the same timezone information to datetime attributes and extras in existing export archives. All that one needs to do for this is to append the +00:00 suffix, which signifies the UTC timezone.
Since the datetime objects were the only types being serialized in the attributes and extras, after the reinstating of the timeonze information, there is no longer a need for the de/serialization dictionaries for each node, stored in node_attributes_conversion and node_extras_conversion, respectively. They are no longer added to new archives and so they can and should be removed from existing archives, reducing the size enormously.
Migration from v0.6 to v0.7, used by verdi export migrate command.
The migration steps are named similarly to the database migrations for Django and SQLAlchemy. In the description of each migration, a revision number is given, which refers to the Django migrations. The individual Django database migrations may be found at:
aiida.backends.djsite.db.migrations.00XX_<migration-name>.py
Where XX are the numbers in the migrations’ documentation: REV. 1.0.XX And migration-name is the name of the particular migration. The individual SQLAlchemy database migrations may be found at:
aiida.backends.sqlalchemy.migrations.versions.<id>_<migration-name>.py
Where id is a SQLA id and migration-name is the name of the particular migration.
-
aiida.tools.importexport.migration.v06_to_v07.
migrate_v6_to_v7
(metadata, data, *args)[source]¶ Migration of export files from v0.6 to v0.7
-
aiida.tools.importexport.migration.v06_to_v07.
migration_data_migration_legacy_process_attributes
(data)[source]¶ Apply migration 0040 - REV. 1.0.40 Data migration for some legacy process attributes.
Attribute keys that are renamed:
_sealed -> sealed
Attribute keys that are removed entirely:
_finished
_failed
_aborted
_do_abort
Finally, after these first migrations, any remaining process nodes are screened for the existence of the process_state attribute. If they have it, it is checked whether the state is active or not, if not, the sealed attribute is created and set to True.
- Raises
CorruptArchive – if a Node, found to have attributes, cannot be found in the list of exported entities.
CorruptArchive – if the ‘sealed’ attribute does not exist and the ProcessNode is in an active state, i.e. process_state is one of (‘created’, ‘running’, ‘waiting’). A log-file, listing all illegal ProcessNodes, will be produced in the current directory.
-
aiida.tools.importexport.migration.v06_to_v07.
remove_attribute_link_metadata
(metadata)[source]¶ Remove Attribute and Link from metadata.json This is not a database migration, but purely to do with the import/export schema.
The “entities” ATTRIBUTE_ENTITY_NAME and LINK_ENTITY_NAME were introduced in v0.3 or v0.4 of the import/export schema. However, they were never used - or the usage has been reverted. They will be removed from metadata.json, slightly simplifying the import functions as well.