aiida.tools.importexport.migration package

Migration export files from old export versions to the newest, used by verdi export migrate command.

aiida.tools.importexport.migration.migrate_recursively(metadata, data, folder)[source]

Recursive migration of export files from v0.1 to newest version, See specific migration functions for detailed descriptions.

Parameters:
  • metadata – the content of an export archive metadata.json file
  • data – the content of an export archive data.json file
  • folder – SandboxFolder in which the archive has been unpacked (workdir)
aiida.tools.importexport.migration.verify_metadata_version(metadata, version=None)[source]

Utility function to verify that the metadata has the correct version number. If no version number is passed, it will just extract the version number and return it.

Parameters:
  • metadata – the content of an export archive metadata.json file
  • version – string version number that the metadata is expected to have

Submodules

Utility functions for migration of export-files.

aiida.tools.importexport.migration.utils.remove_fields(metadata, data, entities, fields)[source]

Remove fields under entities from data.json and metadata.json

Parameters:
  • metadata – the content of an export archive metadata.json file
  • data – the content of an export archive data.json file
  • entities – list of ORM entities
  • fields – list of fields to be removed from the export archive files
aiida.tools.importexport.migration.utils.update_metadata(metadata, version)[source]

Update the metadata with a new version number and a notification of the conversion that was executed

Parameters:
  • metadata – the content of an export archive metadata.json file
  • version – string version number that the updated metadata should get
aiida.tools.importexport.migration.utils.verify_metadata_version(metadata, version=None)[source]

Utility function to verify that the metadata has the correct version number. If no version number is passed, it will just extract the version number and return it.

Parameters:
  • metadata – the content of an export archive metadata.json file
  • version – string version number that the metadata is expected to have

Migration from v0.1 to v0.2, used by verdi export migrate command.

aiida.tools.importexport.migration.v01_to_v02.migrate_v1_to_v2(metadata, data, *args)[source]

Migration of export files from v0.1 to v0.2, which means generalizing the field names with respect to the database backend

Parameters:
  • metadata – the content of an export archive metadata.json file
  • data – the content of an export archive data.json file

Migration from v0.2 to v0.3, used by verdi export migrate command.

aiida.tools.importexport.migration.v02_to_v03.migrate_v2_to_v3(metadata, data, *args)[source]

Migration of export files from v0.2 to v0.3, which means adding the link types to the link entries and making the entity key names backend agnostic by effectively removing the prefix ‘aiida.backends.djsite.db.models’

Parameters:
  • data – the content of an export archive data.json file
  • metadata – the content of an export archive metadata.json file

Migration from v0.3 to v0.4, used by verdi export migrate command.

The migration steps are named similarly to the database migrations for Django and SQLAlchemy. In the description of each migration, a revision number is given, which refers to the Django migrations. The individual Django database migrations may be found at:

aiida.backends.djsite.db.migrations.00XX_<migration-name>.py

Where XX are the numbers in the migrations’ documentation: REV. 1.0.XX And migration-name is the name of the particular migration. The individual SQLAlchemy database migrations may be found at:

aiida.backends.sqlalchemy.migrations.versions.<id>_<migration-name>.py

Where id is a SQLA id and migration-name is the name of the particular migration.

aiida.tools.importexport.migration.v03_to_v04.add_extras(data)[source]

Update data.json with the new Extras Since Extras were not available previously and usually only include hashes, the Node ids will be added, but included as empty dicts

aiida.tools.importexport.migration.v03_to_v04.migrate_v3_to_v4(metadata, data, folder, *args)[source]

Migration of export files from v0.3 to v0.4

Note concerning migration 0032 - REV. 1.0.32: Remove legacy workflow tables: DbWorkflow, DbWorkflowData, DbWorkflowStep These were (according to Antimo Marrazzo) never exported.

aiida.tools.importexport.migration.v03_to_v04.migration_add_node_uuid_unique_constraint(data)[source]

Apply migration: 0014 - REV. 1.0.14, 0018 - REV. 1.0.18 Check that no entries with the same uuid are present in the export file if yes - stop the import process

aiida.tools.importexport.migration.v03_to_v04.migration_base_data_plugin_type_string(data)[source]

Apply migration: 0009 - REV. 1.0.9 DbNode.type content changes: ‘data.base.Bool.’ -> ‘data.bool.Bool.’ ‘data.base.Float.’ -> ‘data.float.Float.’ ‘data.base.Int.’ -> ‘data.int.Int.’ ‘data.base.Str.’ -> ‘data.str.Str.’ ‘data.base.List.’ -> ‘data.list.List.’

aiida.tools.importexport.migration.v03_to_v04.migration_calc_job_option_attribute_keys(data)[source]

Apply migrations: 0023 - REV. 1.0.23 custom_environment_variables -> environment_variables jobresource_params -> resources _process_label -> process_label parser -> parser_name

aiida.tools.importexport.migration.v03_to_v04.migration_code_sub_class_of_data(data)[source]

Apply migrations: 0016 - REV. 1.0.16 The Code class used to be just a sub class of Node, but was changed to act like a Data node. code.Code. -> data.code.Code.

aiida.tools.importexport.migration.v03_to_v04.migration_dbgroup_name_to_label_type_to_type_string(metadata, data)[source]

Apply migrations: 0021 - REV. 1.0.21 Rename dbgroup fields: name -> label type -> type_string

aiida.tools.importexport.migration.v03_to_v04.migration_dbgroup_type_string_change_content(data)[source]

Apply migrations: 0022 - REV. 1.0.22 Change type_string according to the following rule: ‘’ -> ‘user’ ‘data.upf.family’ -> ‘data.upf’ ‘aiida.import’ -> ‘auto.import’ ‘autogroup.run’ -> ‘auto.run’

aiida.tools.importexport.migration.v03_to_v04.migration_dbnode_type_to_dbnode_node_type(metadata, data)[source]

Apply migration: 0030 - REV. 1.0.30 Renaming DbNode.type to DbNode.node_type

aiida.tools.importexport.migration.v03_to_v04.migration_migrate_builtin_calculations(data)[source]

Apply migrations: 0019 - REV. 1.0.19 Remove ‘simpleplugin’ from ArithmeticAddCalculation and TemplatereplacerCalculation type

ATTENTION:

The ‘process_type’ column did not exist before migration 0010, consequently, it could not be present in any export archive of the currently existing stable releases (0.12.*). Here, however, the migration acts on the content of the ‘process_type’ column, which could only be introduced in alpha releases of AiiDA 1.0. Assuming that ‘add’ and ‘templateplacer’ calculations are expected to have both ‘type’ and ‘process_type’ columns, they will be added based solely on the ‘type’ column content (unlike the way it is done in the DB migration, where the ‘type_string’ content was also checked).

aiida.tools.importexport.migration.v03_to_v04.migration_move_data_within_node_module(data)[source]

Apply migrations: 0025 - REV. 1.0.25 The type string for Data nodes changed from data.* to node.data.*.

aiida.tools.importexport.migration.v03_to_v04.migration_process_type(metadata, data)[source]

Apply migrations: 0010 - REV. 1.0.10 Add DbNode.process_type column

aiida.tools.importexport.migration.v03_to_v04.migration_provenance_redesign(data)[source]

Apply migration: 0020 - REV. 1.0.20 Provenance redesign

aiida.tools.importexport.migration.v03_to_v04.migration_remove_dbcomputer_enabled(metadata, data)[source]

Apply migration: 0031 - REV. 1.0.31 Remove DbComputer.enabled

aiida.tools.importexport.migration.v03_to_v04.migration_remove_node_prefix(data)[source]

Apply migrations: 0028 - REV. 1.0.28 Change node type strings: ‘node.data.’ -> ‘data.’ ‘node.process.’ -> ‘process.’

aiida.tools.importexport.migration.v03_to_v04.migration_rename_parameter_data_to_dict(data)[source]

Apply migration: 0029 - REV. 1.0.29 Update ParameterData to Dict

aiida.tools.importexport.migration.v03_to_v04.migration_replace_text_field_with_json_field(data)[source]

Apply migration 0033 - REV. 1.0.33 Store dict-values as JSON serializable dicts instead of strings NB! Specific for Django backend

aiida.tools.importexport.migration.v03_to_v04.migration_trajectory_symbols_to_attribute(data, folder)[source]

Apply migrations: 0026 - REV. 1.0.26 and 0027 - REV. 1.0.27 Create the symbols attribute from the repository array for all TrajectoryData nodes.

Migration from v0.4 to v0.5, used by verdi export migrate command.

The migration steps are named similarly to the database migrations for Django and SQLAlchemy. In the description of each migration, a revision number is given, which refers to the Django migrations. The individual Django database migrations may be found at:

aiida.backends.djsite.db.migrations.00XX_<migration-name>.py

Where XX are the numbers in the migrations’ documentation: REV. 1.0.XX And migration-name is the name of the particular migration. The individual SQLAlchemy database migrations may be found at:

aiida.backends.sqlalchemy.migrations.versions.<id>_<migration-name>.py

Where id is a SQLA id and migration-name is the name of the particular migration.

aiida.tools.importexport.migration.v04_to_v05.migrate_v4_to_v5(metadata, data, *args)[source]

Migration of export files from v0.4 to v0.5

This is from migration 0034 (drop_node_columns_nodeversion_public) and onwards

aiida.tools.importexport.migration.v04_to_v05.migration_drop_computer_transport_params(metadata, data)[source]

Apply migration 0036 - REV. 1.0.36 Drop the column transport_params from the Computer model

aiida.tools.importexport.migration.v04_to_v05.migration_drop_node_columns_nodeversion_public(metadata, data)[source]

Apply migration 0034 - REV. 1.0.34 Drop the columns nodeversion and public from the Node model

Migration from v0.5 to v0.6, used by verdi export migrate command.

The migration steps are named similarly to the database migrations for Django and SQLAlchemy. In the description of each migration, a revision number is given, which refers to the Django migrations. The individual Django database migrations may be found at:

aiida.backends.djsite.db.migrations.00XX_<migration-name>.py

Where XX are the numbers in the migrations’ documentation: REV. 1.0.XX And migration-name is the name of the particular migration. The individual SQLAlchemy database migrations may be found at:

aiida.backends.sqlalchemy.migrations.versions.<id>_<migration-name>.py

Where id is a SQLA id and migration-name is the name of the particular migration.

aiida.tools.importexport.migration.v05_to_v06.migrate_deserialized_datetime(data, conversion)[source]

Deserialize datetime strings from export archives, meaning to reattach the UTC timezone information.

aiida.tools.importexport.migration.v05_to_v06.migrate_v5_to_v6(metadata, data, *args)[source]

Migration of export files from v0.5 to v0.6

aiida.tools.importexport.migration.v05_to_v06.migration_migrate_legacy_job_calculation_data(data)[source]

Apply migration 0038 - REV. 1.0.38

Migrates legacy JobCalculation data to the new process system. Essentially old JobCalculation nodes, which have already been migrated to CalcJobNodes, are missing important attributes process_state, exit_status and process_status. These are inferred from the old state attribute, which is then discarded as its values have been deprecated.

aiida.tools.importexport.migration.v05_to_v06.migration_serialize_datetime_objects(data)[source]

Apply migration 0037 - REV. 1.0.37

Migrates the node attributes and extras from the EAV schema to JSONB columns. Since JSON does not support datetime objects, and the EAV did, existing datetime objects have to be serialized to strings. Just like the database migration they were serialized to the standard ISO format, except that they were first converted to UTC timezone and then the stored without a timezone reference. Since existing datetimes in the attributes and extras in the database were timezone aware and have been migrated to an ISO format string including the timezone information we should now add the same timezone information to datetime attributes and extras in existing export archives. All that one needs to do for this is to append the +00:00 suffix, which signifies the UTC timezone.

Since the datetime objects were the only types being serialized in the attributes and extras, after the reinstating of the timeonze information, there is no longer a need for the de/serialization dictionaries for each node, stored in node_attributes_conversion and node_extras_conversion, respectively. They are no longer added to new archives and so they can and should be removed from existing archives, reducing the size enormously.

Migration from v0.6 to v0.7, used by verdi export migrate command.

The migration steps are named similarly to the database migrations for Django and SQLAlchemy. In the description of each migration, a revision number is given, which refers to the Django migrations. The individual Django database migrations may be found at:

aiida.backends.djsite.db.migrations.00XX_<migration-name>.py

Where XX are the numbers in the migrations’ documentation: REV. 1.0.XX And migration-name is the name of the particular migration. The individual SQLAlchemy database migrations may be found at:

aiida.backends.sqlalchemy.migrations.versions.<id>_<migration-name>.py

Where id is a SQLA id and migration-name is the name of the particular migration.

aiida.tools.importexport.migration.v06_to_v07.migrate_v6_to_v7(metadata, data, *args)[source]

Migration of export files from v0.6 to v0.7

aiida.tools.importexport.migration.v06_to_v07.migration_data_migration_legacy_process_attributes(data)[source]

Apply migration 0040 - REV. 1.0.40 Data migration for some legacy process attributes.

Attribute keys that are renamed:

  • _sealed -> sealed

Attribute keys that are removed entirely:

  • _finished
  • _failed
  • _aborted
  • _do_abort

Finally, after these first migrations, any remaining process nodes are screened for the existence of the process_state attribute. If they have it, it is checked whether the state is active or not, if not, the sealed attribute is created and set to True.

Raises:
  • CorruptArchive – if a Node, found to have attributes, cannot be found in the list of exported entities.
  • CorruptArchive – if the ‘sealed’ attribute does not exist and the ProcessNode is in an active state, i.e. process_state is one of (‘created’, ‘running’, ‘waiting’). A log-file, listing all illegal ProcessNodes, will be produced in the current directory.

Remove Attribute and Link from metadata.json This is not a database migration, but purely to do with the import/export schema.

The “entities” ATTRIBUTE_ENTITY_NAME and LINK_ENTITY_NAME were introduced in v0.3 or v0.4 of the import/export schema. However, they were never used - or the usage has been reverted. They will be removed from metadata.json, slightly simplifying the import functions as well.