| Both sides previous revision Previous revision | |
| user_guide:extend:upgrade_rules [2019/11/13 01:21] – gawrilow | user_guide:extend:upgrade_rules [2019/11/13 15:35] (current) – gawrilow |
|---|
| * merging or splitting big object types, changing the inheritance hierarchy | * merging or splitting big object types, changing the inheritance hierarchy |
| * renaming, merging or splitting applications | * renaming, merging or splitting applications |
| Whatever the reasons for a data model change, we have to provide an automatic data transition for users upgrading their installation which must prevent any data loss. The mechanism is based on a strict versioning scheme and so called upgrade rules. The rest of this page describes the steps a developer must execute every time they introduce an incompatible data model change. The procedure slightly differs depending on whether it deals with the first data model change after the most recent polymake release, or with a subsequent change between two releases. In the following examples we will assume the most recently released version to be ''4.1''. | Whatever the reasons for a data model change, we have to ensure a lossless automatic data transition for users upgrading their installation. The mechanism is based on a strict versioning scheme and so called upgrade rules. The rest of this page describes the steps you as a developer must execute every time you introduce an incompatible data model change. The procedure slightly differs depending on whether the data model change is the first one after the most recent polymake release. Just for illustrative purposes we will assume in the following examples the most recently released version to be ''4.1''. |
| |
| === Introducing the first change after a release === | === Introducing the first change after a release === |
| This situation can be recognized by the current polymake version (stored in a global variable ''$Version'' in ''Polymake.pm'') being equal to the last release version, and consisting of exactly two numbers, e.g. ''4.1'' . | This situation can be recognized by the current polymake version (stored in a global variable ''$Version'' in ''Polymake.pm'') being equal to the last release version, and consisting of exactly two numbers, e.g. ''4.1'' . |
| |
| * Execute a script ''big_object_inventory''. It should create a new file ''upgrades/big_objects-4.1.1'' . It will contain a condensed snapshot of the current data model representing the inheritance and aggregation relationships between all currently defined big object types. The numeric suffix of this filename ''4.1.1'' defines the internal upgrade target version, which is referred to in the next steps. | * Execute a script ''big_object_inventory''. It should create a new file ''upgrades/big_objects-4.1.1'' . It will contain a condensed snapshot of the current data model representing the inheritance and aggregation relationships between all currently defined big object types. The numeric suffix of this filename ''4.1.1'' defines the intermediate upgrade target version, which is referred to in the next steps. |
| * Implement the intended changes to the data model in the rulebase. It's not necessary to implement everything at once, but at least the declarations of changed properties and object types must be brought into the new desired shape. | * Implement the intended changes to the data model in the rulebase. It's not necessary to implement everything at once, but at least the declarations of changed properties and object types must be brought into the new desired shape. |
| * Create a new file ''upgrades/NEXT_RELEASE_VERSION'', e.g. ''upgrades/4.2''. Define appropriate [[#upgrade_rules|upgrade rules]] there, giving them the same internal upgrade target version (''4.1.1''). | * Create a new file ''upgrades/NEXT_RELEASE_VERSION'', e.g. ''upgrades/4.2''. Define appropriate [[#upgrade_rules|upgrade rules]] there, giving them the same intermediate upgrade target version (''4.1.1''). |
| |
| === Introducing a subsequent change === | === Introducing a subsequent change === |
| * Execute a script ''big_object_inventory''. It can decide to create a new inventory file ''upgrades/big_objects-4.1.3'' or to go on with the existing inventory. If it creates a new inventory (it will tell you) don't forget to commit it. | * Execute a script ''big_object_inventory''. It can decide to create a new inventory file ''upgrades/big_objects-4.1.3'' or to go on with the existing inventory. If it creates a new inventory (it will tell you) don't forget to commit it. |
| * Implement the intended changes to the data model in the rulebase. | * Implement the intended changes to the data model in the rulebase. |
| * Add new [[#upgrade_rules|upgrade rules]] to the existing file ''upgrades/4.2'', giving them the target version one above the current one, e.g. ''4.1.3'' . Although polymake automatically sorts the upgrade rules by target version when it loads the file, it's recommended to place the rules in ascending version order, for a better overview. | * Add new [[#upgrade_rules|upgrade rules]] to the existing file ''upgrades/4.2'', giving them the intermediate target version one above the current one, e.g. ''4.1.3'' . Although polymake automatically sorts the upgrade rules by target version when it loads the file, it's recommended to place the rules in ascending target version order, just for a better overview. |
| |
| === Finishing the change === | === Finishing the change === |
| The remaining steps are the same in both scenarios. | The remaining steps are the same in both scenarios. |
| |
| * Set the global variable ''$Version'' to the internal upgrade target version. | * Set the global variable ''$Version'' to the intermediate upgrade target version of your new upgrade rules. |
| * Create enough [[unit_tests#testing_data_upgrade_rules|testcases]] covering all new upgrade rules, make sure the transformation works as intended. Commit all new files: the big object inventory, upgrade rules, and testcases. | * Create enough [[unit_tests#testing_data_upgrade_rules|testcases]] covering all new upgrade rules, make sure the transformation works as intended. Don't forget to commit all new files into the git repo: the big object inventory, upgrade rules, and testcases. |
| * When the implementation of the new feature is complete, run the entire testsuite. All data files used in the testsuite will be upgraded to the target version: commit them. You can also delegate the data file upgrading and commit to Jenkins, just execute a "fast" or "full" test job. | * When the implementation of the new feature is complete, run the entire testsuite. All data files used in the testsuite will be upgraded to the target version; commit them. You can also delegate the data file upgrading and commit to Jenkins, just execute a "fast" or "full" test job on your feature branch project page. |
| |
| === Resolving conflicts === | === Resolving conflicts === |
| |
| It can happen that two developers decide to introduce (different) model changes at the same time. Whoever is first ready with their work can merge it into the master branch as usual. However, if you are the second developer, you should take particular precautions in order to prevent broken data files: | It can happen that two developers decide to introduce (different) model changes at the same time. If you are first ready with your work, you can merge it into the master branch as usual. However, if the second turn is yours, you should take particular precautions in order to prevent broken data files: |
| * Check out the master branch. Rerun the ''big_object_inventory'' script, just to be on a safe side. Most probably, it won't produce a new inventory file, but should it happens, take it along to your feature branch. | * Check out the master branch. Rerun the ''big_object_inventory'' script, just to be on a safe side. Most probably, it won't produce a new inventory file, but should it happen, take it along to your feature branch. |
| * Return to your feature branch and merge the master branch into it. Resolve conflicts in source code, if any, as usual. | * Return to your feature branch and merge the master branch into it. Resolve conflicts in source code, if any, as usual. |
| * Increase the current version one more time. E.g. if you both started at version ''4.1.2'', your `competitor' has committed their changes with the target version ''4.1.3'', now you have to bump it to ''4.1.4''. Assign this new version to your upgrade rules. | * Increase the current version one more time. E.g. if you both started at version ''4.1.2'', your `competitor' has committed their changes with the target version ''4.1.3'', now you have to bump it to ''4.1.4''. Assign this new version to your upgrade rules and to the global variable ''$Version''. |
| * If you have already transformed many (or even all) data files from the testsuite, they might now have merge conflicts or even be smoothly merged but have a wrong version (still ''4.1.3''). If there are any merge conflicts, resolve them taking "theirs", this will effectively revert all your transformations. If you are lucky and your upgrade rules are idempotent (e.g. renaming a property), you can now simply run the testsuite. If not (e.g. you are restructuring the data within the same property) and there is a substantial amount of data files which you are reluctant to revert, you can temporarily modify the code to recognize already transformed data, run the testsuite, and revert the rule code changes. | * If you have already transformed many (or even all) data files from the testsuite, they might now have merge conflicts or even be smoothly merged but now have a wrong version (still ''4.1.3''). If there are any merge conflicts, resolve them by choosing "theirs", this will effectively revert all your transformations. If you are lucky and your upgrade rules are idempotent (e.g. renaming a property), you can now simply run the testsuite. If not (e.g. you are restructuring the data within the same property) and there is a substantial amount of data files which you are reluctant to revert, you can temporarily modify the code to recognize already transformed data and avoid double processing, run the testsuite, and revert the rule code changes. |
| * Commit the merge. | * Commit the merge. |
| |
| ==== Upgrade rules ==== | ==== Upgrade rules ==== |
| |
| Upgrade rules are small perl scripts operating on polymake data in serialized form, that is, on pure perl structures like numbers, strings, anonymous lists and hashes, exactly how you see them in a JSON file. All upgrade rules applicable to data with source version M.N are stored in a file named ''upgrades/M.(N+1)'', that is, after the target version. An upgrade rule file can only contain upgrade rules and generic perl code like subroutines, lexical variables, or require statements, but no other kinds of rules or declarations. | Upgrade rules are perl subroutines operating on polymake data in serialized form, that is, on pure perl structures like numbers, strings, anonymous lists and hashes, exactly how you see them in a JSON file. All upgrade rules applicable to data with source version M.N are stored in a file named ''upgrades/M.(N+1)'', that is, named after the next release version. An upgrade rule file may only contain upgrade rules and generic perl code like subroutines, lexical variables, ''use'' or ''require'' statements, but no other kinds of rules or declarations. |
| |
| At the beginning, an upgrade rulefile should contain a comment block briefly describing all transformations defined there. | By convention, an upgrade rulefile should start with a comment block briefly describing all transformations defined there. |
| Please refer to the existing rules for suitable examples. | Please refer to the existing rules for real life examples. |
| |
| The upgrade rules are applied to matching objects in the increasing order of target versions; the rules with equal target versions are applied in the order of definition in the rulefile. If a rule can be applied to a big object and to some of its subobjects, the parent object will be processed first, while the order of processing its properties is unpredictable. | The upgrade rules are applied to matching objects in the increasing order of target versions; the rules with equal target versions are applied in the order of definition in the rulefile. If a rule can be applied to a big object and to some of its subobjects, the parent object will be processed first, while the order of processing of its properties is unpredictable. |
| |
| An upgrade rule can have one of the following forms: | An upgrade rule can have one of the following forms (M.N.P is the intermediate target version as described in the examples above): |
| upgrade TARGET.VERSION.PATCHLEVEL APP_NAME::BIG_OBJECT_TYPE { | upgrade M.N.P APP_NAME::BIG_OBJECT_TYPE { |
| my ($obj) = @_; | my ($obj) = @_; |
| # some transformations | # some transformations |
| } | } |
| |
| This rule will be called for every big object of the specified type or derived thereof, regardless whether it's a top-level standalone object, a subobject of another big object, or an element of a big object array. The rule may do whatever is necessary, including modifying its type, any properties and their optional attributes, accessible as ''%%$obj->{_type}%%'' resp. ''%%$obj->{PROPERTY_NAME}%%'' and ''%%$obj->{_attrs}->{PROPERTY_NAME}%%''. Please be aware that the explicit ''_type'' element will only be present in top-level objects or when it deviates from the default type for the given subobject. | This rule will be called for every big object of the specified type or derived thereof, regardless whether it's a top-level standalone object, a subobject of another big object, or an element of a big object array. The rule may do whatever is necessary, including modifying its type, any properties and their optional attributes, accessible as ''%%$obj->{_type}%%'' resp. ''%%$obj->{PROPERTY_NAME}%%'' and ''%%$obj->{_attrs}->{PROPERTY_NAME}%%''. Please be aware that the explicit ''_type'' element might be absent in subobjects when their type exactly matches the property declaration. |
| | |
| upgrade TARGET.VERSION.PATCHLEVEL APP_NAME::BIG_OBJECT_TYPE.PROPERTY { | upgrade M.N.P APP_NAME::BIG_OBJECT_TYPE.PROPERTY { |
| my ($obj, $prop_name) = @_; | my ($obj, $prop_name) = @_; |
| # some transformations | # some transformations |
| } | } |
| |
| This rule will be called for every occurrence of the specified property in a big object of specified type or derived thereof. The PROPERTY part can in fact be a path descending arbitrarily deep into a subobject's properties: ''SUBOBJECT.PROPERTY''. The ''$obj'' parameter will point to the direct parent object of the property of interest, not to the object at the top of the hierarchy. The ''$prop_name'' parameter will contain the name of the bottom-most property. The same rule can be reused for several different properties by listing several property paths separated by bars: ''PROPERTY1 | PROPERTY2'' . | This rule will be called for every occurrence of the specified property in a big object of specified type or derived thereof. The PROPERTY part can in fact be a path descending into subobjects: ''SUBOBJECT1.SUBOBJECT2.PROPERTY''. The ''$obj'' parameter will point to the direct parent object of the property of interest, not to the object at the top of the hierarchy. The ''$prop_name'' parameter will contain the name of the bottom-most property. The same rule can be reused for several different properties by listing several property paths separated by bars: ''PROPERTY1 | PROPERTY2'' . |
| |
| Please be aware that the value of any property can also be an ''undef'', the upgrade rule will be called in any case. | When writing transformations of property values, please keep in mind that they can happen to be ''undef''. |
| |
| In both variants, the return value should be ''true'' if something has been changed or ''false'' if everything stayed as before. Please note that the big object type specified in a rule header do not have any type parameters - the rule will be applied to all instances of a parametrized type. | All big object types appearing in an upgrade rule must always be qualified with the application name, including new type names assigned to the ''_type'' element. Big object types in the header may never be written with type parameters, a rule is always applicable to all instances of a parametrized type. |
| | |
| | The return value of an upgrade rule should be ''true'' if something has been changed or ''false'' if everything has stayed as before. |
| |
| When renaming or moving a property into a different place in the object tree, don't forget to move and/or rename its optional attribute too. Because this is a quite common transformation, there are convenience functions ''rename_property'' and ''move_property'' which can be called from the rule body. Moreover, there are two shortcut forms for rules where the renaming or deleting a property is the only action to be performed: | When renaming or moving a property into a different place in the object tree, don't forget to move and/or rename its optional attribute too. Because this is a quite frequent operation, there are convenience functions ''rename_property'' and ''move_property'' which can be called from the rule body. Moreover, there are two shortcut forms for rules where the renaming or deleting a property is the only action to be performed: |
| |
| upgrade TARGET.VERSION.PATCHLEVEL APP_NAME::BIG_OBJECT_TYPE.PROPERTY = rename NEW_PROPERTY; | upgrade M.N.P APP_NAME::BIG_OBJECT_TYPE.PROPERTY = rename NEW_PROPERTY; |
| upgrade TARGET.VERSION.PATCHLEVEL APP_NAME::BIG_OBJECT_TYPE.PROPERTY = delete; | upgrade M.N.P APP_NAME::BIG_OBJECT_TYPE.PROPERTY = delete; |
| | |
| |