Data model in polymake evolves constantly: new big object types, subobjects, and properties show up virtually in every release. As long as only new stuff is being added, the forward compatibility is preserved, so that an old data file can safely be loaded into an arbitrary version of polymake released later. However, sometimes we have to revise old design decisions and change the existing data model in an incompatible way. A non-exhaustive list of compatibility breaking change examples:

• renaming or deleting a property
• moving a property into a subobject
• changing the data type of a property
• changing the serialized representation of a property data type
• renaming or changing the type parameters of a big object type
• merging or splitting big object types, changing the inheritance hierarchy
• renaming, merging or splitting applications

Whatever the reasons for a data model change, we have to ensure a lossless automatic data transition for users upgrading their installation. The mechanism is based on a strict versioning scheme and so called upgrade rules. The rest of this page describes the steps you as a developer must execute every time you introduce an incompatible data model change. The procedure slightly differs depending on whether the data model change is the first one after the most recent polymake release. Just for illustrative purposes we will assume in the following examples the most recently released version to be 4.1.

This situation can be recognized by the current polymake version (stored in a global variable $Version in Polymake.pm) being equal to the last release version, and consisting of exactly two numbers, e.g. 4.1 .

• Execute a script big_object_inventory. It should create a new file upgrades/big_objects-4.1.1 . It will contain a condensed snapshot of the current data model representing the inheritance and aggregation relationships between all currently defined big object types. The numeric suffix of this filename 4.1.1 defines the intermediate upgrade target version, which is referred to in the next steps.
• Implement the intended changes to the data model in the rulebase. It's not necessary to implement everything at once, but at least the declarations of changed properties and object types must be brought into the new desired shape.
• Create a new file upgrades/NEXT_RELEASE_VERSION, e.g. upgrades/4.2. Define appropriate upgrade rules there, giving them the same intermediate upgrade target version (4.1.1).

This situation can be recognized by the current version consisting of three numbers, e.g. 4.1.2 .

• Execute a script big_object_inventory. It can decide to create a new inventory file upgrades/big_objects-4.1.3 or to go on with the existing inventory. If it creates a new inventory (it will tell you) don't forget to commit it.
• Implement the intended changes to the data model in the rulebase.
• Add new upgrade rules to the existing file upgrades/4.2, giving them the intermediate target version one above the current one, e.g. 4.1.3 . Although polymake automatically sorts the upgrade rules by target version when it loads the file, it's recommended to place the rules in ascending target version order, just for a better overview.

The remaining steps are the same in both scenarios.

• Set the global variable $Version to the intermediate upgrade target version of your new upgrade rules.
• Create enough testcases covering all new upgrade rules, make sure the transformation works as intended. Don't forget to commit all new files into the git repo: the big object inventory, upgrade rules, and testcases.
• When the implementation of the new feature is complete, run the entire testsuite. All data files used in the testsuite will be upgraded to the target version; commit them. You can also delegate the data file upgrading and commit to Jenkins, just execute a “fast” or “full” test job on your feature branch project page.

It can happen that two developers decide to introduce (different) model changes at the same time. If you are first ready with your work, you can merge it into the master branch as usual. However, if the second turn is yours, you should take particular precautions in order to prevent broken data files:

• Check out the master branch. Rerun the big_object_inventory script, just to be on a safe side. Most probably, it won't produce a new inventory file, but should it happen, take it along to your feature branch.
• Return to your feature branch and merge the master branch into it. Resolve conflicts in source code, if any, as usual.
• Increase the current version one more time. E.g. if you both started at version 4.1.2, your `competitor' has committed their changes with the target version 4.1.3, now you have to bump it to 4.1.4. Assign this new version to your upgrade rules and to the global variable $Version.
• If you have already transformed many (or even all) data files from the testsuite, they might now have merge conflicts or even be smoothly merged but now have a wrong version (still 4.1.3). If there are any merge conflicts, resolve them by choosing “theirs”, this will effectively revert all your transformations. If you are lucky and your upgrade rules are idempotent (e.g. renaming a property), you can now simply run the testsuite. If not (e.g. you are restructuring the data within the same property) and there is a substantial amount of data files which you are reluctant to revert, you can temporarily modify the code to recognize already transformed data and avoid double processing, run the testsuite, and revert the rule code changes.
• Commit the merge.

Upgrade rules are perl subroutines operating on polymake data in serialized form, that is, on pure perl structures like numbers, strings, anonymous lists and hashes, exactly how you see them in a JSON file. All upgrade rules applicable to data with source version M.N are stored in a file named upgrades/M.(N+1), that is, named after the next release version. An upgrade rule file may only contain upgrade rules and generic perl code like subroutines, lexical variables, use or require statements, but no other kinds of rules or declarations.

By convention, an upgrade rulefile should start with a comment block briefly describing all transformations defined there. Please refer to the existing rules for real life examples.

The upgrade rules are applied to matching objects in the increasing order of target versions; the rules with equal target versions are applied in the order of definition in the rulefile. If a rule can be applied to a big object and to some of its subobjects, the parent object will be processed first, while the order of processing of its properties is unpredictable.

An upgrade rule can have one of the following forms (M.N.P is the intermediate target version as described in the examples above):

upgrade M.N.P APP_NAME::BIG_OBJECT_TYPE {
  my ($obj) = @_;
  # some transformations
  return BOOLEAN;
}

This rule will be called for every big object of the specified type or derived thereof, regardless whether it's a top-level standalone object, a subobject of another big object, or an element of a big object array. The rule may do whatever is necessary, including modifying its type, any properties and their optional attributes, accessible as $obj->{_type} resp. $obj->{PROPERTY_NAME} and $obj->{_attrs}->{PROPERTY_NAME}. Please be aware that the explicit _type element might be absent in subobjects when their type exactly matches the property declaration.

upgrade M.N.P APP_NAME::BIG_OBJECT_TYPE.PROPERTY {
  my ($obj, $prop_name) = @_;
  # some transformations
  return BOOLEAN;
}

This rule will be called for every occurrence of the specified property in a big object of specified type or derived thereof. The PROPERTY part can in fact be a path descending into subobjects: SUBOBJECT1.SUBOBJECT2.PROPERTY. The $obj parameter will point to the direct parent object of the property of interest, not to the object at the top of the hierarchy. The $prop_name parameter will contain the name of the bottom-most property. The same rule can be reused for several different properties by listing several property paths separated by bars: PROPERTY1 | PROPERTY2 .

When writing transformations of property values, please keep in mind that they can happen to be undef.

All big object types appearing in an upgrade rule must always be qualified with the application name, including new type names assigned to the _type element. Big object types in the header may never be written with type parameters, a rule is always applicable to all instances of a parametrized type.

The return value of an upgrade rule should be true if something has been changed or false if everything has stayed as before.

When renaming or moving a property into a different place in the object tree, don't forget to move and/or rename its optional attribute too. Because this is a quite frequent operation, there are convenience functions rename_property and move_property which can be called from the rule body. Moreover, there are two shortcut forms for rules where the renaming or deleting a property is the only action to be performed:

upgrade M.N.P APP_NAME::BIG_OBJECT_TYPE.PROPERTY = rename NEW_PROPERTY;
upgrade M.N.P APP_NAME::BIG_OBJECT_TYPE.PROPERTY = delete;