Data model in polymake evolves constantly: new big object types, subobjects, and properties show up virtually in every release. As long as only new stuff is being added, the forward compatibility is preserved, so that an old data file can safely be loaded into an arbitrary version of polymake released later. However, sometimes we have to revise old design decisions and change the existing data model in an incompatible way. A non-exhaustive list of compatibility breaking change examples:

• renaming or deleting a property
• moving a property into a subobject
• changing the data type of a property
• changing the serialized representation of a property data type
• renaming or changing the type parameters of a big object type
• merging or splitting big object types, changing the inheritance hierarchy
• renaming, merging or splitting applications

Whatever the reasons for a data model change, we have to provide an automatic data transition for users upgrading their installation which must prevent any data loss. The mechanism is based on a strict versioning scheme and so called upgrade rules. The rest of this page describes the steps a developer must execute every time they introduce an incompatible data model change. The procedure slightly differs depending on whether it deals with the first data model change after the most recent polymake release, or with a subsequent change between two releases. In the following examples we will assume the most recently released version to be 4.1.

This situation can be recognized by the current polymake version (stored in a global variable $Version in Polymake.pm) being equal to the last release version, and consisting of exactly two numbers, e.g. 4.1 .

• Execute a script big_object_inventory. It should create a new file upgrades/big_objects-4.1.1 . It will contain a condensed snapshot of the current data model representing the inheritance and aggregation relationships between all currently defined big object types. The numeric suffix of this filename 4.1.1 defines the internal upgrade target version, which is referred to in the next steps.
• Implement the intended changes to the data model in the rulebase. It's not necessary to implement everything at once, but at least the declarations of changed properties and object types must be brought into the new desired shape.
• Create a new file upgrades/NEXT_RELEASE_VERSION, e.g. upgrades/4.2. Define appropriate upgrade rules there, giving them the same internal upgrade target version (4.1.1).

This situation can be recognized by the current version consisting of three numbers, e.g. 4.1.2 .

• Execute a script big_object_inventory. It can decide to create a new inventory file upgrades/big_objects-4.1.3 or to go on with the existing inventory. If it creates a new inventory (it will tell you) don't forget to commit it.
• Implement the intended changes to the data model in the rulebase.
• Add new upgrade rules to the existing file upgrades/4.2, giving them the target version one above the current one, e.g. 4.1.3 . Although polymake automatically sorts the upgrade rules by target version when it loads the file, it's recommended to place the rules in ascending version order, for a better overview.

The remaining steps are the same in both scenarios.

• Set the global variable $Version to the internal upgrade target version.
• Create enough testcases covering all new upgrade rules, make sure the transformation works as intended. Commit all new files: the big object inventory, upgrade rules, and testcases.
• When the implementation of the new feature is complete, run the entire testsuite. All data files used in the testsuite will be upgraded to the target version: commit them. You can also delegate the data file upgrading and commit to Jenkins, just execute a “fast” or “full” test job.

It can happen that two developers decide to introduce (different) model changes at the same time. Whoever is first ready with their work can merge it into the master branch as usual. However, if you are the second developer, you should take particular precautions in order to prevent broken data files:

• Check out the master branch. Rerun the big_object_inventory script, just to be on a safe side. Most probably, it won't produce a new inventory file, but should it happens, take it along to your feature branch.
• Return to your feature branch and merge the master branch into it. Resolve conflicts in source code, if any, as usual.
• Increase the current version one more time. E.g. if you both started at version 4.1.2, your `competitor' has committed their changes with the target version 4.1.3, now you have to bump it to 4.1.4. Assign this new version to your upgrade rules.
• If you have already transformed many (or even all) data files from the testsuite, they might now have merge conflicts or even be smoothly merged but have a wrong version (still 4.1.3). If there are any merge conflicts, resolve them taking “theirs”, this will effectively revert all your transformations. If you are lucky and your upgrade rules are idempotent (e.g. renaming a property), you can now simply run the testsuite. If not (e.g. you are restructuring the data within the same property) and there is a substantial amount of data files which you are reluctant to revert, you can temporarily modify the code to recognize already transformed data, run the testsuite, and revert the rule code changes.
• Commit the merge.

Upgrade rules are small perl scripts operating on polymake data in serialized form, that is, on pure perl structures like numbers, strings, anonymous lists and hashes, exactly how you see them in a JSON file. All upgrade rules applicable to data with source version M.N are stored in a file named upgrades/M.(N+1), that is, after the target version. An upgrade rule file can only contain upgrade rules and generic perl code like subroutines, lexical variables, or require statements, but no other kinds of rules or declarations.

At the beginning, an upgrade rulefile should contain a comment block briefly describing all transformations defined there. Please refer to the existing rules for suitable examples.

The upgrade rules are applied to matching objects in the increasing order of target versions; the rules with equal target versions are applied in the order of definition in the rulefile. If a rule can be applied to a big object and to some of its subobjects, the parent object will be processed first, while the order of processing its properties is unpredictable.

An upgrade rule can have one of the following forms:

upgrade TARGET.VERSION.PATCHLEVEL APP_NAME::BIG_OBJECT_TYPE {
  my ($obj) = @_;
  # some transformations
  return BOOLEAN;
}

This rule will be called for every big object of the specified type or derived thereof, regardless whether it's a top-level standalone object, a subobject of another big object, or an element of a big object array. The rule may do whatever is necessary, including modifying its type, any properties and their optional attributes, accessible as $obj->{_type} resp. $obj->{PROPERTY_NAME} and $obj->{_attrs}->{PROPERTY_NAME}. Please be aware that the explicit _type element will only be present in top-level objects or when it deviates from the default type for the given subobject.

upgrade TARGET.VERSION.PATCHLEVEL APP_NAME::BIG_OBJECT_TYPE.PROPERTY {
  my ($obj, $prop_name) = @_;
  # some transformations
  return BOOLEAN;
}

This rule will be called for every occurrence of the specified property in a big object of specified type or derived thereof. The PROPERTY part can in fact be a path descending arbitrarily deep into a subobject's properties: SUBOBJECT.PROPERTY. The $obj parameter will point to the direct parent object of the property of interest, not to the object at the top of the hierarchy. The $prop_name parameter will contain the name of the bottom-most property. The same rule can be reused for several different properties by listing several property paths separated by bars: PROPERTY1 | PROPERTY2 .

Please be aware that the value of any property can also be an undef, the upgrade rule will be called in any case.

In both variants, the return value should be true if something has been changed or false if everything stayed as before. Please note that the big object type specified in a rule header do not have any type parameters - the rule will be applied to all instances of a parametrized type.

When renaming or moving a property into a different place in the object tree, don't forget to move and/or rename its optional attribute too. Because this is a quite common transformation, there are convenience functions rename_property and move_property which can be called from the rule body. Moreover, there are two shortcut forms for rules where the renaming or deleting a property is the only action to be performed:

upgrade TARGET.VERSION.PATCHLEVEL APP_NAME::BIG_OBJECT_TYPE.PROPERTY = rename NEW_PROPERTY;
upgrade TARGET.VERSION.PATCHLEVEL APP_NAME::BIG_OBJECT_TYPE.PROPERTY = delete;