Differences

This shows you the differences between two versions of the page.

--- user_guide:howto:polydb_tutorial [2019/09/11 15:53] – [Access Crednetials] paffenholz
+++ user_guide:howto:polydb_tutorial [2021/06/19 21:33] (current) – correction of a typo schroeter
@@ Line 1: / Line 1: @@
-====== Introduction ot polyDB version 3 (using json format) ======
+====== Introduction to polyDB ======
-Note: This introduction applies to the version of polyDB in the **branch ''feature/polydb-mongo-views'' of the polymake git**. For the released version please see [[user_guide:howto:poly_db_tutorial|here]]. Also, this tutorial explains how to use the ''polyDB'' from within polymake. If you want to access the data without using ''polymake'' please check [[user_guide:howto:polydb_api|here]].
+This tutorial explains how to access the [[:polydb|polyDB database]] from within polymake using the extension ''polyDB''. It comes bundled with ''polymake'', so there is no need to install extra software, except for the MongoDB.pm perl package. (This tutorial is for polymake version 4 and later. The old version is [[poly_db_tutorial|here]], but this needs also an old version of the database). If you encounter any errors or problems concerning ''polyDB'' or the extension, please don't hesitate to [[https://forum.polymake.org/|ask in the forum]].
-The polyDB extension provides access to the [[:polydb|polyDB database]]. It comes bundled with ''polymake'', so there is no need to install extra software, except for the MongoDB.pm perl package. If you encounter any errors or problems concerning polyDB, please don't hesitate to [[https://forum.polymake.org/|ask in the forum]].
+The ''polymake'' extension is not necessary to use the data. You can access the data also
+  * via the [[https//db.polymake.org|web interface]]
-However, the ''polymake'' extension is not necessary to use the data. You can access the data also
-  * via the [[https//db.polymake.org|web interface]] (note: this currently obtains its data from the old version)
   * using the mongo shell directly or via any gui (see **below** for more details (to be written))
+  * using the [[user_guide:howto:polydb_api|polyDB REST API]].
 Software developers can also include access to polyDB using any of the many MongoDB interfaces and use the data directly in their programs. The few structural assumptions made in the database that you need to follow in your development are explained **below** (to be written).
-===== Learning which Data is Available ====
+===== Initializing Access =====
-The data in polyDB is stored in **collections**, which can be organized in nested **sections**. Access to a certain collection requires the full path to it, i.e. both the (nested) section containing the collection and the collection name. E.g. the collection fo smooth reflexive Fano polytopes up to dimension nine is in the collection ''SmoothReflexive'' contained in the ''Lattice'' subsection of the section ''Polytopes''. The section separator is a dot (''.''), so you find the data in
+You need a **database connection** before you can work with ''polyDB'' (and a working internet connection if you want to access a database that is not stored in your local computer). For the main instance of ''polyDB'' you can just call
+<code>
+  $polydb = polyDB();
+</code>
+If you have a personal account in the database (e.g. if you maintain a collection or work on a provate project), you should pass your username and password:
+<code>
+  $polydb = polyDB(username=><username>, passwd=><password>);
+</code>
+You can also store this in the custom variables <code>$PolyDB::default::db_user</code> and <code>$PolyDB::default::db_pwd</code>... Then you don't have to specify them in the connection method, they will be picked up automatically and are preserved over sessions.
+===== Learning which Data is Available =====
+The data in polyDB is stored in **collections**, which can be organized in nested **sections**. Access to a certain collection requires the full path to it, i.e. both the (nested) section containing the collection and the collection name. E.g. the collection of smooth reflexive Fano polytopes up to dimension nine is in the collection ''SmoothReflexive'' contained in the ''Lattice'' subsection of the section ''Polytopes''. The section separator is a dot (''.''), so you find the data in
   * **section**: ''Polytopes.Lattice''
   * **collection**: ''SmoothReflexive''
-Subsections can also be listed in front of the collection, so the following pairs ''(s,c)'' of section and collection lead to the same data: (''Polytopes'',''Lattice.SmoothReflexive'') and ('' '',''Polytopes.Lattice.SmoothReflexive'').
-The command <code>db_info</code> gives you a list of collections you can read, ordered by sections. Yo can restrict the search by
+The ''info'' method gives you a list of collections you can read, ordered by sections. If you have a private account in the database then this command will also list the additional collections you have been granted access to.
-  * providing a chain of sections and subsections in the argument ''section''. This can also just be an initial substring, so <code>db_info(section=>"Polytopes.La")</code> lists all collections within the section ''Polytopes'' whose first subsection starts with ''La''
+<code>
-  * given a specific section, you can filter the collections by an initial substring, so <code>db_info(section=>"Polytopes.Lattice", collection=>"Smoo")</code> lists all collections within the section ''Polytopes.Lattice'' whose name starts with ''Smoo''.
+$polydb->info();
-If you have access to a not publicly available collection please set the variables ''$PolyDB::default::db_user'' and ''$PolyDB::default::db_pwd'' first to see your private collections in the above list.
+===============
+available polydb collections
+===============
+SECTION: Manifolds
+This database contains combinatorial manifolds
+	COLLECTION: DIM2_3
+	This is a collection of  combinatorial 2-  and	3-manifolds with up to 10 vertices given as triangulations and calculcated by Frank Lutz found at:
+	  http://page.math.tu-berlin.de/~lutz/stellar/mixed.html.
+[...]
+</code>
+You can restrict the search by
+  * providing a chain of sections and subsections in the argument ''section''. This can also just be an initial substring, so <code>$polydb->info(section=>"Polytopes.La")</code> lists all collections within the section ''Polytopes'' whose first subsection starts with ''La''
+  * given a specific section, you can filter the collections by an initial substring, so <code>$polydb->info(section=>"Polytopes.Lattice", collection=>"Smoo")</code> lists all collections within the section ''Polytopes.Lattice'' whose name starts with ''Smoo''.
 ===== Reading Data =====
-There are various functions available to query data:
+Before you can access a collection you need to establish a **connection handle** with the method ''get_collection''. For example, if you want to acess the list of 0/1-polytopes up to combinatorial equivalence you would call
-  * ''db_count'': to count the number of objects with given properties
+<code>
-  * ''db_query'': to get an array of objects with given properties
+$collection=$polydb->get_collection("Polytopes.Combinatorial.01Polytopes");
-  * ''db_cursor'': to obtain a cursor on objects with given properties
+</code>
-  * ''db_aggregate'': to appy complex aggregation pipelines on a collection
+Now you can query this ist the various methods provided:
+  * ''count'': to count the number of objects satisfying the provided query
+  * ''find_one'': to get one example satisfying the provided query
+  * ''find'': to obtain a cursor on objects satisfying the provided query
+  * ''distinct'': to get an array of distinct values for a property among all objects satisfying the provided query
+  * ''aggregate'': to apply complex aggregation pipelines on a collection
-The main argument of the first three functions is a MongoDb query hash. You can use the full MongoDB query syntax as decribed [[https://docs.mongodb.com/manual/tutorial/query-documents/|here]]. Note that ''polymake'' uses the perl interface, so the query should be given as a perl hash instead of a json document (it mostly suffices to use ''=>'' instead of '':''). For some perl examples see [[https://metacpan.org/pod/MongoDB::Tutorial#Retrieving-Documents|here]]. Basic queries for one or more parameter look like<code>{"N_VERTICES"=>10}
+The main argument of the first three functions is a MongoDD query hash. You can use the full MongoDB query syntax as decribed [[https://docs.mongodb.com/manual/tutorial/query-documents/|here]]. Note that ''polymake'' uses the perl interface interface of MongoDB, so the query should be given as a perl hash instead of a json document (it mostly suffices to use ''=>'' instead of '':''). For some perl examples see [[https://metacpan.org/pod/MongoDB::Tutorial#Retrieving-Documents|here]]. Basic queries for one or more parameter look like<code>{"N_VERTICES"=>10}
 {"DIM"=>5, "N_FACETS"=>7}
 </code>
-Bounds or Ranges can be defined with the operators ''&gt'', ''&gte'', ''&lt'' and ''&lte''. For example<code>{"N_VERTICES"=> { "&gte" => 5, "&lte" => 10 } }</code> returns documents where the number of vertices is between five and ten (including the boundaries). More operators can be found [[https://docs.mongodb.com/manual/reference/operator/query/#query-selectors|here]]. You can also query for elements in arrays either somewhere in the array or at a specific position.
+Bounds or ranges can be defined with the operators ''&gt'', ''&gte'', ''&lt'' and ''&lte''. For example<code>{"N_VERTICES"=> { "&gte" => 5, "&lte" => 10 } }</code> returns documents where the number of vertices is between five and ten (including the boundaries). More operators can be found [[https://docs.mongodb.com/manual/reference/operator/query/#query-selectors|here]]. You can also query for elements in arrays either somewhere in the array or at a specific position.
 The last function allows to pass an aggregation pipeline as described [[https://docs.mongodb.com/manual/aggregation/|here]] (note again that the pipeline needs to be passed as a perl hash instead of a json document).
-All queries need the name of the section and collection. You can either pass this via the options ''section'' and ''collection'' or, in particular if you query the same collection several times in a row, via the variables ''$PolyDB:"default::db_section_name'' and ''$PolyDB:"default::db_collection_name''. If you set them via [[documentation:master:core#set_custom|set_custom]].  this is even persistent over ''polymake'' sessions. So a query could look like<code>db_query({"DIM"=>4}, section=>"Polytopes.Lattice", collection=>"SmoothReflexive");</code>or<code>$PolyDB::default::db_section="Polytopes.Lattice";
+All queries need the name of the section and collection. You can either pass this via the options ''section'' and ''collection'' or, in particular if you query the same collection several times in a row, via the variables ''$PolyDB:"default::db_section_name'' and ''$PolyDB:"default::db_collection_name''. If you set them via [[documentation:master:core#set_custom|set_custom]].  this is even persistent over ''polymake'' sessions. So a query could look like<code>$cur=$collection->find({"DIM"=>4}, section=>"Polytopes.Lattice", collection=>"SmoothReflexive");</code>or<code>$PolyDB::default::db_section="Polytopes.Lattice";
 $PolyDB:"default::db_collection="SmoothReflexive";
-db_query({"DIM"=>4});</code>
+$cur=$collection->find({"DIM"=>4});</code>
-''db_query'' returns an array of polytopes, ''db_count'' just counts the number of objects matching your query, while ''db_cursor'' returns a cursor over the objects matching your query. You can iterate over the result via
+''find'' returns a cursor over the list of results, ''count'' just counts the number of objects matching your query. You can iterate over the result of ''find'' via
 <code>
-$c=db_cursor({"DIM"=>4}, section=>"Polytopes.Lattice", collection=>"SmoothReflexive");
+$cur=$collection->find({"DIM"=>4}, section=>"Polytopes.Lattice", collection=>"SmoothReflexive");
-while ( $c->has_next() ) {
+while ( $cur->has_next() ) {
-  $p=$c->next();
+  $p=$cur->next();
   print $p->N_VERTICES;
 }
@@ Line 52: / Line 83: @@
   * ''skip=>$n'': skips the first ''$n'' documents
   * ''limit=>$n'': returns at most ''$n'' documents
-  * ''representative=>1'' returns just one document matching the query
-==== Access Credentials ====
+You can reset the cursor with<code>$cur->reset</code>if you want to iterate over the results again.
-There are two pairs of custom variables for access credentials:
-  * ''$PolyDB::default::db_{user,pwd}'': For a user that has read access (usually set to ''polymake''/''database'' for all public collections). Set this to your private user if you have been granted access to private collections
-  * ''$PolyDB::default::db_collection_admin_{user,pwd}'': For credentials with write access to collections. Note that the first pair is not checked for write access even if the user given there has write access. You must set this pair for write access.
 ===== Inserting new Data =====
@@ Line 64: / Line 91: @@
 Inserting data requires several steps:
-  * You need to prepare a json document with  **meta information** about you collection in the following form<code>{
+  * You need to prepare a json document with  **meta information** about you collection in the following form<code>
-   "description" : "Smooth reflexive polytopes with decomposition structure in dimensions 1 to 9",
+{
-   "contributor" : "Andreas Paffenholz",
+    "_id" : "info.2.1",
-   "maintainer" : "Andreas Paffenholz",
+    "description" : "0/1 Polytopes",
-   "creator" : "Andreas Paffenholz",
+    "collection" : "01Polytopes",
-   "fields" : {
+    "section" : [ "Polytopes", "Combinatorial" ],
-       "ALTSHULER_DET": 1,
+    "creator" : "Oswin Aichholzer",
-       "BALANCED": 1,
+    "contributor" : "Andreas Paffenholz",
-       "VERY_AMPLE" : 1
+    "maintainer" : "Andreas Paffenholz",
-   },
+    "polydb_version" : "2.1",
-   "polydb_version" : "2.1",
+    "packages" : {
-   "packages": {
+        "polymake" : {
-       "polymake" : {
+            "version" : "3.4",
-          "version" : "3.4",
+            "type" : "polytope::Polytope<Rational>"
-          "type" : "polytope::Polytope<Rational>"
+        }
-       }
+    },
+    "uri" : "https://polymake.org"
+}</code>
+  * You need to provide a full json schema describing your data. If you have a polymake object with the data you want, then the function ''create_restrictive_schema'' can help you with this and provide an initial template. Here is part of the schema for 0/1-Polytopes.<code>{
+   "type": "object",
+   "$schema": "http://json-schema.org/draft-07/schema#",
+   "properties": {
+      "SELF_DUAL": {
+         "$ref": "#/definitions/common-Bool"
+      },
+      "VERTICES": {
+         "$ref": "#/definitions/common-Matrix-Rational-NonSymmetric"
+      },
+      "_ns": {
+         "additionalProperties": false,
+         "properties": {
+            "polymake": {
+               "type": "array",
+               "additionalItems": false,
+               "items": [
+                  {
+                     "const": "https://polymake.org"
+                  },
+                  {
+                     "const": "3.5"
+                  }
+               ]
+            }
+         },
+         "type": "object"
+      }
    },
-   "uri" : "http://polymake.org/polytopes/paffenholz/www/fano.html”
+   "additionalProperties": false,
-}</code>
+   "required": [
-  * You need to provide a full json schema describing your data. If you have a polymake object with the data you want, then the function ''create_restrictive_schema'' can help you with this and provide an initial template. In ''polyDB'' the json schema is stored as the ''schema'' entry of a document also specifying the section and collection in the entries ''section'' and ''collection''. Both JSON schemas and MongoDB use ''$'' as a special character to specify functions. This leads to a clash when you try to store a json schema in MongoDB. Hence, in the schema document we replace ''$''$ with ''__'' for storing and restore this when reading the schema.
+      "_ns",
+      "SELF_DUAL",
+      "VERTICES"
+   ],
+   "definitions": {
+      "common-Rational": {
+         "pattern": "^-?(\\d+(/\\d+)?|inf)$",
+         "type": "string"
+      },
+      "common-Matrix-Rational-NonSymmetric": {
+         "type": "array",
+         "items": {
+            "oneOf": [
+               {
+                  "$ref": "#/definitions/common-Vector-Rational"
+               },
+               {
+                  "type": "object",
+                  "properties": {
+                     "cols": {
+                        "type": "integer",
+                        "minimum": 0
+                     }
+                  },
+                  "required": [
+                     "cols"
+                  ],
+                  "additionalProperties": false
+               }
+            ]
+         }
+      },
+      "common-Bool": {
+         "type": "boolean"
+      },
+      "common-Vector-Rational": {
+         "items": {
+            "$ref": "#/definitions/common-Rational"
+         },
+         "type": "array"
+      }
+   }
+}</code>This schema needs one special property ''_polyDB'', which specifies some information on the document. Among the properties you should have<code>"_polyDB": {
+         "required": [
+            "collection",
+            "creation_date",
+            "section",
+            "uri",
+            "version"
+         ],
+         "type": "object",
+         "properties": {
+            "uri": {
+               "type": "string"
+            },
+            "collection": {
+               "type": "string"
+            },
+            "version": {
+               "type": "string",
+               "pattern": "^[0-9]{1,2}.[0-9]{1,2}$"
+            },
+            "creation_date": {
+               "pattern": "^[1-9][0-9]{3}-[0-9]{2}-[0-9]{2}$",
+               "type": "string"
+            },
+            "section": {
+               "type": "string"
+            }
+         }
+      }</code> and should be a required property with an entry also in the properties listed in ''_attr'' as <code>
+      "_attrs": {
+         "additionalProperties": false,
+         "properties": {
+            "_polyDB": {
+               "properties": {
+                  "attachment": {
+                     "const": true
+                  }
+               }
+            }
+         },
+         "type": "object"
+      }</code>
   * If you want you collection to be included in the ''db_info'' command you need a json document describing you collection in the form<code>{
    "collection" : "TOM",
@@ Line 100: / Line 241: @@
    ]
 }</code>
-  * If you want to place this also in a new section, then also this (and all new subsections created) need a description document.
+  * If you want to place this also in a new section, then also this (and all new subsections created) need a description document. However, a description for a section can only be edited by an administrator, so just send us the description and we will add it.
-Meta information, schema and documentation are stored with the functions
+Meta information, schema and documentation are stored with the methods
 <code>
-db_set_collection_meta_information($meta);
+$collection->set_info($meta);
-db_set_collection_schema($schema);
+$collection->set_schema($schema);
-db_write_collection_metadata(file=><file>);
+$collection->set_collection_doc($doc, replace=>true/false, update=>true/false);
 </code>
-where in the first two functions the argument is either a perl hash or the name of a file containing a json document.
+where the first argument is the data as a perl hash.
-Insertion is done with the function ''db_insert''. This function either takes a file, a single ''polymake'' big object or an array of such as first argument and writes this data into the collection specified by the options ''section'' and ''collection''. As for queries you can set these via custom variables and then don't need to specify them in ''db_insert''. Currently you need to specify ''use_schema => 1'' in the command to use the meta information and the json schema you provided. Further options are
+Insertion of data is done with the method ''insert''. This function either takes a file, a single ''polymake'' big object or an array of such as first argument and writes this data into the collection specified by the options ''section'' and ''collection'' (these can also be specified with the same custom variables as for queries). This has some more options, see <code>help "insert";</code>in the polymake shell.
-  * ''type_information'': to specify a different json schema as a perl hash
-  * ''replace'': to replace an existing document
-  * ''noinsert'': For a dry run of the command.
@@ Line 120: / Line 258: @@
 === Starting a new collection ===
-A new collection is started with the command<code>db_admin_initiate_collection(section=><section>, collection=><collection>);</code> where you can omit the two options if you have set the section and collection name with the two custom variables ''$PolyDB::default::db_section_name'' and ''$PolyDB::default::db_collection_name'' before. If the collection should not be public, then also pas the option ''public=>false''.
+A new collection is started with the command<code>$polydb->initiate_collection(section=><section>, collection=><collection>);</code>If the collection should not be public, then also pas the option ''public=>false''. For a public collection the read access role of the new collection is added to the default role ''polymakeUser'' which is granted to every user of ''polyDB''. One can add this later if one wants to build up and test the collection befor making it publicly available.
-If this creates new intermediate subsections you should set the section documentation with <code>db_write_section_metadata(file=><file>);</code> so that the new collection appears in the list printed by ''db_info'' for all users with sufficient permissions.
+If this creates new intermediate subsections you should set the section documentation with <code>$polydb->set_section_doc($doc, section=>...);</code> so that the new collection appears in the list printed by ''db_info'' for all users with sufficient permissions.
-Note that the first command essentially only creates two new roles in MongoDB, one for read access to the collection (and all sections up to the root) and one for write access to the collection (and only to the collection, not to the sections). The actual collections are only created once the first document is written into the collection.
+Note that the first command essentially only creates two new roles in MongoDB, one for read access to the collection (and all sections up to the root) and one for write access to the collection (and only to the collection, not to the sections). The actual collections are only created once the first document is written into the collection. This implies that collections will not be listed with ''db_info'' if any of the intermediate sections has no documentation, as then the collection where this is stored is not created.
-Any user that has the write access rule (which you can assign with ''db_admin_add_user_to_collection'') can insert, delete and modify documents in the collection, the meta information, the schema and to documentation of the collection.
+You can add users for read acces with the method ''add_user_to_collection''. Note that this is only useful if the collection is not public. Any user that has the write access rule (which you can assign with ''$polydb->add_user_to_collection(user=>..., collection=>..., admin=>true'') can insert, delete and modify documents in the collection, the meta information, the schema and to documentation of the collection. Note that in this method the collection must be given fully qualified, e.g. as ''Polytopes.Combinatorial.01Polytopes''.
-=== Access Credentials ===
-There are three pairs of custom variables for access credentials:
-  * ''$PolyDB::default::db_{user,pwd}'': For a user that has read access (usually set to ''polymake''/''database'' for all public collections)
-  * ''$PolyDB::default::db_collection_admin_{user,pwd}'': For credentials with write access to collections
-  * ''$PolyDB::default::db_admin_{user,pwd}'': For MongoDB admin credentials. This is used instead of the collection admin credentials if those are not set.