Concepts
This page describes the concepts that are useful to know when using the Squonk Data Manager, and the words this application uses for them.
Users and roles
A user is someone signed in to the Data Manager. Users are defined in the Keycloak Single Sign On (SSO) system this deployment uses, and you need an account there before you can become one. Once you have one you are potentially able to use any of the Squonk applications. See the Logging in guide.
An account carries roles granted in Keycloak, in two sets: one for the Data Manager and one for the Account Server. Each set is a ladder, and each role can do everything the one below it can:
- evaluator - may take out subscriptions only in its own personal unit, and may create projects there only at the Evaluation tier.
- user - the ordinary role, and the one most accounts hold.
- administrator - the administrator of that service.
The exact role names are chosen by whoever deployed this instance. Your own two are shown in the Account menu. An account that holds neither service's roles can sign in but cannot use the application: Home says the roles are missing, and only an administrator can grant them.
A platform administrator is an account the Account Server reports as carrying administrative privilege across the platform. Actions that only a platform administrator can ever take are hidden from everyone else rather than shown disabled, so an ordinary account never meets them.
Organisations, units and projects
An organisation (e.g. your company or university) is the top of the ownership tree, and it is the identity you work under. The organisation you are working as is shown in the masthead, and it is the only thing about where you are that the application remembers between visits.
Within an organisation are units (e.g. a research group or department). A unit is a billing container: it owns projects and datasets and pays for what they use. A unit is not something you browse in. You choose one while creating a project or uploading a dataset, and at no other time.
A project is where you do some work and collaborate with others, and a project belongs to a unit. A project isolates your work and lets you give specific people access to it.
The project you are working in is always the one in the address bar. There is no separate context to set and nothing that swaps the project underneath a page you already have open, so the page you are looking at, the data it shows, and the link you can share always agree.
There is a Default organisation which all users can belong to. In it you can create your own personal unit — one unit, yours alone — and create projects inside it. You are not given a personal unit automatically; you create it, and the Getting started guide walks through doing so. Alternatively, an organisation can be set up for you and your coworkers, or a complete Squonk 2 environment.
Owners and members
An organisation and a unit each have an owner — the one account recorded as owning it — and a list of members. Neither is a role you hold in a project; they decide what you may create and change in the ownership tree above one:
- Creating a unit in an organisation needs membership or ownership of that organisation.
- Creating a project in a unit needs membership of the unit or of its organisation.
- Deleting a unit is the unit owner's to do.
Two special cases follow from the Default organisation. It does not have members, and the members of a personal unit cannot be changed — it is yours alone by definition.
You do not need to be a member of a unit in order to use a project inside it. Access to a project is granted in the project itself, described next.
Access control
Only you and the people you have added to your project can access the files in it. Projects can be shared with users outside your organisation. In a public project all users are effectively observers.
The roles a person can hold in a project are:
- Creator - the person who created the project. The creator is recorded as a fact of the project and is shown in its Manage section, and carries no authority of its own.
- Administrator - can administer the project, including changing its privacy, its membership, and deleting it. A project can have multiple administrators.
- Editor - can use the project and modify the data in it, and can run work in it.
- Observer - can view the contents of the project, but cannot change the data or run anything.
Where you hold more than one of these, lists and headings state the highest role you hold.
Throughout the application, an ordinary action your roles and memberships do not allow is shown disabled with its reason, rather than removed, so it is clear what a task requires. These markings decide what is offered; the Data Manager and the Account Server decide what is actually permitted.
File access is controlled by standard POSIX permissions, with files typically being read and writable by the user and the group. Each user has a unique user id and each project has a unique group id, with all users added to the project being members of that group. This means that, with the file permissions described above, users added to your project have read-write access to your files. Other users will be able to see the files if the project is a public project but not if it is private.
Workspaces and sections
The application has three workspaces, reached from the main navigation:
- Projects - the index of the projects you can reach, and the way into one.
- Datasets - every dataset you can reach.
- Administration - organisations, units, subscriptions, charges and usage.
A project has four sections: Files, Run, Results and Manage. They are sections of the one project named in the address bar, so nothing you see in them comes from another project.
The project's files
A project can be thought of as a bit of disk space accessible only to users of that project. That disk space is accessible when you run workflows, applications and jobs (see below). You have complete control over what you do in your project.
Files can be added to a project so that you can work on them. Dataset versions can also be attached to a project, and project files can be added as new datasets. Files will also typically be created in your project by the work you run there.
Datasets
Datasets are files that you can work with. You don't need to use datasets to do some work in the Data Manager, but using them has the following benefits:
- Datasets can be shared with other users.
- Datasets can be versioned.
- Metadata is indexed e.g. descriptions and data types of the fields in a SD-file.
- Datasets with molecules (e.g. SD-files) are indexed with the RDKit cartridge, which in future will allow structure searches to be performed to identify the use of molecules, or similar molecules in other datasets.
A dataset is reachable globally rather than from inside one unit, and a version is the thing actually displayed. See the Datasets guide.
Workflows, applications and jobs
Three kinds of thing can be run in a project. Together they are what a project's Run section offers, and each card there states which kind it is.
An application is something that can be launched and run for a period of time (including several
days). The application typically has access to your project's bit of disk space so can work with its
files. Two are currently available: JupyterNotebook, where you can start a notebook to work with
your files — the .ipynb file is stored in your project too, providing a record of your work — and
DataVisualisation, for building interactive plots of your data.
A job is a bit like a very specialised application. Jobs are designed to process data in a very specific way, but allow you to specify the parameters of how the processing is performed. There is a wide range of jobs, and a job is published in one or more versions.
The max-min-picker job illustrates this. It selects a diverse subset of molecules from a set of input molecules, and running it prompts for parameters such as the inputs and the number of molecules to pick.
That particular job is implemented as a Python module using RDKit, but there are lots of ways of creating jobs; a guide to creating them is here. Creating jobs is a role for a developer, but using them once they have been deployed is very easy.
A workflow is a defined sequence of steps run as one thing. Like a job it is launched from a project's Run section with the inputs and options it declares, and its progress is followed step by step under Results.
Like applications, jobs and workflows run as Kubernetes pods. Some jobs are very simple and complete in seconds, whilst others can run complex parallelised workflows and take hours to complete.
Third parties are encouraged to contribute jobs, and there is a mechanism to monetise a job, so that its author is paid when other users run it. See Resource usage below.
Instances, tasks and running workflows
What you launch produces a result, and there are three kinds. An instance is one run of an application or a job. A task is a unit of work the Data Manager carries out on your behalf, such as attaching a dataset version to a project. A running workflow is one execution of a workflow definition.
Together these three are a project's Results. Individually they are never called a "job run", because the three behave differently and a project's Results list can be narrowed to any of them. See the Results guide.
Subscriptions and project tiers
Creating a project takes out a subscription in the unit that owns it, and the subscription is allocated a tier:
- Evaluation - limited quota and all data is public
- Bronze - limited quota and data can be made private
- Silver - like bronze but with more quota
- Gold - like silver but with more quota
Evaluation projects are always public and visible to all users, but are free to use. Projects in other tiers can be made private and restricted to specific users.
A subscription is the Account Server's product. "Product ID" is still shown in technical details and in API contracts, but everywhere in this application it is called a subscription.
A subscription states two figures that bound what can be spent against it:
- Its allowance - the coins included each billing period. Spending beyond the allowance is permitted, but those coins cost more.
- Its limit - the ceiling, which cannot be exceeded at all.
A dataset store is subscribed in the same way, and its allowance is chosen when the subscription is created. An allowance can be increased but never reduced. See the Administration guide.
Resource usage
An organisation deploying a Squonk 2 environment to its own Kubernetes cluster — or having Informatics Matters do so — is already paying for the CPU and memory resources being used. On the public evaluation site, Evaluation tier projects are free, but all data in them is public and so visible to any user, and they have limited resource quotas. Keeping data private, or needing more resources, means subscribing the project to the Bronze, Silver or Gold tiers, which differ mostly in the usage quotas, with Gold having the highest.
Billing is not yet implemented. Once it is, Bronze, Silver and Gold tier projects will carry monthly fees. The responsibility for payment is at the unit or organisation level: for your own personal unit in the Default organisation, that means you.
Usage is measured in Squonk coins (or coins for short). A coin has a monetary value, and a project's subscription carries an allowance of them each billing period, the biggest allowance going to Gold. Coins are consumed by three activities:
- Storage used by your project (e.g. the size of all the files in GBs)
- Execution of workflows, applications and jobs
- Dataset storage
Storage is calculated and recorded at regular periods each day, and the maximum daily usage is what deducts coins from the project's account. A linear projection of what the current usage will cost by the end of the billing month is also made, though it is only an estimate. A project's Manage section states both, as Storage and Predicted spend.
Execution is handled differently. Each time something is run it can consume coins that reflect:
- The CPU, GPU and memory resource needed for execution
- Any IP or licensing costs
Most jobs are free, or cost so little that it has no impact. Some — virtual screening using docking, say — consume a great deal of CPU and so have a real cost of execution. Others carry significant IP value, or use commercial software that has to be paid for, and can cost accordingly. This is what makes the "marketplace" possible: one user provides access to a job or application that is valuable to others, and can be rewarded for it if they wish. The mechanism for defining job usage is currently quite basic, and no payment systems are in place, so no bill can be run up today.
Dataset storage is consumed outside any project. Datasets are optional, but they sit in storage of their own, so a unit's costs include the datasets it holds. This is usually not much of an issue, though datasets of many GBs can affect what a unit spends.
Reaching the allowance does not stop work: coins consumed above it are added to the monthly bill at a higher rate than the tier's own. A subscription that should not run up an unexpected bill can be set not to exceed its allowance, or to exceed it only by a stated number of coins.
Reaching the limit does stop work. Operations that consume coins — adding files, running jobs — are disabled, and the project's Manage section says so, with each blocked control giving that as its reason. You can still read your data and delete files you no longer need, and you can move to a tier with a bigger quota.