Squonk (animal) logo with title text 'Squonk' and subtitle 'Data Manager'

Quick Links

A Squonk2 Deployment

What you see as the Squonk Data Manager when you use it from the web interface is actually a number of components that have been deployed to a Kubernetes cluster. We already have multiple deployments, but examples here refer to our reference site that is running on Amazon Web Services.

Other deployments would have basically the same components but at different URLs.

Systems Overview

Keycloak Single Sign On (SSO)

Authentication for all the Squonk products is handled through a Keycloak server which provides SSO capabilities using the OpenID Connect Identity provider. To do almost anything you must first be authenticated. Once you have signed in you get access to all the systems in that deployment that you are allowed access to without needing to log in again.

Data Manager API

This is an OpenAPI compliant REST API implemented in Python, and available also through a Swagger UI. It provides the majority of the functions that you see in the UI such as managing datasets and the contents of a project, executing jobs and applications and viewing the results.

The Swagger UI for the reference site can be found here.

Account Server API

This is a separate service that handles the concepts of users, organisations, units, accounting and billing. The idea is that multiple applications (e.g Data Manager API) would use the same Account Server to that the usage of a user, organisation and unit is consolidated in one place. A single subscription gets you access to multiple services.

Typically you don't notice the Account Server as a separate application as it would be seemlessly integrated into the client application, which is exactly how it works in the Data Manager web interface - you work with organisations and units but don't notice that this is actually being done through interactions with a separate service.

Account Server is also an OpenAPI compliant REST API implemented in Python, available through a Swagger UI.

The Swagger UI for the reference site can be found here.

Data Manager Web interface

If you are reading this document you are probably already using the web interface. This is a web application implemented using React and NextJS that uses the Data Manager and Account Server APIs. It too is deployed to the Kubernetes cluster. If you're not already there here is the location of the reference site.

Clients for programmatic access

We have created clients for working with the Data Manager API. If you are wanting to work with the API programmatically using this might be much easier than using the API directly, but not all functions of the API are available through the clients.

Python

A Python client has been created.

  • The docs are available here
  • An installable PyPi package can be found here

JavaScript / Node

We have also created a client for accessing the Data Manager API. This is used to create the Data Manager UI. It provides async functions to make request along with react-query hooks to handle query caching, refetching and so on. This package is generated automatically from the OpenAPI schema using Orval, and is fully typed with typescript. Requests are made with Axios which is compatible in NodeJS and not just in the browser.