NERC Data Labs Phase 4

Concepts

• nerc
• wallingford
• software development package
• nerc data labs phase
• innovative use of nerc data
• part of the jasminx proposal nerc
• nerc data labs
• data assets
• novel use of nerc
• use of parallel data cluster services

Location

Description

NERC JASMIN Data Labs development capital plan As part of the JASMINX proposal NERC has provided 120K to enable further developments of the NERC Data Labs project. NRC DataLabs underpins a number of UKRI projects supporting innovative use of NERC data combined with new computational and analytical methods using JASMIN as the host computing platform (Digital Environment and Digital solutions) . In this way, Data Labs leverage novel use of NERC's data assets and reduce the technical barriers to exploitation of JASMIN investment by the wider research community. Currently, NERC Data Labs provide a range of functionality to support research collaboration including:- • Secure sign on and allocation of users to individual "data labs" where development projects are carried out on JASMIN • Flexible data storage using GlusterFS including use of parallel data cluster services such as DASK and Spark • Use of the executable notebook technologies such as Jupyter, Zepplin and R • Hosting of lightweight applications such as RShiny to allow rapid collaboration among diverse research teams • Containerisation of application deployment (e.g. using Docker) so that technologies developed can be more easily moved to other cloud platforms as required. NERC Data Lab development will move the NERC Environmental Data Service toward the paradigm of data centres as data analytics service providers where the variety and volume of data available to researchers require them to have more flexible, cloud-based research environments where analytical methods are move to and run on-demand against the dynamic data resources within the data centre rather than repeatedly downloaded. Hence, the continued investment in Data Labs development is key to enable the NERC EDS to meet new demands of research for new data intensive and transdisciplinary research (e.g. as promoted by UKRI SPF calls). Profile of work 1. Provide a visual workflow engine interface to be able to build, share and publish workflows in DataLabs. Provide an editable visual representation of the workflow. This will be dependent on funds and may have to be a scoping exercise but we can learn from the DAFNI development 2. Improve the user interface so users can see their compute usage and clean up their resources once their jobs have completed 3. Provide tighter coupling and file level control (ACL's) between the Caringo petabyte object store and Datalabs. This will reduce the requirement for users to have a JASMIN account to access the object store and allow access to data stored in the Group Workspaces. This is dependent on JASMIN infrastructure resource. 4. Provide automated sign up to DataLabs 5. Provide dashboarding solutions to enable customisable front ends to Jupyter notebooks using tools such as Panel, Voila and Dash

Award Detail

CPV Codes

• 48983000 - Development software package

Indicators

• Contract is suitable for SMEs.

Other Information

Tessella has been continually active within the climate science, earth observation and environmental science domain for the past twenty years. They have worked extensively with NERC Centres including: British Atmospheric Data Centre (BADC), Centre for Environmental Data Analysis (CEDA) and UK Centre for Ecology and Hydrology (UKCEH). In 2014 Tessella were one of the founding user communities on JASMIN, going on to establish a range of new capabilities within JASMIN's offering, and have since been involved in establishing, defining and implementing Datalabs in collaboration with UKCEH since its inception in 2017. To continue the successful development of DataLabs we require: • An understanding of the existing Datalabs solution in order to minimise project start-up activities, and maximise knowledge sharing within the joint NERC-JASMIN implementation team. • Ability to interface between the science-focussed research community and technically-focussed implementers to define and prioritise activities supporting the key deliverables of this phase. • Ability to work collaboratively with NERC/UKCEH project leaders to initially support definition and prioritisation of a product backlog reflecting the project's key focusses, then provide strategic technical and implementation advice to minimise delivery risk against a novel technology stack. • Demonstrate rigorous software engineering practices combined with a strong devops mentality to ensure Datalabs offers a robust solution capable of supporting a large community of researchers. • Creative, vendor-independent solutions to technical challenges while not losing sight of the core robustness and supportability of the Datalabs offering for many years into the future. • Ability to work collaboratively with STFC internal infrastructure operations team. • Leveraging existing research community Datalabs champions and developing new scientific use cases to showcase Datalabs' capabilities. Tessella have worked with us on this DataLabs development since 2017. They have demonstrated those requirements stated above during the various phases of DataLabs. In addition Tessella have been heavily involved in the technical implementation of the DAFNI project (Data & Analytics Facility for National Infrastructure), a UKRI facility hosted on JASMIN. It is important that we develop methods to link these two analytical platforms and Tessella with a knowledge of both will be able to facilitate this Several previous phases of DataLabs have been procured through competitive tenders and in each case Tessella was the successful bidder. The most recent phase was competed through the Digital Outcomes and Specialists Framework, which includes several thousand suppliers: there were three completed applications for the tender and Tessella offered the best technical solution and the best value for money. It is unlikely that a further competitive tender would