Science Data Processor

SDP logo


The SDP is now is in full implementation mode of the Software Engineering Institute’s (SEI) Views and Beyond approach for the System Architecture.

After conducting various generic as well as SDP tailored Quality Attributes workshops earlier this year, teams identified quality attribute scenarios that are now being translated and documented into “Views”. Currently, the system level, Connector and Component (C&C) and Module decomposition and dependency views are under construction with the former stabilising prior to a formal, official review. Below are the latest draft versions of these views for the SDP.

Figure 1 – System level Connector and Component view – Primary representation

In addition to the System level, work on creating component level view documentation is simultaneously underway. The combination of both the System and component views is a key component within the System level Architecture document which in turn is a critical deliverable within the SDP CDR deliverable package.

The work conducted thus far on the Architecture documentation is pivotal in progressing towards a review milestone in November 2017, and paves the way for future releases within a phased SDP CDR.

Figure 2 – System level Module dependency and decomposition view – Primary representation


Commissioning (building & testing) of the SDP P3 (Performance Prototype Platform) Alaska hardware progressed through March and April 2017. The platform was subsequently made available to the first users during May. The P3 Alaska hardware is a significant and powerful resource for SDP and comprises:

  • 3 control node
  • 29 compute nodes
  • 2 high memory nodes
  • 2 GPU nodes
  • 1 NVMe storage mode
  • 2 SSD storage nodes
  • 5 ARM64 Ceph storage clusters

P3 is enabling vertical prototyping to test performance of products and horizontal prototyping to test interfaces. The findings from its initial usage were discussed at the Rotterdam engineering meetings and will be exposed to the International Supercomputing Conference in Frankfurt during June.

Other areas of activity to highlight are:

  • Significant progress on the SDP algorithm reference library (ARL) written in Python.
  • Further evolution and use of the SDP parametric model for studying opportunities for cost optimisation.
  • Focussed work on data models.
  • Defining and analysis of L3 RAM requirements.
  • Pioneering work using OpenStack for the provision of services, including logging capabilities, to enable performance assessments that will explore the limits of the SDP architecture.
  • Use of the SDP integration prototype (SIP) framework – an activity to study and evolve interfaces representing significant risk in the SDP architecture.
  • Deployment of the SDP SIP onto the recently commissioned performance platform to help explore SDP provisioning and analysis of how (for later sprints) it can allow cross-working with execution framework evaluations and ARL experiments.
  • Evaluation of different execution framework (EF) software at a modelling (of the data flows) level and by direct testing of framework overheads.
  • Building & testing of data transfer environments and tools to explore data delivery options.
  • Investigations of numerical precision requirements and word length for pipelines.
  • Defining quality attributes for SDP system software.Investigations into compute
  • Investigations into compute
  • Node requirements.

SKA Engineering Meeting 2017 – Rotterdam, The Netherlands

The SDP Consortium was well represented at the annual SKA Engineering meeting in Rotterdam with over 60 SDP representatives attending.

The SDP parallel sessions, and the SDP Consortium days, proved valuable for a whole Consortium catch-up on the progress of key work packages as well as providing the opportunity for in-depth face to face discussions on issues that required a smaller target audience for resolution and forward work planning purposes.

The next SDP Consortium F2F meeting date and location is under discussion.

Figure 3 – SDP Consortium Day Rotterdam 2017

Construction Planning

A topic of much discussion during the SDP Consortium day was the SDP Construction plan. The main points of discussion were planning for Construction based on a recently proposed SAFe (scaled agile) framework and the development of a Software Construction plan to align with this and a phased Software Architecture release approach.

Work has begun on the SDP Construction plan with the latest draft version completed in early June. Based on the information gleaned from the discussions mentioned above the plan will be further refined. Refinement work is also planned to commence imminently on the SDP Rollout Plan. The AIV Rollout Plan for LOW has been released and is currently under review by members of the SDP Consortium. This plan will provide valuable input and information for formulation of the SDP Rollout Plan.

SKA and CERN Cooperative Agreement

On the 14th July SKA and CERN joined a Cooperative Agreement that establishes a framework for collaborative projects seeking to address joint challenges in approaching Exascale computing and data storage.

Figure 4 – Dr. Fabiola Gianotti, CERN Director-General, and Prof. Philip Diamond, SKA Director-General, signing a cooperation agreement between the two organisations on Big Data.

The Square Kilometre Array (SKA) project and CERN’s Large Hadron Collider (LHC), the world’s largest particle accelerator, are two of the biggest generators of scientific data on the planet and as such share similar challenges as a result.

Both parties have identified the acquisition, storage, management, distribution, and analysis of scientific data as particularly burning topics to meet the technological challenges.

As part of the agreement, CERN and SKA will hold regular meetings to monitor progress and discuss the strategic direction of their collaboration. They will organise collaborative workshops on specific technical areas of mutual interest and propose demonstrator projects or prototypes to investigate concepts for managing and analysing Exascale data sets in a globally distributed environment. The agreement also includes the exchange of experts in the field of Big Data as well as joint publications.

Initial technical discussions have started and currently focus on OpenStack and in particular scalable bare metal deployments and containers on bare metal. Other areas high on the list are converged storage and Ceph with RDMA.

International Super Computing (ISC) Conference – Frankfurt – June 2017

Shortly after the SKA Engineering meeting concluded in Rotterdam many SKA and SDP representatives made the short trip across the border to discuss, engage and collaborate in all things “SuperComputing” at the International SuperComputing Conference in Frankfurt, Germany.

Selected representatives from the SKA and SDP Consortium held a joint briefing session to inform the vendor community of the latest progress information of the SDP project as we progress towards CDR, formation of the Inter-Governmental Organisation (which will run the observatory), and procurement. The briefing session also included current thinking on the technical directions of the HPC & data analytics aspects of the project and the progress the SDP Consortium has made since the ISC conference in 2016.

Further details on the briefing materials presented can be found at the following url:

Report provided by the SDP consortium