Central Signal Processor


Who are we?

The “Central Signal Processor” (CSP) Consortium is comprised of 13 signatories from 8 countries with more than 10 additional participating organisations. The Consortium includes a rich mixture of engineers, scientists and managers from various academic institutions, industry and government labs spread over 5 continents (see https://www.skatelescope.org/csp/ for more details). As might be expected, it has been a challenge to proceed efficiently with such a diverse and distributed team.

The lead organisation of the Consortium is the National Research Council of Canada (NRC). NRC has contracted MDA Systems Ltd. (MDA) to assist in leading the Consortium.

What are we designing?

The CSP Element includes the design of the hardware and associated firmware/software necessary for the generation of visibilities, searching for new pulsar candidates, and pulsar timing data from the telescope arrays. More background on the CSP can be found in the previous eNews submissions: http://newsletter.skatelescope.org/category/pdf-version-of-enews/

Current Status of Design Activities

Since the last eNews submission in August, the CSP Team has completed another round of costing in September and has been updating the ICDs and progressing the requirements to pave the way to CDR.

The sub-element design teams have continued to progress with detailed design and prototyping as they approach their CDRs. There has been continued activity on the system engineering side (requirements, ICDs, modelling, processes, standards, ILS/RAMs). There are still challenges in finalising the Level 1, 2, and 3 requirements required for efficient progression to CDR.

Key Sub-element Design Development

Local Monitoring and Control (LMC)

The CSP Local Monitoring and Control (LMC) Sub-element is responsible for coordinating all the CSP processing functions according to commands from the Telescope Manager (TM), reporting the overall CSP status based on the reports from the processing sub-elements, and configuring and sequencing the sub-elements. This sub-element is being led by NRC with assistance from INAF and NCRA. The CSP LMC team is actively supporting SKAO-led initiatives to define SKA standards and guidelines for implementation of the SKA Control System and the SKA software engineering process. The CSP LMC team is leading the effort on the definition of states, modes, commands and configuration and contributing to the definition of the design patterns for generation and handling of logs and alarms. Significant progress has been made on the definition ofinterfaces. The INAF team developed a prototype based on the SKA Control Systems Guidelines v1.0; work on the Prototype Report is well under way.

LOW Correlator and Beamformer (Low.CBF)

Since the SKA Engineering meeting in Rotterdam June 2017, the Low.CBF team has continued meetings, scheduled telecons, weekly status updates and ad hoc technical discussions. There has been steady progress in working towards CDR deliverables in parallel to addressing remaining cost control initiatives, assessing ECPs, reviewing requirements and working on internal and external ICDs.

A new FPGA code developer joined AUT in July. Norbert Abel, who came from industry (Endace Ltd) has joined Will Kamp, also from industry, to make-up AUT’s FPGA developer resource contribution. Norbert has been assigned to work on Low.CBF. AUT visited CSIRO in August to meet the Perentie team which was a good opportunity to review the coming two-year work plan, and map out FPGA code development schedules and resource assignments during the period between CDR and C0 start. The visit also coincided with two engineers from ASTRON working on site, Koos Kegel and Leon Hiemstra, working on FPGA code prototyping and System Engineering based CDR documentation deliverables.

Figure 1. Perentie Pre-construction workshop at CSIRO with team members present from AUT, ASTRON and CSIRO. Norbert is third from the left, Will fourth from right.

John Bunton, our Low.CBF & Low System Engineer, attended the Construction Schedule face to face meeting on 4 October. John presented the Low CSP plan for construction and attended a number of planning meetings.

Finalising ICDs and gaining closure for CDR submission has been problematic. The large workload across the consortium and looming deadlines has made it difficult for some to make closing off these documents a high priority, both intra-consortium, inter-consortium and SKAO.

The main prototyping activities over the last three months has been centred around Monitoring and Control (MACE) FPGA and software code integration, MACE software development, development in Tango GUI, Gemini GUI and protocol, Gemini support modules, Automated Register Generation prototyping and documentation and Gemini LRU cooling prototyping.

CDR documentation has progressed despite resources been extended to focus on L2 Rev 4 requirements reviews. This has now been baselined and the Low.CBF has been working on L3 flow-down. This is close to finishing which will enable submission for EA-1 Requirements Specification for sub-element CDR.

Detailed Design document has received great attention in the last month with a three-week workshop at ASTRON. This has progressed well in critical sections of the document and is now being consolidated with supporting sections and reference documents.

Figure 2. Koos Kegel, Agnes Mika, Yuqing Chen, Andre Gunst and John Bunton at the CDR workshop Oct/Nov 2017

The Perentie international collaboration involving AUT, ASTRON and CSIRO received an award from CSIRO for their efforts. The team has achieved much in the past two years and the future looks broader following CDR and into construction.

Figure 3. Grant Hampson and Andre Gunst sharing the CSIRO award for international collaboration.

CDR Element L2 documentation is progressing in parallel to L3 Sub-element documentation. CSIRO Low Systems Engineer is responsible to ensure completion for CDR submission in April 2018, whilst contributing to other CDR documents owned by others in CSP Consortium, e.g. EMI/EMC document, Power Quality documents, Safety Hazard documentation.

MID Correlator and Beamformer (Mid.CBF)

The Mid.CBF Sub-element is led by NRC and is based on a Stratix 10 FPGA solution. This is a joint effort with MDA and NZ Alliance.

With the Cost Control Project complete, the Mid.CBF team is working full speed ahead on the Frequency Slice Architecture design. The team has been working on two fronts: CDR deliverable documents and prototyping. CDR delivery will occur in two stages: a December 11th, 2017 pre-CDR delivery, followed by the complete CDR package delivered on January 22nd, 2018. Two key documents for the pre-CDR delivery, the Detailed Design Document and Signal Model, are nearing completion.

The focus of the Mid.CBF prototyping activities is as follows:

  • Firmware design to progress high risk areas of firmware development
    • Excellent progress made on implementation of the following key IP blocks:
      • X-Part correlator IP block
      • PSS beamformer block
      • Digital Resampler+Delay/Phase tracker IP block
      • Parameterizable Corner Turn in DDR4 IP bock
      • Channelizer and tunable filter IP blocks (Intel developing reference designs)
    • In all cases, resource usage and target clock rates are being met
  • Monitor and Control Software running on embedded processors
    • TANGO successfully running on Arria 10 embedded processor
    • Further progress made on Linux underlying drivers for M&C of FPGA IP blocks and FPGA configuration
    • TANGO devices and GUIs under development to complete M&C software stack
  • Hardware design of TALON-DX Processing Board
    • TALON-DX Processing board contains the following key components:
      • Intel Stratix 10 SX210 System-on-Chip FPGA
      • 4 x DDR4 DIMMs @ rates up to 2666 MT/s
      • 5 x FCI LEAP mid-board optical modules (12x25G bi-direction SERDES links each)
      • 2 x QSFP28 cages for external interfaces
    • Design complete and first prototypes received using an engineering sample Stratix 10 GX280 FPGA (no embedded processor).
    • Fabrication of TALON-DX boards with engineering sample Stratix 10 SX280 FPGAs (with embedded processor) planned for December.
  • Mechanical/thermal modelling and design work
    • Custom 2U 19” rack mount server box containing
      • 2 x TALON-DX processing boards
      • 1 x Simple power distribution / isolation module
      • 1+1 Redundant COTS ATX power supply
      • 4 x hot-swappable fans (LRU can function using only 3)
    • Detailed modelling of air-cooling which is now the baseline design.
    • Detailed design for the TALON LRU packaging.
    • Prototype server boxes being manufactured and expected in November

This past quarter, the hardware design for the TALON-DX board completed and two prototypes have been built using engineering sample Stratix 10 GX280 FPGAs (no embedded processor). The two boards have been received and are currently going through a “bring-up” procedure. So far, power rails have been verified and FPGA has been successfully programmed. SERDES/Optical Modules and DDR4 testing will commence shortly.

The Mid.CBF Team is on track for Sub-element CDR submission in January 2018, with the review to follow in March.

Figure 4. TALON-DX Signal processing board populated with DDR4, FCI LEAP Mid-board Optical Modules and QSFP28 100GBASE4-SR Modules on the bench at NRC.

Pulsar Search Engine (PSS)

The Pulsar Search Engine is a large sub-element of the CSP used to search for pulsars and fast transients that will have almost identical instances for both SKA Mid and Low.The design team is led by the University of Manchester, University of Oxford and the Max Planck Institute for Radio Astronomy supported by input from INAF Italy, NZ Alliance, ATC Edinburgh, and ASTRON.

In the last few months, we have been concentrating on preparing our documentation for CDR.

Our prototype PSS cluster, protoNIP, is deployed in the Karoo Array Processing Building. We have been developing the infrastructure and the software to be able to deploy our entire software stack across the nodes, another crucial learning step for the construction phase. This has been done in conjunction with SKA-SA who have provided excellent support.

We are continuing our work on the end-to-end pipelining of our software, this has enabled us to put together two complete pipelines, one for single pulse search and the other for time domain acceleration pulsar search. These are implemented in Cheetah and will soon be rolled out on protoNIP.

Our test vector machines and continuous integration process are now using a Virtual Private Network that allows us to virtually place our testing servers, physically located at our different institutes, on to the same virtual network. We are using this infrastructure to demonstrate how collaborative code development, continuous integration, and careful unit testing, allow us to develop high quality code across an international collaboration. We have also developed a set of web interfaces to enable the easy access to the test vectors for ourselves and collaborators.

Our work with Covnetics on the development of a version of the Fourier domain acceleration search on a FPGA based board continues. They have made excellent progress in the convolution modules and data throughput and extraction across the PCIe interface. We are interacting with them by reviewing intermediate documents and providing test vectors and cross-checking the output results. We are also working on the installation of the firmware on boards in Linux based PCs ready for use with protoNIP.

Pulsar Timing Engine (PST)

The Pulsar Timing Sub-element (PST) will perform high-fidelity, high-precision timing observations of known pulsars for both Low and Mid telescopes. The primary task performed by this instrument is phase-coherent dispersion removal, a computationally intensive algorithm that requires performing many large Fast Fourier Transform operations in real time. The PST design is based on commodity off-the-shelf (COTS) hardware with graphics processing unit (GPU) accelerators, and an early version of this solution is currently being commissioned at the MeerKAT telescope. We recently completed successful benchmarks of our prototype software on a system composed of an IBM POWER8 multiprocessor, NVIDIA P100 GPUs with dual-lane NVLink 1.0 high-speed interconnect, and Mellanox ConnectX-4 100 Gb Ethernet. These tests demonstrate that the PST design and prototype implementation are sufficiently flexible to adapt to future COTS architectures and exploit the advantages of procuring cutting-edge hardware during the construction phase. By the time this article is published, the PST team will have delivered all of our design artefacts for Critical Design Review in early February 2018.

Path to CDR

Overall, the CSP Consortium has made good progress since August. The focus is to “freeze” the requirements and ICDs to support efficient progression to sub-element CDRs followed by element level CDR. The team is focused and working hard to prepare the required materials.

Report provided by the CSP consortium