Central Signal Processor


Who are we?

The “Central Signal Processor” (CSP) Consortium is comprised of 13 signatories from 8 countries with more than 10 additional participating organizations. The Consortium includes a rich mixture of engineers, scientists and managers from various academic institutions, industry and government labs spread over 5 continents (see https://www.skatelescope.org/csp/ for more details). As might be expected, it has been a challenge to proceed efficiently with such a diverse and distributed team.

The lead organisation of the Consortium is the National Research Council of Canada (NRC). NRC has contracted MDA Systems Ltd. (MDA a Maxar Company) to assist in leading the Consortium.

What are we designing?

The CSP Element includes design of the hardware and associated firmware/software necessary for the generation of visibilities, searching for new pulsar candidates, and pulsar timing data from the telescope arrays. More background on the CSP can be found in the previous eNews submissions: http://newsletter.skatelescope.org/category/pdf-version-of-enews/

Current Status of Design Activities

Since the last eNews submission in November the CSP Team has completed four out of the five sub-element CDRs, produced another costing update, updated various ICDs and progressed the requirements to pave the way to CSP CDR.

The sub-element design teams have continued to progress with detailed design and prototyping as they finalized their deliverables for their sub-element CDRs. There has been continued activity on the system engineering side (requirements, ICDs, modeling, processes, standards, ILS/RAMs). There are still challenges in finalizing the Level 1, 2, and 3 requirements required for efficient progression to CDR.

Key Sub-element Design Development

Local Monitoring and Control (LMC)

The CSP Local Monitoring and Control (LMC) Sub-element is responsible for coordinating all the CSP processing functions according to commands from the Telescope Manager (TM), reporting the overall CSP status based on the reports from the processing sub-elements, and configuring and sequencing the sub-elements. This sub-element is being led by NRC with significant contribution from INAF and assistance from NZA, MDA and NCRA. The CSP LMC team provided a significant contribution to the SKAO-led initiatives to define SKA standards and guidelines for implementation of the SKA Control System and the SKA software engineering process, in particular the definition of the SKA Control Model (including states, modes, commands and configuration) and the design patterns for generation and handling of logs and alarms. Significant progress has been achieved on the CDR Deliverables. The INAF team finalized prototyping activities and submitted the prototype report in February 2018. The Interface Control Documents for the interface with TM and with other CSP sub-elements have been re-structured to comply with the SKA TANGO API design patterns, updated to comply with the latest version of the CBF, PSS and PST designs (as submitted for respective sub-element CDRs), and signed. The CSP LMC requirements have been updated to comply with the CSP Requirement Specification Rev 5 and released for internal CSP review. Significant progress has been achieved on the verification requirements which are currently being reviewed by the CSP LMC team. The risk registry and cost have been updated in February 2018, as per the latest SKAO instructions. All this preparatory work allowed the CSP LMC team to finalize and firm-up the design and achieve significant progress on the Detailed Design Document.

LOW Correlator and Beamformer (Low.CBF)

Since the last newsletter in November 2017 much progress has been achieved on the Low.CBF sub-element, both at the formal documentation level as well as the prototyping level. The team pulled together and worked on their CDR submission containing approximately ten core documents as well as a Work In Progress (WIP) package containing more than 40 documents. The core documents of the submission were requirements and their verification, detailed design, prototyping, signal modelling, development plan, risk, cost, logistics and safety. The WIP file contained mainly prototyping result documents. The documents were completed and submitted on the 26 January 2018 to allow the reviewers the time to review and comment before the face-to-face meeting on March 8, 9 and 13th. Over 400 Observations Action Registers (OARs) were recorded, most of which were addressed before the meeting, and some of the hairier ones discussed during the meeting.

Figure 1. Gijs Schoonderbeek (ASTRON) presenting the new monolithic liquid cooled Gemini heatsink at the Low.CBF Sub-element CDR

The Low.CBF Sub-element CDR was conducted at the SKAO headquarters. Alain Baudry and Stephen Ord were the external reviewers and provided valuable feedback on the design. Thank you also to the Mid.CBF team who had conducted a cross-review which added perspective to the review. The structure and content of the review package was commended and supported an effective review. Wallace Turner was the chair and he led us through the process efficiently. Philip Gibbs was our secretary. The review went very well ending with the initial panel feedback indicating that the final report (due in a few weeks) should indicate a pass with some OAR actions to resolve by element CDR. All in all it was a very pleasurable experience and valuable to receive feedback.

Figure 2. Side by side view of the Low.CBF (left) and Mid.CBF (right) FPGA based processing boards. There are many similarities between both solutions.

The CSP consortium ran a busy day at the SKAO on March 12th. The Low.CBF team attended various splinter sessions and further progressed CSP element level design, interface details and future project plans. Topics including element level requirements, LFAA interface, RFI mitigation, power quality, EMI/EMC, logistics and plan towards element CDR.

Although the sub-element CDR goal has been achieved, there is still work remaining closing out the remaining OARs, as well as the CSP element level CDR deliverables.

Figure 3. The Low.CBF Sub-element CDR team celebrating their success! From left to right: Grant Hampson, Yuqing Chen, Koos Kegel, Gijs Schoonderbeek, Andrew Brown, Agnes Mika and John Bunton.

In addition the team has been progressing the prototyping of the Low.CBF system. A Low.CBF Integration Test Facility (ITF) has been established at CSIRO with a liquid-air cooled rack, networking, server, automatic power transfer switch and liquid-to-liquid heat exchanger. Much has been already been learnt from the ITF and will continue to inform the design as the solution becomes more complete. A single Gemini FPGA LRU has been installed in the rack which can be controlled and monitored through a Python Command Line Interface (CLI). The server talks to the FPGA using a custom Gemini data protocol, for which an Automatic Register Generation System (ARGS) has been established. ARGS creates a transparent layer between the FPGA memory map and the interfacing software.

The Gemini board is also now being revised with the release of HBM enabled FPGAs. It is anticipated that the new LRU will be in the system later this year. To keep the team cohesive and moving in the same direction the Low.CBF team is having a face-to-face meeting in April to create a plan and specification of the Early Perentie Assembly (EPA). This will initially focus on writing FPGA code and testing of external interfaces using emulators.

MID Correlator and Beamformer (Mid.CBF)

The Mid.CBF Sub-element is led by NRC and is based on a Stratix 10 FPGA solution. This is a joint effort with MDA and NZ Alliance.

This past quarter, Mid.CBF delivered all sub-element CDR documents by the scheduled deadline of January 22, 2018. The package was reviewed by a panel consisting of five SKAO reviewers and three external reviewers, Jonathan Weintroub, David Fort, and Larry D’Addario. A three day review was held at SKAO in Manchester, March 5 – 7, 2018, involving presentations by the team and spirited discussion of the raised OARs. The team would like to thank all reviewers involved for their time and the valuable feedback on the review package. The review concluded with Mid.CBF receiving a full pass.

Figure 4. Mid.CBF CDR participants included six members of the Mid.CBF team, three external reviewers, several reviewers and observers from SKAO. Three members of the Low.CBF team also reviewed the Mid.CBF design package and attended the Mid.CBF CDR.

The status of key prototyping activities is as follows:

  • Firmware design to progress high risk areas of firmware development
    • Excellent progress made on implementation of the following key IP blocks:
      • Full cross-connect correlator IP block
      • PSS beamformer block
      • Digital Resampler+Delay/Phase tracker IP block
      • Parameterizable Corner Turn in DDR4 IP bock
      • Channalizer and tunable filter IP blocks (Intel developing reference designs)
      • In all cases, resource usage and target clock rates are being met
    • Design work started on the top-level correlator FPGA design for the FSP-Part
  • Monitor and Control Software running on embedded processors
    • TANGO successfully running on Arria 10 embedded processor
    • Linux underlying drivers for M&C of FPGA IP blocks and FPGA configuration complete
    • TANGO devices and GUIs for LED blinking IP block complete
    • Complete top to bottom M&C stack complete and functioning on Arria 10
    • Porting software to Stratix 10 embedded processors well underway
  • Hardware design of TALON-DX Processing Board
    • TALON-DX Processing board contains the following key components:
      • Intel Stratix 10 SX210 System-on-Chip FPGA
      • 4 x DDR4 DIMMs @ rates up to 2666 MT/s
      • 5 x FCI LEAP mid-board optical modules (12x25G bi-direction SERDES links each)
      • 2 x QSFP28 cages for external interfaces
    • First prototypes received and board bring-up complete
    • 26G and 100GbE tests completed with excellent bit error rates results
    • Fabrication of TALON-DX boards with engineering sample Stratix 10 SX280 FPGAs (with embedded processor) complete and boards are ready to start testing
  • Mechanical/thermal modelling and design work
    • Custom 2U 19” rack mount server box containing:
      • 2 x TALON-DX processing boards
      • 1 x Simple power distribution / isolation module
      • 1+1 Redundant COTS ATX power supply
      • 4 x hot-swappable fans (LRU can function using only 3)
    • Detailed modelling of air-cooling which is now the baseline design
    • Detailed design for the TALON LRU packaging
    • First prototype server boxes received and testing underway

With the Mid.CBF Sub-element CDR review successfully completed, the focus now shifts to supporting CSP Element CDR. The Mid.CBF team will take on a significant role in writing the CSP_Mid side of CSP CDR documentation set, while coordinating with the PSS, PST and LMC teams to ensure a complete, high quality review package is available on schedule. In parallel, the team will be addressing OARs from the Mid.CBF CDR and making any required updates to the documents.

Figure 5. The air-cooled TALON LRU custom 2U 19” rack mount server box containing two mechanical model TALON-DX boards for testing purposes. Fiber optical and copper Gigabit Ethernet connections are on the front panel with redundant hot-swappable power supplies and fans at the rear. The Mid.CBF TALON system will consist of 370 TALON LRUs.

Pulsar Search Engine (PSS)

The PSS has also been through the sub-element CDR process since the last newsletter, in fact they were first. The submission was achieved on time in December 2017. The total data pack submitted by the whole Non-imaging team included 12 core documents supported by 30 documents with a total of 1581 pages and additional spreadsheets. It was also supported by more than 150,000 lines of code all available through repositories for consideration and review. In January 2018 we received a few hundred OARs on our submission, covering the dozen primary documents. We had an intense week in which we had to review and respond to all of the OARs leading up to the CDR itself which was on the 29th and 30th January. The two-day review was supported by a presentation which helped illuminate and clarify some of the OAR issues that had been raised.

The LOW/MID.PSS Sub-element CDR was conducted at the SKAO headquarters and was externally reviewed by Cees Bassa and Matthew Kerr. They provided very valuable feedback on the design. Considerable effort was also put in by the LOW/MID.PST team in their cross-review which will harmonise the post-OAR documentation for both sub-elements. The structure and content of the review package was commended and supported an effective review, in particular the presentation which supported the review and provide early visibility of the OAR resolutions. Wallace Turner was the chair and he led us through the process effectively and efficiently. Philip Gibbs was our secretary and provide very good feedback, in particular on risks. The review went very well ending with the initial panel feedback indicating that the final report (due in a few weeks) should indicate a pass with some OAR actions to resolve by element CDR. These include minor amendments needed to some of our documents to highlight some design choices and the supporting evidence for them. This information was available in the document pack but needed to be made more visible. We also needed to provide more information on the so-called realizable model as part of the Signal Model.

As well as undertaking our own review the PSS team provided a cross-review of the PST documentation stack. This too was extremely valuable in seeing areas where our own documentation could be enhanced.

Since the sub-element CDR we have been working hard on addressing the OARs. This has included expanding the range of simulations undertaken for the Signal Model and folding in the Realisable Model elements that were previously in our proto-typing documents. We have also been updating our DDD and continuing to update requirements across the various levels. Prototyping has also continued apace. The Cheetah and Rabbit pipeline environments have continued to develop significantly with new functionality added as well as further abstraction allowing them to be used more widely. Integrating updated versions of the Single Pulse Search and Fourier Domain Acceleration Search pipelines in Cheetah has continued too. The first stage of the FPGA based prototype FDAS pipeline, which includes all elements up to the harmonic sum, has now been completed by Covnetics and is currently being tested with the appropriate test vectors. This has been developed for the Arria 10 based boards that are part of the protoNIP prototype located in the Karoo. We are currently working with Covnetics to define the second stage of work which will look at including the Harmonic Summing and also probably scaling to Stratix 10. Work on protoNIP has also continued and we are currently finalising the build, OS and software distribution systems for it. Ready for roll out of the prototype codes in time for element CDR.

Pulsar Timing Engine (PST)

The Pulsar Timing Sub-element (PST) will perform high-fidelity, high-precision timing observations of known pulsars for both Low and Mid telescopes. The primary task performed by this instrument is phase-coherent dispersion removal, a computationally intensive algorithm that requires performing many large Fast Fourier Transform operations in real time. The PST design is based on commodity off-the-shelf (COTS) hardware with graphics processing unit (GPU) accelerators, and an early version of this solution is currently in use at the MeerKAT telescope. In February, the PST design passed sub-element Critical Design Review and the design team is currently addressing all comments and questions raised by the review panel. We are particularly grateful to Matthew Kerr and Cees Bassa for undertaking additional responsibilities as external reviewers of the PST design documents.

Path to CDR

Overall, the CSP Consortium has made good progress since November. The focus now is to perform the cleanup following the sub-element CDRs and progress the Level 2 CSP CDR materials. We look forward to a successful CSP CDR meeting in September.

Report provided by the CSP consortium