UCAM 046/15 For the provision of SKA-Science Data Processor Object Storage Software and NVRAM Storag

Descriptions

The Square Kilometre Array (SKA, http://www.skatelescope.org/) is a major international project to build a next generation radio astronomy observatory. The anticipated construction cost of the full SKA is about 2 000 000 000 EUR. When completed, the SKA will be, by a large margin, the largest and most sensitive low-frequency radio observatory ever built.The design of the main elements of the SKA is being carried out by International Consortia, with the SKA Organisation coordinating the design process. The SKA Organisation is also the overall design authority for the SKA. The SKA Organisation is a UK company limited by guarantee and is based in the UK at Jodrell Bank in Cheshire.An integral part of the observatory is processing the data from the streams produced by digital signal processing into calibrated, usable science products. This processing is the responsibility of the ‘Science Data Processor’ (SDP) element of the SKA which is led by the University of Cambridge. The ‘Science Data Processing Consortium’ includes members on five continents and a mixture of Universities and government organisation and institutes.The deployment of the telescopes in SKA will be phased, with the first phase (SKA1) due to commence procurement in 2017. Until 2017 the SKA project is in a pre-production phase during which elements are undergoing design and technology is brought to production readiness level. The observatory will consist of telescopes which will be sited in Southern Africa and Western Australia. There will be a computing facility for Science Data Processing on each of the continents.The purpose of the work is to assess readiness of technology developments in storage for use in the SDP compute clusters. The applications running in the SDP clusters have an extreme need for high performance read I/O. The details of a preliminary design of the SDP can be in the Bibliography below. Additional information on software can also be found here.The SDP clusters may well have a hybrid storage system comprising ultra-fast node-local storage leveraging NVRAM, fast island-wide shared storage spanning a limited collection of nodes (e.g. 2 racks of computers may comprise an ‘island’) while a third storage system will provide an archive in which the science products will be stored. Given current industry trends, object stores are of considerable interest.The purpose of this project is to establish if combinations of existing technologies are feasible to achieve the following goals:— Can an application simultaneously use NVRAM storage as part of an object store and as memory class storage?— Can a fast RDMA interconnect be leveraged to access an object storage backend?— Layer a layout-aware I/O-interface, for example ADIOS, on an object store and demonstrate good performance?