November 12, 2022
The Edinburgh International Data Facility (EIDF) has combined Spectra Logic tape storage with Cerebras wafer-scale AI so that archived datasets can be mined for insights.
Spectra’s cold data storage tape technology is being used as the fuel for the new AI Wafer-Scale Engine tech from Cerebras, the CS-1, which comes with 400,000 cores built from 1.2 trillion transistors and a 100 petabit/sec interconnect. The Cerebras technology is built with an entire silicon wafer making the central processor, rather than cutting it up into individual chips. These Spectra and Cerebras systems are sited at the EIDF – part of Edinburgh Parallel Computing Centre (EPCC) at the University of Edinburgh.
EIDF is a regional data facility for use by researchers and data scientists. It houses regional, national, and international datasets stored in two Spectra TFinity tape libraries. Each is four frames (racks) in size and houses 2,020 tape slots and 24 LTO-8 tape drives, for a maximum 24PB in capacity. EIDF is actually using 20PB of that capacity.
Sitting between the Cerebras AI engine and Spectra archive stores are a mix of several HPE Apollo servers and one Superdome Flex server with 20PB ClusterStor E1000 Lustre parallel filesystem and Ceph block-access storage, with NVMe SSDs. The Superdome Flex server has 24x Xeon CPUs accessing a massive 18TB pool of memory, 102TB of SSD capacity, and 12 NICs providing 1.2Tbit/s bandwidth to the CS-1. The Apollos are running HPE’s Ezmeral Container Platform software with Kubernetes support and this enables self-service AI/ML applications for EIDF scientists, with use of accelerators such as GPUs.
EIDF uses HPE’s Data Management Framework software to enable hierarchical, tiered storage management with automated data movement between the TFinity tape libraries and the servers’ storage.
These TFinity libraries can scale out to 45 frames and store an exabyte of compressed LTO-8 data. If EIDF moves to LTO-9 then a a 45-frame TFinity could store an exabyte of uncompressed data. One library is a remote one and it is connected to the main site by a Bridgeworks WANrockIT link. This is a Fibre Channel to Fibre Channel connection over EIDF’s IP WAN which has been upgraded to handle 40Gbit/s transfers.
Mark Parsons, Director of EPCC, University of Edinburgh, said: “EIDF is combining computing and data resources to create a facility that will allow users to leverage data to innovate throughout their organizations. The TFinity tape systems provides the scalability that this project demands as it develops over the next decade.”
We are told that the EIDF is supporting Edinburgh in its ambition to become the data capital of Europe and provides a rich environment of data science, AI computing, and long-term data management services for many partners in the public, private, and academic sectors.
