HPC Cluster Cloud Deployment with Bright Computing and Elastifile

On-Prem to Cloud Deployment of Elastifile 

Cloud Bursting?  Looking to extend HPC with Bright and Elastifile? Here are some points of consideration when using Bright Cluster Manager Portal and Elastifile as the extended file system for the datacenter to a cloud like Amazon Web Services (AWS) or Azure.

Elastifile can be deployed as bare metal, VM or container locally, and uses EC2 instances in the cloud.  It is a file system construct on top of an underlying object environment with file data characterized by: file metadata (File MD), File Mapping data objects (512 broken into 128 up to 4K data object OID), and file data objects.  Elastifile’s directory encoding using DirMD and DirListing OIDs are stored and accessed via hash table.  Variable length objects are used for both file data and metadata, however, only metadata utilizes caching.

In terms of security, there are no issues as it is the same filesystem replicated by CloudConnect and accessed via Network File System (NFS).

Both Ephemeral storage or Elastic Block Store (EBS) can be used.  Ephemeral storage being the higher cost, higher performance, non-persistent option compared to EBS.

Elastifile’s software architecture consists of three layers: protocol, metadata, and data.  The following diagrams are a guide to it’s underlying cloud architecture.

 

With Elastifile’s infrastructure designed to support optimization of cloud and hybrid cloud environments; Bright Cluster Manager can further simplify the process of deploying to the cloud.  Bright integrated with Elastifile allows organizations to harness the power of cloud without risking impact to current on-prem operations.  The following diagrams outline the cloud compute interaction and workload example using Bright and Elastifile.

Defined within SLURM Cloud Queue:

POWER ON CLOUD NODES
...Wait for notification...
NEED ELASTIFILE COMMAND TO CONFIRM \cloud DIR SYNC
...Wait for notification...
Run qsub job
...Wait for notification...
NEED ELASTIFILE COMMAND TO SYNC \cloud BACK
...Email user when complete...

 

 

 

 

By | 2018-07-20T08:34:18+00:00 July 20th, 2018|High Performance Computing|Comments Off on HPC Cluster Cloud Deployment with Bright Computing and Elastifile