In vjcitn/benchOOM: benchmarking out-of-memory strategies for R

Introduction

HDF Server is a Python-based web service that can be used to send and receive HDF5 data using an HTTP-based REST interface.

Setup the HDF server

Local Installation

Python packages required: NumPy 1.10.4 or later h5py 2.5 or later tornado 4.0.2 or later watchdog 0.8.3 or later * requests 2.3 or later (for client tests)

a.) Install Anaconda

  conda create -n h5serv python=2.7 h5py tornado requests pytz
  activate h5serv
  pip install watchdog

b.) Clone the hdf5-json project:

  git clone https://github.com/HDFGroup/hdf5-json.git
  cd hdf5-json/
  python setup.py install

c.) Clone the h5serv project:

 git clone https://github.com/HDFGroup/h5serv.git
 cd h5serv/server/
 python app.py

The server would start running. This would be indicated by the output - Starting event loop on port: 5000

AWS Instance

a.) Launch an AWS Instance

b.) Perform the above mentioned steps to install the HDF server in the instance.

c.) Run the server

  cd h5serv/server
  python app.py

The server would start running. This would be indicated by the output - Starting event loop on port: 5000

Verification

To verify that the h5serv was installed correctly:

1. Open a new terminal[for local installation] / Launch the AWS instance again [for installation on AWS instance]

2. Run Anaconda command prompt

  source activate h5serv
  cd h5serv/test
  python testall.py

This would run a number of tests to verify the installation.

Working with the HDF Server on an AWS Instance

Currently, the HDF server has been installed and is running on : AWS Public AMI id: ami-4e77ac58

a.)With this AMI, launch an AWS instance. ( ssh -i /path/to/keyfile.pem ec2-user@publicipoftheinstance )

  source activate h5serv
  cd hdf5-json/h5serv/server/
  python app.py

b.)Launch the instance again in a separate terminal.

  cd scripts
  python tall_test.py

Get data on the server

For example, if we want to get the assays.h5 file (for geuFPKM data) from our local machine onto the server running on an instance:

  scp -i/path/keyfile.pem /path/assays.h5 ec2-user@publicipoftheinstance:/hdf5-json/h5serv/data/

The file placed in the /data folder gets picked up by the server and it generates a domain name. e.g: for the assays.h5 file, domain name : assays.hdfgroup.org.

vjcitn/benchOOM documentation built on April 19, 2021, 11:20 p.m.