In charlotte-ngs/lbgfs2020: Livestock Breeding and Genomics (Fall 2020)

knitr::opts_chunk$set(echo = TRUE)

Disclaimer

So far the exercise platform was run on a bare-metal installation of the rstudio server on a rented computing machine. This causes a lot of maintenance work, mainly after the semester when the course is over. The main work consists of stepping through user accounts and deleting them. To reduce this burden, we are looking for an alternative way which causes less administrative work.

Alternatives

The alternatives that we are testing consist of running an rstudio server inside of a docker container. There are different options for this.

One container instance per user
Single container instance with multiple users inside of that one instance

One Container Instance per User

From the point of view of separation and isolation of the data and configuration for the different users, it might be easier to set up a separate container for each user. The advantage of this approach is that we do not need any user management inside of the container. Furthermore, the home directory or any given subdirectory could be volume-bound to a directory on the host machine which would make the backing up of the user generated data much easier. The volume-binding would allow to do the deployment of new exercises and solutions on the host directly without running any commands inside of the docker container. The recovery of the whole system after a crash can be done by just re-starting the individual containers. All the user data would still be on the host which has already a running backup system in place.

But this solution requires a lot of work to be done on the host. The different running containers must be monitored and the setup process requires many details to be correct. Furthermore every instance must be bound to a different host-port and this must be communicated to the users which is error prone. It is much easier to have the same link for all the users which is not possible with this solution.

The following commands show how this can be run.

docker run -d -p 10087:8787 -e PASSWORD=pvrpass -v /Users/pvr/lbgfs2020/home/pvr:/home/rstudio --name pvr_rstudio rocker/verse

This pulls the docker image 'rocker/verse' and starts the container under the name 'rstudio'. The option '-d' tells docker to run the image in 'detached' mode. The status of the container can be checked with

docker ps -a

The running rstudio server can be accessed in the browser with http://localhost:10087

More users can be added by changing the host port and by changing the mapped volumen on the host. Adding users alice and bob might look as follows.

docker run -d -p 10088:8787 -e PASSWORD=alicepass -v /Users/pvr/lbgfs2020/home/alice:/home/rstudio --name alice_rstudio rocker/verse
docker run -d -p 10089:8787 -e PASSWORD=bobpass -v /Users/pvr/lbgfs2020/home/bob:/home/rstudio --name bob_rstudio rocker/verse

The usernames are taken from a list that is given as input. Then we loop over the usernames and start a docker container with a generated password. This password is then written to a file that can be used in a message to the students.

cd misc/20200918_exercise_platform
cp ~/Google\ Drive/Vorlesungen/LBG/FS2020/BelegungenLerneinheit_751630500L_Herbstsemester_2020.csv .
RESULTFILE=students_lbgfs2020.txt
if [ -f "$RESULTFILE" ];then rm $RESULTFILE;fi
cat BelegungenLerneinheit_751630500L_Herbstsemester_2020.csv | grep -v 'E-Mail' | while read s
do 
  name=$(echo $s | cut -d ',' -f1)
  firstname=$(echo $s | cut -d ',' -f2 | cut -d ' ' -f1)
  email=$(echo $s | cut -d ',' -f5)
  user=$(echo $email | cut -d '@' -f1)
  echo ${user},${email},${name},${firstname} >> $RESULTFILE
done

A test student is created manually.

echo pvr,peter.vonrohr@usys.ethz.ch,von Rohr,Peter > test_students_lbgfs2020.txt

This information can now be used for creating the docker instance.

#LBGHOME=/Users/pvr/lbgfs2020/home
LBGHOME=/home/quagadmin/lbgfs2020/home
RSTPORT=10087
cat test_students_lbgfs2020.txt | while read s
do
  user=$(echo $s | cut -d ',' -f1)
  email=$(echo $s | cut -d ',' -f2)
  name=$(echo $s | cut -d ',' -f3)
  firstname=$(echo $s | cut -d ',' -f4)
  pass=`tr -dc A-Za-z0-9_ < /dev/urandom | head -c8`
  dcmd="docker run -d -p $RSTPORT:8787 -e PASSWORD=$pass -v $LBGHOME/${user}:/home/rstudio --name ${user}_rstudio rocker/verse"
  echo $dcmd
  docker run -d -p $RSTPORT:8787 -e PASSWORD=$pass -v $LBGHOME/${user}:/home/rstudio --name ${user}_rstudio rocker/verse
  sudo chmod -R 777 $LBGHOME/${user}
  printf "To: $email \nSubject: Exercise Platform \n\nDear $firstname $name \nThis e-mail contains the required information for the exercise platform. \n\nPort: $RSTPORT \nPassword: $pass \n\nBest regards, Peter" > ${user}_email.txt
  # firewall
  sudo ufw allow $RSTPORT/tcp
  RSTPORT=$((RSTPORT+1))
done
# real students
RSTPORT=10088
cat students_lbgfs2020.txt | while read s
do
  user=$(echo $s | cut -d ',' -f1)
  email=$(echo $s | cut -d ',' -f2)
  pass=`tr -dc A-Za-z0-9_ < /dev/urandom | head -c8`
  dcmd="docker run -d -p $RSTPORT:8787 -e PASSWORD=$pass -v $LBGHOME/${user}:/home/rstudio --name ${user}_rstudio rocker/verse"
  echo $dcmd
  docker run -d -p $RSTPORT:8787 -e PASSWORD=$pass -v $LBGHOME/${user}:/home/rstudio --name ${user}_rstudio rocker/verse
  sudo chmod -R 777 $LBGHOME/${user}
  printf "To: $email \nSubject: Exercise Platform \n\nDear $firstname $name \nThis e-mail contains the required information for the exercise platform. \n\nPort: $RSTPORT \nPassword: $pass \n\nBest regards, Peter" > ${user}_email.txt
  # firewall
  sudo ufw allow $RSTPORT/tcp
  RSTPORT=$((RSTPORT+1))
  sleep 2
done

New student

LBGHOME=/home/quagadmin/lbgfs2020/home
cd /home/quagadmin/lbgfs2020
echo 'adrian.bertschi,adrian.bertschi@bsse.ethz.ch,Bertschi,Adrian' > new_student_lbgfs2020.txt
RSTPORT=10096
cat new_student_lbgfs2020.txt | while read s
do
  user=$(echo $s | cut -d ',' -f1)
  email=$(echo $s | cut -d ',' -f2)
  pass=`tr -dc A-Za-z0-9_ < /dev/urandom | head -c8`
  dcmd="docker run -d -p $RSTPORT:8787 -e PASSWORD=$pass -v $LBGHOME/${user}:/home/rstudio --name ${user}_rstudio rocker/verse"
  echo $dcmd
  docker run -d -p $RSTPORT:8787 -e PASSWORD=$pass -v $LBGHOME/${user}:/home/rstudio --name ${user}_rstudio rocker/verse
  sudo chmod -R 777 $LBGHOME/${user}
  printf "To: $email \nSubject: Exercise Platform \n\nDear $firstname $name \nThis e-mail contains the required information for the exercise platform. \n\nPort: $RSTPORT \nPassword: $pass \n\nBest regards, Peter" > ${user}_email.txt
  # firewall
  #sudo ufw allow $RSTPORT/tcp
  RSTPORT=$((RSTPORT+1))
  sleep 2
done

Stopping all instances

cat test_students_lbgfs2020.txt | while read s
do
  user=$(echo $s | cut -d ',' -f1)
  docker stop ${user}_rstudio
done
cat students_lbgfs2020.txt | while read s
do
  user=$(echo $s | cut -d ',' -f1)
  docker stop ${user}_rstudio
  sleep 2
done

docker rm $(docker ps --filter "status=exited" -q)

Sending E-Mails

#cat pvr_email.txt | ssmtp peter.vonrohr@usys.ethz.ch

#LBGHOME=/home/quagadmin/lbgfs2020/home
if [ ! -d "email_sent" ]; then mkdir -p email_sent;fi
cat test_students_lbgfs2020.txt | while read s
do
  user=$(echo $s | cut -d ',' -f1)
  email=$(echo $s | cut -d ',' -f2)
  echo " * Sending ${user}_mail.txt to $email ..."
  cat ${user}_email.txt | ssmtp $email
  sleep 2
  mv ${user}_email.txt email_sent
done
# students
cat students_lbgfs2020.txt | while read s
do
  user=$(echo $s | cut -d ',' -f1)
  email=$(echo $s | cut -d ',' -f2)
  echo " * Sending ${user}_mail.txt to $email ..."
  cat ${user}_email.txt | ssmtp $email
  sleep 2
  mv ${user}_email.txt email_sent
  sleep 2
done

Clone

LBGHOME=/home/quagadmin/lbgfs2020/home
cat test_students_lbgfs2020.txt | while read s
do
  user=$(echo $s | cut -d ',' -f1)
  cd $LBGHOME/$user
  echo " * Cloning repo for $user ..."
  git clone https://github.com/charlotte-ngs/lbgfs2020.git -b rexpf
  sleep 2
done
# students
LBGHOME=/home/quagadmin/lbgfs2020/home
cat students_lbgfs2020.txt | while read s
do
  user=$(echo $s | cut -d ',' -f1)
  cd $LBGHOME/$user
  echo " * Cloning repo for $user ..."
  git clone https://github.com/charlotte-ngs/lbgfs2020.git -b rexpf
  sleep 2
done

# new student
LBGHOME=/home/quagadmin/lbgfs2020/home
cd /home/quagadmin/lbgfs2020
cat new_student_lbgfs2020.txt | while read s
do
  user=$(echo $s | cut -d ',' -f1)
  cd $LBGHOME/$user
  echo " * Cloning repo for $user ..."
  git clone https://github.com/charlotte-ngs/lbgfs2020.git -b rexpf
  sleep 2
done

Single Container - Multiple Users

A single container is set up and running. Inside this container, separate accounts for all the users are created. The advantage of this solution is that the preparatory work on the host is minimal and all users can access the single container via the same link. Users do not have to worry about knowing their own port or their private link to their instance.

The problem with this solution is that the home directories of the individual user accounts cannot be volume-bound to a directory on the host machine. At least this is what is described under https://github.com/rocker-org/rocker/wiki/Using-the-RStudio-image. If the volume-binding is not possible, the backing-up of data and recoveries after a crash of the system are tedious tasks. Furthermore with this approach, users would need to remember usernames and passwords.