5. Developer guide

This developer guide is meant for software developers who would like to understand REANA source code and contribute to it.

5.1. Local development workflow

REANA cluster is composed of several micro-services with multiple independent source code repositories. reana is a helper development script that facilitates working with multiple repositories for local development and integration testing purposes.

5.1.1. reana

Run REANA development and integration commands.

How to configure your environment:


$ export REANA_SRCDIR=~/project/reana/src
$ export REANA_GITHUB_USER=tiborsimko

How to prepare your environment:


$ # prepare directoru that will hold sources
$ mkdir $REANA_SRCDIR && cd $REANA_SRCDIR
$ # install reana developer helper script
$ mkvirtualenv reana
$ pip install git+git://github.com/reanahub/reana.git#egg=reana
$ # run ssh-agent locally to simplify GitHub interaction
$ eval "$(ssh-agent -s)"
$ ssh-add ~/.ssh/id_rsa

How to fork and clone all REANA repositories:


$ reana git-fork -c ALL
$ eval "$(reana git-fork -c ALL)"
$ reana git-clone -c ALL

How to install latest master REANA cluster and client CLI scripts:


$ workon reana
$ reana install-client
$ reana install-cluster

How to compile and deploy latest master REANA cluster:


$ minikube start --kubernetes-version="v1.11.2" --vm-driver=kvm2
$ eval $(minikube docker-env)
$ reana docker-build
$ reana docker-images
$ pip install reana-cluster
$ reana-cluster -f reana-cluster-latest.yaml init

How to set up your shell environment variables:


$ eval $(reana setup-environment)

How to run full REANA example using a given workflow engine:


$ reana run-example -c reana-demo-root6-roofit -w serial -s 10

How to test one component pull request:


$ cd reana-job-controller
$ reana git-checkout -b . 72 --fetch
$ reana docker-build -c .
$ kubectl delete pod -l app=job-controller

How to test multiple component branches:


$ reana git-checkout -b reana-job-controller 72
$ reana git-checkout -b reana-workflow-controller 98
$ reana git-status
$ reana docker-build
$ kubectl delete pod -l app=job-controller
$ kubectl delete pod -l app=workflow-controller

How to release and push cluster component images:


$ reana git-clean
$ reana docker-build --no-cache
$ # we should now run one more test with non-cached ``latest``
$ # once it works, we can tag and push
$ reana docker-build -t 0.3.0.dev20180625
$ reana docker-push -t 0.3.0.dev20180625
$ # we should now make PR for ``reana-cluster.yaml`` to use given tag
reana [OPTIONS] COMMAND [ARGS]...

Commands

docker-build

Build REANA component images.

docker-images

List REANA component images.

docker-pull

Pull REANA component images from DockerHub.

docker-push

Push REANA component images to DockerHub.

docker-rmi

Remove REANA component images.

git-checkout

Check out local branch corresponding to a…

git-clean

Clean REANA source repository code tree.

git-clone

Clone REANA source repositories from GitHub.

git-diff

Diff checked-out REANA local source code…

git-fetch

Fetch REANA upstream source code repositories…

git-fork

Display commands to fork REANA source code…

git-push

Push REANA local repositories to GitHub…

git-status

Report status of REANA source repositories.

git-upgrade

Upgrade REANA local source code repositories.

help

Display usage help tips and tricks.

install-client

Install latest REANA client Python package…

install-cluster

Install latest REANA cluster Python package…

run-example

Run given REANA example with given workflow…

setup-environment

Display commands to set up shell environment…

version

Return REANA version.

You can use --help option to see the detailed help for each command.

5.2. Debugging

In order to debug a REANA component, you first have to install REANA cluster in the development mode (see reana-cluster documentation). Once you have done this, you have to build the image of the component you are working on in development mode and we restart the corresponding pod:

$ cd src/reana-server
$ # if we are not connected to the minikube docker daemon we should do it
$ eval "$(minikube docker-env)"
$ docker build . --build-arg DEBUG=true -t reanahub/reana-server:latest
$ kubectl delete pod --selector=app="server"

Let us now introduce wdb breakpoint as the first instruction of the first instruction of the get_analyses() function located in reana_server/rest/analyses.py:

_images/setting-the-breakpoint.png

We can check that the code has been in fact updated and make a request to the component:

$ kubectl logs --selector=app="server"

 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 310-304-952
172.17.0.1 - - [15/Feb/2018 12:43:49] "GET /api/ping HTTP/1.1" 200 -
 * Detected change in '/code/reana_server/rest/analyses.py', reloading
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 310-304-952
$ curl "192.168.99.100:30659/api/analyses?organization=default&user=00000000-0000-0000-0000-000000000000"

After doing that we can go to the wdb dashboard (you can get wdb address using reana-cluster get wdb).

_images/wdb-active-sessions.png

And finally select the debugging session.

_images/wdb-debugging-ui.png

Limitations

It is not possible to get live code updates in workflow engine components since celery option –autoreload doesn’t work and it is deprecated. To debug celery right now:

  • Set breakpoint: import wdb; wdb.set_trace()
  • Kill the workflow engine container: kubectl delete pod cwl-default-worker-2461563162-r4hgg