5. Developer guide

This developer guide is meant for software developers who would like to understand REANA source code and contribute to it.

5.1. Local development workflow

REANA cluster is composed of several micro-services with multiple independent source code repositories. reana-dev is a helper development script that facilitates working with multiple repositories for local development and integration testing purposes.

5.1.1. reana-dev

Run REANA development and integration commands.

How to configure your environment:

$ export REANA_SRCDIR=~/project/reana/src
$ export REANA_GITHUB_USER=tiborsimko

How to prepare your environment:

$ # prepare directory that will hold sources
$ # create new virtual environment
$ virtualenv ~/.virtualenvs/myreana
$ source ~/.virtualenvs/myreana/bin/activate
$ # install reana-dev developer helper script
$ pip install git+git://github.com/reanahub/reana.git#egg=reana
$ # run ssh-agent locally to simplify GitHub interaction
$ eval "$(ssh-agent -s)"
$ ssh-add ~/.ssh/id_rsa

How to fork and clone all REANA repositories:

$ reana-dev git-fork -c ALL
$ eval "$(reana-dev git-fork -c ALL)"
$ reana-dev git-clone -c ALL

How to install latest master REANA cluster and client CLI scripts:

$ reana-dev install-client
$ reana-dev install-cluster

How to compile and deploy latest master REANA cluster:

$ # install minikube and set docker environment
$ minikube start --kubernetes-version="v1.11.2" --vm-driver=kvm2
$ eval $(minikube docker-env)
$ # option (a): cluster in production-like mode
$ reana-dev docker-build
$ reana-cluster -f reana-cluster-latest.yaml init
$ # option (b): cluster in developer-like debug-friendly mode
$ reana-dev docker-build -b DEBUG=true
$ reana-cluster -f reana-cluster-dev.yaml init

How to set up your shell environment variables:

$ eval $(reana-dev setup-environment)

How to run full REANA example using a given workflow engine:

$ reana-dev run-example -c reana-demo-root6-roofit -w serial -s 10

How to test one component pull request:

$ cd reana-job-controller
$ reana-dev git-checkout -b . 72 --fetch
$ reana-dev docker-build -c .
$ reana-dev kubectl-delete-pod -c .

How to test multiple component branches:

$ reana-dev git-checkout -b reana-job-controller 72
$ reana-dev git-checkout -b reana-workflow-controller 98
$ reana-dev git-status
$ reana-dev docker-build
$ reana-dev kubectl-delete-pod -c reana-job-controller
$ reana-dev kubectl-delete-pod -c reana-workflow-controller

How to release and push cluster component images:

$ reana-dev git-clean
$ reana-dev docker-build --no-cache
$ # we should now run one more test with non-cached ``latest``
$ # once it works, we can tag and push
$ reana-dev docker-build -t 0.3.0.dev20180625
$ reana-dev docker-push -t 0.3.0.dev20180625
$ # we should now make PR for ``reana-cluster.yaml`` to use given tag
reana-dev [OPTIONS] COMMAND [ARGS]...



You can use --help option to see the detailed help for each command.

5.2. Debugging

In order to debug a REANA component, you first have to install REANA cluster in the development mode (see reana-cluster documentation). Once you have done this, you have to build the image of the component you are working on in development mode and we restart the corresponding pod:

$ cd src/reana-server
$ reana-dev docker-build -c . -b DEBUG=true
$ reana-dev kubectl-delete-pod -c .

Let us now introduce wdb breakpoint as the first instruction of the first instruction of the get_workflows() function located in reana_server/rest/workflows.py:


We can check that the code has been in fact updated and make a request to the component:

$ kubectl logs --selector=app="server"

DB Created.
 * Serving Flask app "/code/reana_server/app.py" (lazy loading)
 * Environment: production
 WARNING: Do not use the development server in a production environment.
 Use a production WSGI server instead.
 * Debug mode: on
 * Running on (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 221-564-335
$ curl $REANA_SERVER_URL/api/workflows?access_token=$REANA_ACCESS_TOKEN

After doing that we can go to the wdb dashboard (you can get wdb address using reana-cluster get wdb).


And finally select the debugging session.



It is not possible to get live code updates in workflow engine components since celery option –autoreload doesn’t work and it is deprecated. To debug celery right now:

  • Set breakpoint: import wdb; wdb.set_trace()
  • Kill the workflow engine container: kubectl delete pod cwl-default-worker-2461563162-r4hgg