5. Developer guide

This developer guide is meant for software developers who would like to understand REANA source code and contribute to it.

5.1. Local development workflow

REANA cluster is composed of several micro-services with multiple independent source code repositories. reana is a helper development script that facilitates working with multiple repositories for local development and integration testing purposes.

5.1.1. reana

Run REANA development and integration commands.

How to configure your environment:


$ export REANA_SRCDIR=~/project/reana/src
$ export REANA_GITHUB_USER=tiborsimko

How to prepare your environment:


$ # prepare directory that will hold sources
$ mkdir $REANA_SRCDIR && cd $REANA_SRCDIR
$ # create new virtual environment
$ virtualenv ~/.virtualenvs/myreana
$ source ~/.virtualenvs/myreana/bin/activate
$ # install reana-dev developer helper script
$ pip install git+git://github.com/reanahub/reana.git#egg=reana
$ # run ssh-agent locally to simplify GitHub interaction
$ eval "$(ssh-agent -s)"
$ ssh-add ~/.ssh/id_rsa

How to fork and clone all REANA repositories:


$ reana-dev git-fork -c ALL
$ eval "$(reana-dev git-fork -c ALL)"
$ reana-dev git-clone -c ALL

How to install latest master REANA cluster and client CLI scripts:


$ reana-dev install-client
$ reana-dev install-cluster

How to compile and deploy latest master REANA cluster:


$ minikube start --kubernetes-version="v1.11.2" --vm-driver=kvm2
$ eval $(minikube docker-env)
$ reana-dev docker-build
$ reana-dev docker-images
$ pip install reana-cluster
$ reana-cluster -f reana-cluster-latest.yaml init

How to set up your shell environment variables:


$ eval $(reana-dev setup-environment)

How to run full REANA example using a given workflow engine:


$ reana-dev run-example -c reana-demo-root6-roofit -w serial -s 10

How to test one component pull request:


$ cd reana-job-controller
$ reana-dev git-checkout -b . 72 --fetch
$ reana-dev docker-build -c .
$ reana-dev kubectl-delete-pod -c .

How to test multiple component branches:


$ reana-dev git-checkout -b reana-job-controller 72
$ reana-dev git-checkout -b reana-workflow-controller 98
$ reana-dev git-status
$ reana-dev docker-build
$ reana-dev kubectl-delete-pod -c reana-job-controller
$ reana-dev kubectl-delete-pod -c reana-workflow-controller

How to release and push cluster component images:


$ reana-dev git-clean
$ reana-dev docker-build --no-cache
$ # we should now run one more test with non-cached ``latest``
$ # once it works, we can tag and push
$ reana-dev docker-build -t 0.3.0.dev20180625
$ reana-dev docker-push -t 0.3.0.dev20180625
$ # we should now make PR for ``reana-cluster.yaml`` to use given tag
reana [OPTIONS] COMMAND [ARGS]...

Commands

docker-build
docker-images
docker-pull
docker-push
docker-rmi
git-checkout
git-clean
git-clone
git-diff
git-fetch
git-fork
git-push
git-status
git-upgrade
help
install-client
install-cluster
kubectl-delete-pod
run-example
setup-environment
version

You can use --help option to see the detailed help for each command.

5.2. Debugging

In order to debug a REANA component, you first have to install REANA cluster in the development mode (see reana-cluster documentation). Once you have done this, you have to build the image of the component you are working on in development mode and we restart the corresponding pod:

$ cd src/reana-server
$ # if we are not connected to the minikube docker daemon we should do it
$ eval "$(minikube docker-env)"
$ docker build . --build-arg DEBUG=true -t reanahub/reana-server:latest
$ kubectl delete pod --selector=app="server"

Let us now introduce wdb breakpoint as the first instruction of the first instruction of the get_workflows() function located in reana_server/rest/workflows.py:

_images/setting-the-breakpoint.png

We can check that the code has been in fact updated and make a request to the component:

$ kubectl logs --selector=app="server"

DB Created.
 * Serving Flask app "/code/reana_server/app.py" (lazy loading)
 * Environment: production
 WARNING: Do not use the development server in a production environment.
 Use a production WSGI server instead.
 * Debug mode: on
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 221-564-335
$ curl $REANA_SERVER_URL/api/workflows?access_token=$REANA_ACCESS_TOKEN

After doing that we can go to the wdb dashboard (you can get wdb address using reana-cluster get wdb).

_images/wdb-active-sessions.png

And finally select the debugging session.

_images/wdb-debugging-ui.png

Limitations

It is not possible to get live code updates in workflow engine components since celery option –autoreload doesn’t work and it is deprecated. To debug celery right now:

  • Set breakpoint: import wdb; wdb.set_trace()
  • Kill the workflow engine container: kubectl delete pod cwl-default-worker-2461563162-r4hgg