5. Developer guide

This developer guide is meant for software developers who would like to understand REANA source code and contribute to it.

5.1. Local development workflow

REANA cluster is composed of several micro-services with multiple independent source code repositories.

The main source code repository contains a Makefile which allows you to quickly clone all the necessary repositories and kick-start your REANA platform developments locally.

You can simply type make to see the available options and usage scenarios.

Description:

  This Makefile facilitates building and testing REANA on a local Minikube cluster.
  Useful for personal development and CI testing scenarios.

Available commands:

  build             Build REANA client and cluster components.
  ci                Perform full Continuous Integration build and test cycle. [main function]
  clone             Clone REANA source code repositories locally.
  deploy            Deploy/redeploy previously built REANA cluster.
  example           Run one or several demo examples.
  prefetch          Prefetch interesting Docker images. Useful to speed things later.
  setup             Prepare local host virtual environment and Minikube for REANA building and deployment.
  teardown          Destroy local host virtual environment and Minikube. All traces go.
  test              Run unit tests on the REANA package.

Configuration options:

  BUILD_TYPE              Is it a dev build or a release build? [default=dev]
  BUILD_ARGUMENTS         Space separated list of build arguments. [e.g. "COMPUTE_BACKENDS=htcondorcern" no build arguments are passed by default]
  CLUSTER_FLAGS           Which values need to be passed to Helm? [e.g. "debug.enabled=true,components.reana_ui.enabled=true"; no flags are passed by default]
  DEMO                    Which demo example to run? [e.g. "reana-demo-helloworld"; default is several]
  EXCLUDE_COMPONENTS      Which REANA components should be excluded from the build? [e.g. reana-ui,reana-message-broker]
  GITHUB_USER             Which GitHub user account to use for cloning for Minikube? [default=anonymous]
  INSTANCE_NAME           Which name/prefix to use for your REANA instance? [default=reana]
  MINIKUBE_CPUS           How many CPUs to allocate for Minikube? [default=2]
  MINIKUBE_DISKSIZE       How much disk size to allocate for Minikube? [default=40g]
  MINIKUBE_DRIVER         Which vm driver to use for Minikube? [default=virtualbox]
  MINIKUBE_MEMORY         How much memory to allocate for Minikube? [default=3072]
  MINIKUBE_KUBERNETES     Which Kubernetes version to use with Minikube? [default=v1.16.3]
  TIMECHECK               Checking frequency in seconds when bringing cluster up and down? [default=5]
  TIMEOUT                 Maximum timeout to wait when bringing cluster up and down? [default=300]
  SERVER_URL              Setting a customized REANA Server hostname? [e.g. "https://example.org"; default is Minikube IP]

Examples:

  # Example 1: set up personal development environment
  $ GITHUB_USER=johndoe make setup clone prefetch

  # Example 2: build and deploy REANA in production mode
  $ make build deploy

  # Example 3: build REANA except REANA-UI and REANA-Message-Broker
  $ EXCLUDE_COMPONENTS=r-ui,r-m-broker make build

  # Example 4: build and deploy REANA in development mode (with live code changes and debugging)
  $ CLUSTER_FLAGS=debug.enabled=true make build deploy

  # Example 5: build and deploy REANA with a custom hostname including REANA-UI
  $ CLUSTER_FLAGS=components.reana_ui.enabled=true SERVER_URL=https://example.org make build deploy

  # Example 6: run one small demo example to verify the build
  $ DEMO=reana-demo-helloworld make example

  # Example 7: run several small examples to verify the build
  $ make example

  # Example 8: perform full CI build-and-test cycle
  $ make ci

  # Example 9: perform full CI build-and-test cycle for a given release before publishing
  $ GITHUB_USER=reanahub BUILD_TYPE=release make ci
  $ # If everything goes well you can publish all the images to DockerHub
  $ reana-dev docker-push -t auto -u reanahub

  # Example 10: perform full CI build-and-test cycle in an independent cluster
  $ mkdir /tmp/nightly && cd /tmp/nightly
  $ git clone https://github.com/reanahub/reana && cd reana
  $ INSTANCE_NAME=nightly make ci
  $ INSTANCE_NAME=nightly make teardown
  $ cd /tmp && rm -rf /tmp/nightly

In addition, REANA comes with a reana-dev helper development script that simplifies working with multiple repositories during local development and integration testing. You can use --help option to see the detailed usage instructions.

5.2. Debugging

In order to debug a REANA component, you first have to mount REANA’s source code into Minikube’s /code directory and then build and re-deploy in development mode:

$ minikube mount $(pwd)/..:/code
$ CLUSTER_FLAGS=debug.enabled=true make build deploy

Let us now introduce wdb breakpoint as the first instruction of the first instruction of the get_workflows() function located in reana_server/rest/workflows.py:

_images/setting-the-breakpoint.png

We can check that the code has been in fact updated and make a request to the component:

$ kubectl logs --selector=app="server" -c server

DB Created.
 * Serving Flask app "/code/reana_server/app.py" (lazy loading)
 * Environment: production
 WARNING: Do not use the development server in a production environment.
 Use a production WSGI server instead.
 * Debug mode: on
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
 * Restarting with stat
 * Debugger is active!
 * Debugger PIN: 221-564-335

$ curl $REANA_SERVER_URL/api/workflows?access_token=$REANA_ACCESS_TOKEN

After doing that we can go to the wdb dashboard:

$ firefox http://`minikube ip`:31984
_images/wdb-active-sessions.png

And finally select the debugging session.

_images/wdb-debugging-ui.png

5.3. Port forwarding

If you ever need to access one specific microservice via HTTP there is a Kubernetes command that can help. The port-forward command connects a local port on the machine to a port on a Kubernetes pod. It directs the traffic reaching the local port to the pod port through an HTTP connection. Example:

$ kubectl port-forward --address 0.0.0.0 <pod_name> <local_port>:<pod_port>

The --address flag defines the local IP address to listen on. Using 0.0.0.0 makes the connection listen to all local IP addresses.