Development

Setting up the Devel Environment

To develop vallex-tools the following tools are needed or recommended.

  • A Python interpreter version at least 3.6; If your system provides one, you are good to go. However, when developing python applications for different versions of Python, it is helpful to have different interpreter versions and be able to switch between them. We therefore recommend installing pyenv which takes care of this.

  • A Python script installer We use several python programs which are only available as pypi packages. While they could theoretically be installed using pip, this would probably lead to dependency problems. We therefore strongly recommend installing a program which takes care of this. There are currently two such programs: pipsi and pipx. The first one is older (and available in system repos for later versions of ubuntu) but it is unmaintained. We therefore recommend installing pipx. To install a program with either of them run pipx/pipsi install PROGRAM. This will create a separate python virtual environment and install the program there. Finally it will link the program binary into ~/.local/bin

  • A Python dependency manager & build tool Poetry is used to take care of python dependencies and building the vallex-tools package. We use it to create a new python virtual environment in $REPO_ROOT/.venv and install all packages listed in poetry.lock into it. The vallex-cli program needs this environment to run.

  • A Git commit hook manager Pre-commit is used to manage commit hooks for the vallex-tools repository. The commit hooks are not strictly necessary, but are strongly recommended Before every commit they run lint programs (to check for python/javascript coding style), a static type checker to check that your code has correctly typed variables, and finally it runs a battery of (mostly) unit-test to test that your code does not, introduce old bugs.

  • A Javascript engine & package manager The NodeJS javascript engine & Yarn javascript package manager are used to build the frontend files. The frontend is written using the vue framework and is split into multiple files. These are then combined using javascript a compiler into a single javascript file which is put into $REPO_ROOT/vallex/server/frontend/dist*/. The compilation process is run using yarn (which also takes care of installing all the necessary javascript packages and the javascript compiler). Yarn is itself written in javascript so it needs a javascript engine (NodeJS in our case) to run.

  • A Selenium browser automation tools The Selenium browser automation tools are used to run automated tests of the web interface. The selenium python package is included in the dev-dependencies, however it additionally needs the browsers (Chromium & Firefox) to be installed and browser drivers for selenium. Also a virtual framebuffer X11 server is helpful to run the tests in headless mode (without opening a browser window).

We now include some basic instructions to install these tools on Debian/Ubuntu based systems. The repository also has a script which tries to do all of this automatically. It has been tested on Ubuntu 16.04 and 18.04. Run it with --help to see its options.

$ ./dev-setup.sh

Python interpreter

If Python 3.6 or later is not available, we use pyenv to install it

# Clone the pyenv repo
$ git clone https://github.com/pyenv/pyenv.git ~/.pyenv

# Define the environment variable PYENV_ROOT
$ echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bash_profile
$ echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bash_profile

# Add pyenv init to your shell
$ echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n  eval "$(pyenv init -)"\nfi' >> ~/.bash_profile

# Restart the shell & source profile
$ exec "$SHELL"
$ . ~/.bash_profile

# Install packages for building Python
$ sudo apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev

# Install & activate Python 3.6
$ pyenv install 3.6.8

Python package manager

  1. Install pipsi (for installing python scripts from PyPi) and then poetry (for managing dependencies):

    $ sudo apt install pipsi
    $ pipsi install poetry
    

    and put ~/.local/bin into your PATH

  2. Install dev dependencies by going to the root of the repo (or wherever Pipfile is located in the repo) and run:

    $ poetry install --no-root
    

Git commit hook manager

$ pipsi install pre-commit    # install the pre-commit command
$ pre-commit install          # run this in the root folder of the repo
                              # to enable the commit-hooks for the repo

Javascript engine & package manager

  1. Install Node.js (see https://github.com/nodesource/distributions):

    $ DISTRO=$(lsb_release -s -c)
    $ curl -sSL https://deb.nodesource.com/gpgkey/nodesource.gpg.key | sudo apt-key add -
    $ echo "deb https://deb.nodesource.com/node_11.x $DISTRO main" | sudo tee  /etc/apt/sources.list.d/nodesource.list
    $ echo "deb-src https://deb.nodesource.com/node_11.x $DISTRO main" | sudo tee  -a /etc/apt/sources.list.d/nodesource.list
    $ sudo apt-get update
    $ sudo apt-get install nodejs=11.\*
    $ mkdir ~/.npm-packages
    $ echo "prefix=$HOME/.npm-packages" >> ~/.npmrc
    

    and put ~/.npm-packages/bin into your PATH.

    Note that currently only version ``11.*`` is known to work. Some later versions seem to have trouble compiling the node-sass dependency.

  2. Install the Yarn javascript package manager:

    $ npm install -g yarn
    
  3. Install the javascript packages

    $ cd vallex/server/frontend && yarn install
    

4. We also recommend to install the Vue.js devtools extension for Chrome or Firefox.

Selenium automation tools

  1. Install Xvfb, Chromium, Firefox and their drivers using system packages:

    $ sudo apt update && sudo apt install xvfb chromium-browser chromedriver firefox firefox-geckodriver
    

Development tasks

If you have make (e.g. GNU Make), the Makefile in the root of the project provides targets for running the following (and more) development tasks. The default target prints out a list of available targets and short description of each.

Create vallex/__version__

Both the vallex-cli and the frontend gui display version information which they find in $REPO_ROOT/vallex/__version__. After cloning the repo for the first time and whenever you commit to git you should run $REPO_ROOT/deploy/update_version.sh to update the version information.

Re-compile javascript

Every time any of the javascript code (including any .vue , .css and other related files) is changed, you have to run the following in order for the GUI to reflect these changes:

$ cd vallex/server/frontend && yarn build

Debugging the frontend

Note that vallex-cli uses the optimized javascript code compiled by running yarn build. However, for debugging, the optimized code is useless.

Run either of the following two lines

$ cd vallex/server/frontend && yarn serve
$ cd vallex/server/frontend && yarn run serve

This will start a webserver at http://localhost:8080/0/, serving the static (non-optimized javascript) files, i.e. the frontend. Then we need to run the python backend server by issuing vallex-cli web. Opening http://localhost:8080/0/ should allow you to debug the frontend; there is automatic livereload for the frontend files (but you have to restart vallex-cli web whenever you change any of the python files).

Git branching structure

When committing or creating pull requests, please obey the following rules:

  • All changes that apply to all projects using PyVallex should go on the master branch (if you create a feature branch during development, we prefer that you rebase it on top of master and then fast-forward master to its top rather than creating a merge commit).

  • The master branch must be always deployable (pass all tests etc.)

  • Individual instances of PyVallex have their own branches, e.g. nomvallex_production (production environment for https://quest.ms.mff.cuni.cz/pyvallex). These branches merge in master as needed.

  • Never allow (multiple copies of) a single commit to appear on multiple branches; in other words, if you are tempted to cherry-pick, you probably should rebase (so that the commit appears on the most general branch where you want it, typically master) and then merge that branch into the other branches.

  • We recommend using gitk for exploring the branch structure.

Code Quality Control

Before submitting changes please make sure that your code

  1. passes all tests (run make test), if some code fails static checks (mypy) for good reasons, you can suppress warnings on a single line by putting the # type: ignore comment on that line)

  1. conforms to PEP8 code style (autopep8 is your friend)

Ideally, you can put all of these tests into pre-commit hooks and reject any commits which fail, e.g. using pre-commit, the repo has a .pre-commit-config.yaml included. To install pre-commit, provided you installed pipsi as recommended above, just run:

$ pipsi install pre-commit    # install the pre-commit command
$ pre-commit install          # run this in the root folder of the repo
                              # to enable the commit-hooks for the repo

To permanently disable the commit hooks in a repository checkout, run

$ pre-commit uninstall

Continuous Integration

Each time the master branch is pushed to on gitblab, a suite of jobs is executed in gitlab CI. These jobs are defined in the file .gitlab-ci.yml and include tests as well as deployment scripts and documentation builds.

Troubleshooting the CI pipeline

An error occurred while installing pyqtwebengine==5.12.1

fails with

[pipenv.exceptions.InstallError]: ['ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.'

then you have to add the newly encountered hash to the file Pipfile.lock. (This problem is caused by the lack of consideration on the part of pyqtwebengine developers, who release new versions without bumping the version number.)

Releasing PyPi packages

To release a new version of the PyPi package do the following:

  • Update the CHANGELOG.rst with the changes since the last release and commit the file. Running make changelog will output a template based on commit logs;

  • Run all tests make test and make sure that they pass

  • tag the release using git tag release/vVERSION_NUMBER where VERSION_NUMBER is the new version number following semver conventions

  • build the pypi package using make build_package, test it using twine check dist/* and then test that it is installable:

$ cd /tmp
$ mkdir test
$ cd test
$ virtualenv -p `which python3` venv
$ . venv/bin/activate
$ pip install $REPO/dist/vallex_tools*$VERSION_NUMBER*.whl
$ deactivate
$ rm -rf venv
$ virtualenv -p `which python3` venv
$ . venv/bin/activate
$ pip install $REPO/dist/vallex-tools*$VERSION_NUMBER*.tar.gz
$ deactivate
  • upload the package to PyPi using twine upload -s dist/* (eventually, this will probably be done automatically by the CI; I just need to figure out how to securely gpg sign the packages in the CI)

Howto Guides

See Howto for miscellaneous guides showing how different things are done in the code.

Indices and tables