Development

Setting up the Devel Environment

To develop vallex-tools the following tools are needed or recommended.

  • A Python interpreter version at least 3.6; If your system provides one, you are good to go. However, when developing python applications for different versions of Python, it is helpful to have different interpreter versions and be able to switch between them. We therefore recommend installing pyenv which takes care of this.

  • A Python script installer We use several python programs which are only available as pypi packages. While they could theoretically be installed using pip, this would probably lead to dependency problems. We therefore strongly recommend installing a program which takes care of this. There are currently two such programs: pipsi and pipx. The first one is older (and available in system repos for later versions of ubuntu) but it is unmaintained. We therefore recommend installing pipx. To install a program with either of them run pipx/pipsi install PROGRAM. This will create a separate python virtual environment and install the program there. Finally it will link the program binary into ~/.local/bin

  • A Python dependency manager & build tool Poetry is used to take care of python dependencies and building the vallex-tools package. We use it to create a new python virtual environment in $REPO_ROOT/.venv and install all packages listed in poetry.lock into it. The vallex-cli program needs this environment to run.

  • A Git commit hook manager Pre-commit is used to manage commit hooks for the vallex-tools repository. The commit hooks are not strictly necessary, but are strongly recommended Before every commit they run lint programs (to check for python/javascript coding style), a static type checker to check that your code has correctly typed variables, and finally it runs a battery of (mostly) unit-test to test that your code does not, introduce old bugs.

  • A Javascript engine & package manager The NodeJS javascript engine & Yarn javascript package manager are used to build the frontend files. The frontend is written using the vue framework and is split into multiple files. These are then combined using javascript a compiler into a single javascript file which is put into $REPO_ROOT/vallex/server/frontend/dist*/. The compilation process is run using yarn (which also takes care of installing all the necessary javascript packages and the javascript compiler). Yarn is itself written in javascript so it needs a javascript engine (NodeJS in our case) to run.

  • A Selenium browser automation tools The Selenium browser automation tools are used to run automated tests of the web interface. The selenium python package is included in the dev-dependencies, however it additionally needs the browsers (Chromium & Firefox) to be installed and browser drivers for selenium. Also a virtual framebuffer X11 server is helpful to run the tests in headless mode (without opening a browser window).

We now include some basic instructions to install these tools on Debian/Ubuntu based systems. The repository also has a script which tries to do all of this automatically. It has been tested on Ubuntu 16.04 and 18.04. Run it with --help to see its options.

$ ./dev-setup.sh

Python interpreter

If Python 3.6 or later is not available, we use pyenv to install it

# Clone the pyenv repo
$ git clone https://github.com/pyenv/pyenv.git ~/.pyenv

# Define the environment variable PYENV_ROOT
$ echo 'export PYENV_ROOT="$HOME/.pyenv"' >> ~/.bash_profile
$ echo 'export PATH="$PYENV_ROOT/bin:$PATH"' >> ~/.bash_profile

# Add pyenv init to your shell
$ echo -e 'if command -v pyenv 1>/dev/null 2>&1; then\n  eval "$(pyenv init -)"\nfi' >> ~/.bash_profile

# Restart the shell & source profile
$ exec "$SHELL"
$ . ~/.bash_profile

# Install packages for building Python
$ sudo apt-get install -y make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev xz-utils tk-dev libxml2-dev libxmlsec1-dev libffi-dev liblzma-dev

# Install & activate Python 3.6
$ pyenv install 3.6.8

Python package manager

  1. Install pipsi (for installing python scripts from PyPi) and then poetry (for managing dependencies):

    $ sudo apt install pipsi
    $ pipsi install poetry
    

    and put ~/.local/bin into your PATH

  2. Install dev dependencies by going to the root of the repo (or wherever Pipfile is located in the repo) and run:

    $ poetry install --no-root
    

Git commit hook manager

$ pipsi install pre-commit    # install the pre-commit command
$ pre-commit install          # run this in the root folder of the repo
                              # to enable the commit-hooks for the repo

Javascript engine & package manager

  1. Install Node.js (see https://github.com/nodesource/distributions):

    $ DISTRO=$(lsb_release -s -c)
    $ curl -sSL https://deb.nodesource.com/gpgkey/nodesource.gpg.key | sudo apt-key add -
    $ echo "deb https://deb.nodesource.com/node_11.x $DISTRO main" | sudo tee  /etc/apt/sources.list.d/nodesource.list
    $ echo "deb-src https://deb.nodesource.com/node_11.x $DISTRO main" | sudo tee  -a /etc/apt/sources.list.d/nodesource.list
    $ sudo apt-get update
    $ sudo apt-get install nodejs=11.\*
    $ mkdir ~/.npm-packages
    $ echo "prefix=$HOME/.npm-packages" >> ~/.npmrc
    

    and put ~/.npm-packages/bin into your PATH.

    Note that currently only version ``11.*`` is known to work. Some later versions seem to have trouble compiling the node-sass dependency.

  2. Install the Yarn javascript package manager:

    $ npm install -g yarn
    
  3. Install the javascript packages

    $ cd vallex/server/frontend && yarn install
    

4. We also recommend to install the Vue.js devtools extension for Chrome or Firefox.

Selenium automation tools

  1. Install Xvfb, Chromium, Firefox and their drivers using system packages:

    $ sudo apt update && sudo apt install xvfb chromium-browser chromedriver firefox firefox-geckodriver
    

Development tasks

If you have make (e.g. GNU Make), the Makefile in the root of the project provides targets for running the following (and more) development tasks. The default target prints out a list of available targets and short description of each.

Create vallex/__version__

Both the vallex-cli and the frontend gui display version information which they find in $REPO_ROOT/vallex/__version__. After cloning the repo for the first time and whenever you commit to git you should run $REPO_ROOT/deploy/update_version.sh (current master) to update the version information. You can set-up a post-commit hook to do this: create a file $REPO_ROOT/.git/hooks/post-commit with the following content (don’t forget to replace $REPO_ROOT with the correct path:

#!/bin/bash
echo
echo -e "\e[33mvallex/__version__ (before running post-commit hook)\e[0m"
cat $REPO_ROOT/vallex/__version__
echo
$REPO_ROOT/deploy/update_version.sh
echo
echo -e "\e[33mvallex/__version__ (after running post-commit hook)\e[0m"
cat $REPO_ROOT/vallex/__version__
echo
echo

It is also meaningful to save the same script as post-merge (to update the version after any merge, including that resulting from running git pull), post-rewrite (to update the version after running git rebase), and post-checkout (to update the version info when you change branches).

However, the $REPO_ROOT/deploy/update_version.sh (current master) script also updates version information in the pyproject.toml file (current master), which is versioned in git, but the changes on the version="0.0" line should not be committed. To ignore changes on one particular line, we set up a git filter. Add this to your $REPO_ROOt/.git/config file:

[filter "gitignore"]
  clean = "sed 's/version=.*#gitignore\\s*$/version=\"0.0\"#gitignore/'"
  smudge = cat

or run this from the command line:

git config filter.gitignore.smudge cat
git config filter.gitignore.clean "sed 's/version=.*#gitignore\\s*$/version=\"0.0\"#gitignore/'"

(See commit 51342e2 for the related files in the repository.)

If, after setting up the configuration as suggested, git status reports that pyproject.toml is modified but git diff does not report any difference, running git add --renormalize might solve the problem. (See, e.g., an answer on Stack Overflow, a possible explanation and a much longer explanation of what’s going on.)

Re-compile javascript

Every time any of the javascript code (including any .vue , .css and other related files) is changed, you have to run the following in order for the GUI to reflect these changes:

$ cd vallex/server/frontend && yarn build

Debugging the frontend

Note that vallex-cli uses the optimized javascript code compiled by running yarn build. However, for debugging, the optimized code is useless. Instead, run the backend separately by issuing vallex-cli web; it will be started at http://localhost:8080 (or, more precisely, on the first free port >= 8080). Once the backend is ready (including parsing the lexicon files), run either of the following two lines

$ cd vallex/server/frontend && yarn serve
$ cd vallex/server/frontend && yarn run serve

This will start a webserver at http://localhost:8081/0/ (or maybe on the first free port >=8080), serving the static (non-optimized javascript) files, i.e. the frontend; there is automatic livereload for the frontend files (but you have to restart vallex-cli web whenever you change any of the python files). Both of these steps (starting backend AND frontend) are performed by the script quality/debug-frontend.sh or by issuing make webdev. However, in this way, the frontend and the backend run on different ports, which leads to some problems with accepting cookies. This problem has been addressed in commit 67f76fc; this and possibly other commits needed only during development are placed on the development branch.

Git branching structure

When committing or creating pull requests, please obey the following rules:

  • All changes that apply to all projects using PyVallex should go on the master branch (if you create a feature branch during development, we prefer that you rebase it on top of master and then fast-forward master to its top rather than creating a merge commit).

  • The master branch must be always deployable (pass all tests etc.)

  • Individual instances of PyVallex have their own branches, e.g. nomvallex_production (production environment for https://quest.ms.mff.cuni.cz/pyvallex). These branches merge in master as needed.

  • Never allow (multiple copies of) a single commit to appear on multiple branches; in other words, if you are tempted to cherry-pick, you probably should rebase (so that the commit appears on the most general branch where you want it, typically master) and then merge that branch into the other branches.

  • We recommend using gitk for exploring the branch structure.

Code Quality Control

Before submitting changes please make sure that your code

  1. passes all tests (run make test), if some code fails static checks (mypy) for good reasons, you can suppress warnings on a single line by putting the # type: ignore comment on that line)

  1. conforms to PEP8 code style (autopep8 is your friend)

Ideally, you can put all of these tests into pre-commit hooks and reject any commits which fail, e.g. using pre-commit, the repo has a .pre-commit-config.yaml included. To install pre-commit, provided you installed pipsi as recommended above, just run:

$ pipsi install pre-commit    # install the pre-commit command
$ pre-commit install          # run this in the root folder of the repo
                              # to enable the commit-hooks for the repo

To permanently disable the commit hooks in a repository checkout, run

$ pre-commit uninstall

Continuous Integration

Each time the master branch is pushed to on gitblab, a suite of jobs is executed in gitlab CI. These jobs are defined in the file .gitlab-ci.yml and include tests as well as deployment scripts and documentation builds.

Troubleshooting the CI pipeline

An error occurred while installing pyqtwebengine==5.12.1

fails with

[pipenv.exceptions.InstallError]: ['ERROR: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.'

then you have to add the newly encountered hash to the file Pipfile.lock. (This problem is caused by the lack of consideration on the part of pyqtwebengine developers, who release new versions without bumping the version number.)

Releasing PyPi packages

To release a new version of the PyPi package do the following:

  • Update the CHANGELOG.rst with the changes since the last release and commit the file. Running make changelog will output a template based on commit logs;

  • Run all tests make test and make sure that they pass

  • tag the release using git tag release/vVERSION_NUMBER where VERSION_NUMBER is the new version number following semver conventions

  • build the pypi package using make build_package, test it using twine check dist/* and then test that it is installable:

$ cd /tmp
$ mkdir test
$ cd test
$ virtualenv -p `which python3` venv
$ . venv/bin/activate
$ pip install $REPO/dist/vallex_tools*$VERSION_NUMBER*.whl
$ deactivate
$ rm -rf venv
$ virtualenv -p `which python3` venv
$ . venv/bin/activate
$ pip install $REPO/dist/vallex-tools*$VERSION_NUMBER*.tar.gz
$ deactivate
  • upload the package to PyPi using twine upload -s dist/* (eventually, this will probably be done automatically by the CI; I just need to figure out how to securely gpg sign the packages in the CI)

Howto Guides

See Howto for miscellaneous guides showing how different things are done in the code.

Indices and tables