Production Environment¶

Installation¶

Requirements¶

Python >= 3.6

NodeJs (> 11.1, < 12.1), Yarn

NGINX

git (and, optionally, svn for the lexicons repo)

Setup¶

First setup some environment variables to be used in the later steps:

$ VALLEX_USER=vallex
$ VALLEX_HOME=/home/www/vallex
$ SVN_USER=ufalr
$ SVN_PASS=
$ SVN_REPO_URL=https://svn.ms.mff.cuni.cz/svn/vallex/trunk
$ GIT_REPO_URL=
$ REPO_SUBDIR=aktualni_data/data-txt

Create a new user

# Create a user
$ sudo adduser --disabled-login --home $VALLEX_HOME $VALLEX_USER
$ sudo setfacl -dm"u:$USER:rxw" $VALLEX_HOME
$ sudo setfacl -dm"u:$VALLEX_USER:rxw" $VALLEX_HOME
$ sudo setfacl -m"u:$VALLEX_USER:rxw" $VALLEX_HOME
$ sudo setfacl -m"u:$USER:rxw" $VALLEX_HOME

Create the directory layout in the users’ home directory

$ cd $VALLEX_HOME

# ommit the last dir if using git for lexicon data
$ mkdir -p bin config _updates lexicons/$REPO_SUBDIR

# Clone the vallex-tools repo
$ git clone https://gitlab.com/Verner/pyvallex.git $VALLEX_HOME/code

## Get the lexicons
##
## Git instrucions:
#
# git clone --depth 1 --no-checkout --filter=blob:none $GIT_URL lexicons
# cd lexicons
# git checkout master -- $REPO_SUBDIR
#
## Svn instructions:
#
# sudo apt install subversion
# cd $VALLEX_HOME/lexicons/$REPO_SUBDIR
# svn co --username $SVN_USER --password $SVN_PASS $SVN_REPO_URL/$REPO_SUBDIR .

Create the python virtual environment and install the python dependencies

$ virtualenv --python=python3 $VALLEX_HOME/code/.venv
$ . $VALLEX_HOME/code/.venv/bin/activate
$ pip install poetry
$ cd $VALLEX_HOME/code
$ poetry install --no-root --no-dev
$ deactivate

Compile the frontend javascript

$ cd $VALLEX_HOME/code/vallex/server/frontend
$ yarn install
$ yarn build-server

Setup NGINX & systemd

# Install the systemd service definition
$ sudo cp $VALLEX_HOME/code/deploy/production/vallex.service /etc/systemd/system
$ sudo cp $VALLEX_HOME/code/deploy/production/update-lexicons.service /etc/systemd/system
$ sudo cp $VALLEX_HOME/code/deploy/production/update-lexicons.timer /etc/systemd/system

# Link in the admin scripts and make them executable
$ for scr in vallex-cli reload-code-changes.sh update-lexicons.sh; do
$   ln -s $VALLEX_HOME/code/deploy/production/$i $VALLEX_HOME/bin
$   chmod a+x $VALLEX_HOME/bin/$i
$ done

# Install the environment file
$ cp $VALLEX_HOME/code/deploy/production/vallex.env $VALLEX_HOME/config
$ sudo ln -s $VALLEX_HOME/config/vallex.env /etc/default/vallex

# also, modify it appropriately for your env, in particular, adding correct
# SVN_PASS and SVN_USER values

# Copy the nginx fragment to the nginx sites config directory
$ sudo cp $VALLEX_HOME/code/deploy/production/vallex.nginx /etc/nginx/sites-available/

# Allow nginx & vallex access to the run directory
$ sudo setfacl -dm"u:www-data:rwx" $VALLEX_HOME/run
$ sudo setfacl -dm"u:$VALLEX_USER:rwx" $VALLEX_HOME/run
$ sudo setfacl -m"u:$VALLEX_USER:rwx" $VALLEX_HOME/run
$ sudo setfacl -m"u:www-data:rwx" $VALLEX_HOME/run

7. Add the line include /etc/nginx/sites-available/vallex.nginx; into the server block of your site NGINX config. Now cross your fingers and run:

$ sudo systemctl enable update-lexicons
$ sudo systemctl enable vallex

Directory Layout of vallex users’ home¶

code¶

The git repo containing vallex-tools code.

code/.venv¶

The python virtual environment containing the required packages. To activate it in your shell, run source code/.venv/bin/activate.

run¶

A directory containing runtime files: gunicorn.sock (this is how nginx communicates with vallex server process), web-db (the “parsed” lexicon data shown by the service), vallex.log the server logs (see also journalctl --unit=vallex.service).

bin¶

Contains vallex-cli executable and helper scripts for administering the service

reload-code-changes.sh
should be run without sudo by a user with sudo privileges whenever the code directory changes (e.g. after pulling changes from git); note that it does not run git pull
update-lexicons.sh
should be run to update the lexicons (runs svn up or git pull, to update the lexicons dir; doesn’t restart the service), should be run without sudo by a user with write access to the lexicons directory and to run/web-db.

config¶

Contains the service configuration

vallex.env
environment variables; these are sourced by several of the admin scripts as well as by the systemd service
server.ini
the vallex server configuration (containing, e.g., the list of lexicons to load)

lexicons¶

A repository of the lexicon data

Maintenance¶

Updating the vallex-tools code.¶

Updating the running instance after you have edited the code or run `git pull` consists of the following steps:

update the version file vallex/__version__:

$ cd $VALLEX_HOME/code
$ ./deploy/update_version.sh

compile the frontend javascript files if they have changed:

$ cd $VALLEX_HOME/code/vallex/server/frontend
$ yarn install        # installs possible new required packages
$ yarn build-server   # compiles the javascript files

install possible new python requirements

$ cd $VALLEX_HOME/code
$ . $VALLEX_HOME/code/.venv/bin/activate
$ pip install poetry                  # poetry is not a dev dependency, the next step will uninstall it
$ poetry install --no-root --no-dev   # install dependencies needed for production (no devel dependencies)

migrate the database, if there were any changes to the db schema

$ $VALLEX_HOME/bin/vallex-cli web migrate_db

restart the vallex service

$ sudo systemctl reload vallex.service

The script $VALLEX_HOME/bin/reload-code-changes.sh runs all of these steps; it must be run with sudoer’s rights (in order to be able to call systemctl).

Manually updating the lexicon data¶

# Use svn/git to update the data repository
$ cd $VALLEX_HOME/lexicons
$ if ! svn up --username $SVN_USER --password $SVN_PASS; then                          # &> $UPDATE_LOG
$    git fetch --all                 # &>> $UPDATE_LOG
$    git reset --hard origin/master  # &>> $UPDATE_LOG
$ fi;

# Load any changes into the db.
# NOTE: if the db contains any changes to the lexicons (currently unimplemented)
# they will be saved

$ $VALLEX_HOME/bin/vallex-cli web sync_db
$ chmod g+w $VALLEX_HOME/run/web-db

Update the list of lexicons¶

When the list of lexicons to show in the ui needs updating (so that exactly those specified in the config are shown) the following needs to be run:

# NOTE: if the db contains any changes to the lexicons (currently unimplemented)
# they will be lost

$ $VALLEX_HOME/bin/vallex-cli web reset_db

# Because you have run the preceding command under your own username,
# the user `vallex` (i.e. the running instance) cannot write it.
# This can be solved by either running the above command as
# sudo -u vallex $VALLEX_HOME/bin/vallex-cli web reset_db
# or by additionally running
$ chmod g+w $VALLEX_HOME/run/web-db

Getting the logs¶

There are two kinds of logs.

First kind: all console output from the vallex.service is saved to system logs (this is handled by systemd). These can be displayed by running the command

$ sudo journalctl -xe --unit=vallex.service

# Or, for `tail -f` like functionality
$ sudo journalctl -f -xe --unit=vallex.service

The second logs are saved in a file specified in the server config ($VALLEX_HOME/config/server.ini). They are the output of the log calls (see Logging (Debug Output)).

Troubleshooting¶

1. Vallex service fails to start The first thing to do is to run

$ sudo systemctl status vallex.service

this should confirm that the service is not running and also print some basic error logs (see also Getting the logs for more logs). The following are typical problems:

Permissions. The service is run as the $VALLEX_USER user and it needs read access to everything under $VALLEX_HOME/code, $VALLEX_HOME/lexicons and to $VALLEX_HOME/config/server.ini; and read+write access to the $VALLEX_HOME/run subdirectory and all files contained in there.

About

Contents

Useful Links