info@thalesians.com 0208 200 2000

Software Requirements

Software Requirements

You are asked to bring your own laptop to our trainings. If no laptop is available, we recommend that you work in a pair with a Delegate who has access to a laptop. We encourage collaboration during the tutorials.

Python and Machine Learning (ML) courses

We expect that the delegates will be using the Anaconda Python distribution. One of its advantages is the ready availability of data science and machine learning libraries.

Please note that there are compatibility issues between the latest version of Anaconda, 2018.12 Python 3.7 version, and two of the deep learning libraries: TensorFlow and Keras. As far as we know these issues haven’t been resolved yet.

We therefore recommend that the delegates install the earlier, Python 3.6-based versionAnaconda3 version 5.2.0, which can be obtained from https://repo.anaconda.com/archive/

Depending on  the operating system, you will need to download and install one of the following packages:

  • Anaconda3-5.2.0-Linux-ppc64le.sh
  • Anaconda3-5.2.0-Linux-x86.sh
  • Anaconda3-5.2.0-Linux-x86_64.sh
  • Anaconda3-5.2.0-MacOSX-x86_64.pkg
  • Anaconda3-5.2.0-MacOSX-x86_64.sh
  • Anaconda3-5.2.0-Windows-x86.exe
  • Anaconda3-5.2.0-Windows-x86_64.exe

If the operating system is 64-bit, we recommend that the delegates use the 64-bit version. Thus, for 64-bit Windows OS, you would download and install Anaconda3-5.2.0-Windows-x86_64.exe

While it is possible to work with other Python distributions, the delegates are warned that incompatibilities may cause problems during the training.

If a particular library is missing, it can be installed into the Anaconda Python distribution using the command-line pipconda, or the visual Anaconda Navigator. Here we describe the process for Anaconda Navigator:

  • Go to Anaconda Navigator (in Windows it is accessible through the Start menu);
  • Go to the “Environments” on the left side of the screen;
  • Search for the missing package in “Search Packages” among “Not Installed” ones (use lowercase, e.g. “keras”);
  • Tick the packages and press “Apply” button for the package to be installed.

Python libraries

The following Python libraries should be installed by the delegates:

  • NumPy
  • Pandas
  • pandas-datareader
  • SciPy
  • StatsModels
  • Matplotlib (PyLab)
  • Seaborn
  • Plotly
  • Scikit-learn
  • TensorFlow
  • Keras

By default, all of these libraries are included in the Anaconda distribution, except for

  • pandas-datareader
  • Plotly
  • TensorFlow
  • Keras

These can be installed from the Anaconda Navigator.

You can use our python-module-tester.ipynb Jupyter notebook to test that all the libraries have been successfully installed into your Python distribution: https://github.com/thalesians/commons/blob/master/software-requirements/python-module-tester.ipynb

kdb+/q training courses

A current version of kdb+/q is required (at the time of writing, version 3.6). It can be dowloaded from
https://kx.com/connect-with-us/download/

Note that there are two versions: the 64-bit Personal Edition (on-demand) and the 32-bit Personal Edition.

The 32-bit Personal Edition will be sufficient for most of the lectures and tutorials, although some lectures cover integration with software products requiring the 64-bit Personal Edition (on-demand). You are therefore advised to download both these editions.

kdb+/q-related technologies

Your particular training course may or may not cover the following kdb+/q related technologies – please check the syllabus. Here you will have instructions for downloading and installing all of them.

exxeleron qPython

exxeleron qPython can be found here: https://github.com/exxeleron/qPython

To install it into your Python distribution, use your Python’s distribution’s

pip install qpython

In particular, we use the Anaconda Python distribution (see notes on Python above), and you can run pip from (Your anaconda distribution)/Scripts.

To test your exxeleron qPython installation, you can use our
https://github.com/thalesians/commons/blob/master/software-requirements/exxeleron-qpython-tester.ipynb

pyQ

Again, we are assuming that you have Python and pip.

In particular, we use the Anaconda Python distribution (see notes on Python above), and you can run pip from (Your anaconda distribution)/Scripts.

pip install pyq

Hopefully you will see something like:

Collecting pyq
 Downloading https://files.pythonhosted.org/packages/53/de/195ce13d50784489365fee87d1e7d637fa2e42f3c5ae359d56c13595c93c/pyq-4.1.4.tar.gz (100kB)
 100% |****| 102kB 931kB/s
 Building wheels for collected packages: pyq
 Running setup.py bdist_wheel for pyq … done
 Stored in directory: C:\Users\paul\AppData\Local\pip\Cache\wheels\16\7c\07\999fd505f26f98cec0c082b5b265b512ce3958667d525cbc54
 Successfully built pyq
 Installing collected packages: pyq
 Successfully installed pyq-4.1.4

embedPy

embedPy does what it says on the tin: enables one to embed (inline) Python code within q code.

Currently it is supported only on the 64-bit version of kdb+/q.

Go to https://github.com/KxSystems/embedPy

Download the file tests/requirements.txt and save this file somewhere.

Go to (Your Anaconda distribution)/Scripts and run

pip install -r (Path to)/requirements.txt

replacing (Path to) with the actual path to requirements.txt that you have downloaded.

Make sure all requirements are installed successfully.

Then go to the releases page on https://github.com/KxSystems/embedPy and download the latest release (say embedPy_windows-1.3.1.zip).

Say your (64-bit) distribution is installed under C:/q.

From embedPy archive, copy p.q and p.k to C:/q.

From embedPy archive, copy p.dll (or p.so, etc., whichever way this library is called on your operating system) to C:/q/w64 (or …/q/m64 – depending on your operating system).

Also put test.q in C:/q/w64 (or …/q/m64 – depending on your operating system).

Make sure that the Python executable, python.exe on Windows, is on your system PATH. On Windows, we launch C:/q/w64/q.exe from C:/q/w64/q.bat, which we have created:

set PATH=C:\Programs\Win64\Anaconda\V4.4.0_3.6;%PATH%
q.exe

Launch q.exe

From the q console, load embedPy like so:

q)\l p.q

where q) is a prompt, you don’t have to type it in. You do, however, have to type in p) to tell the system that you are about to enter some Python code. And so,

q)p)print(3+5)

produces

8

To test whether embedPy is working properly,

q)\l test.q

NB! If all is well, the system will exit without an error message, so you’ll have to launch kdb+/q again if you want to continue your session!

jupyterq

The Jupyter Notebook is an open-source web application that allows you to create and share documents that contain live code, equations, visualisations, and narrative text.

Uses include: data cleaning and transformation, numerical simulation, statistical modelling, data visualisation, machine learning, and more.

jupyterq is a Jupyter kernel for kdb+/q, which enables you to use q in Jupyter notebooks.

Currently it is supported only on the 64-bit version of kdb+/q.

We shall also assume that you are working with the Anaconda Distribution (see above).

You also need embedPy, which we have already introduced (see above), to run jupyterq.

Go to jupyterq home page (https://github.com/kxsystems/jupyterq), go to the releases page, download the release archive and unzip it. (In our case, it is jupyterq_windows-1.1.3.zip.)

We prepend the following lines to the start of install.bat contained in the unzipped release archive:

set PATH=C:\Programs\Win64\Anaconda\V4.4.0_3.6;%PATH%
set PATH=C:\Programs\Win64\Anaconda\V4.4.0_3.6\Scripts;%PATH%
set QHOME=C:\q

since we need to have python and jupyter on the system PATH and also need to set the QHOME environment variable.

Run install.bat. Hopefully it completes without errors.

Then launch the Jupyter Notebook:

C:\Programs\Win64\Anaconda\V4.4.0_3.6\Scripts\jupyter-notebook.exe --notebook-dir=S:\dev

When the browser window appears with the Jupyter Notebook webapp, when you click on the New drop-down list close to the top, you should see Q (kdb+) among the choices.

kdb+tick

Go to the repository containing kdb+tick (https://github.com/KxSystems/kdb-tick), click on “Clone or download”, and then “Download ZIP”. You should obtain kdb-tick-master.zip.

Let us say that you q installation is located in C:/q, and the binary itself is under C:/q/w32 (although it could be ~/q and ~/q/m64, etc., so replace the directories below accordingly).

From kdb-tick-master.zip, copy tick.q to C:/q, so you end up with C:/q/tick.q.

From kdb-tick-master.zip, copy the entire directory and its contents tick to C:/q, so you end up with C:/q/tick. (That directory will contain r.q and u.q.)

Go to the repository containing the kdb+/q cookbook (https://github.com/KxSystems/cookbook), click on “Clone or download”, and then “Download ZIP”. You should obtain cookbook-master.zip.

From cookbook-master.zip, copy the entire directory and its contents start to C:/q, so you end up with C:/q/start (and the files that it contains, e.g. trades.q, etc.).

From the directory C:/q/start/tick, copy sym.q to under C:/q/tick, so you end up with C:/q/tick/sym.q.

If you are on Windows, edit the batch files under C:/q/start/tick/win (namely feed.bathlcv.batlast.batrdb.batshow.batticker.battq.batvwap.bat), so they contain the correct path to q (e.g. “c:\q“) and the correct path to q binary (e.g. “w32“, “w64“, whatever it is in your case).

If you are on a different operating system, then apply the same to …/q/start/tick/run1.sh.

Make sure that you can then run all kdb+tick sample instances successfully: on Windows, by launching C:/q/start/tick/run.bat; on other operating systems, by launching …/q/start/tick/run.sh.