Setup OS Requirements Debian

This section describes how to perform the setup for Debian Linux 10. The Peek platform is designed to run on Linux.

Please read through all of the documentation before commencing the installation procedure.

Note

These instructions are for Debian 10, AKA Buster

Installation Objective

This Installation Guide contains specific Debian Linux 10 operating system requirements for the configuring of synerty-peek.

Required Software

Some of the software to be installed requires internet access. For offline installation some steps are required to be installed on another online server for the files to be packaged and transferred to the offline server.

Below is a list of all the required software:

  • Python 3.9.x

  • Postgres 12.x

Suggested Software

The following utilities are often useful.

  • rsync

  • git

  • unzip

Optional Software

  • Oracle Client

Installing Oracle Libraries is required if you intend on installing the peek agent. Instruction for installing the Oracle Libraries are in the Online Installation Guide.

  • FreeTDS

FreeTDS is an open source driver for the TDS protocol, this is the protocol used to talk to the MSSQL SQLServer database.

Installation Guide

Follow the remaining section in this document to prepare your debian operating system for to run the Peek Platform.

The instructions on this page don’t install the peek platform, that’s done later.

Install Debian 10 OS

This section installs the Debian 10 64bit Linux operating system.

Create VM

Create a new virtual machine with the following specifications

  • 4 CPUs

  • 8gb of ram

  • 60gb of disk space

Install OS

Download the debian ISO, navigate to the following site and click amd64 under netinst CD image

Download Debian


Mount the ISO in the virtual machine and start the virtual machine.

Run through the installer manually, do not let your virtual machine software perform a wizard or express install.

Staring Off

At the Debian GNU/Linux installer boot menu screen, select:

Install

At the Select a language screen, select:

English

At the Select your location screen, select the appropriate location.


At the Configure the keyboard screen, select the appropriate keyboard, or leave as default.


At the Configure the network screen, enter your desired hostname or:

peek

At the Configure the network screen, enter your desired domain, or:

localdomain

At the Setup users and passwords screen, watch for the following prompts, replace <root_password> and <peek_password> with your desired passwords.

Setup users and passwords screen prompts

Prompt

Enter :

Root password

<root_password>

Re-enter password to verify

<root_password>

Full name for the new user

Peek Platform

Username for your account

peek

Choose a password for the new user:

<peek_password>

Re-enter password to verify:

<peek_password>


On the Configure the clock screen, select your desired timezone.


Partition Table

On the Partition disks screen, select:

Manual

Then, select the disk, it will look similar to:

SCSI3 (0,0,0) (sda) - 32.2 GB VMware ...

Then it will prompt to Create new empty partition table on this device?, select:

<Yes>

We’ll be creating three partitions, /boot / and swap. For a heavily used production server you may want to create more virtual disks and separate out /var, /home, and /tmp. With one file system per disk.

Having one file system per disk removes the need for the overhead of LVM, and the VM software can still expand the disk and filesystem as required.

/boot

Select the following disk from the menu:

pri/log **.* GB   FREE SPACE

Enter the following responses to the prompts

/boot partition prompts part1

Prompt

Enter :

How to user this free space

Create a new partition

New partition size

500m

Type for the new partition

Primary

Location for the new Partition

Beginning

At the Partition Settings prompt, enter the following:

/boot partition prompts part2

Prompt

Enter :

Use as:

Ext2 file system

Mount point

/boot

Done setting up the partition

swap

Select the following disk from the menu:

pri/log **.* GB   FREE SPACE

Enter the following responses to the prompts

swap partition prompts part1

Prompt

Enter :

How to user this free space

Create a new partition

New partition size

4g

Type for the new partition

Primary

Location for the new Partition

Beginning

At the Partition Settings prompt, enter the following:

swap partition prompts part2

Prompt

Enter :

Use as:

swap

Done setting up the partition

/ (root)

The root file system is created at the end of the disk, ensuring that if we use the VM software to expand the virtual disk, this is the file system that will be expanded.

The default guided install doesn’t do this.


Select the following disk from the menu:

pri/log **.* GB   FREE SPACE

Enter the following responses to the prompts

swap partition prompts part1

Prompt

Enter :

How to user this free space

Create a new partition

New partition size

100%

Type for the new partition

Primary

Location for the new Partition

Beginning

At the Partition Settings prompt, enter the following:

swap partition prompts part2

Prompt

Enter :

Use as

Ext4 journaling file system

Mount point

/

Reserved blocks

1%

Done setting up the partition


All done, select:

Finish partitioning and write changes to disk

At the Write the changes to disk? prompt, Select:

<Yes>

Finishing Up

On the Configure the package manager screen, select the location closest to you.


At the Debian archive mirror, select your preferred site.


At the HTTP proxy information prompt, select:

<Continue>

The installer will now download the package lists.


At the Configure popularity-contest screen, select:

<No>

Note

It’d be good to select <Yes>, but as Peek is an enterprise platform, it’s most likely installed behind a corporate firewall.


At the Software selection screen, select the following, and deselect all the other options:

  • SSH server

  • standard system utilities

Optionally, select a desktop environment, Peek doesn’t require this. “MATE” is recommended if one is selected.


The OS will now install, it will take a while to download and install the packages.


At the Install the GRUB boot loader on a hard disk screen, select:

<Yes>


At the Device for boot loader installation prompt, select:

/dev/sda

At the Finish the installation screen, select:

<Continue>

Deconfigure the Debian ISO from DVD drive in the VM software.


The OS installtion is now complete.

SSH Setup

SSH is this documentations method of working with the Peek Debian VM.

SSH clients are availible out of the box with OSX and Linux. There are many options for windows users, This documentation recommends MobaXterm is used for windows as it also supports graphical file copying.

This document assumes users are familair with what is required to use the SSH clients for connecting to and copying files to the Peek VM.

If this all sounds too much, reinstall the Peek OS with a graphical desktop environment and use that instead of SSH.

Note

You will not be able to login as root via SSH by default.


Login to the console of the Peek Debian VM as root and install ifconfig

with the following command:

apt-get install net-tools

Run the following command:

ifconfig

Make note of the ipaddress, you will need this to SSH to the VM. The IP addresss will be under eth0, second line, inet addr.


Install sudo with the following command:

apt-get install sudo

Give Peek sudo privielges with the following command:

usermod -a -G sudo peek

You must now logout from the root console.

Login as Peek

Login to the Debian VM as the peek user, either via SSH, or the graphical desktop if it’s installed.

Important

All steps after this point assume you’re logged in as the peek user.

Configure Static IP (Optional)

If this is a production server, it’s more than likely that you want to assign a static IP to the VM, Here is how you do this.


Edit file /etc/network/interfaces

Find the section:

allow-hotplug eth0
iface eth0 inet dhcp

Replace it with:

auto eth0
iface eth0 inet static
    address <IPADDRESS>
    netmask <NETMASK>
    gateway <GATEWAY>

Edit the file /etc/resolv.conf, and update it.

  1. Replace “localdomain” with your domain

  2. Replace the IP for the nameserver with the IP of you DNS. For multiple name servers, use multiple nameserver lines.

    domain localdomain
    search localdomain
    nameserver 172.16.40.2
    

Installing General Prerequisites

This section installs the OS packages required.

Note

Run the commands in this step as the peek user.


Install the C Compiler package, used for compiling python or VMWare tools, etc:

PKG="gcc make linux-headers-amd64"
sudo apt-get install -y $PKG

Install some utility packages:

PKG="rsync unzip wget git"

sudo apt-get install -y $PKG

Install the Python build dependencies:

PKG="build-essential curl git m4 ruby texinfo libbz2-dev libcurl4-openssl-dev"
PKG="$PKG libexpat-dev libncurses-dev zlib1g-dev libgmp-dev libssl-dev"
sudo apt-get install -y $PKG

Install the Postgres build dependencies:

PKG="bison flex"
PKG="$PKG libreadline-dev python-dev"
sudo apt-get install -y $PKG

Install Libs that some python packages link to when they install:

# For the cryptography package
PKG="libffi-dev"

sudo apt-get install -y $PKG

Install Libs required for LDAP:

PKG="libsasl2-dev libldap-common libldap2-dev"

sudo apt-get install -y $PKG

Install Libs that database access python packages link to when they install:

# For Shapely and GEOAlchemy
PKG="libgeos-dev libgeos-c1v5"

# For the PostGresQL connector
PKG="$PKG libpq-dev"

# For the SQLite python connector
PKG="$PKG libsqlite3-dev"

sudo apt-get install -y $PKG

Install Libs that the oracle client requires:

# For LXML and the Oracle client
PKG="libxml2 libxml2-dev"
PKG="$PKG libxslt1.1 libxslt1-dev"
PKG="$PKG libaio1 libaio-dev"

sudo apt-get install -y $PKG

Cleanup the downloaded packages

sudo apt-get clean

Installing VMWare Tools (Optional)

This section installs VMWare tools. The compiler tools have been installed from the section above.


In the VMWare software, find the option to install VMWare tools.


Mount and unzip the tools

sudo rm -rf /tmp/vmware-*
sudo mount /dev/sr0 /mnt
sudo tar xzf /mnt/VM*gz -C /tmp
sudo umount /mnt

Install the tools with the default options

cd /tmp/vmware-tools-distrib
sudo ./vmware-install.pl -f -d

Cleanup the tools install

sudo rm -rf /tmp/vmware-*

Reboot the virtual machine.

sudo shutdown -r now

Keep in mind, that if the static IP is not set, the IP address of the VM may change, causing issues when reconnecting with SSH.

Preparing .bashrc

Warning

Open ~/.bashrc insert the following at the start, before:

# If not running interactively, don't do anything

If you do not place the below code before that line, it will not be parsed.

##### SET THE PEEK ENVIRONMENT #####
# Setup the variables for PYTHON and POSTGRESQL
export PEEK_PY_VER="3.9.1"
export PEEK_TSDB_VER="1.7.4"
export PGDATA=~peek/pgdata/12

export PATH="$HOME/opt/bin:$PATH"
export LD_LIBRARY_PATH="$HOME/opt/lib:$LD_LIBRARY_PATH"

# Set the variables for the platform release
# These are updated by the deploy script
export PEEK_ENV=""
[ -n "${PEEK_ENV}" ] && export PATH="${PEEK_ENV}/bin:$PATH"

Warning

Restart your terminal to get the new environment.

Compile and Install Python 3.9.1

The Peek Platform runs on Python. These instructions download, compile and install the latest version of Python.


Download and unarchive the supported version of Python:

cd
source .bashrc
wget https://github.com/python/cpython/archive/v${PEEK_PY_VER}.zip
unzip v${PEEK_PY_VER}.zip
cd cpython-${PEEK_PY_VER}

Configure the build:

./configure --prefix=/home/peek/opt/ --enable-optimizations --enable-shared

Make and Make install the software:

make install

Cleanup the download and build dir:

cd
rm -rf cpython-${PEEK_PY_VER}
rm v${PEEK_PY_VER}.zip

Symlink the python3 commands so they are the only ones picked up by path:

cd /home/peek/opt/bin
ln -s pip3 pip
ln -s python3 python
cd

Test that the setup is working:

RED='\033[0;31m'
GREEN='\033[0;32m'
NC='\033[0m' # No Color

SHOULD_BE="/home/peek/opt/bin/python"
if [ `which python` == ${SHOULD_BE} ]
then
    echo -e "${GREEN}SUCCESS${NC} The python path is right"
else
    echo -e "${RED}FAIL${NC} The python path is wrong, It should be ${SHOULD_BE}"
fi

SHOULD_BE="/home/peek/opt/bin/pip"
if [ `which pip` == ${SHOULD_BE} ]
then
    echo -e "${GREEN}SUCCESS${NC} The pip path is right"
else
    echo -e "${RED}FAIL${NC} The pip path is wrong, It should be ${SHOULD_BE}"
fi

Upgrade pip:

pip install --upgrade pip

synerty-peek is deployed into python virtual environments. Install the virtualenv python package:

pip install virtualenv

The Wheel package is required for building platform and plugin releases:

pip install wheel

Install PostgreSQL

Install the relational database Peek stores its data in. This database is PostgreSQL 12.

Note

Run the commands in this step as the peek user.

Download the PostgreSQL source code

PEEK_PG_VER=12.5
SRC_DIR="$HOME/postgresql-${PEEK_PG_VER}"

# Remove the src dir and install file
rm -rf ${SRC_DIR} || true
cd $HOME

wget https://ftp.postgresql.org/pub/source/v${PEEK_PG_VER}/postgresql-${PEEK_PG_VER}.tar.bz2
tar xjf postgresql-${PEEK_PG_VER}.tar.bz2

cd ${SRC_DIR}

Configure and build PostGresQL

export CPPFLAGS=" -I`echo $HOME/opt/include/python*m` "
export LDFLAGS=" -L$HOME/opt/lib "

./configure \
      --disable-debug \
      --prefix=$HOME/opt \
      --enable-thread-safety \
      --with-openssl \
      --with-python


make -j4

make install-world

# this is required for timescale to compile
cp ${SRC_DIR}/src/test/isolation/pg_isolation_regress ~/opt/bin

Remove install files to clean up the home directory

# Remove the src dir and install file
cd
rm -rf ${SRC_DIR}*

Initialise a PostgreSQL database

#Refresh .bashrc so initdb can find postgres
source .bashrc

initdb --pgdata=$HOME/pgdata/12 --auth-local=trust  --auth-host=md5

Tune the postgresql.conf

F="$HOME/pgdata/12/postgresql.conf"

sed -i 's/max_connections = 100/max_connections = 200/g' $F

Make PostgreSQL a service :

Note

This will require sudo permissions

Run the following command

touch postgresql-12.service

F=postgresql-12.service

cat <<"EOF" | sed "s,\$HOME,`echo ~peek`,g" > $F
[Unit]
Description=PostgreSQL 12 database server
After=syslog.target
After=network.target

[Service]
Type=forking
User=peek
Group=peek

# Location of database directory
Environment=PGDATA=$HOME/pgdata/12

# Disable OOM kill on the postmaster
OOMScoreAdjust=-1000
Environment=PG_OOM_ADJUST_FILE=/proc/self/oom_score_adj
Environment=PG_OOM_ADJUST_VALUE=0

ExecStart=$HOME/opt/bin/pg_ctl -D ${PGDATA} start
ExecStop=$HOME/opt/bin/pg_ctl -D ${PGDATA} stop
ExecReload=/bin/kill -HUP $MAINPID
KillMode=mixed
KillSignal=SIGINT


# Do not set any timeout value, so that systemd will not kill postmaster
# during crash recovery.
TimeoutSec=0

[Install]
WantedBy=multi-user.target
EOF

sudo mv $F /etc/systemd/system/postgresql-12.service

Reload the daemon

sudo systemctl daemon-reload

Install CMake

Download CMake source code:

PEEK_CMAKE_VER=3.19.2
SRC_DIR="$HOME/CMake-${PEEK_CMAKE_VER}"
wget https://github.com/Kitware/CMake/archive/v${PEEK_CMAKE_VER}.zip

unzip v${PEEK_CMAKE_VER}.zip
cd ${SRC_DIR}

Compile CMake from source:

./configure --prefix=$HOME/opt

make -j6 install

# Remove the src dir and install file
cd
rm -rf ${SRC_DIR}*
rm v${PEEK_CMAKE_VER}.zip

Install PostgreSQL Timescaledb

Next install timescaledb, this provides support for storing large amounts of historical data.

www.timescale.com


Download the timescaledb source code

PEEK_TSDB_VER=1.7.4

cd
wget https://github.com/timescale/timescaledb/archive/${PEEK_TSDB_VER}.zip
unzip ${PEEK_TSDB_VER}.zip
cd timescaledb-${PEEK_TSDB_VER}

Install the packages:

export CPPFLAGS=`pg_config --cppflags`
export LDFLAGS=`pg_config --ldflags`

# Bootstrap the build system
./bootstrap -DAPACHE_ONLY=1

# To build the extension
cd build && make

# To install
make install

# Cleanup the source code
cd
rm -rf ${PEEK_TSDB_VER}.zip
rm -rf timescaledb-${PEEK_TSDB_VER}

Add the timescale repository:

curl -s https://packagecloud.io/install/repositories/timescale/timescaledb/script.deb.sh | sudo bash

Install timescaledb-tune:

sudo apt install -y timescaledb-tools

Tune the database:

PGVER=12
FILE="$HOME/pgdata/${PGVER}/postgresql.conf"
timescaledb-tune -quiet -yes -conf-path ${FILE} -pg-version ${PGVER}

Start PostgreSQL:

systemctl enable postgresql-12 --now

Finish PostgreSQL Setup

Finish configuring and starting PostgreSQL.


Allow the peek OS user to login to the database as user peek with no password

F=$HOME/pgdata/12/pg_hba.conf
cat | sudo tee $F <<EOF
# TYPE  DATABASE        USER            ADDRESS                 METHOD
local   all             peek                                    trust

# "local" is for Unix domain socket connections only
local   all             all                                     peer
# IPv4 local connections:
host    all             all             127.0.0.1/32            md5
# IPv6 local connections:
host    all             all             ::1/128                 md5
EOF

Create the database:

createdb -O peek peek

Set the PostgreSQL peek users password:

psql -d peek -U peek <<EOF
\password
\q
EOF

# Set the password as "PASSWORD" for development machines
# Set it to a secure password from https://xkpasswd.net/s/ for production

Note

If you already have a database, you may now need to upgrade the timescale extension.

psql peek <<EOF
ALTER EXTENSION timescaledb UPDATE;
EOF

Cleanup traces of the password:

[ ! -e ~/.psql_history ] || rm ~/.psql_history

Grant PostgreSQL Peek Permissions

The PostgreSQL server now runs parts of peeks python code inside the postgres/postmaster processes. To do this the postgres user needs access to peeks home directory where the peek software is installed.

Grant permissions

sudo chmod g+rx ~peek

Install Worker Dependencies

Install the parallel processing queue we use for the peek-worker-service tasks.

Note

Run the commands in this step as the peek user.

Install redis and rabbitmq

sudo apt-get install -y redis-server rabbitmq-server
sudo apt-get clean

Enable the RabbitMQ management plugins:

sudo rabbitmq-plugins enable rabbitmq_mqtt
sudo rabbitmq-plugins enable rabbitmq_management
sudo service rabbitmq-server restart

Increase the size of the redis client queue

BEFORE="client-output-buffer-limit pubsub 64mb 16mb 90"
AFTER="client-output-buffer-limit pubsub 32mb 8mb 60"
sudo sed -i "s/${BEFORE}/${AFTER}/g" /etc/redis/redis.conf

sudo systemctl restart redis

Install Oracle Client (Optional)

The oracle libraries are optional. Install them where the agent runs if you are going to interface with an oracle database.


Warning

Open ~/.bashrc insert the following at the start, before:

# If not running interactively, don't do anything

If you do not place the below code before that line, it will not be parsed.

# Setup the variables for ORACLE
export LD_LIBRARY_PATH="/home/peek/oracle/instantclient_21_1:$LD_LIBRARY_PATH"
export ORACLE_HOME="/home/peek/oracle/instantclient_21_1"

Source the new profile to get the new variables:

source ~/.bashrc

Make the directory where the oracle client will live

mkdir /home/peek/oracle

Download the following from oracle.

The version used in these instructions is 21.1.0.0.0.

  1. Download the ZIP “Basic Package” instantclient-basic-linux.x64-21.1.0.0.0.zip from http://www.oracle.com/technetwork/topics/linuxx86-64soft-092277.html

  2. Download the ZIP “SDK Package” instantclient-sdk-linux.x64-21.1.0.0.0.zip from http://www.oracle.com/technetwork/topics/linuxx86-64soft-092277.html

Copy these files to /home/peek/oracle on the peek server.


Extract the files.

cd ~/oracle
unzip instantclient-basic-linux.x64-21.1.0.0.0.zip*
unzip instantclient-sdk-linux.x64-21.1.0.0.0.zip*

Install FreeTDS (Optional)

FreeTDS is an open source driver for the TDS protocol, this is the protocol used to talk to a MSSQL SQLServer database.

Peek needs this installed if it uses the pymssql python database driver, which depends on FreeTDS.


Warning

Open ~/.bashrc insert the following at the start, before:

# If not running interactively, don't do anything

If you do not place the below code before that line, it will not be parsed.

# Setup the variables for FREE TDS
export LD_LIBRARY_PATH="/home/peek/freetds:$LD_LIBRARY_PATH"

Warning

Restart your terminal you get the new environment.


Install FreeTDS:

sudo apt-get install -y freetds-dev

Create file freetds.conf in ~/freetds and populate with the following:

mkdir ~/freetds
cat > ~/freetds/freetds.conf <<EOF

[global]
    port = 1433
    instance = peek
    tds version = 7.4

EOF

If you want to get more debug information, add the dump file line to the [global] section Keep in mind that the dump file takes a lot of space.

[global]
    port = 1433
    instance = peek
    tds version = 7.4
    dump file = /tmp/freetds.log

What Next?

Refer back to the How to Use Peek Documentation guide to see which document to follow next.