Setup a development environment¶
When developing new applications you will need a basic setup containing all the required components.
Because most server components react best when running on linux (or other unix alternative), we choose to setup a virtual machine for all backend operations. Frontend tasks are using OPNsense ® as a base, for which we install a separate virtual machine.
The actual services can be executed on the developers client machine.
Note
When executing command’s on a shell, the actual command will be shown after the prompt (e.g. :~$ :~# ). Results may be shortened for readability using a list points (…..).
Setup a backend VM¶
Our preference for the backend server components is Ubuntu, we use version 20.04 (LTS) to install all needed server components. Without the framework code as a service/daemon we need the following services installed on our machine:
RabbitMQ (messaging)
PostgreSQL (relation database)
Cassandra (NoSQL database)
SSH (remote access / copy files)
The least expensive option to install a virtual machine on your host would probably be virtual box , which has installers available for Windows, Mac and linux and is free to use.
VirtualBox is available under the GPL version 2 license, except the available extension pack (which uses a more restricted license). For our purposes the GPL version is enough to work with.
Download and install Ubuntu 22.04 LTS¶
The first step is to install a basic linux server, using the iso file available from Ubuntu.
It’s usually a good idea to allocate enough memory and disk space to this server (> 2 GB Memory, > 20 GB disk). The network configuration which is used by default on VirtualBox (and many other VM’s) is NAT, this creates an internal network on your machine which can’t be reached from other machines and will use your machines interface address to access the internet/network.
Note
Because VirtualBox won’t allow access from our host system using its own address, we need to add another network adapter to access it from our host, the type of this adapter should be “Host Only”.
Tip
To add another network in VirtualBox, go to the settings of the virtual machine and search for “Network” in the top bar, then choose adapter 2, next enable and select “Host-Only”.
Most options during the install you can keep on their defaults, as soon as you enter “Software selection” it’s a good idea to add “OpenSSH server” to the list.
A more detailed installation manual for Ubuntu can be found on their website .
Login to Ubuntu¶
During installation you have created an account, which can now be used to login on the console in the vm. Our first goal is to determine the ip address this machine was provided.
Tip
For quite some administration tasks you need to be logged in as root, Ubuntu uses sudo to switch between your user and root. Use sudo before a statement to impersonate the root user, in case you want a root shell just type “sudo -i” and press Enter.
Log in and execute ip a on a console,
root@ubuntu:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp0s5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 00:1c:42:9e:83:ca brd ff:ff:ff:ff:ff:ff
inet 10.7.96.100/24 brd 10.211.55.255 scope global dynamic enp0s5
valid_lft 1372sec preferred_lft 1372sec
inet6 fdb2:2c26:f4e4:0:21c:42ff:fe9e:83ca/64 scope global dynamic mngtmpaddr noprefixroute
valid_lft 2591791sec preferred_lft 604591sec
inet6 fe80::21c:42ff:fe9e:83ca/64 scope link
valid_lft forever preferred_lft forever
Note
Our second (host-only) network interface won’t be configured at this time. Use netplan to add the un-configured interface to /etc/netplan/xx-cloud-init.yaml then reboot.
Update and add package repositories¶
Now we know the ip address of our virtual machine and its accessible using ssh (as chosen during setup), we can login to the machine using putty or any other ssh client.
Note
Don’t forget to login using your created user name and the ip address found in the steps above, you can’t login using root as long as there’s no password configured for the root user.
Install RabbitMQ¶
Install RabbitMQ and required packages:
## Install Essential Dependencies
root@ubuntu:~# apt-get update -y
root@ubuntu:~# apt-get install curl gnupg -y
## Enable apt HTTPS Transport
root@ubuntu:~# apt-get install apt-transport-https
root@ubuntu:~# apt-get install curl gnupg apt-transport-https -y
## Add Repository Signing Keys
## Team RabbitMQ's main signing key
root@ubuntu:~# curl -1sLf "https://keys.openpgp.org/vks/v1/by-fingerprint/0A9AF2115F4687BD29803A206B73A36E6026DFCA" | sudo gpg --dearmor | sudo tee /usr/share/keyrings/com.rabbitmq.team.gpg > /dev/null
## Community mirror of Cloudsmith: modern Erlang repository
root@ubuntu:~# curl -1sLf https://ppa1.novemberain.com/gpg.E495BB49CC4BBE5B.key | sudo gpg --dearmor | sudo tee /usr/share/keyrings/rabbitmq.E495BB49CC4BBE5B.gpg > /dev/null
## Community mirror of Cloudsmith: RabbitMQ repository
root@ubuntu:~# curl -1sLf https://ppa1.novemberain.com/gpg.9F4587F226208342.key | sudo gpg --dearmor | sudo tee /usr/share/keyrings/rabbitmq.9F4587F226208342.gpg > /dev/null
root@ubuntu:~#
To add the apt repository to the source list directory (under /etc/apt/sources.list.d), use:
tee /etc/apt/sources.list.d/rabbitmq.list <<EOF
## Provides modern Erlang/OTP releases from a Cloudsmith mirror
##
deb [signed-by=/usr/share/keyrings/rabbitmq.E495BB49CC4BBE5B.gpg] https://ppa1.novemberain.com/rabbitmq/rabbitmq-erlang/deb/ubuntu jammy main
deb-src [signed-by=/usr/share/keyrings/rabbitmq.E495BB49CC4BBE5B.gpg] https://ppa1.novemberain.com/rabbitmq/rabbitmq-erlang/deb/ubuntu jammy main
## Provides RabbitMQ from a Cloudsmith mirror
##
deb [signed-by=/usr/share/keyrings/rabbitmq.9F4587F226208342.gpg] https://ppa1.novemberain.com/rabbitmq/rabbitmq-server/deb/ubuntu jammy main
deb-src [signed-by=/usr/share/keyrings/rabbitmq.9F4587F226208342.gpg] https://ppa1.novemberain.com/rabbitmq/rabbitmq-server/deb/ubuntu jammy main
EOF
After updating the list of apt sources it is necessary to run apt-get update:
root@ubuntu:~# apt-get update
Then install the RabbitMQ package with:
## Install Erlang packages
root@ubuntu:~# apt-get install -y erlang-base \
root@ubuntu:~# erlang-asn1 erlang-crypto erlang-eldap erlang-ftp erlang-inets \
root@ubuntu:~# erlang-mnesia erlang-os-mon erlang-parsetools erlang-public-key \
root@ubuntu:~# erlang-runtime-tools erlang-snmp erlang-ssl \
root@ubuntu:~# erlang-syntax-tools erlang-tftp erlang-tools erlang-xmerl
## Install rabbitmq-server and its dependencies
root@ubuntu:~# apt-get install rabbitmq-server -y --fix-missing
Check status of the RabbitMQ service
root@ubuntu:~# systemctl status rabbitmq-server
Setup RabbitMQ¶
This VM is only used for development and can’t be reached from other hosts then the development machine itself, so we choose to set the password to something simple for testing purposes (user: admin, passwd: admin).
The next steps take care of the required changes to rabbitmq and make sure we can access the user interface from a webrowser.
root@ubuntu:~# rabbitmqctl add_user admin admin
Adding user "admin" ...
root@ubuntu:~# rabbitmqctl set_user_tags admin administrator
Setting tags for user "admin" to [administrator] ...
root@ubuntu:~# rabbitmqctl set_permissions -p / admin ".*" ".*" ".*"
Setting permissions for user "admin" in vhost "/" ...
root@ubuntu:~# rabbitmqctl delete_user guest
Deleting user "guest" ...
root@ubuntu:~# rabbitmq-plugins enable rabbitmq_management
Enabling plugins on node rabbit@backend:
rabbitmq_management
The following plugins have been configured:
rabbitmq_management
rabbitmq_management_agent
rabbitmq_web_dispatch
Applying plugin configuration to rabbit@backend...
The following plugins have been enabled:
rabbitmq_management
rabbitmq_management_agent
rabbitmq_web_dispatch
started 3 plugins.
root@ubuntu:~# systemctl restart rabbitmq-server.service
Enable all feature flags (see also feature-flag-ground-rules )
root@ubuntu:~# rabbitmqctl enable_feature_flag all
Don’t forget to open port 5672 and 15672 when blocked by a firewall. For example with ufw:
root@ubuntu:~# ufw allow 5672
Rule added
Rule added (v6)
root@ubuntu:~# ufw allow 15672
Rule added
Rule added (v6)
Now we should be able to access the admin interface using the following url (replace 10.7.96.103 for your actual ip address):
Try to login using admin/admin so we know for sure our message queue is functional.
Install Postgres¶
First configure apt to get access to Postgres repositories.
root@ubuntu:~# sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list'
root@ubuntu:~# wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
root@ubuntu:~# apt-get update
Install Postgres, Postgis and other required librearies
root@ubuntu:~# apt-get -y install postgresql
root@ubuntu:~# apt -y install postgis
Setup Postgres¶
For Postgres we’re also using a simple (not very secure) setup, for the same reasons as the rabbitmq above.
root@ubuntu:~# echo "host all all 0.0.0.0/0 md5" >> /etc/postgresql/16/main/pg_hba.conf
root@ubuntu:~# echo "listen_addresses= '*'" >> /etc/postgresql/16/main/postgresql.conf
root@ubuntu:~# systemctl restart postgresql.service
Next we’re going to create a user valuea using the same password.
root@ubuntu:~# sudo -u postgres psql
psql (16.2 (Ubuntu 16.2-1.pgdg22.04+1))
Type "help" for help.
postgres=# CREATE USER valuea WITH PASSWORD 'valuea';
CREATE ROLE
postgres=# create database valuea encoding='utf8' LC_COLLATE = 'en_US.UTF-8' LC_CTYPE = 'en_US.UTF-8' owner='valuea' template=template0;
CREATE DATABASE
postgres=# GRANT ALL PRIVILEGES ON DATABASE valuea to valuea;
GRANT
postgres=# \q
root@ubuntu:~#
Add the postgis extension for geometry support.
root@ubuntu:~# sudo -u postgres psql valuea
psql (14.1 (Ubuntu 14.1-2.pgdg20.04+1))
Type "help" for help.
valuea=# CREATE SCHEMA postgis;
CREATE SCHEMA
valuea=# ALTER DATABASE valuea SET search_path=public, postgis, contrib;
ALTER DATABASE
valuea=# \connect valuea;
You are now connected to database "valuea" as user "postgres".
valuea=# CREATE EXTENSION postgis SCHEMA postgis;
CREATE EXTENSION
valuea=# CREATE EXTENSION postgis_topology;
CREATE EXTENSION
postgres=# \q
root@ubuntu:~#
And give user valuea acces to postgis
root@ubuntu:~# sudo -u postgres psql -c "grant all on schema postgis to valuea" valuea
Don’t forget to open port 5432 when blocked by a firewall. For example with ufw:
root@ubuntu:~# ufw allow postgres
Rule added
Rule added (v6)
To test your setup, you can download and install pgadmin4 and configure a connection to our newly deployed server.
Optional: cassandra install and setup¶
First install the repository keys, using:
root@ubuntu:~# curl https://www.apache.org/dist/cassandra/KEYS | apt-key add -
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 244k 100 244k 0 0 130k 0 0:00:01 0:00:01 --:--:-- 130k
OK
And fetch all updates:
root@ubuntu:~# apt update
Get:1 http://ppa.launchpad.net/webupd8team/java/ubuntu xenial InRelease [17.6 kB]
Get:2 http://security.ubuntu.com/ubuntu xenial-security InRelease [102 kB]
....
Reading package lists... Done
Next make sure we’re using the latest software version:
root@ubuntu:~# apt upgrade
Reading package lists... Done
.........
Do you want to continue? [Y/n]
Install required packages¶
We now need to install postgresql, rabbitmq and cassandra including some build requirements in case we do want to use our framework on this vm later on.
root@ubuntu:~# apt install -y oracle-java8-installer cassandra
Reading package lists... Done
Building dependency tree
.....
root@ubuntu:~# apt install -y rabbitmq-server postgresql-9.5
Reading package lists... Done
Building dependency tree
.....
root@ubuntu:~# apt install -y postgis postgresql-9.5-postgis-2.2
Reading package lists... Done
Building dependency tree
.....
root@ubuntu:~# apt install -y python-pip build-essential python-dev libev4 libev-dev python-psycopg2 python-lxml
Reading package lists... Done
Building dependency tree
.....
Setup Cassandra¶
Cassandra is used for its capability to store data distributed across different nodes, but for our test setup we will only use one. The configuration file for Cassandra is stored at /etc/cassandra/cassandra.yaml.
cluster_name: 'ValueA'
seed_provider:
- class_name: org.apache.cassandra.locator.SimpleSeedProvider
parameters:
# seeds is actually a comma-delimited list of addresses.
# Ex: "<ip1>,<ip2>,<ip3>"
- seeds: "192.168.56.101"
# listen_address: localhost
listen_interface: enp0s8
# rpc_address: localhost
rpc_interface: enp0s8
enable_user_defined_functions: yes
Replace enp0s8 for the interface serving the “host-only” network and 192.168.56.101 with the address used on the interface and restart the daemon (remove old data in the process), using:
root@ubuntu:~# rm -rf /var/lib/cassandra/*
root@ubuntu:~# systemctl restart cassandra.service
Wait some time and check if the “cluster” is running:
root@ubuntu:~# nodetool status
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN 192.168.56.101 112.1 KiB 256 100.0% d058a639-6d27-495c-8b51-6ec355eebb42 rack1
To access the cluster from the local server we need to install the cassandra-driver using pip:
root@ubuntu:~# pip install -U cassandra-driver
Collecting cassandra-driver
Downloading cassandra-driver-3.15.tar.gz (464kB)
100% |████████████████████████████████| 471kB 705kB/s
.....
Successfully installed cassandra-driver-3.15 futures-3.2.0 six-1.11.0
Note
Version may differ, at the time of this writing the current version was 3.15.
Next login to the cluster and create a keyspace.
root@ubuntu:~# export CQLSH_NO_BUNDLED=TRUE
root@ubuntu:~# cqlsh 192.168.56.101
Connected to ValueA at 192.168.56.101:9042.
[cqlsh 5.0.1 | Cassandra 3.11.3 | CQL spec 3.4.4 | Native protocol v4]
Use HELP for help.
cqlsh> CREATE KEYSPACE valuea WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };
cqlsh> exit;
Note
Our cluster contains only one server, hence the replication factor should be equal to 1.
Setup a Frontend VM¶
Our frontend uses a standard OPNsense system extended with ValueA core components.
Download and install OPNsense¶
Go to https://opnsense.org/download/ and download the latest DVD image for amd64 from a mirror nearby.
With the installer available, we’re ready to setup another virtual machine (BSD -> Freebsd / 64 bit), which shouldn’t be very large. A machine with about 1 GB memory and 15 GB disk space should be enough. Assuming we’re using Virtualbox with NAT it’s a good idea to add an extra host only network before we begin, it’s not required to define two networks, but it is practical to be able to use the internet for updates.
Tip
For easy installation make sure the host-only network is the first interface and the second interface is of type NAT, so the default interfaces are setup correctly in OPNsense and you don’t have to switch anything.
Next unpack the file, connect it to the virtual machine and start the installer.
Note
When the initial prompt is shown, invoke the installer with user installer and password opnsense
Keep all options standard until a new root password can be provided, normally you should be good using all the standard options (Guided installation).
Note
The default password is opnsense, if you leave the password empty it will fallback to default.
The last step of the installer contains of a reboot, after this step you should detach the (iso) installer.
On initial boot the wan interface should be connected using NAT, the lan interface uses a static address which we should change, login using root and choose menu option 2 (Set interface IP address), then select 1 (LAN) and setup using dhcp.
Note
Do not revert to http when asked by the configurator, after changing the address to dhcp you should see the newly acquired address in the console.
Initial configuration¶
To be able to upload files easily it’s advisable to enable SSH access, to do so we’re going to login to the web interface using a webbrowser (https://<ip address>/).
Tip
To exit the initial install wizard, click on the logo in the upper left corner.
To enable SSH, goto System -> Settings -> Administration and “enable secure shell, permit root and permit password login”.
Install ValueA core¶
Copy os-valuea-core-x.y.txz to the firewall using winscp or similar and install the required packages:
Note
When connected to OPNsense using ssh, you will be presented with a menu. Choose 8 to select a shell (needed to input the commands above)
Note
Use pkg search php to search which major php version is used, then install the required packages for the same package (e.g. php71)
pkg update
pkg search php
pkg install php[version]-bcmath php[version]-mbstring php[version]-pecl-mailparse
killall php-cgi
pkg add -f os-valuea-core-x.y.txz
When the plugin is installed, we can configure our connection to RabbitMQ using the credentials provided earlier (username: admin, password: admin, hostname: ip address of linux machine) in http://<ipaddress OPNsense>/ui/valueacore/.
Note
Be aware that as long as there’s no framework broker connected, we can’t test our setup. Learn by example contains the steps needed to test the connection.
Configure ValueA backend server¶
When using the backend server as a broker, instead of a local broker, additional configuration is needed.