Central MQTT broker – configure OpenVPN server

In following articles I'll describe another part of my IoT project – building infrastructure for data exchange. Remember from my first post, this infrastructure consists of two main parts: central MQTT broker and bunch of local brokers.

First few articles will be about configuring central MQTT broker. It requires a computer with public IP address. Rent a VPS is pretty good choice for this purpose. Operating system which runs on your VPS doesn't really matter. In my case, I'm using Debian 8 with systemd. Configuration steps may differ for other system setup.

Encrypted connection

Security is one of my major concerns. Each local MQTT broker must establish a connection with central broker over SSL encrypted tunnel. There are basically two possibilities:

1. Configure mosquitto to use encrypted connection.
2. Establish VPN tunnel between hosts and configure central broker to listen on VPN interface only.

Second way have more benefits. This approach delegates responsibility for encryption into another piece of software. It is also possible to run other services at VPN interface, such as HTTP server for some diagnostic applications. Only downside is that VPN means more complex configuration.

For this tutorial, I use virtual system running in VirtualBox. I assigned two network interfaces to it. One in NAT mode (access to the Internet) and second connected to host-only network. This is for reason that I don't want to use my real public IP addresses in this tutorial. I can access my virtual machine at 192.168.56.102.

Install OpenVPN

First step is install VPN daemon. Best choice for me is to use OpenVPN. It is open source and well known VPN solution with huge community. To install OpenVPN issue following command:

root@buben-vps:~# apt-get install openvpn

And that's it. VPN server software is installed. Now comes the hard part.

Certificates and keys

To establish an encrypted connection with VPN server, we need an encryption key. We also needs some third party authority which verifies, that encryption key belongs to authorized user.

In another words, to establish connection we have to create encryption key for both server and client, create certificate authority (CA) to sign keys and create certification requests. This is generally done using openssl library tool with the same name – openssl. But create all these things with openssl command requires lot of commands with many parameters. For that reason EasyRSA was created, which simplifies this procedure.

Following steps should be done on some secured system dedicated for storing CA and key signing. It is recommended to keep this computer in locked room and completely disconnected from the network. For purpose of this article, I create all files on my /tmp directory.

First of all download EasyRSA from GitHub:

buben@ca:/tmp$ git clone https://github.com/OpenVPN/easy-rsa

Next, change working directory and create configuration file:

buben@ca:/tmp$ cd easy-rsa/easyrsa3
buben@ca:/tmp/easy-rsa/easyrsa3$ cp vars.example vars
buben@ca:/tmp/easy-rsa/easyrsa3$ nano vars

Now edit configuration values. Most important are following variables:

#set_var EASYRSA_REQ_COUNTRY     "US"
#set_var EASYRSA_REQ_PROVINCE    "California"
#set_var EASYRSA_REQ_CITY        "San Francisco"
#set_var EASYRSA_REQ_ORG "Copyleft Certificate Co"
#set_var EASYRSA_REQ_EMAIL       "me@example.net"
#set_var EASYRSA_REQ_OU          "My Organizational Unit"

Just uncomment them and assign them appropriate literals. If you want to omit some of these values (EASYRSA_REQ_OU in my case) fill it with “.” (dot).

After EasyRSA is configured, you can initialize public key infrastructure (PKI):

buben@ca:/tmp/easy-rsa/easyrsa3$ ./easyrsa init-pki

After that, empty pki should look like this:

buben@ca:/tmp/easy-rsa/easyrsa3$ tree pki/
pki/
├── private
└── reqs

Building key files

Now you can build certificate authority and key files. First thing is CA:

buben@ca:/tmp/easy-rsa/easyrsa3$ ./easyrsa build-ca

Note: using Easy-RSA configuration from: ./vars
Generating a 2048 bit RSA private key
.......................+++
.....+++
writing new private key to '/tmp/easy-rsa/easyrsa3/pki/private/ca.key.cAsM68O5hi'
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Common Name (eg: your user, host, or server name) [Easy-RSA CA]:buben-ca

CA creation complete and you may now import and sign cert requests.
Your new CA certificate file for publishing is at:
/tmp/easy-rsa/easyrsa3/pki/ca.crt

Command build-ca creates CA files encrypted with pass phrase. You can append nopass argument to create plaintext CA. I recommend you to protect it. Command also asks for common name. It should be some meaningful string, as the prompt suggests.

Next, create server key and certificate request file:

buben@ca:/tmp/easy-rsa/easyrsa3$ ./easyrsa gen-req buben-vps nopass

Note: using Easy-RSA configuration from: ./vars
Generating a 2048 bit RSA private key
.+++
..............+++
writing new private key to '/tmp/easy-rsa/easyrsa3/pki/private/buben-vps.key.3pklh1AD07'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Common Name (eg: your user, host, or server name) [buben-vps]:

Keypair and certificate request completed. Your files are:
req: /tmp/easy-rsa/easyrsa3/pki/reqs/buben-vps.req
key: /tmp/easy-rsa/easyrsa3/pki/private/buben-vps.key

In this case I recommend you to use nopass argument. Otherwise you will have to manually type a pass phrase every time you want to start a VPN server, which is nonsence.

Command again asks for common name. In case of server key, it must match the server hostname. This is the way how the client verifies that it is authenticating with correct machine. If common name doesn't match to server hostname, certificate verification fails.

Next, sign server request with your CA:

buben@ca:/tmp/easy-rsa/easyrsa3$ ./easyrsa sign-req server buben-vps

Note: using Easy-RSA configuration from: ./vars


You are about to sign the following certificate.
Please check over the details shown below for accuracy. Note that this request
has not been cryptographically verified. Please be sure it came from a trusted
source or that you have verified the request checksum with the sender.

Request subject, to be signed as a server certificate for 3650 days:

subject=
    commonName                = buben-vps


Type the word 'yes' to continue, or any other input to abort.
  Confirm request details: yes
Using configuration from /tmp/easy-rsa/easyrsa3/openssl-1.0.cnf
Enter pass phrase for /tmp/easy-rsa/easyrsa3/pki/private/ca.key:
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
commonName            :ASN.1 12:'buben-vps'
Certificate is to be certified until Sep 12 14:53:09 2025 GMT (3650 days)

Write out database with 1 new entries
Data Base Updated

Certificate created at: /tmp/easy-rsa/easyrsa3/pki/issued/buben-vps.crt

If CA is encrypted, command asks for its pass phrase.

Last step is create Diffie-Hellman parameters for initial key exchange:

buben@ca:/tmp/easy-rsa/easyrsa3$ ./easyrsa gen-dh

Note: using Easy-RSA configuration from: ./vars
Generating DH parameters, 2048 bit long safe prime, generator 2
This is going to take a long time
........................... [ … snip … ] ........++*++*

DH parameters of size 2048 created at /tmp/easy-rsa/easyrsa3/pki/dh.pem

And that's all! PKI hierarchy now should look like this:

buben@ca:/tmp/easy-rsa/easyrsa3$ tree pki
pki
├── ca.crt
├── certs_by_serial
│   └── 01.pem
├── dh.pem
├── index.txt
├── index.txt.attr
├── index.txt.old
├── issued
│   └── buben-vps.crt
├── private
│   ├── buben-vps.key
│   └── ca.key
├── reqs
│   └── buben-vps.req
├── serial
└── serial.old

Configure OpenVPN

Okay, VPN server is installed on the VPS and we also have generated CA and key files on our dedicated system. Now is time to create configuration and start VPN tunnel.

OpenVPN configuration files can't be stored anywhere. Best practice is store it in /etc/openvpn directory. I like to create separate directories for CA, keys and configuration files (both server and client, if necessary). My configuration skeleton looks like this:

root@buben-vps:~# tree /etc/openvpn
/etc/openvpn
├── ca_certificates
├── certs
└── server

Copy CA, DH and key files to VPS:

buben@ca:/tmp/easy-rsa/easyrsa3$ scp pki/ca.crt pki/dh.pem root@192.168.56.102:/etc/openvpn/ca_certificates
buben@ca:/tmp/easy-rsa/easyrsa3$ scp pki/issued/buben-vps.crt pki/private/buben-vps.key root@192.168.56.102:/etc/openvpn/certs

Now create configuration file:

root@buben-vps:~# nano /etc/openvpn/server/buben-vps.conf

And paste following lines:

port 1194
proto tcp
dev tun

ca /etc/openvpn/ca_certificates/ca.crt
cert /etc/openvpn/certs/buben-vps.crt
key /etc/openvpn/certs/buben-vps.key
dh /etc/openvpn/ca_certificates/dh.pem

server 10.9.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt

keepalive 10 120
comp-lzo
persist-key
persist-tun
verb 4

Now you can run a server from console:

root@buben-vps:~# openvpn --cd /etc/openvpn/server --config buben-vps.conf

Systemd unit

By default, openvpn package creates its systemd unit file at /lib/systemd/system/openvpn@.service which looks like this:

root@buben-vps:~# cat /lib/systemd/system/openvpn@.service
[Unit]
Description=OpenVPN connection to %i
PartOf=openvpn.service
ReloadPropagatedFrom=openvpn.service

[Service]
Type=forking
ExecStart=/usr/sbin/openvpn --daemon ovpn-%i --status /run/openvpn/%i.status 10 --cd /etc/openvpn --config /etc/openvpn/%i.conf
ExecReload=/bin/kill -HUP $MAINPID
WorkingDirectory=/etc/openvpn

[Install]
WantedBy=multi-user.target

Personally, I don't like forking services. So I created alternative unit file at /etc/systemd/system/openvpn-server@.service:

root@buben-vps:~# cat /etc/systemd/system/openvpn-server@.service
[Unit]
Description=OpenVPN service for %I
After=network.target
Documentation=man:openvpn(8)
Documentation=https://community.openvpn.net/openvpn/wiki/Openvpn23ManPage
Documentation=https://community.openvpn.net/openvpn/wiki/HOWTO

[Service]
Type=simple
ExecStart=/usr/sbin/openvpn --cd /etc/openvpn/server --config %i.conf
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

This unit file runs openvpn process at foreground, changes its working diretory at /etc/openvpn/server and loads appropriate configuration file. It also doesn't create any status file. If service crashes for any reason, it will be restarted after 5 seconds.

After creating this file, issue following command to reload all systemd units:

root@buben-vps:~# systemctl daemon-reload

Now you can instantiate and enable service to automatically start at boot up:

root@buben-vps:~# systemctl enable openvpn-server@buben-vps.service

And start the service:

root@buben-vps:~# systemctl start openvpn-server@buben-vps.service

Congratulations! VPN server is up and running.You can check for tunnel interface using following command:

root@buben-vps:~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:c5:b6:d8 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic eth0
       valid_lft 86381sec preferred_lft 86381sec
    inet6 fe80::a00:27ff:fec5:b6d8/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:e5:ef:bf brd ff:ff:ff:ff:ff:ff
    inet 192.168.56.102/24 brd 192.168.56.255 scope global dynamic eth1
       valid_lft 1203sec preferred_lft 1203sec
    inet6 fe80::a00:27ff:fee5:efbf/64 scope link
       valid_lft forever preferred_lft forever
4: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 100
    link/none
    inet 10.9.0.1 peer 10.9.0.2/32 scope global tun0
       valid_lft forever preferred_lft forever

Test the tunnel

If you want to test the VPN tunnel, you can configure OpenVPN test client for it. At firs, create key files:

buben@ca:/tmp/easy-rsa/easyrsa3$ ./easyrsa gen-req test nopass
buben@ca:/tmp/easy-rsa/easyrsa3$ ./easyrsa sign-req client test

Now create client configuration skeleton and copy appropriate key files. Let's say at /tmp/vpntest. Configuration skeleton with empty config file should look like this:

buben@test:~$ tree /tmp/testvpn/
/tmp/testvpn/
├── ca_certificates
│   └── ca.crt
├── certs
│   ├── test.crt
│   └── test.key
└── client
    └── test.conf

Note that client doesn't use dh.pem file.

Next edit client configuration file:

buben@test:~$ nano /tmp/testvpn/client/test.conf

And paste following content:

client
dev tun
proto tcp
remote 192.168.56.102 1194
resolv-retry infinite
nobind
persist-key
persist-tun

ca /tmp/testvpn/ca_certificates/ca.crt
cert /tmp/testvpn/certs/test.crt
key /tmp/testvpn/certs/test.key

comp-lzo
verb 4

Once you have your client configured, you can establish a VPN tunnel:

buben@test:~$ sudo openvpn --cd /tmp/testvpn/client/ --config test.conf

And verify that your tunnel is working:

buben@test:~$ ping -c 5 10.9.0.1
PING 10.9.0.1 (10.9.0.1) 56(84) bytes of data.
64 bytes from 10.9.0.1: icmp_seq=1 ttl=64 time=0.413 ms
64 bytes from 10.9.0.1: icmp_seq=2 ttl=64 time=0.412 ms
64 bytes from 10.9.0.1: icmp_seq=3 ttl=64 time=0.413 ms
64 bytes from 10.9.0.1: icmp_seq=4 ttl=64 time=0.425 ms
64 bytes from 10.9.0.1: icmp_seq=5 ttl=64 time=0.394 ms

--- 10.9.0.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3999ms
rtt min/avg/max/mdev = 0.394/0.411/0.425/0.020 ms

To be continued

Now we have created VPN connection for IoT network. In the next article I'll describe MQTT broker configuration.