čtvrtek 16. června 2016
Monitoring swimming pool with mqopen
pátek 10. června 2016
ESP8266 node firmware update
- Added service topics - I added various service MQTT topics which node sends into network when it connects. This is useful for statistics purposes. debugging or for searching nodes with obsolete firmware or hardware versions.
- Device class - I divided nodes into two basic classes:
- Sensor - Sensor device sends data to network.
- Reactor - Reactor devices reacts to data from sensors and take some actions. I have already implemented some prototype code, but this part is in the main development.
- Asynchronous sensors - I implemented abstraction layer for synchronous and asynchronous sensors. Synchronous sensor periodically sends data. Asynchronous ones waits for some event and sends data only when event happens (with retain bit set). This feature takes advantage of MQTT to save bandwith and improve response time for a other devices. I'm building a door sensor which reports every time when door status changes (open / closed).
- Initial data publish - Nature of asynchronous sensor it that it sends data only when event happens. I implemented logic which sends sensor state when node becomes online, if the state is relevant.
- Various bug fixes - I fixed some bugs in the code and hopefully improved device reliability.
čtvrtek 19. května 2016
mqopen project
I'm very happy to introduce you my work as mqopen project.
I created small website and wiki pages with images, build instructions and other tutorials for open source community.
Hope you'll like it.
úterý 29. září 2015
Central MQTT broker – mosquitto
Let's continue in configuring central MQTT broker. In previous article I described the OpenVPN part. Now is time for MQTT broker itself.
Broker implementation is mosquitto. It is open source software, compliant with MQTT version 3.1 and 3.1.1. First of all, lets install it:
root@buben-vps:~# apt-get install mosquitto
Configuration
Central broker should be configured with following attributes:
- Listen on VPN interface – This is what I discussed if previous post. To ensure that only authorized clients can access broker, it must listen on tunnel interface only.
- Client authentication – Even if key owners has access to broker, I like to add username/password protection. Credentials are send in plain text, but the are encrypted in VPN tunnel for outside world.
Broker configuration file should look like this:
#persistance setting persistence true persistence_location /var/lib/mosquitto/ # listen on localhost interface bind_address localhost # listen on VPN interface listener 1883 10.9.0.1 # User authentication password_file /etc/mosquitto/mosquitto.passwd allow_anonymous false
irst part specifies persistence directory. This is useful when clients uses messages with retain feature and broker is restarted for some reason. Persistence will save retain messages over broker restarts.
Next part defines listen interfaces. Default listener listen on localhost interface, second one listen on VPN interface. I want to listen on localhost for some data processing services which runs on the same machine.
Finally, last part defines user authentication. I use this feature for possibility to grant access to my VPN tunnel for other devices, but ensure that they cannot mess my MQTT data exchange. Unfortunately, authentication is required to clients from all interfaces. It would be great to require authentication for clients at VPN interface only.
Password file
To authenticate MQTT users, we need to store user names and passwords somewhere. Mosquitto comes with mosquitto_passwd utility for managing password files. To create new file issue following commad:
root@buben-vps:~# mosquitto_passwd -c /etc/mosquitto/mosquitto.passwd testuser
Command asks you for password and store it in hashed form in appropriate file. Unfortunately, this utility cannot accept password via command line argument, so using some generated password is quite complicated.
I also noticed, that command can accept -U option, which updates password file to use hashed password instead of plain text. I don't know why, but it re-hash already hashed passwords. This makes this feature little bit useless.
systemd unit
By default, mosquitto doesn't use standard systemd unit. It is started using traditional System V script located at /etc/init.d/mosquitto. You can check this by following command:
root@buben-vps:~# systemctl status mosquitto
● mosquitto.service - LSB: mosquitto MQTT v3.1 message broker
Loaded: loaded (/etc/init.d/mosquitto)
Active: active (running) since Tue 2015-09-29 14:41:55 CEST; 9s ago
CGroup: /system.slice/mosquitto.service
└─3037 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
I like modern systemd units, which runs services in foreground. So let's create is. Stop current mosquitto service at first:
root@buben-vps:~# systemctl stop mosquitto root@buben-vps:~# update-rc.d mosquitto remove root@buben-vps:~# rm /etc/init.d/mosquitto
Now create systemd unit file:
root@buben-vps:~# nano /etc/systemd/system/mosquitto.service
And paste following content:
[Unit] Description=MQTT v3.1 message broker After=network.target openvpn-server@buben-vps.service Requires=network.target openvpn-server@buben-vps.service [Service] Type=simple ExecStart=/usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf Restart=always [Install] WantedBy=multi-user.target
Lets talk about systemd unit configuration directives little bit. Requires directive specifies that this unit needs active network.target unit and openvpn-server@buben-vps.service unit. If any unit listed there is stopped or fails, this unit will be stopped as well. After directive defines activation order. Activation of this unit is delayed until all units listed at After directive are started up.
Both After and Requires directives has same values. network.target tells that service should be started after network related stuff is initialized (such as initialization of TCP/IP stack). This dependecy is much more important for shut-down procedure. It instructs systemd that this service should be terminated before network connectivity. This ensures that broker can nicely close all its connections. openvpn-server@buben-vps.service defines dependency on service which provides VPN connectivity. Mosquitto is configured to listen at VPN tunnel IP address, so this service must be started up before mosquitto gets activated.
Other configuration directives are self explanatory. If you are not sure about their meaning, check systemd documentation.
After unit file is created, reload systemd configuration, enable unit and start it up:
root@buben-vps:~# systemctl daemon-reload root@buben-vps:~# systemctl enable mosquitto root@buben-vps:~# systemctl start mosquitto
And that's it. Central MQTT broker is up and running. In next articled I'll descibe building of different kinds of local MQTT brokers.
čtvrtek 17. září 2015
Central MQTT broker – configure OpenVPN server
In following articles I'll describe another part of my IoT project – building infrastructure for data exchange. Remember from my first post, this infrastructure consists of two main parts: central MQTT broker and bunch of local brokers.
First few articles will be about configuring central MQTT broker. It requires a computer with public IP address. Rent a VPS is pretty good choice for this purpose. Operating system which runs on your VPS doesn't really matter. In my case, I'm using Debian 8 with systemd. Configuration steps may differ for other system setup.
Encrypted connection
Security is one of my major concerns. Each local MQTT broker must establish a connection with central broker over SSL encrypted tunnel. There are basically two possibilities:
- Configure mosquitto to use encrypted connection.
- Establish VPN tunnel between hosts and configure central broker to listen on VPN interface only.
Second way have more benefits. This approach delegates responsibility for encryption into another piece of software. It is also possible to run other services at VPN interface, such as HTTP server for some diagnostic applications. Only downside is that VPN means more complex configuration.
For this tutorial, I use virtual system running in VirtualBox. I assigned two network interfaces to it. One in NAT mode (access to the Internet) and second connected to host-only network. This is for reason that I don't want to use my real public IP addresses in this tutorial. I can access my virtual machine at 192.168.56.102.
Install OpenVPN
First step is install VPN daemon. Best choice for me is to use OpenVPN. It is open source and well known VPN solution with huge community. To install OpenVPN issue following command:
root@buben-vps:~# apt-get install openvpn
And that's it. VPN server software is installed. Now comes the hard part.
Certificates and keys
To establish an encrypted connection with VPN server, we need an encryption key. We also needs some third party authority which verifies, that encryption key belongs to authorized user.
In another words, to establish connection we have to create encryption key for both server and client, create certificate authority (CA) to sign keys and create certification requests. This is generally done using openssl library tool with the same name – openssl. But create all these things with openssl command requires lot of commands with many parameters. For that reason EasyRSA was created, which simplifies this procedure.
Following steps should be done on some secured system dedicated for storing CA and key signing. It is recommended to keep this computer in locked room and completely disconnected from the network. For purpose of this article, I create all files on my /tmp directory.
First of all download EasyRSA from GitHub:
buben@ca:/tmp$ git clone https://github.com/OpenVPN/easy-rsa
Next, change working directory and create configuration file:
buben@ca:/tmp$ cd easy-rsa/easyrsa3 buben@ca:/tmp/easy-rsa/easyrsa3$ cp vars.example vars buben@ca:/tmp/easy-rsa/easyrsa3$ nano vars
Now edit configuration values. Most important are following variables:
#set_var EASYRSA_REQ_COUNTRY "US" #set_var EASYRSA_REQ_PROVINCE "California" #set_var EASYRSA_REQ_CITY "San Francisco" #set_var EASYRSA_REQ_ORG "Copyleft Certificate Co" #set_var EASYRSA_REQ_EMAIL "me@example.net" #set_var EASYRSA_REQ_OU "My Organizational Unit"
Just uncomment them and assign them appropriate literals. If you want to omit some of these values (EASYRSA_REQ_OU in my case) fill it with “.” (dot).
After EasyRSA is configured, you can initialize public key infrastructure (PKI):
buben@ca:/tmp/easy-rsa/easyrsa3$ ./easyrsa init-pki
After that, empty pki should look like this:
buben@ca:/tmp/easy-rsa/easyrsa3$ tree pki/ pki/ ├── private └── reqs
Building key files
Now you can build certificate authority and key files. First thing is CA:
buben@ca:/tmp/easy-rsa/easyrsa3$ ./easyrsa build-ca Note: using Easy-RSA configuration from: ./vars Generating a 2048 bit RSA private key .......................+++ .....+++ writing new private key to '/tmp/easy-rsa/easyrsa3/pki/private/ca.key.cAsM68O5hi' Enter PEM pass phrase: Verifying - Enter PEM pass phrase: ----- You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Common Name (eg: your user, host, or server name) [Easy-RSA CA]:buben-ca CA creation complete and you may now import and sign cert requests. Your new CA certificate file for publishing is at: /tmp/easy-rsa/easyrsa3/pki/ca.crt
Command build-ca creates CA files encrypted with pass phrase. You can append nopass argument to create plaintext CA. I recommend you to protect it. Command also asks for common name. It should be some meaningful string, as the prompt suggests.
Next, create server key and certificate request file:
buben@ca:/tmp/easy-rsa/easyrsa3$ ./easyrsa gen-req buben-vps nopass Note: using Easy-RSA configuration from: ./vars Generating a 2048 bit RSA private key .+++ ..............+++ writing new private key to '/tmp/easy-rsa/easyrsa3/pki/private/buben-vps.key.3pklh1AD07' ----- You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter '.', the field will be left blank. ----- Common Name (eg: your user, host, or server name) [buben-vps]: Keypair and certificate request completed. Your files are: req: /tmp/easy-rsa/easyrsa3/pki/reqs/buben-vps.req key: /tmp/easy-rsa/easyrsa3/pki/private/buben-vps.key
In this case I recommend you to use nopass argument. Otherwise you will have to manually type a pass phrase every time you want to start a VPN server, which is nonsence.
Command again asks for common name. In case of server key, it must match the server hostname. This is the way how the client verifies that it is authenticating with correct machine. If common name doesn't match to server hostname, certificate verification fails.
Next, sign server request with your CA:
buben@ca:/tmp/easy-rsa/easyrsa3$ ./easyrsa sign-req server buben-vps
Note: using Easy-RSA configuration from: ./vars
You are about to sign the following certificate.
Please check over the details shown below for accuracy. Note that this request
has not been cryptographically verified. Please be sure it came from a trusted
source or that you have verified the request checksum with the sender.
Request subject, to be signed as a server certificate for 3650 days:
subject=
commonName = buben-vps
Type the word 'yes' to continue, or any other input to abort.
Confirm request details: yes
Using configuration from /tmp/easy-rsa/easyrsa3/openssl-1.0.cnf
Enter pass phrase for /tmp/easy-rsa/easyrsa3/pki/private/ca.key:
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
commonName :ASN.1 12:'buben-vps'
Certificate is to be certified until Sep 12 14:53:09 2025 GMT (3650 days)
Write out database with 1 new entries
Data Base Updated
Certificate created at: /tmp/easy-rsa/easyrsa3/pki/issued/buben-vps.crt
If CA is encrypted, command asks for its pass phrase.
Last step is create Diffie-Hellman parameters for initial key exchange:
buben@ca:/tmp/easy-rsa/easyrsa3$ ./easyrsa gen-dh Note: using Easy-RSA configuration from: ./vars Generating DH parameters, 2048 bit long safe prime, generator 2 This is going to take a long time ........................... [ … snip … ] ........++*++* DH parameters of size 2048 created at /tmp/easy-rsa/easyrsa3/pki/dh.pem
And that's all! PKI hierarchy now should look like this:
buben@ca:/tmp/easy-rsa/easyrsa3$ tree pki pki ├── ca.crt ├── certs_by_serial │ └── 01.pem ├── dh.pem ├── index.txt ├── index.txt.attr ├── index.txt.old ├── issued │ └── buben-vps.crt ├── private │ ├── buben-vps.key │ └── ca.key ├── reqs │ └── buben-vps.req ├── serial └── serial.old
Configure OpenVPN
Okay, VPN server is installed on the VPS and we also have generated CA and key files on our dedicated system. Now is time to create configuration and start VPN tunnel.
OpenVPN configuration files can't be stored anywhere. Best practice is store it in /etc/openvpn directory. I like to create separate directories for CA, keys and configuration files (both server and client, if necessary). My configuration skeleton looks like this:
root@buben-vps:~# tree /etc/openvpn /etc/openvpn ├── ca_certificates ├── certs └── server
Copy CA, DH and key files to VPS:
buben@ca:/tmp/easy-rsa/easyrsa3$ scp pki/ca.crt pki/dh.pem root@192.168.56.102:/etc/openvpn/ca_certificates buben@ca:/tmp/easy-rsa/easyrsa3$ scp pki/issued/buben-vps.crt pki/private/buben-vps.key root@192.168.56.102:/etc/openvpn/certs
Now create configuration file:
root@buben-vps:~# nano /etc/openvpn/server/buben-vps.conf
And paste following lines:
port 1194 proto tcp dev tun ca /etc/openvpn/ca_certificates/ca.crt cert /etc/openvpn/certs/buben-vps.crt key /etc/openvpn/certs/buben-vps.key dh /etc/openvpn/ca_certificates/dh.pem server 10.9.0.0 255.255.255.0 ifconfig-pool-persist ipp.txt keepalive 10 120 comp-lzo persist-key persist-tun verb 4
Now you can run a server from console:
root@buben-vps:~# openvpn --cd /etc/openvpn/server --config buben-vps.conf
Systemd unit
By default, openvpn package creates its systemd unit file at /lib/systemd/system/openvpn@.service which looks like this:
root@buben-vps:~# cat /lib/systemd/system/openvpn@.service [Unit] Description=OpenVPN connection to %i PartOf=openvpn.service ReloadPropagatedFrom=openvpn.service [Service] Type=forking ExecStart=/usr/sbin/openvpn --daemon ovpn-%i --status /run/openvpn/%i.status 10 --cd /etc/openvpn --config /etc/openvpn/%i.conf ExecReload=/bin/kill -HUP $MAINPID WorkingDirectory=/etc/openvpn [Install] WantedBy=multi-user.target
Personally, I don't like forking services. So I created alternative unit file at /etc/systemd/system/openvpn-server@.service:
root@buben-vps:~# cat /etc/systemd/system/openvpn-server@.service [Unit] Description=OpenVPN service for %I After=network.target Documentation=man:openvpn(8) Documentation=https://community.openvpn.net/openvpn/wiki/Openvpn23ManPage Documentation=https://community.openvpn.net/openvpn/wiki/HOWTO [Service] Type=simple ExecStart=/usr/sbin/openvpn --cd /etc/openvpn/server --config %i.conf Restart=always RestartSec=5 [Install] WantedBy=multi-user.target
This unit file runs openvpn process at foreground, changes its working diretory at /etc/openvpn/server and loads appropriate configuration file. It also doesn't create any status file. If service crashes for any reason, it will be restarted after 5 seconds.
After creating this file, issue following command to reload all systemd units:
root@buben-vps:~# systemctl daemon-reload
Now you can instantiate and enable service to automatically start at boot up:
root@buben-vps:~# systemctl enable openvpn-server@buben-vps.service
And start the service:
root@buben-vps:~# systemctl start openvpn-server@buben-vps.service
Congratulations! VPN server is up and running.You can check for tunnel interface using following command:
root@buben-vps:~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:c5:b6:d8 brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic eth0
valid_lft 86381sec preferred_lft 86381sec
inet6 fe80::a00:27ff:fec5:b6d8/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:e5:ef:bf brd ff:ff:ff:ff:ff:ff
inet 192.168.56.102/24 brd 192.168.56.255 scope global dynamic eth1
valid_lft 1203sec preferred_lft 1203sec
inet6 fe80::a00:27ff:fee5:efbf/64 scope link
valid_lft forever preferred_lft forever
4: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 100
link/none
inet 10.9.0.1 peer 10.9.0.2/32 scope global tun0
valid_lft forever preferred_lft forever
Test the tunnel
If you want to test the VPN tunnel, you can configure OpenVPN test client for it. At firs, create key files:
buben@ca:/tmp/easy-rsa/easyrsa3$ ./easyrsa gen-req test nopass buben@ca:/tmp/easy-rsa/easyrsa3$ ./easyrsa sign-req client test
Now create client configuration skeleton and copy appropriate key files. Let's say at /tmp/vpntest. Configuration skeleton with empty config file should look like this:
buben@test:~$ tree /tmp/testvpn/
/tmp/testvpn/
├── ca_certificates
│ └── ca.crt
├── certs
│ ├── test.crt
│ └── test.key
└── client
└── test.conf
Note that client doesn't use dh.pem file.
Next edit client configuration file:
buben@test:~$ nano /tmp/testvpn/client/test.conf
And paste following content:
client dev tun proto tcp remote 192.168.56.102 1194 resolv-retry infinite nobind persist-key persist-tun ca /tmp/testvpn/ca_certificates/ca.crt cert /tmp/testvpn/certs/test.crt key /tmp/testvpn/certs/test.key comp-lzo verb 4
Once you have your client configured, you can establish a VPN tunnel:
buben@test:~$ sudo openvpn --cd /tmp/testvpn/client/ --config test.conf
And verify that your tunnel is working:
buben@test:~$ ping -c 5 10.9.0.1 PING 10.9.0.1 (10.9.0.1) 56(84) bytes of data. 64 bytes from 10.9.0.1: icmp_seq=1 ttl=64 time=0.413 ms 64 bytes from 10.9.0.1: icmp_seq=2 ttl=64 time=0.412 ms 64 bytes from 10.9.0.1: icmp_seq=3 ttl=64 time=0.413 ms 64 bytes from 10.9.0.1: icmp_seq=4 ttl=64 time=0.425 ms 64 bytes from 10.9.0.1: icmp_seq=5 ttl=64 time=0.394 ms --- 10.9.0.1 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 3999ms rtt min/avg/max/mdev = 0.394/0.411/0.425/0.020 ms
To be continued
Now we have created VPN connection for IoT network. In the next article I'll describe MQTT broker configuration.
pátek 4. září 2015
MQTT node software
In last article I described a hardware aspects of my IoT endpoint device. Lets look at it software now.
Software must handle DHT22 sensor for measurements, implement enc28j60 chip driver, TCP/IP stack for it and MQTT client for sending data to network. My first requirement is to use plain C instead of using any Arduino software. Reason for this is it gives me lot more possibilities, smaller code footprint (I assume) and more convenient way to fixing a bugs. Also once a software is done, it can be very easily reused in another projects.
TCP/IP stack
Most challenging part of node software is TCP/IP stack. Fortunately, there is already quite good implementation called avr-uip, which is uIP project ported to AVR platform.
uIP is minimalistic TCP/IP stack written by Adam Dunkels designed for very small embedded systems. It is designed as event-based framework. It requires some learning, but it's not too hard.
My friend from our local hackerspace already made very similar project which uses this TCP/IP stack. So his work can be used as good reference point.
Only downside is that avr-uip is very outdated, so I had to make some changes to it and fix some bugs. But it works now.
Get it work
With working TCP/IP stack is very easy to get data from the AVR over the network. I found some another projects which implement enc28j60 driver, MQTT client and DHT22 driver, forked them and merged them into my software. I had to create man state machine and after some refactoring and bug fixes my node started to sending MQTT data.
You can find the code on the projects GitHub page. If you want to build your own MQTT node, please read attached readme file for configuration instructions.
Future features
I want to create more user-friendly device, which doesn't need to be manually configured for each build. Main features should be some DHCP client code to dynamically assign node IP address.
Another major feature should be DNS client to resolve local broker IP address from DNS server.
This could allow to develop very cheap plug and play device which doesn't require any configuration. One challenging aspect is very limited resources of used MCU, which has only 2kB of RAM.
úterý 28. července 2015
Simple MQTT node
In the previous article I described a basic idea of building my network of interconnected devices. In next few articles I'll describe building of very simple sensor with network interface, which can send data using MQTT protocol.
Node requirements:
- Use well known Arduino hardware
- Software written in plain C
- Cheap components
- Scalable design
- Easy to use
MCU
Because I don't want to create my own board, I have to use some existing one. For that reason I decided to use well known Arduino Nano board with atmega328p MCU. I have lot of these in my shelf. This MCU has 32KBytes of flash storage for code, 1KByte of EEPROM as non-volatile storage and 2KBytes of RAM.
I have plan to use Arduino Pro Mini for final product. It costs about 2.50 USD on ebay.
Another advantage of using Arduino hardware is lot of existing code for it. I can easily connect some components, upload existing sketch and check, if my wiring is correct. After verifying that hardware is connected properly, I could start writing my own software. This safes me lot of work by finding bugs in both hardware and software.
Software
Although I'm using Arduino hardware, I'm not using any Arduino software. I decided to implement all software logic in plain C instead.
I thing that Arduino is really great for learning of programming microcomputers or to make quick proof of concept sketch. But its code is quite messy, not efficient and it limits many hardware possibilities. So for making some serious project it is insufficient.
Use of plain C has many advantages. You have complete control about memory usage and its optimization, interrupts, timers, power consumption and other things. You have to also create Makefile, which brings another control in code configuration, compilation and optimization.
Ethernet controller
There are two common Ethernet controller chips. WIZnet w5100 and Microchip enc28j60. W5100 is chip used in official Arduino Ethernet Shield. It has its own TCP/IP stack implemented in hardware and it is well supported by community. But it is twice as expensive as the enc28j60 chip. Because I want to make many of these devices, price is crucial. I had already some enc28j60 modules in my shelf.
Using enc28j60 chip requires more work in software. It requires to implement own TCP/IP stack in software layer. But it seems that this work is already done by someone else.
Sensors
For development purposes I decided to make a node only with DHT-22 (also known as AM2302) sensor. This sensor can measure relative humidity and temperature in 2s time period. DHT-22 is more expensive variant of DHT-11, which has very limited accuracy and measurement range.
I used this sensor mainly for development. Final software should be capable to use any other sensor or device.
Wiring
Finally, wiring of my prototype looks like this.
Next time I'll describe node software and building MQTT brokers.






