Prosit Neujahr 2016

Das war es also das Jahr 2015, draussen explodieren die Raketen und verkünden das Ende. Ja, mein Blog ist noch jung und ich möchte mich allen Lesern bedanken und freue mich schon auf das nächste Jahr. Wie man auch mit 30 keine Biografie macht, macht man auch ein paar Beiträgen noch keinen Jahresrückblick. Aber man fasst gute Vorsätze zu fassen, schmiedet große Pläne und teilt diese mit seiner Leserschaft.

Also, für das neue Jahr habe ich mir vorgenommen die Updaterate von ca. einem Beitrag pro Woche beizubehalten. Ich hoffe das gelingt denn ich mache das hier schließlich nur als Hobby und bin in meinem Brotberuf mitunter auch sehr eingespannt. Außerdem habe ich vor demnächst eine englische Übersetzung meines Blogs anzubieten, in der dann die Beiträge ein paar Tage versetzt auch auf Englisch erscheinen. Vielleicht gelingt es mir auch irgendein Cross-Over Projekt mit einer ganz anderen Welt einzufädeln. Mich würde irgend etwas mit Kunst interessieren also, wenn wer eine Idee hat und einen Partner für das Technische sucht dann meldet euch.

Auf der Softwareseite steht ein Projekt mit Erlang auf dem Raspi sowie ein Projekt mit MATLAB/Simulink auf dem Programm. Bei Erlang gefällt wie das Thema Verteilung dort gehandhabt wird und außerdem ist es mal ein Exot unter den Programmiersprachen. Mathworks macht in letzter Zeit viel in Richtung Raspberry, Arduino, LEGO Mindstorms usw und das möchte ich mir mal ansehen. Was es genau wird weiß ich selber noch nicht also lasst euch einfach überraschen.

Auf der Hardwareseite habe ich noch nicht so konkrete Pläne. Ich habe mir ein Lillypad besorgt und suche noch eine Idee dafür, aber ansonsten ist noch kein Plan in der Pipeline. Solltet ihr Ideen haben oder eine Sache von der Ihr denkt “Hey das würde mich mal interessieren” dann nehmt euch ein Herz und schreibt mir, ich freue mich immer über den Austausch mit Geistesbrüdern.

So, nun gehts aber auch für mich ans feiern und es bleibt mir euch etwas zu wünschen. Da man als Techniker nur all zu gut weiß wie elementar das Thema Scheitern zum Leben dazu gehört wünsche ich euch ein Prosit Neujahr 2016 und wünsche euch für das Jahr 2016, dass ihr aus euren Fehlern und Irrwegen die richtigen Lehren ziehen könnt.

 

Irrigation – Part 4: Automatic Database Backup

I just went trough a Scottish shower of emotions. IT people know this very well. I had the plan to “just” activate the eth0 interface on my RaspberryPI to get if off my WLAN. Unfortunately that was not so easy. For a moment I was afraid I had to power it off. I previous cases this caused a damaged file system. I was lucky, there was no such problem but I realized that I don’t have any backup of my database at all.

I’m not that kind of guy that has several servers running 24/7 in his house so I decided to put the backup into my Dropbox. I create a folder called rpi2backup in my Dropbox and a folder /tmp/dbbackup/ on my Raspberry. For uploading my backup file I use the uploader from Andrea Faprizi. After installing the uploader I just follow the instruction of the installer. I don’t know exactly why but I had to do the procedure twice.

The next step is the MariaDB backup. I will not reinvent the wheel here so I just follow the tutorial here. I have modify the script in the tutorial a bit to upload the backup automatically after creation.

Now I will sleep better.

Irrigation– Part 3: Visualization

For sure I want to see the measuring values from by automatic plant irrigation. But it is a bit annoying to use my database tools all the time. That’s the reason for me to make a small visualisation. I want to see the actual values as text and two trends. One showing the last 24h and the other showing the last week.

I have already my webserver so the drug of choice for me is making a website. But that’s not the only reason. It is becoming state of the art to make front-ends with web technologies. The advantages are obvious, simple deployment, platform independent (especially mobile devices). Furthermore web technologies have gained power over the last years (e.g. web socket, HTML5, SVG). Also rendering performance improved enough to render trends fluidly.

I always develop and test queries separately in Workbench IDE. For a better readability in my trends I compress the data a bit. The 24h data shows the average per hour. The data for the last week shows the average per quarter day. The value for soil moisture is scaled 1024 = 100%.

Usually I would put the queries in stored procedures and use them as they offer some advantages but unfortunately the mysqli does not support stored procedures returning multiple row sets. Therefore you find the queries hard-coded in the PHP script.

For displaying the trend data I use a 3rd party charting library. I use Chart.js in this case. It is for sure not the most powerful library but it is slim, fast, free and open source  (MIT license).

That’s the code for the website

The result is a nice website that also works great on mobile devices.

Datenanzeige

Irrigation- Part 2: Data Recording

I use a Raspberry PI 2 as server for my data recording. Additionally you need a SD card, a power supply (I use just a USB cable to connect it to my access point) and if needed a WLAN stick. I strongly recommend to use a case e.g. like this one

First of all the Raspberry needs an operating system (raspbian) installed. There are plenty of manuals for that in the internet. As you have seen in the Arduino project I send my data using a web server. Writing a small socket server would have been the more lightweight solution but it is really a tough job to program a service that runs 24/7 stable and robust. Therefore I like to use some kind of a well-known and tested middleware. For data recording the combination of Apache webserver, PHP and MySQL (LAMP) is an ideal solution for me. There are many tutorials for setting up such a system in the internet (I used this one). Instead of using the “real” MySQL I use the open source successor MariaDB.  I just replace

apt-get install apache2 apache2-utils php5 libapache2-mod-php5 php5-mysql mysql-server mysql-client phpmyadmin -y

by

apt-get install apache2 apache2-utils php5 libapache2-mod-php5 php5-mysql mariadb-server mysql-client phpmyadmin -y

MariaDB and MySQL are compatible on a wide scale. So you can use almost all the tools you use for MySQL also for MariaDB.

To improve my tool chain I also install a samba server on the Raspberry. You find a tutorial here. I develop on a Windows platform and the Samba server enables me to mount the wwwroot of the raspberry as network share. So I can simply develop on this share without taking a detour using FTP. For developing on the MariaDB you can use the installed PHPMyAdmin or you install the MySQL Workbench instead. The Workbench throws an error when connecting to a MariaDB but it’s working without any problems. It’s a matter of taste which way you prefer. Ich like the Workbench because I’m used to work in a real IDE for database development so the PHPMyAdmin feels a bit unfamiliar for me.

After setting up the Raspberry and the development tool chain you can create a database. I call my database simply “templogg”

Then I  create a table called “tbllogging”

The column “dt” is a datetime column and acts as my primary key. The columns temp, humidity and soil are data type real and take the measuring values. As you could see in the Arduino code, the PHP script, called “newrecord.php” is missing. I put the script directly in the wwwroot.

The script requires the MySQL extension in the php.ini.

The mysqli extension is not enabled by default, so the php_mysqli.dll DLL must be enabled inside of php.ini. In order to do this you need to find the php.ini file (typically located in c:\php), and make sure you remove the comment (semi-colon) from the start of the line extension=php_mysqli.dll, in the section marked [PHP_MYSQLI]. – http://php.net/manual/en/mysqli.installation.php

After that I try my script. I call the URL with some test data. You can do that with any browser.

http://raspberryip/newrecord.php?temp=33.00&humidity=44.00&soil=55.00

Then I check if the data has been recorded correctly.

When it works I delete the test data.

Now I start my Arduino and check if the recording works fine with that too. If that’s the case the prototype will do measurements on a houseplant for the next weeks.

Prototype in Aktion

In the meanwhile I will make a small website to display my measurements without using my database IDE.

Irrigation- Part 1: A Prototype

In a first step I will make a prototype for my sensor board. Usually you build a prototype just on a breadboard. But I would like to use my prototype for some weeks to see how my sensors work and how much energy is needed. Therefore I use a prototype circuit board instead.

My shopping list for that:

Additionally I need:

  • A prototype circuit board
  • 3 pcs. 10k Resistors (can be even bigger as long aa all 3 are equal)
  • 1 electrolytic capacitor (>500uF)
  • 1 Transisotr (e.g. BC547C)
  • 1 3K3 Resistor
  • a pin header
  • a jumper
  • a soldering iron

I know that there a many statements about the right soldering iron in the internet but for me personally a cheap soldering station is more than enough. There is also a good desoldering pump coming with the station and every beginner will need it. If you have too much money then spend it in a quite good solder. As power supply I use a USB powerbank that I’ve received as a giveaway. You can also get one for less than 10€ (e.g. here)

I try to do not solder items directly when building a prototype. Therefore I use pin headers. The disadvantage is that the prototype is not that robust but the advantage is I can easily borrow some parts without taking the soldering iron.

The ESP8266 just takes 3.3V so I take the USB to TTL converter also as voltage converter. Be sure to put the jumper to 3.3V. To safe energy I will put my ESP in deep-sleep. To prevent my sensors taking power when they are not used I use a transistor to power them on only when needed. The ESP needs quite a high current when connecting to the WLAN. I use an electrolytic capacitor parallel to the voltage supply. Sizing of that is no rocket science, it should be somewhere between 400uF and 2200uF (6-10V). For programming the ESP the CH_PD Pin has to be put on ground. Some recommend a button for that but I prefer a jumper.

The manual for the ESP8266 says that for wake up from deep-sleep pin 16 has to be connected to CH_PD. Unfortunately for some reasons this is not working with mine. I use the RESET for the wakeup instead. This prevents using some options for wakeup but I don’t care for the prototype.

 

Der Prototyp in EAGLE
Prototype in EAGLE

It is a puzzle to place all the components on the board. I tried to save as much space because I had a small leftover to be used.

Meine Prototyen Platine
My final prototype

For programming my ESP I use the Arduio IDE. You have to install the ESP board in the IDE. Add in File->Properties->Additional Board Manager URLs: http://arduino.esp8266.com/stable/package_esp8266com_index.json  Select Board „…“ > Board Manager… , choose esp8266 and press install. After that you can program the ESP like any other Arduino. For programming the jumper has to be set on starting the board. Sometimes the upload fails with some strange error messages. If this happens I simply restart the ESP and try it again. Remember to remove the jumper after uploading the software. Wake-up from deep-sleep is not working when the jumper ist set.

 

Automatic Plant Irrigation System

I love home-grown vegetables. Eat tomatoes straight from the bush and you know how tomatoes have to taste like. I grow my tomatoes in buckets on my balcony and it really works fine. Unfortunately I’m a bit lazy in watering so my harvest was not as good as it could be in the last years.

therefore I decided to make an automatic plant irrigation system. I will use ESP8266 together with temperature/humidity and soil humidity measurements and send the actual values with WLAN to my data central, a Raspberry Pi. The measuring unit should be completely autonomous so they will be quipped with small solar panels each. The Raspberry will be used for data recording and works as controller for the watering unit. For watering I will properly need some pumps because unfortunately I have no chance to make the watering by gravity only.

Last but not least, it should not cost too much!

Goodbye Magic Cards – Part 2: PHP Script

After preparing the data last time I need to do the script. I use PHP because I like the language and I’m quite fit in PHP programming. Additionally it can be used on Windows and Linux equally. This time I use the  Netbeans IDE, even if a simple text pad would be sufficient as well. For sure such a program could be done in any other programming language (such as Phyton, C#, ..) as well.

Similar last time I must have a look on the HTML code. I search for the tags and can find them easily. Now I have to search for some code snippets that enable me to identify the single columns of the table. While doing such work I always have an empty text file open in my Notepad++ and copy interesting info right into that file. I can copy the string out of this file whenever I need it later on.

Then I make the script. Is has to run through each line of my CSV file and call the target website with the parameters. The script then parse the website and writes the parameters together with the price in a new CSV file.

I use REGEX for parsing the website. Some programmers don’t really like regex because it is so mighty that it can be seen as separate programming language. Honestly it is a bit complicated at the beginning but it is ideal for such a job and there are some helpful resources. For e.g. I strongly suggest to use https://regex101.com/ for developing and testing the expressions.

To make the code run the following extension have to be enabled in the php.ini

  • extension=php_openssl.dll
  • extension=php_curl.dll

Additionally “allow_url_fopen = On” has to be set.

 

The result is a nice CSV file with all the information I need. I open it in my spreadsheet program and can easily build the sum.

Goodbye Magic Cards – Part 1: Reverse Engineering, Notepad Action and Spreadsheet Magic

I want to make a small program to get the value of my Magic Card collection. The script will automatically calls the information from a 3rd party website. Therefore I just had to sit down for approx. 2 hours and enter the card names into a spreadsheet application. As a miser I use OpenOffice.Calc for private. For my personal needs that’s more than enough even if its scope cannot compete with Microsoft Excel.

I enter all the card names and their edition in my table. The edition is necessary because it might have an impact on the card’s value.

OpenOffice Calc Tabellenblatt
OpenOffice Calc Table

Now I can start to search for a suitable website to get the price information. To keep it simple I search for a site that hands over the parameters in the site URL. Therefore I use the site’s search function so see how it is working.

In my case I just search for the first of my cards with the flowery name “Woodripper” out of “Nemesis” series. That’s the sites response

Testabfrage
Test query

Looking a the parameters after the “/?” I understand that the card’s name is handed over in clear text: “cardName=Woodripper”. The variable names are easy to understand so I see that idExpansion must be the series. Unfortunately there is no clear text, it’s coded in an indentifier.The other variables are search options that are not useful for me at the moment. I also see that the result is displayed in a nice HTML table and that will simplify parsing later on.

My next job is to find out how the idExpansion relates to the name of the series. The series names are shown in the website, therefore this information has to somewhere be in the HTML code. I have a look on the source code of the search page. This is pretty simple and works in all browsers similar. You have to right-click somewhere on the website and select the item called like “display source code”. Unfortunately I get the result almost unformatted. That’s not unusual but at the first look I cannot see anything interested. It would be possible to use an advanced HTML editor to beautify my code but I use my beloved Notepad++ instead.

I copy the complete code into a new file and search for the string “Nemesis”. I find the  search string embedded intags. Understanding the syntax of option tags I know that they will tell me what I have searched for: the ID and the clear text.

In my example the option tags look like this:

<option  value=”32″>Nemesis</option>

It is <option value=”idExpansion”>clear text</option>

I change the formatting of the code using the search and replace function of Notepad++. I add a line break (\r\n) after each </option> tag.

Umformatieren
Change the formatting

When I scroll down the document now I can easily find the data I’m searching for. Now I just delete everything before the firstand after the lasttag. Now the document contains the payload only. I now also delete the first row because the option ALL has no use for me.

Anfang meiner Nutzdaten
Beginning of the payload

Now I have to put it into a format that my spreadsheet application can work with. I use again the search and replace function and replace <option value =” with an empty string, “>with a tabulator (\t) and with an empty string. Now I have a nice tabstop separated text file.

Tabstopp getrennter Text
tabstop separated text file

Now it’s time for a bit of Excel-Magic (I know politically correct it should be called “Openoffice.Calc-Magic or even better “Spreadsheet-Application-Magic” but that sounds to weird to me).  I import the text file into a separate tab and add the series ID in my card list using VLOOKUP.

Importierte Zuordnungsliste
Imported series id lookup table
Liste mit Verweis
complete list

Now I can save my file in ODT format first and then I save it additionally in CSV format. Now I the date is prepared for the next step. Next time I will continue with the script.

Goodbye Magic Cards – The Script