Copenhagen

Turning a spare Dreamplug mini-PC into a remote-controllable audio player

My company gave away a couple of abandoned spare devices previously used for evaluation purposes. So, I got hands on a dreamplug, a socket-plugin sized mini computer (ARM based) with decent power, low energy consumption, integrated dual Ethernet, WiFi, Bluetooth and, most important, an optical audio output – in other words: a Raspberry PI with a housing and the IO I was missing. The digital output was the primary reason for using it as an addon to my stereo.

My stereo was lacking the capability to play mp3/mp4/aac/lossless files without hassle, which means playing music without having to perform any manual action after importing a new CD I bought. I was not satisfied with my current solution, which consisted of a TerraTec NOXON2audio with an attached USB pen drive, as unfortunately it never ran perfectly stable with USB drives with a capacity greater than 64GB. When we developed the device at TerraTec, even 8GB USB drives were beyond “are enough for everybody/anybody wants to pay” capacity.

My other option was using an empty C.A.R. 4000 housing from the 2000s, fit a Raspberry PI and a USB DAC (or PI Audio) inside, hook up the buttons and the display to the GPIOs and build a beautiful HiFi size player. Even if this approach was very tempting, a couple of circumstances made the expected bill of material too high to take the retrofit project into consideration. First, the 3″ display. The original CAR4000 uses a parallel powertip display, which takes a lot of GPIO space and requires me to develop a new driver from scratch. New 3″ displays are too expensive or take too much GPIOs aswell, so I would have to use shift registers for connecting the control buttons, which also are originally designed to be connected to the controller via ADC – the Raspberry PI does not have an ADC, so I would have had to add an ADC, modify the board or use fancy timing code. None of them was a satisfying option, so I suspended the project.

Simply exchanging any of the gear in my living room was the least acceptable version. My oldschool AVR, built in France in 1999, may not have any retail value anymore, but such a great performance that it simply would not make sense to replace it.

So, using the dreamplug as a hidden addon perfectly was the most viable and promising option in the situation described above.

Use case

The amplifier offers both switched and unswitched power outlets on the rear, I plugged the Dreamplug to the unswitched one and connected the digital output to the digital in of the amp, the 3.5mm jack should go to my multiroom speaker system.  This makes the entire solution feel like an upgrade for the unit, not an additional device, and I came to the opinion that reducing the numbers of units in my stereo is not a bad thing.

Software stack

Originally, I wanted to build a new Hifi size unit with a display, buttons, incremental encoders and more fancy things. As I have a lot of old tablet computers and mobile phones, it became more attractive to relocate the user interface to an external unit.

Regarding the usecase, music player daemon (MPD) was the fitting solution: It offers a network interface for remote control, a media library, and various input and output plugins. Unlike common clones of platforms such as Google Music, such as Ampache or Subsonic, MPDs primary concern is using the audio output facilities of the machine it is installed on. This results into the following software stack on the device:

  • Debian Jessie Linux
  • Libertas WiFi drivers for the Marvell WiFI chipset
  • Pulseaudio
  • MPD
  • Alsatools for mixer controls
  • sshd for maintainance and file transfer
  • rsync for music library synchronization
  • udev for automounting
  • Golang runtime for a web service that switches the 3.5mm jack on and off to control multiroom

The dreamplug

Unlike the Raspberry pi, the unit does not provide a HDMI output, the only way to get a console is using a UART port. The device has a 4-pin mini connector, the pinout is:

1     GND
2     RX
3     TX
4     3.3V

I used a fan connector from an old VGA card, connected it to a 5V->3.3V serial level converter and used a terminal with the settings 115k/8N1. From there, it was possible to interact with the u-boot bootloader and write a Debian Jessie image to the flash.

Music Player Deaemon (MPd)

Just a few changes to the configurations are neccessary:

To mount the pen drive automatically, udev comes into play:

Mobile Phone control

Among many options, I choose M.A.L.P (for Android) because of its featureset, the decent UI and the fact that it is open source and free of ads. M.A.L.P requires an initial configuration consisting of the server address and the password. Afterwards, an old spare tablet computer became the control unit for the media player.

Multiroom

My home has got ceiling-mounted speakers in all rooms, with a central terminal which I can connect to an external audio source. My amplifier offers a Zone2 functionality, unfortunately it does not work when the main speakers are connected via bi-wiring/bi-amping, which is the case in my setup. So, I needed another solution. In case of the dreamplug, two facts made the decision on the solution very easy:

  • The analogue 3.5mm jack is capable of driving passive speakers with an acceptable performance
  • The analogue output uses a separate Pulseaudio mixer control, so I can use the digital out to the amp and the analogue multiroom out independently

The only challenge left was turning the analogue out on and off. Simply pulling the cable was not acceptable. So I decided on writing a slim golang webservice which provided an endpoint for setting the analog output volume and an Angular2 app to provide a simple user interface.

The golang webservice

Thanks to systemd, the inclusion of the service into the system start process is easy:

The Angular2 application

The Angular2 application just provides a frontend for the webservice, which consists of two simple on/off buttons.

 

 

 

After installing the Angular2 application, serving it with Nginx, proxy_passing the /api/v1 url segments to the golang service, the device presented a welcome page with the option to turn multiroom on and off.

 

Podcasting with MPD

Although MPD can archive much, it has no native support for Podcasts, but this functionality can easily by added by taking advantage of the playlisting feature. Here, I created a simple Golang task that retrieves a podcast (using the awesome Podfeed Go library) and converts it into a m3u playlist. This service is triggered by cron.hourly and piped to a file named according to the podcast title inside the MPD playlist folder.

 

Somewhere in Northern Germany

Message Queue Telemetry Transport (MQTT)

After various encounters and two projects which employed MQTT, I thought it was a good idea to write a summary about my experiences and my point of view. As my new job is very close to the IoT and Industry 4.0 sector, it did not take much time to encounter the hype.

Once upon a time

MQTT is, like AMQP, a messaging protocol for distributed systems, meaning that a node can pass a message to one or multiple other nodes. Having used RabbitMQ and AMQP for years, this principle is nothing seriously new to me, and especially as a RMQ user, a core point of MQTT does not sound very exciting at the first glance – its simplicity. In AMQP, I have a whole bunch of functionality to get messages from A to B, including advanced routing and the, pardon me, awesome exchange functionality. MQTT lacks all of them – in exchange for a very little basic footprint, which makes it very tempting for embedded systems and little devices (such as sensors…) which do not possess much memory.

In the basic version of MQTT, developed in the late 1990s, the functionality was just publishing a Message M from a client C to a Broker B, which would dispatch it to all Clients S0..Sn which expressed interest in the type of message. A message M was defined as {Topic T, Payload P, QOS Q}, both parts of the tuple were represented as String. A broker B would wait for a set of Subscriptions S0..Sn, do just do a simple strcmp on any new message and dispatch the message to all clients which subscribed to the topic of a new message in the first place. Using a simple QOS mechanism, the publisher of a message was able to specify the way the Broker should deal with messages on a per-message basis: fire-and-forget (and do not care), deliver to at least one client, deliver to >1 clients.

On the transport side, MQTT could be sent through plain wire (such as a RS-232 / RS-485 line or even teletype), or via TCP/IP, of course unencryted and unauthorized.

So, the protocol was plain simple and a real minimalistic approach to setting up a message exchange, by no ways applicable to our needs in the year 2016 and should be forgotten, right?

Wrong.

My first applications with MQTT

My first experience with MQTT was in 2010, when I used it as a transport protocol for sensor data over a RS-485 line.

The project was a greenfield project, but the controller in the sensor was very limited, and my Atmel AtTiny had much more to care about than parsing, verifying and deserializing fancy XML data even for the transport layer. Using full-blown frameworks such as Apache Thrift was not an option aswell, but MQTT was deployed very fast. Using topic names “config.sensitivity”, “config.sensorType”, “sensor[0].value” and so on made the evaluation of data very easy, not to say trivial. On a 2-pin cable inside a machine cabinet, I also did not care very much about security, so it just made my live easier and saved me some hours of work to implement my own protocol. Cool, job done.

Then came the second, which is mainly used for telemetry and event exchange with other devices, using more advanced features of todays MQTT implementations:

  • TLS/SSL
  • Server and client certificates, automatically rolled out during device production, respectively orchestration of the broker server – which means: Authentication by certificates. Of course MQTT also supports plain username/passwords.
  • Hierarchical topics. Topics are not only strings anymore – following the convention of naming them in url-pattern style, such as devices/<id>/events/updates/caseTemperature, it is possible to perform registrations on a hierarchical basis. For example, a subscriber registered for devices/<id>/* would also receive messages published to sub-topics such as the above.
  • Which brings us to Access control lists, supported by a subset of the mainstream brokers. For example, why should a sensor be able to subscribe to its own readings? Or, more relevant, why shouldn’t one share a MQTT broker with other clients, as long as they are not likely to interfere?
  • Distributed, scalable brokers.

Which means, one could even use MQTT on the internet nowadays without being fired.

Implementations

Client

On the client side, I mainly used Eclipse Paho. After writing a Autocloseable Fluent-API facade for it, it was quite pleasant to work with.

Examples (Java)

Connect to a Broker

Or with SSL:

Publishing a message:

Simple publish-receive test:

 

Broker

On the broker side, I used Moquette (easily embeddable into own Java applications) and Mosquitto. As there are excellent documentations on both, I will not go into details here.

Conclusion

We have been through many approaches to make computers talk to each other. We have been through the SOAP-Ceremony, the CORBA-Hell, and now we even manage to write beautiful RESTful webservices. We also laughed about Windows NT messaging. Personally, I like to model communication architectures using RabbitMQ, but I also have to confess that often a simple messaging protocol without any advanced features would have done the job.

On the other hand, I do not think much of hypes and consultants who try to make a fortune on selling the next best thing. So, I will just add MQTT to my toolbox, be happy with the current exciting project which employs MQTT for M2M and telemetry, and, as usual, choose the right tool for the job, with or without a fancy title, next time again. As MQTT will gain more use-cases just because of simplicity, I do not think this will be our last encounter.

Etretat revisited

Tenerife

Test-Driving the Go Language

When I completed my compiler design studies, I felt it was not appropriate for me to give my exercise projects any fancy names and put them on Github. With this in mind, I was used to question any new language which came out of nowhere and wait for a number of readings on my technology radar before considering to deal with it.

With a number of significant readings from Google Go, I decided to give it a pleasant test drive.

The programming language Go, originally a 20%-Project initiated by a Google employee, quickly found a position in my mind – a slick alternative to C/C++ for smaller systems and fast backend services.  Being used to Java/Spring Boot and .net/Nancy, Go could be interesting to me if there is any gap it could fill – maybe a gap I was yet to discover.

To find out, I built a simple echo service which covers some aspects I would usually face on backend services:

  • HTTP Server
  • JSON
  • Commandline Arguments or Environment Variables
  • Logging (ELK friendly, please)
  • Easy to integrate in CI/CD environments

Setting up Go

Setting up go takes 5 minutes on UMTS or 4 hours on a crappy hotel WiFi. Download the package, install, create your workspace and start coding. I used a text editor for writing Go code, afterwards I noticed that there is an awesome set of plugins for Atom.

Right after creating the workspace one thing becomes obvious – Go loves conventions, convention over configuration and implicit declarations. The workspace is organized in a way that seems comfortable to Java developers, because it works perfectly if you organize all your projects and external dependencies in the form of Java namespaces.

Java developers will, on the other hand, not be comfortable with the missing concept of classes and the implicit private/public declarations: function names in camelCase are private, while their PascalCased counterparts are public.

The Echo Service

Lets build a simple Echo Service, which just listens to HTTP requests and echoes them back, while writing comprehensive logs and following a set of practices which appear a good idea in Java.

Enter Go..

Main file

 

Service Handler

 

Compiling

or, to build to GoHome/bin:

Go generates fat binaries which include all their dependencies, which means that deployment is a little bit easier, regardless of the bigger files.

Test Run

 

Summary

I needed about 3 hours to learn the basic concepts of go, get my workspace running and develop the trivial service above. Not too far away from Spring Boot when leaving the Java language out of consideration. To me, Go looks more than promising. I will definitively consider it as an alternative to C on my next project which requires a small and slick microservice backend for tasks I can quickly replace with an alternative implementation.

Iceland

Oslo

Building an iTunes Web Interface in 2 Hours with Java and Spring Boot

In my mind, the Spring Framework is positioned right next to the “Code less, do more” sign, promising to cut redundancy and boilerplate code and letting the developer focus on the actual work that needs to be done – just the same motivation as JEE. In this tutorial, without intending do encourage any kind of deathmatch between JEE and Spring,  I am going to serve the contents of my iTunes Repository through a Spring RESTful Webservice and render it through an AngularJs frontend. The reason why I chose Spring for this tutorial is one really interesting project from the Spring world – Spring Boot, a framework which allows the developer to build and deploy self-contained standalone webservices very quickly.

Overall System Architecture

Exposing a music library through a web service can be achieved via various approaches, the selection of the initial way to take is based on the following considerations:

  • The repository is maintained by iTunes, but talking about separation of concerns, this fact should be transparent to any consumer of the Music Library Access Layer. So we need a kind of interface or facade/ACL around the iTunes specific functionality to avoid an unintended vendor lock-in.
  • Even inside the iTunes domain, the task of acquiring the content does not neccessarily mean using walking one single path. The most trivial way is taking advantage of the predictable folder structure, but taking the way along the sqlite database or the XML file is also viable and, from the features provided, superior to fetching the folder structure.
  • iTunes Libraries can (and probably will) be huge, whily querying them is expensive. So, caching or importing into a faster database would be a good idea.

Considering the above, the chosen approach for the scope of this tutorial is:

  • Creating an interface which is ignorant of the type of media storage and extraction strategy used
  • Implementing the trivial filebased approach to get a working result quickly – if the overall solution is good, we will switch to a better one down the road.

From the decision above, the following modules are derived:

  1. Library Reader (de.m9d.edu.m3po.musicDirectoryReader) – Reads the Library and provides the content through a standard interface
  2. Web Service (de.m9d.edu.m3po.musicDirectoryService) – Exposes the Library Reader through a RESTful webservice
  3. HTML5/JavaScript Frontend – Let there be humans!

Basic Project Setup

I use Maven for the task of collecting and wiring dependencies, however the same could be achieved with any other package management solution (such as Gradle) aswell.

I will be using Spring Boot’s “fat jar” approach, so the project will compile to a self-containing jar file that includes all dependencies, including an embedded Tomcat – so there is no need to setup an own application server. Building a .war file for any dinosaur application server can be achieved with just one little tweak in the Maven Configuration.

The basic version which I describe in this tutorial uses the following technology stack:

  • Spring Boot Web Starter
  • Spring Boot HATEOAS Starter (love Richardson Maturity Model)
  • Spring Boot Starter AOP (Aspect oriented Programming Support)
  • Spring Security for User Authentification
  • Immutables.io (as I also develop in c#, I was a little jealous about the property system)
  • JSONdoc for automatic API documentation and Test client
  • WADL homebrewn controller to provide a WADL file for testing suites, such as SoapUI
  • Spring Unit Testing support
  • Mockito for mocking Components (outside of Spring)
  • Angular.js for the JavaScript Frontend
  • Angular-material for the JavaScript User Interface

The full version (introduced in the next 2 hours sessions) also uses:

  • Neo4J Graph Database
  • Neo4J Spring support
  • Spring Boot Actuator for Health Monitoring
  • Logstash, Elasticsearch and Kibana for Log analysis

My weapons of choice concerning the general toolchain are:

  • Netbeans IDE (8.1)
  • Maven
  • Jenkins CI
  • SonarQube Code Inspection
  • Node, Bower and Gulp. Stop clinging to yesterday, JavaScript!
  • Git and Phabricator for versioning and housekeeping.

Components

iTunes Interface

in iTunes, we are mainly dealing with Artists, Albums/Compilations and Tracks. For the purpose of retrieving those entities, a possible minimal Interface is:

Sitting in front of a C# IDE everyday aswell, I appreciate the properties and class initializers for the great job they do in cleaning up code from unnecessary IDE generated bloat – we really should depracate mandatory getters and setters in 2015. In plain Java8, there is no such feature available, but there are libraries which provide that feature. I use Lombok and Immutables, usually I prefer Immutables, because it does not cause issues when using Spring AOP in the same project.

Expected Behavior of the repository reader:

 

Webservice

The webservice consumes the Media Library Reader and provides a HTTP/REST interface. With just a little Spring power, this task is quite easy.

Webservice Controller

In Spring, a class has to follow a simple pattern in order to become a REST service, adding the @RestController annotation suffices. Together with some documentation for my favorite auto-documentation tool and stripping the Javadoc header, the class below does the task of exposing the artists list of a provided music repository to any REST client.

Keeping JSONDoc annotations and actual service code in the same class might raise concerns – one could even extract an interface from the class and put the entire documentation there. This did not happen here, because:

  • I think that documentation and upper level service belong together – as close as possible
  • There are more annotations to be put here later – having them all in one place might be a good idea
  • Why not feel the urge to modularize the RestControllers themselves as soon as possible?

The @ApiVersion is a qualifier, which will be used for client-side version selection later.

Logging Aspects

The controller should not necessarily have to deal with the environment-specific way of logging – it simply is not its concern. Thanks to Spring AOP and the Pointcut API, a basic logging aspect can be implemented as below – Spring AOP will kindly intercept the requests matching the AspectJ expression in the pointcut annotations.

HATEOAS Support (the glory of REST)

In the SpringBoot/HATEOAS tutorial, the assembly of the Resource is implemented in the controller. This might be fine for smaller projects, such as our iTunes Server, but me, I would feel guilty in terms of Single Responsibility Principle Violations if I would actually take this approach. Fortunately, Spring provides an Interface for a Entity to Resource conversion, so I just implement the Adapters outside and let Spring autowire them into my controller.

For example, assembling a HATEOAS compliant resource from an instance of Artist could be implemented as:

Annotated by @Component, it is trivial to autowire the desired implementation of the Resource Assembler using Springs Dependency Injection.

Initializing the Repository Reader

Again, Spring offers many ways to wire a working instance of the Repository reader to the Webservice controller. In this example, I will use the direct approach of providing a builder for a reader instance, which just returns an iTunes file-based reader. Being annotated with @Configuration and @Bean, Spring will use the getLibraryReader() method to resolve the outgoing dependency the REST Controller states during construction.

 

Run it

After a maven clean build and run, the server is waiting or requests. Using a plain HTTP client or a full-blown test suite, such as SoapUI, we can retrieve artist, album and track lists:

The logfile is also very concise and a fantastic light snack for logstash kv{} operator:

(timestamp) TRACE 9192 — [m3potp:1] d.m.e.m.m.MusicApiLoggingAspect: client=0:0:0:0:0:0:0:1 http.status=200 method=GET uri=/api/v1/artists, message=found 771 artists

Summary

The service is now able to expose a media library. In the next part, we will add Spring Boot Actuator for Health Monitoring and Spring Security for authentification.