Month: January 2015

How to create a vagrant base-box for VirtualBox

Introduction

What is vagrant?

Vagrant is a tool to run and maintain lightweight and flexible development environments. It is often used in software companies to enable all of their developer to work with the same standardized virtual machines that fit to their production environments. If it is configured right, you can simply type vagrant up and you have a running dev environment.

Why do we need a base-box

To be able to work with vagrant as the management system of your development environment there are two parts necessary. One is the environment configuration based on chef or puppet recipes. These recipes will provide a list of packages and set-ups to configure the virtual machine according to their current needs. For example you can install a web-server and MySQL, set up the user accounts and get it up running.

The base-box itself contains only the very rudimentary system. In our example it is a nearly plain Ubuntu Server 14.04. This image will be used to ground all of our development systems. Vagrant boots it up and runs the recipes from chef or puppet to install the needed software.

Create the base-box

Software requirements

To set up a base-box for using with vagrant we at least need vagrant itself and VirtualBox. You can grab all of them by following the next links. As you might see there is also a link to download the actual Ubuntu Server image. This will be needed to install the operating system of your base-box.

I will not write about the installation and configuration of VirtualBox but it should be self-explanatory.

Another way to install vagrant and VirtualBox on Mac OSX is to use homebrew in addition with cask. Brew should be known by every advanced user of OSX and cask is a simple addition to not only install command-line tools but also to handle full GUI software packages. With it you can install software like VirtualBox or IntelliJ Idea for example.

The following command will install our necessary software.

Hardware settings of the virtual machine

My specifications.

Harddisk
  • VDI self growing image (43Gb)
Chipset
  •  PIIX3
RAM
  • 512Mb
 Network
  •  eth0: NAT
  • eth1: Host-only
 Hardware virtualization
  •  VT-x / AMD-V
  • Nested Paging
 Video
  •  12Mb
  • no 2D / 3D acceleration
 Audio
  • none

The following screens will guide you through the set up of the virtual machine.

Installation of the base system

With the Ubuntu Server image mounted to your virtual disc drive, boot up the machine using the start button and press F12 followed by pressing c to select the CD as your boot device. The Ubuntu installation wizard will pop up and asks you for your preferred language and keyboard settings. This part can be configured the way you like it. At the point where you will be asked for the partition layout, select use the whole disk  but without the logical volume manager (LVM). This is convenient and fits to all of our needs.

The next step is setting up the primary user account. For vagrant to be able to run, this user has to be vagrant set up with the same as password.

  • username: vagrant
  • password:  vagrant

We will leave the next step with no automatic updates and proceed onto the software selection. The only prerequisite vagrant has for a base-box is the presence of a ssh-server. After selecting the openssh-server press return to continue. It will take a few minutes to complete the package installation and setting up the system.

That’s it. After ejecting the disk image from the virtual cd-drive we can continue with our last step.

Configure the base-system

We already installed the openssh-server during the ubuntu installation wizard but it is not yet configured to be used by vagrant.

Log in into your newly created machine with the credentials we set up earlier.

Ubuntu first login

Ubuntu first login

Vagrant uses an insecure keypair to be able to log in to your machine. To set up this key we have to copy its public part to the virtual machine. First we need to determine the ip-address of the system by running ifconfig.

ifconfig

ifconfig

With this address in mind we can user the following command to add the key to the server.

If this does not work, simply paste the content of the vagrant.pub file into the VMs vagrant users home folder under .ssh/authorized_keys with the following commands.

Paste the content and press ctrl + o  followed by ctrl + x. For openssh to be able to user this key it needs to have a specific combination of access right. Run the following commands to fix them.

The next step is very important because vagrant uses the sudo command to do all its stuff as a privileged user but does not want to type a password every time it uses it. To configure sudo passwordless type sudo visudo and add the following line right behind the privileges of the root user.

And also edit the line about the sudo user group to be able to user it without a password.

Install the guest additions

To get the best out of your virtual machine experience it is recommended to install to VirtualBox guest additions.

Before you can do this, there are a few packages needed to be able to compile the kernel packages.

The next step is to mount the guest additions image to your machine by going to the main menu, select the devices tab and click on Insert Guest Additions CD Image followed by typing into your machines console the next command.

Finally run the script that matches to your selected system architecture. In my case it is a 64bit system so i had to use the following command.

Install puppet or chef

Last but not least we need to install one of the environment configuration management clients. If you prefer chef or puppet is completely up to you. I will use puppet because of some company dependencies.

If everything succeeds we are ready to go.

Packaging the box

This step sounds like one of the hardest but it is very simple because vagrant provides us with all of the needed tools. We only have to type one command and vagrant will provide you with a base-box in the current working directory. The base-box-14.04 is the name of your virtual machine.

Final result.

Download my base-box

If your are interested in using my newly created base-box image, simply download it from the following link.

Sources

 

First impressions of the carambola 2

What is a Carambola 2

The Carambola 2 is a wireless module based on a Qualcomm/Atheros AR9331 SoC and is build by 8devices a company from Vilnius, Lithuania. They are specialized in small wireless devices for a small budget but with enormous connectivity and flexibility.

Specifications

CPU
  • AR9331, 400MHZ
Memory
  • Flash: 16Mb
  • RAM: 64Mb DDR2
Frequency
  • 2.4 GHz
Max output power
  • 21 dBm
Wireless standard
  • 802.11 bgn
Antenna (port)
  • U.FL connector
Power supply
  • 3.3V
  • Consumption: 0.5W
Interfaces
  • USB (host / slave)
  • serial port
  • 2x ethernet
  • i2S
  • SLIC
  • SPDIF
  • 23x GPIO
Size
  • 28 x 38mm
Operating system
  • OpenWrt

More information can be found on the homepage of 8devices/carambola-2.

Why did i buy a carambola 2?

Normally this question has no direct answer for me. I own more than a dozen SoC / micro-controller boards. There are all kinds of Raspberry Pis (Type A, B, B+ etc.), a BananaPi and a hand full of  Arduino based controller. Most of them are stored in a shelve beside my desk and are awaiting there final mission. Some others got used for home-automation and my connected media-system. I will write more about their applications later on this blog.

The carambola 2 instead was bought with a final destination to be used for. It will be the heart of a multi-copter controlling unit. If everything works as expected i will be able to control all kind of drones with this device (… plus a few radio connectors) and only a tablet or smartphone. I can’t tell all information by now, but there will be more here soon.

Unboxing

It took the package about two weeks to be delivered from Lithuania and cost me about 40€. Because of Lithuania also being part of the european union there were no problems with customs and a delivery to Germany. As you might see on the pictures, i decided to not only buy the carambola 2 but also the developers-board. It is a lot more handy to test and develop with this extension board as to solder all of my connectors directly to the SoC. The final device won’t be build on the developers board because it has to much interfaces and is way to large. The unboxing itself was very unspectacular, the package only contains the board and a small merchandize puzzle which you can see on the top left corner of the picture.

Unboxing the Carambola 2

Unboxing the Carambola 2

Getting started

Wiring the carambola2

To work with the board you have to connect at least to wires. The mini-USB interface to power it up and a network cable to get a connection. You have to use the first ethernet interface to use the device as a router and obtain an ip-address for your computer via dhcp (it is the one on the right side if you place it on the desk, the USB-ports looking towards you). If you would like to use the carambola 2 as a client in your current network, use the left ethernet interface. It is configured by default to obtain an ip-address from your home router. You have to get the assigned address from your router network overview page.

Wired carambola2

Wired carambola2

First boot of OpenWrt

After connecting the mini-USB cable to a free port on your computer or a stand-alone power-unit it will take only a few seconds to boot the device. To access the configuration of OpenWrt you can user several methods. By default the username is root and the password is admin. The default ip-address of the carambola 2 is 192.168.1.1. It can be changed afterwards at the configuration interface or if you build the kernel by yourself during menuconfig.

LuCi browser interface

Open up the following Url in your preferred Browser.

Connect via ssh

OpenWrt ssh

OpenWrt via ssh

Advanced

Build your first firmware on OSX

This part is not necessary if you do not need any special kernel configuration or packages. But i will  show you how to compile your own image just in case you will need it for later development. Go to configuration of the OpenWrt system if you are not interested in compiling a kernel.

Disk image creation

Because OpenWrt needs a case-sensitive filesystem and Mac OSX by default provides only a case-insensitive one, we have to create disk image and attach it to the system.

Needed packages

OpenWrt needs a hand full of packages to be build. I will not explain how to install XCode framework or how to get homebrew running, but will provide you a list of necessary build packages.

This could take a few minutes (approximate 45min), be patient.

Because of OSX already providing getopt we have to temporarly prepend our newly installed gnu-getopt package to PATH.

If you had awk installed previously you have to unlink it and link gawk afterwards.

Get the latest sources from github

From now on we are working inside of the mounted volume /Volume/OpenWrt.

Update current package list from 8devices repository

First we have to generate the image configuration. Type the following command and leave everything untouched, just exit.

menuconfig OpenWrt

menuconfig OpenWrt

At least for the gnu-gettext package the compiler needs another version as the one provided by Mac OSX. You did install the right one with the above brew install command nut unfortuniatly this homebrew is keg-only. This means you can’t use it simply linking it. There is only a –force option to be able to link it, but this could harm the stability of your system. Homebrew offers a special build envorinoment for problems like this. Just type the following command and go on inside of this environment.

Now we can run the build command. If you are using a multi-core CPU, define the command with the number of cores you would like to user after the -j param. I have an old Core2Duo, that’s why i used the -j 2. These param defines the number of jobs being processed simultanously.

Install the firmware

Copy the firmware to your device using one of these methods.

Web interface

This method is the most recommended one for users new with OpenWrt.

Log in into the Luci web interface and go to System -> Backup / Flash Firmware. If you changed something of the configuration, i recommend to backup your config using the generate archive button under the backup section.

The new firmware can be flashed by pressing the flash image button near to the bottom of the page. Uncheck the keep settings” box if you would like to reset all settings to kernel defaults.

Flash and backup interface of OpenWrt

Flash and backup interface of OpenWrt

Advanced methods

The above method is by far the most convenient and easy one for every person new to OpenWrt. There are a few other methods that could be used to flash a new firmware but i do not recommend them. If you are interested in learning more about them, just visit the developer wiki over at the 8devices homepage.

Sources

Bootstrap DS215j

Introduction

The DS215j is the newest NAS device of the Synology DiskStation family.

DS215j connected

DS215j connected

I bought mine two weeks ago to serve the following functions

  • TimeMachine server for 2 MacBooks
  • Photo storage with external synced backup
  • DVB-T streaming and recording server (tvheadend or VideoStation)
  • automated tv-episode download and serving (SickRage)

Not all of these functions are working by now because of some issues with the new hardware architecture. The DS215j is based on a Marvell Armada 750 SoC, which is pretty new. By now, Synology released their toolchain and the guys of SynoCommunity have build a lot of packages to be installed via the graphical package manager. But for me, the DiskStation is not only the GUI but i also needed shell access combined with a few tools, the station does not provide be default.

This is why i tried and managed to install ipkg.

Optware bootstrap

There are no bootstrap packages for the DS215j by now but the Marvell Armada 375 is somewhat compatible with the Marvell Kirkwood mv6281 binaries. We can use them to set up our optware environment. If you see any problems with the provided binaries, please feel free to add a comment.

Set up ipkg

We have to download the bootstrap code and set up its environment.

Create the optware root directory

After extracting the above tar-archive a folder /opt has been created. This folder will be deleted every time you update your DSM. To prevent this problem, we have to move the content to a folder underneath /volume1/.

Old versions of the bootstrap script used a mount-point to bind /opt but this is not save any more because there have been some issues with the unmount during shutdown. Linking is the better solution.

Add /opt to PATH

Add the following line to /root/.profile or if you are using a user-account to get SSH access /home/username/.bashrc or the ZSH equivalent /home/username/.zshrc.

Set up the init-scripts

If you would like to install services, you want to start at boot-time, you have to set up these startup script.

Create the folder where the script will be placed.

Paste the following code into /usr/local/etc/rc.d/optware.sh.

Make it executable.

Thats it. If you have done everything right, you can reboot your DiskStation and log-in again to install your first packages.

Install packages with ipkg

Terminal

First we have to update the packages list and after that we install three necessary tools.

Graphical way

If you are not the guy who likes it to work within a terminal-session, there is an app for it, running inside the DSM to manage your ipkg packages.

Open the Package-Center. Press Settings -> Package Sources -> Add and paste the following repository URL.

Now you can browse the newly added repository under the Community Tab inside of the Package-Center. Install iPKGui and launch the app afterwards from the main menu. It could be possible that the app has some issues with its name, but this is only a design flaw.

iPKGui icon

iPKGui icon

Sources

Golang and reflections

Introduction

Reflection is the ability of a program to examine its own structure. In go we can use it for example to build dynamically accessable structs.

How-to use reflect

I will give you with two examples and a hopefully easy-to-understand explanation of them.

Simple usage

At hikewith.me we use a simple way of reflection to map dynamically specified keys to a set-up value. With this we can map database-columns to a given key and generate SQL queries according to the given input instead of predefining everything. In our database, field are named another way then in our backend code (go). For example the first-name of a user will be saved in the column first_name but the field in our User-struct is named FirstName. We need a way to map these connection dynamically for updating the user data and keeping our code simple.

Field-map struct

These struct type defines the connection between our used struct field and the columns found in our database table. FirstName, LastName etc. are the field names used in our User struct. The assigned value will be the table column. It will be loaded from a json-file (see JSON configuration).

User struct

This type stores the information about a user. If he changes some of his data we use this struct to update the dataset in our database.

JSON configuration

As already noted, we store the mapping between columns and keys in a JSON-file that will be loaded during the initial start of our backend.

Configuration loader

To load the JSON represented field configuration we use the gotamer-cfg-loader. You can get it using the following line.

The usage is really simple. Initialize the var to save the loaded configuration as type of your FieldMapping struct and afterwards simply load the file content into this struct.

Field-mapper

This is where the magic happens. The method will be called with the field name as param and returns the value of the field. First it reflects the list of all fields of the FieldMapping struct after this we can do an indirect reflect to get value of the field.

A more generic version

The following code is a more generic version of my code from above. It is not bound to a specific type but can be used with any kind of type to get a fields value. I will show how to use this function within the more advanced example.

Advanced usage

In this second example, i will show how to compare two structs of the same type with each other and get a diff. With this diff-struct we update our user data-set.

Compare struct

It is a simple key-value-store that will be used in a slice to collect all diffs.

Compare two structs of the same type

This function will be called from within a new-user-struct like this newuserdata.compareUpdate(olduserdata). The reflection will be used in the first place to get the number of fields in this struct. With this we can iterate over all fields and use the above defined getFieldByIndex function to get the value of each field and compare them to the corresponding value of the old-user-struct. The last step is to add the compare struct to our returned slice if there was a difference.