Simplified KabelDeutschland streaming proxy

Review

A few days go i wrote about How to reverse engineer the KabelDeutschland API. Attached to this post there was a PHP script hosted on github to get your own streaming proxy running.

After testing and spreading the word with some people i came to the decision that the procedure was far too complicated. The need of a webserver and the large number of configuration options were unnecessary.

That’s why i build a single binary version, written in go.

Limitation

This tool does not remove the restrictions made by KabelDeutschland. You will need an internet and a TV contract with them. It only give you the ability to use their streams besides the iOS app.

Download

Binaries for all major platforms can be found at the shared folder.

Shared Folder at mega.co.nz (mirror at dropbox)

Select the binary suitable for your machine.

Tested environments

  • Macbook Pro Mid 2012, OSX 10.10.2
  • Synology DS215j (arm)
  • RaspberryPi (arm)
  • Windows XP 32bit

Please drop me a line if it works or not on your system.

Installation

Grab the config.json and edit it with your customer-service-center credentials. You can also change the port if you like.

On windows, double-click the executable file. On Linux/Mac you have to use your terminal, navigate to the folder you downloaded the binary and type the following command.

If you got a permission denied error, you have to set the executable bit for the binary

On all systems you will see the same output if everything went fine.

You will see the list of interfaces, the proxy listens to. All of them are ready to use to serve the channel playlist.

On certain Windows systems, it is not possible to display the interface list. The proxy will work nevertheless. Simply paste your known IP-Adress or one of your loopback addresses.

Installation on a Synology Diskstation

By now the easiest way to install the KabelDeutschland proxy is to download the arm-version of the binary and put it to your home-folder on the diskstation. Put the edited config.json next to it. Now you have to log into your nas  using ssh as your user and run the following command.

This creates a new screen, starts the proxy and immediately detaches the screen. That means the process runs in the background.

In order to be able to use the screen command, your box has to be bootsrapped. You can find the instructions to do this here: Bootrap the DS215j. Installation of screen will be done with this command as root.

You can connect to the proxy using the IP-address of your diskstation. http://[diskstation]:8787.

To stop the process, attach to the screen and press control+c. Attaching works using this command.

Predefined package

Synology has its own package manager and I’ll try to build a repository containing the proxy as a installable package. This will provide a way more easier solution. I will keep you updated.

Usage

Copy one of the links form the terminal output and paste it into your desired media player. It should load the playlist automatically. In VLC it looks like on the screenshot. You can now switch between the channels. The playlist contains links pointing to the proxy. This link handles the license generation, KabelDeutschland used to verify the stream.

How to reverse-engineer the KabelDeutschland tv-streaming API

The main intention

As a paying customer for many years, with both cable-tv and cable-internet, i found it very frustrating to not be able to watch tv on my smartphone or tablet but only on my tv. Back in 2014 my provider KabelDeutschland pulled out an tv-app for iOS that enables exactly this functionality but only for Apple mobile-devices. I always hoped that they would release an Android version shortly after but until today every customer-service-request regarding this topic leads to a “it will be available somewhere in the future” and “we can’t provide it because of some technical difficulties”. As a software developer and hardware engineer i know that this answer is just nonsense. If it runs in iOS there shouldn’t be any problem to get it up and running on Android and even on desktop.

Besides the intention to simply prove, that it is possible to get the streaming working on other devices, i had a more direct need. I bought a whole home-cinema setup including an Epson EH-TW 6100 beamer, an Onkyo Dolby Atmos capable surround-system and an Amazon Fire TV. I was not able to watch TV on this setup, because i had no DVB-C receiver to attach it to the system. But being able to install XBMC/Kodi paired with the IPTV plugin, it would be easy to do so, if there were any streaming sources.

Important Information

You need to have two contracts with KD. One for cable-tv and one for internet over cable. The second one is necessary because the streams are only available inside of their own network.

If you like to go further please be sure to met both dependencies.

Reverse-engineer the API

Final conclusion

I am aware that this part of this article should be at the end but i think some of you are not that interested in the hard details on how to do a man-in-the-middle-attack to uncover the KD API. So i decided to write this first. You can grab the fully functional streaming-proxy script as a repository in my github account: edi-design/kd-streaming-proxy. There is a README on how to get it up and running but i will provide a more detailed version below.

But by now i will continue with the description of the whole information-gathering-process.

Needed hardware and tools

Setup Burp and the iDevice

The first step is to download Burp and start it using the command line. Because Burp is written in Java you need at least the Java Runtime Environment.

Open your preferred console and type the following command to start Burp.

You will the following window. First we need to disable the intercept-function, push the button until it says intercept is of.

Burp Suite - intercept is off

Burp Suite – intercept is off

Next we go to the Options Tab and change the interface from 127.0.0.1:8080 to *:8080. This can be done by clicking the edit-button and select the All interfaces checkbox.

Burp interfaces

Burp interfaces

This is the complete part of configuring Burp. Now we switch over to our iPhone and connect it to the same network the computer running Burp is connected to. The next step is configuring Burp as proxy for this network. Go to Settings -> Wifi -> click on your Network -> HTTP Proxy Manual and add your computers IP-Adress as server.

iPhone proxy configuration

iPhone proxy configuration

Because part of the communication between the app and the KabelDeutschland API is SSL-encrypted, we need to install the Burp-provided SSL certificate. This provides us with the possibility to read the encrypted communication in plain text.

Open Safari on your iPhone and browse to http://burp. Click on CA Certificate. This will download the cert and asks you if you want to install it. Ignore the possible security risk. You can delete it afterwards.

Capturing some data

After the preparation has been finished we can now start the TV-app and collect the data it sends and gets from their backend services.

Burp should now look like this.

Captured data

Captured data

Analyzing the data

Starting to analyze the data by reading every call with its request and response, step by step.

The first call before even asking for credentials goes against an getconfig endpoint. It provides the app with all the necessary information regarding the api.

The response.

In this data we see the JsonGateway, this is the main entry-point. The second relevant information is the initObj. If you take a look at all further calls this object is needed as a mandatory param to identify against their api.

The sign-in

Before we can do any further calls, we need to have a valid session. The app does a SSOSignIn request to gather information about a SiteGuid and a DomainId. Both of these have to be appended to our formerly captured initObj. At this point you need your KD-CSC credentials. These are typically username and password you need to view your contract or invoice. KD use it to determine wether you are a cable customer or not.

The response.

Get the channel-list

Since we now have all the information to query the channel-list endpoint, lets do it. Finding the call inside of the Burp result is easy, search for GetChannelMediaList. Thanks to KD almost all of there methods are self-explanatory.

The list of all possible tv-channel will be returned, including a streaming link. I will only show an excerpt of it, because the whole response is nearly 3000 lines of json. The first element in my case contains all information about Das Erste (ARD).

Watching the stream?

Seeing the output above, it should be very easy to watch the stream. We’ve got an url that looks like a valid stream, but if you click on it, it gives you an access denied.

After some more digging into the log of Burp, i saw that there was another call named GetLicensedLinks, always called after selecting a channel to view. It provides as response the valid streaming link.

This call takes, next to the obvious initObj, two other params, the FileId and the Link. Both can be gathered from the above channel-list call.

Seeing the response, there is our most-wanted information.

You can paste one of the two urls into VLC media player and you will be able to watch the channel.

Conclusion

Because the whole process is a bit tricky and time-consuming, especially if you just want to sit down and watch tv, thats why i provided a little PHP-script at github (https://github.com/edi-design/kd-streaming-proxy). This script will do all of the work for you. You will need a webserver running inside of your home, because of the mandatory KD internet contract. Calling the script without params will provide you with the download of a playlist, containing all of the channels provided. This playlist contains links to the script itself with the channel id as param. Every time it gets called, it generate a new valid licensed link and redirects to it, that enables you to watch the stream.

I hope it is easy enough to set up and handle. As a simple webserver i can recommend the Synology NAS or a raspberry pi.

If you have any questions, feel free to ask.

pyLoad in a Debian chroot on the DS215j

Introduction

During my last post about the DS215j from Sinology i introduced the DiskStation and provided a way to bootstrap the NAS. Lately i found myself in the situation that it was not enough to be able to install packages using bootstrap and ipkg. I’d like to have a working pyLoad installation to be able to handle all of my downloading belongings. The provided DownloadStation was not capable of handling so called OneClickHoster especially the captcha request.

Because of this reason i needed an alternative. It was very easy to find some, pyLoad. It was not that easy to get it running on my DS215j. There are precompiled .ipk packages of pyLoad on their homepage but they are not working because of an issue with python.

I will provide the instructions to install pyLoad from ipkg even if it is not runnable. Because if someone has the same problems, he will find this article and sees the better solution.

Install pyLoad

From ipkg sources

This part of the article won’t provide you a running pyLoad instance by now. Maybe there will be a python2.7 fix for the following issues.

Needed packages

First we need a list of packages to be able to run pyLoad. The following command will install them. You have to log-in into your DiskStation using root or a user able to use sudo.

Download the pyLoad package

The precompiled package will be downloaded directly from the homepage of pyLoad and installed afterwards.

Running pyLoadCore

During the first start, pyLoad should place its default configuration under ~/.pyLoad, which it does, but nothing more. The -s parameter starts the initial setup process.

The command finished exactly after it was started. This couldn’t be the way it should work. I dug a little bit deeper and ran the following command to be able to see a few debug messages.

This provided me with this truncated output.

That message indicates a problem with some module. Some googling showed me, there is a problem with pyLoad an python2.5, which is default, running the python command. Knowing about this i tried python2.6, getting the same result. Only python2.7 gave me another result. But not the one i was expecting.

The needed pycurl is currently not available for python2.7 on arm architecture.

Now it was time to give up this way and switch to a more powerful but advanced solution.

Using a Debian chroot environment

What is chroot

Chroot is short for change root and means to change the root directory of a process running on a linux system. The process is encapsulated inside this environment and can’t access files and folders outside of it. With a few device mount-points we are able to run a whole linux system inside of this sandbox and can use all of the benefits of it without changing the host operating system. Chroot is a simple sandboxing and virtualization approach.

Remove pyLoad ipkg installation

Before we are getting started with our chroot installation, we will clean the mess we did earlier.

Chroot package from SynoCommunity

The guys over at SynoCommunity are providing a fully functional debian wheezy chroot environment. If you have added their repository into your DSM Package Center, simply install the debian-chroot package that can be found under the community tab.

This will take a few minutes install because of its size. After the installation is finished, click the action dropdown and select run. The next steps will be performed via ssh on your DiskStation.

Set up your chroot environment

The first things we have to do, is to mount your Volume and to add all of your users. The last step is needed to prevent file-mode issues between the host system and your chroot.

Mount your volume

To enter your new environment, the package provides a simple script that could be called with some params. We will run the following commands to enter the chroot as root user and create a folder to mount your volume.

Now we have to edit the script we called seconds ago and add a few lines to handle the mounting of our volume. I will post the whole modified script with adding descriptions in just before the lines i added.

Now we can stop and restart the environment to be able to use our mounted volume.

Generate locales and set timezone

By default Debian has no configured locale packages. We have to install them and set them up according to our needs.

Add the users and groups

To be able to write and read onto your data you have to set up a group inside your DSM that has full access to your volume and contains all of the users that should have access from inside your chroot.

Adding a new group is quiet simple. Open your DSM, start the Control Panel, click on groups and create a new one. Mine is called chroot and has access to all of my shared folders.

chroot group on dsm

chroot group on dsm

For the next step we have to leave the DSM because there is no way to get the needed data from there. On your DiskStation shell, type the following commands and note the user id of your user added to the chroot-group and the id of the group itself.

My user id is 1026 and the group id of chroot is 65536. With this information we can log into our debian environment.

Now we are adding the user, in my case it is andre, and the group to the chroot environment. The home folder of the user is my DSM home folder. This is very convenient because now i’m able to share files and configurations that are stored inside of it between the host and the sandboxed system. Another benefit of it is, if you plan to remove the debian-chroot, all files will be kept inside of this folder even after removing the package.

If everything went fine you can check the following command and compare the output.

Install pyLoad in chroot

The installation is simple, paste the following commands and you are ready to go.

You can now run the pyLoad setup process. But for me it was not suitable to run it as root, so i decided to change it to my local user andre. The following steps will show how to do it.

This starts the configuration wizard that will guide you through all necessary parts with ease.

After the config part you can run it directly with removing the -s part from the command. We will start it inside of a screen, to be able to detach the session and keep it up and running.

Autostart pyLoad on DiskStation boot

Because of your chroot-environment not being able to start and stop services by its own during boot-up or shutdown, we have to provide a little script that works from the inside of your DiskStation and is able to do this task.

Log in via ssh as root on your DiskStation and create a file using the following command.

Paste these code into the file.

The script will be launched every time the NAS will boot or shutdown and runs some commands inside your chroot to start or stop your services.

Thats it, you are now able to use pyLoad and it will start and stop automatically.

Use pyLoad

There are many ways to interact with the pyLoad-daemon. The most common way is to use the web-interface. Simply open http://{ip-of-your-ds}:8080 in your preferred browser and you will see some of the following screens. All of the configuration you did earlier during the set-up process can be changed under the config tab.

Sources

 

 

 

How to create a vagrant base-box for VirtualBox

Introduction

What is vagrant?

Vagrant is a tool to run and maintain lightweight and flexible development environments. It is often used in software companies to enable all of their developer to work with the same standardized virtual machines that fit to their production environments. If it is configured right, you can simply type vagrant up and you have a running dev environment.

Why do we need a base-box

To be able to work with vagrant as the management system of your development environment there are two parts necessary. One is the environment configuration based on chef or puppet recipes. These recipes will provide a list of packages and set-ups to configure the virtual machine according to their current needs. For example you can install a web-server and MySQL, set up the user accounts and get it up running.

The base-box itself contains only the very rudimentary system. In our example it is a nearly plain Ubuntu Server 14.04. This image will be used to ground all of our development systems. Vagrant boots it up and runs the recipes from chef or puppet to install the needed software.

Create the base-box

Software requirements

To set up a base-box for using with vagrant we at least need vagrant itself and VirtualBox. You can grab all of them by following the next links. As you might see there is also a link to download the actual Ubuntu Server image. This will be needed to install the operating system of your base-box.

I will not write about the installation and configuration of VirtualBox but it should be self-explanatory.

Another way to install vagrant and VirtualBox on Mac OSX is to use homebrew in addition with cask. Brew should be known by every advanced user of OSX and cask is a simple addition to not only install command-line tools but also to handle full GUI software packages. With it you can install software like VirtualBox or IntelliJ Idea for example.

The following command will install our necessary software.

Hardware settings of the virtual machine

My specifications.

Harddisk
  • VDI self growing image (43Gb)
Chipset
  •  PIIX3
RAM
  • 512Mb
 Network
  •  eth0: NAT
  • eth1: Host-only
 Hardware virtualization
  •  VT-x / AMD-V
  • Nested Paging
 Video
  •  12Mb
  • no 2D / 3D acceleration
 Audio
  • none

The following screens will guide you through the set up of the virtual machine.

Installation of the base system

With the Ubuntu Server image mounted to your virtual disc drive, boot up the machine using the start button and press F12 followed by pressing to select the CD as your boot device. The Ubuntu installation wizard will pop up and asks you for your preferred language and keyboard settings. This part can be configured the way you like it. At the point where you will be asked for the partition layout, select use the whole disk  but without the logical volume manager (LVM). This is convenient and fits to all of our needs.

The next step is setting up the primary user account. For vagrant to be able to run, this user has to be vagrant set up with the same as password.

  • username: vagrant
  • password:  vagrant

We will leave the next step with no automatic updates and proceed onto the software selection. The only prerequisite vagrant has for a base-box is the presence of a ssh-server. After selecting the openssh-server press return to continue. It will take a few minutes to complete the package installation and setting up the system.

That’s it. After ejecting the disk image from the virtual cd-drive we can continue with our last step.

Configure the base-system

We already installed the openssh-server during the ubuntu installation wizard but it is not yet configured to be used by vagrant.

Log in into your newly created machine with the credentials we set up earlier.

Ubuntu first login

Ubuntu first login

Vagrant uses an insecure keypair to be able to log in to your machine. To set up this key we have to copy its public part to the virtual machine. First we need to determine the ip-address of the system by running ifconfig.

ifconfig

ifconfig

With this address in mind we can user the following command to add the key to the server.

If this does not work, simply paste the content of the vagrant.pub file into the VMs vagrant users home folder under .ssh/authorized_keys with the following commands.

Paste the content and press ctrl + o  followed by ctrl + x. For openssh to be able to user this key it needs to have a specific combination of access right. Run the following commands to fix them.

The next step is very important because vagrant uses the sudo command to do all its stuff as a privileged user but does not want to type a password every time it uses it. To configure sudo passwordless type sudo visudo and add the following line right behind the privileges of the root user.

And also edit the line about the sudo user group to be able to user it without a password.

Install the guest additions

To get the best out of your virtual machine experience it is recommended to install to VirtualBox guest additions.

Before you can do this, there are a few packages needed to be able to compile the kernel packages.

The next step is to mount the guest additions image to your machine by going to the main menu, select the devices tab and click on Insert Guest Additions CD Image followed by typing into your machines console the next command.

Finally run the script that matches to your selected system architecture. In my case it is a 64bit system so i had to use the following command.

Install puppet or chef

Last but not least we need to install one of the environment configuration management clients. If you prefer chef or puppet is completely up to you. I will use puppet because of some company dependencies.

If everything succeeds we are ready to go.

Packaging the box

This step sounds like one of the hardest but it is very simple because vagrant provides us with all of the needed tools. We only have to type one command and vagrant will provide you with a base-box in the current working directory. The base-box-14.04 is the name of your virtual machine.

Final result.

Download my base-box

If your are interested in using my newly created base-box image, simply download it from the following link.

Sources

 

Bootstrap DS215j

Introduction

The DS215j is the newest NAS device of the Synology DiskStation family.

DS215j connected

DS215j connected

I bought mine two weeks ago to serve the following functions

  • TimeMachine server for 2 MacBooks
  • Photo storage with external synced backup
  • DVB-T streaming and recording server (tvheadend or VideoStation)
  • automated tv-episode download and serving (SickRage)

Not all of these functions are working by now because of some issues with the new hardware architecture. The DS215j is based on a Marvell Armada 750 SoC, which is pretty new. By now, Synology released their toolchain and the guys of SynoCommunity have build a lot of packages to be installed via the graphical package manager. But for me, the DiskStation is not only the GUI but i also needed shell access combined with a few tools, the station does not provide be default.

This is why i tried and managed to install ipkg.

Optware bootstrap

There are no bootstrap packages for the DS215j by now but the Marvell Armada 375 is somewhat compatible with the Marvell Kirkwood mv6281 binaries. We can use them to set up our optware environment. If you see any problems with the provided binaries, please feel free to add a comment.

Set up ipkg

We have to download the bootstrap code and set up its environment.

Create the optware root directory

After extracting the above tar-archive a folder /opt has been created. This folder will be deleted every time you update your DSM. To prevent this problem, we have to move the content to a folder underneath /volume1/.

Old versions of the bootstrap script used a mount-point to bind /opt but this is not save any more because there have been some issues with the unmount during shutdown. Linking is the better solution.

Add /opt to PATH

Add the following line to /root/.profile or if you are using a user-account to get SSH access /home/username/.bashrc or the ZSH equivalent /home/username/.zshrc.

Set up the init-scripts

If you would like to install services, you want to start at boot-time, you have to set up these startup script.

Create the folder where the script will be placed.

Paste the following code into /usr/local/etc/rc.d/optware.sh.

Make it executable.

Thats it. If you have done everything right, you can reboot your DiskStation and log-in again to install your first packages.

Install packages with ipkg

Terminal

First we have to update the packages list and after that we install three necessary tools.

Graphical way

If you are not the guy who likes it to work within a terminal-session, there is an app for it, running inside the DSM to manage your ipkg packages.

Open the Package-Center. Press Settings -> Package Sources -> Add and paste the following repository URL.

Now you can browse the newly added repository under the Community Tab inside of the Package-Center. Install iPKGui and launch the app afterwards from the main menu. It could be possible that the app has some issues with its name, but this is only a design flaw.

iPKGui icon

iPKGui icon

Sources

Golang and reflections

Introduction

Reflection is the ability of a program to examine its own structure. In go we can use it for example to build dynamically accessable structs.

How-to use reflect

I will give you with two examples and a hopefully easy-to-understand explanation of them.

Simple usage

At hikewith.me we use a simple way of reflection to map dynamically specified keys to a set-up value. With this we can map database-columns to a given key and generate SQL queries according to the given input instead of predefining everything. In our database, field are named another way then in our backend code (go). For example the first-name of a user will be saved in the column first_name but the field in our User-struct is named FirstName. We need a way to map these connection dynamically for updating the user data and keeping our code simple.

Field-map struct

These struct type defines the connection between our used struct field and the columns found in our database table. FirstName, LastName etc. are the field names used in our User struct. The assigned value will be the table column. It will be loaded from a json-file (see JSON configuration).

User struct

This type stores the information about a user. If he changes some of his data we use this struct to update the dataset in our database.

JSON configuration

As already noted, we store the mapping between columns and keys in a JSON-file that will be loaded during the initial start of our backend.

Configuration loader

To load the JSON represented field configuration we use the gotamer-cfg-loader. You can get it using the following line.

The usage is really simple. Initialize the var to save the loaded configuration as type of your FieldMapping struct and afterwards simply load the file content into this struct.

Field-mapper

This is where the magic happens. The method will be called with the field name as param and returns the value of the field. First it reflects the list of all fields of the FieldMapping struct after this we can do an indirect reflect to get value of the field.

A more generic version

The following code is a more generic version of my code from above. It is not bound to a specific type but can be used with any kind of type to get a fields value. I will show how to use this function within the more advanced example.

Advanced usage

In this second example, i will show how to compare two structs of the same type with each other and get a diff. With this diff-struct we update our user data-set.

Compare struct

It is a simple key-value-store that will be used in a slice to collect all diffs.

Compare two structs of the same type

This function will be called from within a new-user-struct like this newuserdata.compareUpdate(olduserdata). The reflection will be used in the first place to get the number of fields in this struct. With this we can iterate over all fields and use the above defined getFieldByIndex function to get the value of each field and compare them to the corresponding value of the old-user-struct. The last step is to add the compare struct to our returned slice if there was a difference.