Sunday 15 November 2015

Python on Cloud Foundry

Cloud Foundry is an open source cloud application platform. It can be installed on an own Server or bought as a service from a company. I myself tried cloud foundry provided by Swisscom. The advantage of the service is on one hand the integrated third party tools like database connections.

This post just describes how to get your python application running on already installed cloud foundry platform, not the installation of the platform. The demo application is a basic cloud interface based on the falcon framework for cloud API.

Preparations

To generate the dependencies for your cloud environment you need pip and virtualenv, which can be installed like this.

sudo apt-get install python-pip, python-virtualenv

Since the demo application uses a falcon cloud API we need falcon. Cython is used to improved the falcon's performance and gunicorn is the http server to run the falcon application on.

sudo pip install cython, falcon, gunicorn

Setup for the project

The whole project for the cloud has to be saved in one folder. To make the installation of the application on the cloud easier are we providing a requirements.txt file and a manifest.yml. Besides this is the folder containing the source for the application, in our case the file server.py.

The manifest.yml has to be written manually. It contains the information about the application. A very simple manifest can you see here.

---
applications:
- name: Example
  memory: 256M
  command: "gunicorn server:app"

The requirements.txt file can be generated automatically using virtualenv. For this open terminal, and create a new virtual environment for our cloud application. Within this virtual environment we install all the required packets to run our application, which is in our case again cython, falcon and gunicorn. With the pip freeze command its possible to print the currently installed packets into the requirements file.

virtualenv env
source env
pip install --upgrade cython falcon gunicorn
pip freeze > example/requirements.txt
deactivate
rm -r env

Upload the application to the cloud

To be able to upload your project to the cloud you need to install the command line interface to the cloud. You can download the one for your system here. Go to the folder containing your application and log into your cloud with the following command and enter your password.

cf login -a https://your-cloud.com -u myusername

Within the cloud you have to change to the space which will contain your app later. For this switch to it with the following command.

cf target -o myusername -s DEVSPACE

Now you can push your application to the cloud using the cf push command. You don't need to specify any more parameters in the push command since we already defined them all in the manifest.yml.

cf push
That's already it. For everyone wanting a starting point for their falcon application is here a simple hello world application for the falcon.
import falcon

class HelloWorld(object):
    def on_get(self, req, resp):
        resp.body = "hello world"
        resp.status = falcon.HTTP_200

app = falcon.API()
helloWorld = HelloWorld()
app.add_route('/helloworld', helloworld)
More support for your first falcon project is available on the official tutorial.

Sources

Monday 21 September 2015

OpenCV with CUDA on Ubuntu 14.04

This post describes how to install OpenCV with CUDA support on a PC with Ubuntu 14.04. To install OpenCV with CUDA support, CUDA needs to be installed in a first step. If you already have a CUDA installation you can jump to the OpenCV installation.

The installation was tested on Ubuntu 14.04 LTS with CUDA 7.5 and OpenCV 3.0

First Step: Installation of CUDA

The current CUDA version is 7.5. It can be downloaded from the Nvidia developers page (CUDA-Toolkit).

Installation of the CUDA Toolkit

sudo dpkg -i cuda-repo-ubuntu1404_7.5-18_amd64.deb
sudo apt-get update
sudo apt-get install cuda

Environmental Variables

The CUDA toolkit needs to be added to the path and library path so it can be used by other applications. This can be done in the .bashrc file or in the /etc/environment file. In our case it is defined in the /etc/environment file.

PATH="...:/usr/local/cuda/bin"
LD_LIBRARY_PATH="/usr/local/cuda/lib64"

Test Installation

To load the new environmental variables restart the PC. Afterwards it should be possible to generate the provided samples with the command

cuda-install-samples-7.5.sh .
With the example deviceQuery it is possible to verify the CUDA installation, which is run as followed.
cd NVIDIA-CUDA-7.5_Samples/1_Utilities/deviceQuery
make
./deviceQuery

This should return an output similar to the following.

./deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "Quadro K2200"
  CUDA Driver Version / Runtime Version          7.5 / 7.5
  CUDA Capability Major/Minor version number:    5.0
  Total amount of global memory:                 4095 MBytes (4294246400 bytes)
  ( 5) Multiprocessors, (128) CUDA Cores/MP:     640 CUDA Cores
  GPU Max Clock rate:                            1124 MHz (1.12 GHz)
  Memory Clock rate:                             2505 Mhz
  Memory Bus Width:                              128-bit
  L2 Cache Size:                                 2097152 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
  Maximum Layered 1D Texture Size, (num) layers  1D=(16384), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(16384, 16384), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 1 copy engine(s)
  Run time limit on kernels:                     Yes
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  Device supports Unified Addressing (UVA):      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 3 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 7.5, CUDA Runtime Version = 7.5, NumDevs = 1, Device0 = Quadro K2200

Result = PASS

Second Step: Installation of OpenCV

Requirements OpenCV

sudo apt-get install libopencv-dev build-essential checkinstall cmake pkg-config yasm libtiff4-dev libjpeg-dev libjasper-dev libavcodec-dev libavformat-dev libswscale-dev libdc1394-22-dev libxine-dev libgstreamer0.10-dev libgstreamer-plugins-base0.10-dev libv4l-dev python-dev python-numpy libtbb-dev libqt4-dev libgtk2.0-dev libfaac-dev libmp3lame-dev libopencore-amrnb-dev libopencore-amrwb-dev libtheora-dev libvorbis-dev libxvidcore-dev x264 v4l-utils

Get OpenCV Repository

Clone the official OpenCV repository from github. It will create the folder opencv and copy the repository in it.

git clone https://github.com/Itseez/opencv.git
It will take a moment to download the repository.

Build OpenCV

cd opencv
mkdir release
cd release
cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local -D WITH_TBB=ON -D BUILD_NEW_PYTHON_SUPPORT=ON -D WITH_V4L=ON -D INSTALL_C_EXAMPLES=ON -D INSTALL_PYTHON_EXAMPLES=ON -D BUILD_EXAMPLES=ON -D WITH_QT=ON -D WITH_OPENGL=ON -D ENABLE_FAST_MATH=1 -D CUDA_FAST_MATH=1 -D WITH_CUBLAS=1 ..
Check the configuration summary in the end. Make sure that CUDA support is activated with the correct version and CUBLAS is activated as well.
--     Use Cuda:                    YES (ver 7.5) 
--     Use OpenCL:                  YES
--
--   NVIDIA CUDA
--     Use CUFFT:                   YES
--     Use CUBLAS:                  YES
--     USE NVCUVID:                 NO
--     NVIDIA GPU arch:             20 21 30 35
--     NVIDIA PTX archs:            30
--     Use fast math:               YES

Install OpenCV

If the configuration looks fine can the installation be started. The installation can take up to a few hours.
make
sudo make install

Configure the library search path

Create the file /etc/ld.so.conf.d/opencv.conf and add the following line.
/usr/local/lib
Add the following line to the ~/.bashrc file

PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/usr/local/lib/pkgconfig
Reload the bashrc file for the current session

source ~/.bashrc

Sources

Sunday 26 April 2015

Ubuntu daemon

Creating a service on Ubuntu 14.04

Creating a daemon on Ubuntu is quite simple. To run your own daemon scripts on boot-up can be done in the following steps:

Create a daemon in /etc/init.d

The daemon to be run needs to be in the init.d folder. Ubuntu provides a template with the daemon /etc/init.d/skeleton. Therefore copy the skeleton script and name it however you like.

sudo cp /etc/init.d/skeleton /etc/init.d/mydaemon

To be able to run the daemon it it necessary to make it executable

sudo chmod +x /etc/init.d/mydaemon

Adapt the script mydaemon to match your functionality, in the example the executable is in your $HOME/bin directory. The lines to be adapted are highlighted.

#! /bin/sh
### BEGIN INIT INFO
# Provides:          mydaemon
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: My Daemon
# Description:       This file starts my daemon
### END INIT INFO

# Author: Foo Bar 
#
# Please remove the "Author" lines above and replace them
# with your own name if you copy and modify this script.

# Do NOT "set -e"

# PATH should only include /usr/* if it runs after the mountnfs.sh script
PATH=/sbin:/usr/sbin:/bin:/usr/bin:/home/user/bin
DESC="my daemon"
NAME=mydaemon
DAEMON= mydaemon
DAEMON_ARGS="--options"
PIDFILE=/var/run/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME

Afterwards try to start and stop your daemon with

sudo service mydaemon start
sudo service mydaemon stop

Start daemon at boot

To automatically start the daemon at boot time it is necessary to add it to the boot list. This is done with the following command:

update-rd.d mydaemon default 97 03

If you want to remove the daemon again use the following command:

update-rd.d -f mydaemon remove