Interesting Research Problem for Sensor Networks and Smart Spaces

Sensor technology in combination with pervasive computation and communication can significantly contribute to improvements in our way of living. Devising (decentralized) solutions for the construction and deployment of large-scale wireless (sensor) systems requires concentration on the application of geospatial social networking which id used for monitoring and managing crowds of people, with the aim of maintaining safety and comfort for the individuals. The central research question is how to devise a large-scale collaborative network of highly mobile, miniature, wireless devices collecting, disseminating, aggregating, and processing information to provide instant feedback to its users for the purpose of maintaining their safety and comfort.

The goal is to achieve significant progress in scaling up solutions for the deployment of wireless distributed sensor/actuator systems. These systems consist of a large collection of sensor/actuator devices that need to be shared by high-level applications. The intention is to close the gap between the networks and the applications. Some aspects of this gap are ease of installation, (self-)configuration and maintenance; ease of application development and deployment, hiding complexities of the network; and harnessing the behavior of a large network of relatively primitive nodes (constraining the emergent behavior).

This would require a cross disciplinary research effort in the following areas:

• Technical research on wireless (sensor) networking
• Research on geospatial social network data gathering and analysis
• Research on human-systems interaction (including management of privacy aspects)

An edited extract from http://ewids.distributed-systems.net/index.html.

Software Frameworks

Introduction

Definition: “A framework is mainly a product for developers used as an integrated development environment that facilitates, supports, guides, confines, and helps the developers in building application in a well defined domain.” Shai Ben-Yehuda.

A Framework is a package of software libraries that abstracts computer software by wrapping them in a well-defined and reusable application programming interface (API). In other words low level API are abstracted with high level API so that programmers comfortably deal with integrated and managed low level features without understanding their peculiarities and still keeping their code standardized and correct. However, frameworks contain key distinguishing features that separate them from normal libraries.

For example, a web application framework may provide user session management, data storage, and a templating system. A desktop application framework may provide user interface functionality and widgets (common GUI elements). Almost all frameworks control at least some aspect of an application’s flow of execution.

Why Frameworks

• Frameworks are for conceptual integrity
• Reducing/Abstracting complexity
• Reducing development time and effort
• Reduce possibility of bugs and errors
• Control the behavior for standardization

Features

Software frameworks have these distinguishing features that separate them from libraries or normal user applications:

1. Inversion of control – The overall program’s flow of control is not dictated by the caller, but by the framework.
2. Default behavior – A framework has a default behavior, which must actually be some useful behavior and not a series of no-ops.
3. Extensibility – A framework can be extended by the user usually by selective overriding or specialized by user code providing specific functionality.
4. Non-modifiable framework code – The framework code, in general, is not allowed to be modified. Users can extend the framework, but not modify its code.

Types

Following are some of the types of frameworks that we find in our development works:

• Development: Eclipse. .NET, Java Development Kit, Mac’s Cocoa (AppKit and Foundation Kit)
• Application runtime: Mac’s Cocoa Runtime, Windows Common Language Runtime, Android, Java Runtime Environment
• Web Application: Apache-PHP
• Security: RSBAC, GFAC, FLASK
• Many others for specific application domain

Fundamentally, there are two kinds of frameworks:

• A full-stack framework provides a single set of components, covering the entire spectrum of features needed to build the application. They ensure that the components work well with one another. Examples are Android, Django and Zend frameworks.
• A glue framework provides a set of adapters and interface code. They are suppose to handle many different components at any part of the software stack. They ensure that the adapters will work well with any possible combinations of components. Examples are FLASK, GFAC, TurboGears, Twisted and Pylons.

There will always be framework developers who prefer the full-stack and there will always be framework developers who prefer glue. There will also always be developers in each camp. There will always be frameworks that can either be glue or full stack depending on the need.

There is no perfect framework. There are only different frameworks that suit different kinds of needs.

Bibliography

1. http://en.wikipedia.org/wiki/Software_framework
2. http://docforge.com/wiki/Framework
3. http://www.b-list.org/weblog/2007/feb/19/python-framework-design/
4. http://c2.com/cgi/wiki?SoftwareFramework

An email response to my family – Here for my students and friends

My brother sent this to a discussion on Islamic Financial System that lead to further discussion on the efforts of our Ulema-e-karaam:

Persistent people begin their success where others end in failure.

~Edward Eggleston

Following is my response in a philosophical mindset:

So true! Where does one get such strength and solid thought as flesh and material surely perish as it is beyond human power to control it. After baba’s death I decided to sort out how I could accept death? How I can perish with being sure I am a winner as a whole? The answer was to face myself and fix internal errors where perception and understanding is twisted with ones desires and misconceptions in the society. This can ruin both this life and the eternal one whereas both are important for a sane believer! Death is expected anytime so we have to make sure how we want to progress and what state of mind and heart is required during this long process so that we can embrace death and the eternal life with a strong desire anytime through this progress.

I made dua to Allah to lead me to the truth and give me eternal success. I made an honest and unbiased effort to find the best of whats possible. I found out things that would make me go through mental shocks for a long time till I could sort it out towards a positive meaning and action plan. In Aasil’s case I learned that the system is screwed up full time and my only way to fix it with all my responsibilities is to live effectively with least possible dependence on it.

At the moment I am learning various specifics of existence and it’s responsibilities, while I am clear at the conceptual level as a servant of my lord and a responsible family member apart from being part of the umma and human race. I am working out weaknesses and major problems prioritized on weight-age and practicality.

Desire is a powerful strength and it can be used to achieve some certain important goals. If you keep an insight on eternal life in your decisions than you are surely going to win in case you are honest to yourself. Material losses will matter less while eternal will be more important and it is based on the decisions, intentions and effort made! A proper balance is required because material losses also affect eternal life.

Above is an integral part of the juice of my learning and experiences but I do accept that I have been taking family  and friends’ issues emotionally at times rather than practically. At the cost of some emotional and spiritual damages I have progressed a lot but it could have been a better experience if I had been more careful by discussing things with my sheikh and teacher before reacting to them. And a lot is still to be covered!

It would be nice if you guys also share such stuff so that we can strengthen our synergy easily. The future of our family, race, umma and the greater humanity depends on our best possible effort. We need to correct ourselves as well as deal our external matters in an organized way.

Love.

–

Shahbaz

Blink App in NesC

In this second article on TinyOS and NesC programming we will visit some of the earlier concepts that we introduced in the first article. This time we will look at them from a more technical perspective so that we can understand how the Blink application fits onto the platform that enables it. Our use case application example, Blink, is basically equivalent to the “helloWorld” application that we use to learn PC targeted languages. But in our case the mote (sensor/actuator device) does not provide a keyboard or screen! Therefore we use the LEDs to blink in a manner that we command in our program.

nesC (pronounced “NES-see”) is an extension to the C programming language designed to use the structuring concepts and execution model of TinyOS. TinyOS is an event-driven operating system designed for sensor network nodes that have very limited resources (e.g., 8K bytes of program memory, 512 bytes of RAM).

The basic concepts behind nesC are:

• Separation of construction and composition: programs are built out of components, which are assembled (“wired”) to form whole programs. Components have internal concurrency in the form of tasks. Threads of control may pass into a component through its interfaces. These threads are rooted either in a task or a hardware interrupt.
• Specification of component behaviour in terms of set of interfaces. Interfaces may be provided or used by components. The provided interfaces are intended to represent the functionality that the component provides to its user, the used interfaces represent the functionality the component needs to perform its job.
• Interfaces are bidirectional: they specify a set of functions to be implemented by the interface’s provider (commands) and a set to be implemented by the interface’s user (events). This allows a single interface to represent a complex interaction between components (e.g., registration of interest in some event, followed by a callback when that event happens). This is critical because all lengthy commands in TinyOS (e.g. send packet) are non-blocking; their completion is signaled through an event (send done). By specifying interfaces, a component cannot call the send command unless it provides an implementation of the sendDone event.Typically commands call downwards, i.e., from application components to those closer to the hardware, while events call upwards. Certain primitive events are bound to hardware interrupts.
• Components are statically linked to each other via their interfaces. This increases runtime efficiency, encourages rubust design, and allows for better static analysis of programs.
• nesC is designed under the expectation that code will be generated by whole-program compilers. This should also allow for better code generation and analysis.

The image below shows the structure (components and their wiring) of a simple application, Blink, that blinks an LED once a second:

Following is the Blink.nc configuration:

/*
 *  Copyright (c) 2000-2002 The Regents of the University  of California.
 *  and Copyright (c) 2002 Intel Corporation
 *  All rights reserved.
 */
configuration Blink {
}
implementation {
  components Main, BlinkM, ClockC, LedsC;

  Main.StdControl -> BlinkM.StdControl;
  BlinkM.Clock -> ClockC;
  BlinkM.Leds -> LedsC;
}

Following is BlinkM.nc module:

/*
 *  Copyright (c) 2000-2002 The Regents of the University  of California.
 *  and Copyright (c) 2002 Intel Corporation
 *  All rights reserved.
 *
 * Implementation for Blink application.  Toggle the red LED when the
 * clock fires.
 */
module BlinkM {
  provides interface StdControl;
  uses interface Clock;
  uses interface Leds;
}
implementation {
  bool state; /* the state of the red LED (on or off)   */

  /* Initialize the component.
   * @return Always returns SUCCESS */
  command result_t StdControl.init() {
    state = FALSE;
    call Leds.init();
    return SUCCESS;
  }

  /* Start things up.  This just sets the rate for the clock component.
   * @return Always returns SUCCESS */
  command result_t StdControl.start() {
    return call Clock.setRate(TOS_I1PS, TOS_S1PS);
  }

  /* Halt execution of the application.
   * This just disables the clock component.
   * @return Always returns SUCCESS */
  command result_t StdControl.stop() {
    return call Clock.setRate(TOS_I0PS, TOS_S0PS);
  }

  /* Toggle the red LED in response to the Clock.fire event.
   * @return Always returns SUCCESS */
  event result_t Clock.fire() {
    state = !state;
    if (state) {
      call Leds.redOn();
    } else {
      call Leds.redOff();
    }

    return SUCCESS;
  }
}

Bibliography:

http://nescc.sourceforge.net/

http://tinyos.net

Basics of TinyOS and NesC

TinyOS is an operating system for low power embedded devices, especially wireless sensor devices. It is open source with a strong developer community. Specifically it is a system, runtime environment, for nesC programs running on Mote hardware. Its features include:

• Resource management.
• Selected components are linked into program at compile time.
• Written in nesC and C.
• All time-consuming commands are non-blocking.

Some salient components are:

Analogue to digital conversion Cryptography Data logging File system I2C communication LED control Memory allocation

Random number generation Routing Sensor board input Serial communication (wired and wireless) Timers Watchdog timer

Programming TinyOS with NesC

NesC is a dialect of C with the basic unit of code being a component. Components connect via interfaces and the process is known as wiring.

Components and Interfaces

A nesC application consists of one or more components assembled, or wired, to form an application executable. Components define two scopes: one for their specification which contains the names of their interfaces, and a second scope for their implementation. A component provides and uses interfaces. The provided interfaces are intended to represent the functionality that the component provides to its user in its specification; the used interfaces represent the functionality the component needs to perform its job in its implementation.

Interfaces are bidirectional: they specify a set of commands, which are functions to be implemented by the interface’s provider, and a set of events, which are functions to be implemented by the interface’s user. For a component to call the commands in an interface, it must implement the events of that interface. A single component may use or provide multiple interfaces and multiple instances of the same interface.

The set of interfaces which a component provides together with the set of interfaces that a component uses is considered that component’s signature.

Modules and Configurations

There are two types of components in nesC: modules and configurations. Modules provide the implementations of one or more interfaces. Configurations are used to assemble other components together, connecting interfaces used by components to interfaces provided by others. Every nesC application is described by a top-level configuration that wires together the components inside.

We will continue by looking at some simple examples in the next article.

Parse, Traverse and Search XML Docs

In this series of articles you will learn XML processing with SAX and DOM. We will make sure that the practical examples are simple yet they give you the right amount of details to dive into professional development. It will be easier to get your hands on available training source code. I am not going to use any other example based on this source code in my following web services’ tutorials. The web service example code uses obsolete libraries. The code can be used for other good purposes based on standard Python libraries.

I assume that you are already familiar with Python interpreter, packages and modules and have some fluency with the language and syntax. I also assume that you have good understanding of data structures and you have used them in programming tasks.

Parsing XML

Parsing an XML document is very simple. We use the “binary.xml” from the given example source code to demonstrate this.

>>> from xml.dom import minidom
>>> binaryxml = minidom.parse('~/diveintopython/py/kgp/binary.xml')
>>> binaryxml
<xml.dom.minidom.Document instance at 0xb777c38c>
>>> print binaryxml.toxml()
<?xml version="1.0" ?>
<grammar>
<ref id="bit">
  <p>0</p>
  <p>1</p>
</ref>
<ref id="byte">
  <p><xref id="bit"/><xref id="bit"/><xref id="bit"/><xref id="bit"/>\
<xref id="bit"/><xref id="bit"/><xref id="bit"/><xref id="bit"/></p>
</ref>
</grammar>

First we have imported minidom module from xml.dom package. minidom.parse takes one argument and returns a parsed representation of the XML document.

The object returned from minidom.parse is a Document object. It is a descendant of the Node class. This Document object is the root level of a complex tree like structure of interlocking Python objects that completely represents the XML document you passed to minidom.parse.

toxml is a method of the Node class (and is therefore available on the Document object you got from minidom.parse. toxml prints out the XML that this Node represents. For the Document node, this prints out the entire XML document.

Traversal

To understand traversal and nodes try the following example code.

>>> binaryxml.childNodes
[<xml.dom.minidom.DocumentType instance at 0xb777c68c>, <DOM Element: grammar at 0xb777c6ec>]
>>> binaryxml.childNodes[0]
<xml.dom.minidom.DocumentType instance at 0xb777c68c>
>>> binaryxml.firstChild
<xml.dom.minidom.DocumentType instance at 0xb777c68c>

Every Node has a childNodes attribute, which is a list of the Node objects. A Document always has only one child node, the root element of the XML document (in this case, the grammar element). To get the first, and in this case last as well, child node, just use regular list syntax. Remember, there is nothing special going on here; this is just a regular Python list of regular Python objects. Getting the first child node of a node is a common activity. The Node class has a firstChild attribute, which is synonymous with childNodes[0]. (There is also a lastChild attribute, which is synonymous with childNodes[-1].)

>>> grammarElement = xmldoc.firstChild
>>> print grammarElement.toxml()
<grammar>
<ref id="bit">
  <p>0</p>
  <p>1</p>
</ref>
<ref id="byte">
  <p><xref id="bit"/><xref id="bit"/><xref id="bit"/><xref id="bit"/>\
<xref id="bit"/><xref id="bit"/><xref id="bit"/><xref id="bit"/></p>
</ref>
</grammar>

The toxml method is defined in the Node class therefore it is available to any (XML) node and not just the Document element.

>>> grammarElement.childNodes
[<DOM Text node "\n">, <DOM Element: ref at 17533332>, \
<DOM Text node "\n">, <DOM Element: ref at 17549660>, <DOM Text node "\n">]
>>> print grammarElement.firstChild.toxml()
>>> print grammarElement.childNodes[1].toxml()
<ref id="bit">
  <p>0</p>
  <p>1</p>
</ref>
>>> print grammarElement.childNodes[3].toxml()
<ref id="byte">
  <p><xref id="bit"/><xref id="bit"/><xref id="bit"/><xref id="bit"/>
<xref id="bit"/><xref id="bit"/><xref id="bit"/><xref id="bit"/></p>
</ref>
>>> print grammarElement.lastChild.toxml()

The XML in binary.xml seems to have two elements/nodes in grammar the two ref elements. But you’re missing something! The carriage returns … After the '<grammar>' and before the first '<ref>' there is a carriage return and this counts as a child node of the grammar element. Similarly there is a carriage return after each '</ref>'. These count as child nodes. Therefore grammar.childNodes has actually a list of 5 objects: 3 Text objects and 2 Element objects.

The first child is a Text object representing the carriage return after the '<grammar>' tag and before the first '<ref>' tag. The second child is an Element object representing the first ref element. The fourth child is an Element object representing the second ref element. The last child is a Text object representing the carriage return after the '</ref>' end tag and before the '</grammar>' end tag.

>>> grammarElement
<DOM Element: grammar at 19167148>
>>> refElement = grammarElement.childNodes[1]
>>> refElement
<DOM Element: ref at 17987740>
>>> refElement.childNodes
[<DOM Text node "\n">, <DOM Text node "  ">, <DOM Element: p at 19315844>, \
<DOM Text node "\n">, <DOM Text node "  ">, \
<DOM Element: p at 19462036>, <DOM Text node "\n">]
>>> pElement = refElement.childNodes[2]
>>> pElement
<DOM Element: p at 19315844>
>>> print pElement.toxml()
<p>0</p>
>>> pElement.firstChild
<DOM Text node "0">
>>> pElement.firstChild.data
u'0'

As you saw in the previous example, the first ref element is grammarNode.childNodes[1], since childNodes[0] is a Text node for the carriage return. The ref element has its own set of child nodes, one for the carriage return, a separate one for the spaces, one for the p element, and so forth. You can even use the toxml method here, deeply nested within the document. The p element has only one child node (you can’t tell that from this example, but look at pNode.childNodes if you don’t believe me), and it is a Text node for the single character '0'. The .data attribute of a Text node gives you the actual string that the text node represents. But what is that 'u' in front of the string?

The 'u' means that the string has unicode encoding unlike the default ASCII in Python environment. The reason is that web services standardize encoding with unicode and that is the same reason why 'u' precedes  '0'.

Searching for elements

Traversing XML documents by stepping through each node can be tedious. If you want to find and get to an element in a hige XML document then there is a smart way to do it. getElementsByTagName.

>>> from xml.dom import minidom
>>> binaryxml = minidom.parse('binary.xml')
>>> refList = binaryxml.getElementsByTagName('ref')
>>> refList
[<DOM Element: ref at 136138108>, <DOM Element: ref at 136144292>]
>>> print refList[0].toxml()
<ref id="bit">
  <p>0</p>
  <p>1</p>
</ref>
>>> print refList[1].toxml()
<ref id="byte">
  <p><xref id="bit"/><xref id="bit"/><xref id="bit"/><xref id="bit"/>\
<xref id="bit"/><xref id="bit"/><xref id="bit"/><xref id="bit"/></p>
</ref>

getElementsByTagName takes one argument, the name of the element you wish to find. It returns a list of Element objects corresponding to the XML elements that have that name. In this case you have two ref elements.

>>> firstRef = refList[0]
>>> print firstRef.toxml()
<ref id="bit">
<p>0</p>
<p>1</p>
</ref>

The first object in your refList is the 'bit' ref element. You can use the same getElementsByTagName method on this Element to find all the <p> elements within the 'bit' ref element. The getElementsByTagName method returns a list of all the elements it found (in this case 2).

>>> pList = firstRef.getElementsByTagName("p")
>>> pList
[<DOM Element: p at 136140116>, <DOM Element: p at 136142172>]
>>> print pList[0].toxml()
<p>0</p>
>>> print pList[1].toxml()
<p>1</p>>>> pList = xmldoc.getElementsByTagName("p")
>>> pList
[<DOM Element: p at 136140116>, <DOM Element: p at 136142172>, <DOM Element: p at 136146124>]
>>> pList[0].toxml()
'<p>0</p>'
>>> pList[1].toxml()
'<p>1</p>'
>>> pList[2].toxml()
'<p><xref id="bit"/><xref id="bit"/><xref id="bit"/><xref id="bit"/>\
<xref id="bit"/><xref id="bit"/><xref id="bit"/><xref id="bit"/></p>'

Note carefully that the difference between this and the previous example. Previously, you were searching for p elements within firstRef, but here you are searching for p elements within binaryxml, the root-level object that represents the entire XML document. This does find the p elements nested within the ref elements within the root grammar element. The first two p elements are within the first ref (the ‘bit’ ref). The last p element is the one within the second ref (the ‘byte’ ref).

This article reduces theory and provides more focus on techniques by example. This tutorial has been derived from Dive into Python, Chapter 9.

Interesting Research Problem for Mobile Networks

Mobile network operations are very resource-intensive, and with a growing number of network elements and technologies, the cost will escalate unless a higher level of process automation is introduced.

Network operators and system providers have been talking self-configuration, self-optimization, self-healing, and self-protection for more than a decade (see e.g. IBMs “Autonomic Vision & Manifesto” from 2001). Progress has been made, but the inherent uncertainty of a mobile network has been a huge obstacle in moving from research to practical applications. It often turns out that only the human brain is able to handle the complexity which comes from the size of the network as well as from uncertainty in e.g. aggregated data (details are removed to keep the amount of data manageable), randomness (e.g. a football match causes a sudden amount of network traffic), missing data (for various reasons data is not always available for all network elements), and conflicting data (data indicating multiple problems).

The solution for automated troubleshooting by 2operate released in 2009 was a breakthrough and a major step towards self-healing. By utilizing technology originally developed for automation of medical diagnostic problems, 2operate was able to provide a level of automation in the mobile network troubleshooting processes which was higher than any competing solution in the market. The key was a technology which was sufficiently strong to cope with the complexity and mentioned sources of uncertainty. The key technology used by 2operate is Bayesian network (or more generally, Probabilistic Graphical Models = PGMs).

This can be further exploited for self-healing and self-protection. Troubleshooting is becoming increasingly complex with increased network complexity from new communications technology (e.g. LTE) which must co-exist and interact with old technology (e.g. GSM and UMTS). Partly overlapping with troubleshooting is network security (some problems may be due to intruders with a desire to prevent or steal communication in the network). Security is becoming increasingly important in the network management community. With more automated operations (self-healing, self-optimization, self-configuration), the network will be more vulnerable to intruders, and automated detection and healing is necessary to keep costs down and to reduce downtime.

• Investigate the problem domain: Which types of technical and security problems are foreseen? How are these prevented today?
• Research and Development in self-healing/self-protection solution: What are the entities to monitor in an LTE network? What can be observed for an entity (indicators)? What can be concluded (root causes)?

Extract from a PHD Call for Application.

Installing and Configuring UEC’s Eucalyptus

In this article you will be dealing with the installation and configuration of a very basic Cloud topology with UEC. You will cover advance features and configurations after I cover an article on integration of the enterprise application with the cloud. I assume that the reader is a seasonal network and system administrator in Linux based environments. You should have at least one machine with hardware based virtualization as KVM is going to be used. This tutorial material should not be used at production systems at all because it is totally an experimental setup.The above diagram shows a minimalistic production topology. For simplicity and hardware constrained experimentation you can use only one server machine to host both CC and NC and this is the same use case that we have in this tutorial.

The core eucalyptus packages of UEC are:

• eucalyptus-cloud – Cloud controller
• eucalyptus-cc – Cluster controller
• eucalyptus-walrus – Walrus (S3)
• eucalyptus-sc – Storage controller
• eucalyptus-nc – Node controller

If you are not aware of the architecture of Eucalyptus then please read Introduction to Private Cloud Computing with UEC’s Eucalyptus.

At this point you need to have a Ubuntu Server CD/DVD to install UEC.

Cluster Controller Installation

1. At boot menu select “Install Ubuntu Enterprise Cloud”
2. During installation, select Cluster installation. NC installation can be delayed easily.
3. Select the same “eth0″ interface as public and private interface.
4. Specify some arbitrary IP address range for Public IPs – 192.168.1.101 to 192.168.1.200. This can later be changed in /etc/eucalyptus/eucalyptus.conf.

Post-Installation

1. Install system updates.

 $ sudo apt-get update
 $ sudo apt-get dist-upgrade

Network Configuration

At this point you start considering the NC. It expects a bridge so you will create a bridge and add eth0 as an interface. For this purpose you will need brctl command from bridge-utils package.

Configure the bridge for static IP with the following configuration in /etc/network/interfaces.

auto eth0
 iface eth0 inet manual
auto br0
 iface br0 inet static
 address cc.uec.shaz
 netmask 255.255.255.0
 network uec.shaz.0
 broadcast uec.shaz.255
 gateway uec.shaz.1
 bridge_ports eth0
 bridge_fd 9
 bridge_hello 2
 bridge_maxage 12
 bridge_stp off

Node Controller Installation

1. Install Node Controller

$ sudo apt-get install eucalyptus-nc

Web Console

The CC can be accessed via a web console with https://cc.uec.shaz:8443. You will now be able to:

• download the account credentials
• view currently installed images
• make available or disable the installed images
• browse through the store and install available images
• add/edit/delete/enable/disable users
• change the CC configuration
• access external links of Ubuntu and Eucalyptus community
• register the cloud with Landscape or RightScale

Retrieving Credentials

After installing CC and NC, users of the cloud will need to retrieve their credentials. This can be done either from a web browser or from a command line.
For command line use the following commands:

$ mkdir -p ~/.euca
$ chmod 700 ~/.euca
$ cd ~/.euca
$ sudo euca_conf --get-credentials mycreds.zip
$ unzip mycreds.zip
$ cd -

Using Your Credentials

After obtaining the credentials, you need to setup EC2 API and AMI tools on the server using X.509 certificates. For this, source the included “eucarc” file to set up the Eucalyptus environment

$ . ~/.euca/eucarc

This can be added to the ~/.bashrc file, so that the Eucalyptus environment is set up automatically when you log in.

$ echo "[ -r ~/.euca/eucarc ] && . ~/.euca/eucarc" >> ~/.bashrc

Node Registration

Execute following command to register the node:

$ sudo euca_conf --no-rsync --discover-nodes

The above command will prompt for adding the discovered nodes

New node found on cc.uec.shaz; add it? [Yn]

Now euca-describe-availability-zones cleanly reports the node registration

$ euca-describe-availability-zones verbose
AVAILABILITYZONE        uec-cloud       cc.uec.shaz
AVAILABILITYZONE        |- vm types     free / max   cpu   ram  disk
AVAILABILITYZONE        |- m1.small     0002 / 0002   1    128     2
AVAILABILITYZONE        |- c1.medium    0002 / 0002   1    256     5
AVAILABILITYZONE        |- m1.large     0001 / 0001   2    512    10
AVAILABILITYZONE        |- m1.xlarge    0001 / 0001   2   1024    20
AVAILABILITYZONE        |- c1.xlarge    0000 / 0000   4   2048    20

Install store images

You cab install image available from the store. After installing from the store euca-describe-images shows the image details.

$ euca-describe-images
IMAGE   eri-0D4D117E    image-store-1268999725/ramdisk.manifest.xml     adminavailable       public  x86_64  ramdisk
IMAGE   eki-F8BF1114    image-store-1268999725/kernel.manifest.xml      adminavailable       public  x86_64  kernel
IMAGE   emi-E2861098    image-store-1268999725/image.manifest.xml       adminavailable       public  x86_64  machine

Testing

Before running an instance of this image, you should first create a keypair that can be used to log into the instance. Create a SSH keypair using the command:

if [ ! -e ~/.euca/mykey.priv ]; then
    touch ~/.euca/mykey.priv
    chmod 0600 ~/.euca/mykey.priv
    euca-add-keypair mykey > ~/.euca/mykey.priv
fi

The default security group should be configured to allow SSH access (port 22). If you plan to use a Windows VM allow access to RDP as well.

$ euca-describe-groups
$ euca-authorize default -P tcp -p 22 -s 0.0.0.0/0
$ euca-authorize default -P tcp -p 3389 -s 0.0.0.0/0
$ euca-describe-groups
GROUP   admin   default default group
PERMISSION      admin   default ALLOWS  tcp     3389    3389    FROM    CIDR0.0.0.0/0
PERMISSION      admin   default ALLOWS  tcp     22      22      FROM    CIDR0.0.0.0/0

From euca-describe-images, get the EMI ID.

$ euca-describe-images
IMAGE   eri-0D4D117E    image-store-1268999725/ramdisk.manifest.xml     adminavailable       public  x86_64  ramdisk
IMAGE   eki-F8BF1114    image-store-1268999725/kernel.manifest.xml      adminavailable       public  x86_64  kernel
IMAGE   emi-E2861098    image-store-1268999725/image.manifest.xml       adminavailable       public  x86_64  machine

Now launch the instance.

$ euca-run-instances emi-E2861098 -k mykey

Sample eucalyptus.conf

EUCALYPTUS=”/”
EUCA_USER=”eucalyptus”
DISABLE_DNS=”Y”
ENABLE_WS_SECURITY=”Y”
LOGLEVEL=”DEBUG”
CC_PORT=”8774″
SCHEDPOLICY=”ROUNDROBIN”
POWER_IDLETHRESH=”300″
POWER_WAKETHRESH=”300″
NODES=” cc.uec.shaz”
NC_SERVICE=”axis2/services/EucalyptusNC”
NC_PORT=”8775″
HYPERVISOR=”kvm”
INSTANCE_PATH=”/var/lib/eucalyptus/instances”
VNET_PUBINTERFACE=”br0″
VNET_PRIVINTERFACE=”br0″
VNET_BRIDGE=”br0″
VNET_DHCPDAEMON=”/usr/sbin/dhcpd3″
VNET_DHCPUSER=”dhcpd”
VNET_MODE=”SYSTEM”

# network configuration from the input configuration file
#VNET_MODE=”MANAGED-NOVLAN”
#VNET_SUBNET=”172.19.0.0″
#VNET_NETMASK=”255.255.0.0″
#VNET_DNS=”10.10.6.3″
#VNET_ADDRSPERNET=”32″
#VNET_PUBLICIPS=”A.B.C.xxx-A.B.C.yyy”

You should be able to get both SYSTEM and MANAGED_NOVLAN modes. Remember to do a cleanstop and cleanstart of eucalyptus services between changing network modes.

$ sudo service eucalyptus stop CLEAN=1
$ sudo service eucalyptus-nc stop CLEAN=1
$ sudo service eucalyptus start CLEAN=1
$ sudo service eucalyptus-nc start CLEAN=1

This tutorial should get you started for a basic configuration without understanding much at this early stage. The objective here is to get you started for further experimentation and learning. With my two more planned tutorials you will be able to understand the architecture more. Until next tutorial you are expected to practice the web console and play around with configurations and images.

For more details please visit Ubuntu UEC Community and Beginners Guide to UEC.

Introduction to Private Cloud Computing with Eucalyptus UEC

In this article I will introduce the reader to private cloud computing, while discuss the nature of a private cloud with UEC based Eucalyptus as a case study to maintain order.

Private Cloud

A private cloud is a cloud computing infrastructure implemented as a service provider rather using a public cloud as a consumer of the service.

The main thing that make private clouds apart from a commercial public cloud is where the infrastructure implemented and how it’s deployed and maintained. A private cloud is typically hosted on the company servers within their own network infrastructure and administrative domain.

The advantage with a private cloud is direct control over every aspect of the cloud’s design and implementation and most importantly the way security is implemented.

Ubuntu Enterprise Cloud

Ubuntu Enterprise Cloud (UEC) is a package stack of applications from Canonical (best with an Ubuntu server). UEC includes a number of open source tools to manage the infrastructure, apart from Eucalyptus. UEC makes cloud computing a set of friendly activities.

By default UEC uses KVM. You would want to consider Xen based setup if you do not have any VT extensions. It would be better to go for a pure Eucalyptus server configuration but still Xen based kernels are difficult to find these days! You might survive Xen based setup on Fedora12.

Eucalyptus

“Elastic Utility Computing Architecture for Linking Your Programs To Useful Systems” is a GPL Licensed software, which provides tooling to create and manage a private cloud that can even be accessed as public cloud. It provides Amazon EC-2 compatible platform and S3 compatible storage platform. It  makes its services available through EC2/S3 compatible APIs. Therefore the client tools for Amazon Web Services (AWS) can be used with it as well.

Features:

• Interface compatibility with EC2
• Simple installation and deployment using Rocks
• Simple set of extensible cloud allocation policies
• Overlay functionality requiring no modification to the target Linux environment
• Basic administrative tools for system management and user accounting
• Configure multiple clusters with private network addresses into a single Cloud
• Portability

Components of Eucalyptus

Eucalyptus has 3 major components:

• Cloud Controller (CLC): includes the front-end services and the Walrus storage system.
• Cluster Controller (CC):  provides support for the virtual network overlay
• Node Controller (NC): that interacts with VT to manage individual VMs.

And for storage management the following two components are utilized:

• Storage Controller (SC): provide persistent block storage for the instances
• Walrus Storage Controller (WSC): provide persistent and simple storage service

Note: Packages provided by Ubuntu are: eucalyptus-cloud, eucalyptus-cc and eucalyptus-nc, eucalyptus-sc, eucalyptus-walrus.

Component Model: http://open.eucalyptus.com/book/export/html/4263

Node Controller (NC)

NC is a VT enabled server capable of running KVM as the hypervisor. UEC automatically installs KVM when the user chooses to install the UEC node. The VMs running on the hypervisor that are controlled by UEC are called instances.

The NC runs on each node and controls the life cycle of instances running on the node. The NC interacts with the OS and the hypervisor running on the node, while on the other hand it also interacts with the Cluster Controller (CC).

NC queries the OS on the node to discover the node’s resources. These are the number of cores, memory, disk space. It also monitors the state of the VM instances running on the node to propagate this data to the CC.

Features:

• Collection of data related to the resource availability and utilization on the node and reporting the data to CC
• Instance life cycle management

Cluster Controller (CC)

CC manages one or more Node Controllers to deploys and manages instances on them. CC also manages the networking for the instances running on the Nodes under certain types of networking modes of Eucalyptus.We will have to delay such advance topics for a while!

CC communicates with CLC and NCs as a managed middle tier.

Features:

• Receive CLC requests to deploy instances
• Select NCs for deploying the instances on
• Control virtual network availability to the instances
• Audit the registered NCs and report accordingly to CLC

Walrus Storage Controller (WS3)

It provides a persistent and simple storage service. WS3 uses REST and SOAP APIs, which are compatible with S3 API.

Features:

• Store machine images
• Store snapshots
• Store and serve files using S3 API

It should be considered as a simple file storage system.

Storage Controller (SC)

It provides persistent block storage for the instances. SC is similar to the Elastic Block Storage service from Amazon Web Services.

Features:

• Creates and manages persistent block storage devices
• Creates snapshots of volumes

Cloud Controller (CLC)

CLC is the front end to the entire cloud infrastructure. It provides EC2/S3 compliant web services interface to the client tools. It also interacts with the rest of the components of the Eucalyptus infrastructure. CLC also provides a web interface for users to manage delegated aspects of the UEC infrastructure.

Features:

• Monitor resources of the cloud infrastructure. This includes hypervisor nodes and the cluster controllers that manage the hypervisor nodes.
• Resource arbitration
• Monitor running instances

CLC has a comprehensive knowledge of the state of the cloud with respect to availability and usage of resources primarily.

—————————————————————————————————————

For more details visit http://open.eucalyptus.com/book/export/html/4263

In the next article we will cover installation and configuration.

Introduction to Amazon Web Services by Recluze and Toqeer, CSRDU

Getting Started with Amazon EC2 – Install, Configure, Connect.

Auto-scale with Amazon Elastic Compute Cloud Ec2.

Introduction to Amazon Web Services.