How it works

You need to install “email-ext” plugin https://wiki.jenkins-ci.org/display/JENKINS/Email-ext+plugin for your jenkins first, and follow the guideline to configure the basic setup first.

Now this plugin sopport html template using jelly/js/groovy, jelly looks complex to me, so I decided to choose groovy (http://groovy.codehaus.org/), which is an agile and dynamic language for the JVM.

In order to use groovy customized html template, we need do extra

Create template directory

The template directory doesn’t exist, need to be created first, you can copy the template file from github directly

$ mkdir $JENKINS_HOME\email-templates # put the template inside

Enable groovy template inside project

Then in the jenkins, configure the post action in the job like below

Bugs in groovy

Unfortunately there are some bugs in the default template, and I provided one fix (issue #13191) there. Others needs more time to fix. Don’t worry, since we will redesign the html anyway, don’t need so care about the old one.

Anyway, you should receive html report in ugly format like below

Design my own html

Inspired by travis-ci’s report, I want to better html report from jenkins as well, below is my sketech for this.

• news: is used in header for general remindings
• status/main/detail: are all build information from job
• support: extra line for support information, like email, faq, etc ..

With the help from my friend @haojii, we got final html/css like below

I don’t want to go detail on the groovy, you just download the artifact: groovy-html-larry.template from my cloned email-ext repo@github and give a try.

It works for git, and skip some information in groovy standard template since it covers more cases, anyway it works for me and it is good enough.

DIY

You can tune with html file groovy-html-larry.html on your css and structure and apply to groovy template when it works

Summary

Better email report will help team to find their information quickly, and make it fun.

This is the first learning from travis-ci.org, and there are more ideas will be copied to jenkins.

Leave comments here if it works for you.

In short http://www.travis-ci.org gives me the new thinking of CI, and I think it is the evoluation as well, originally I think CloudBee (jenkins behind it) http://www.cloudbees.com/ is great to put development into cloud. But that is just next normal step, not evolution (at least in CI part).

Let’s see how traditional CI works

Normal CI server like Jenkins

Everytime we talk about CI, we will have 2 steps in mind

- prepare the CI server with needed 3pp installed.
- CMer will create the job for related tasks towards code repository

Looks good and smoothly, and we are used to it.

It had some limitations

Multi platform or package dependance

It will be quite trickly if you want your software to be built and verified in different OS, mostly we need prepare the same machines, if you are good at jenkins, then master/slave mode is used.

Then go further for multi 3pp and different dependance, then this make environment preparation is very hard, since the matrix will be large.

You may think about cross compiler, one solution for building.

Surely as I demoed before, you can use vagrant/virtualbox inside jenkins server, yes, it is good, just not elegant

Jobs are managed by CMer

It is named jobs to each build task, and since it is central, mostly only CMer had permission to create jobs and let CI server run for you, it covers mainly on master/main branch. Since server is limited, running CI for your private branch is expensive on adminstrative.

And who understand best on building and testing environment ? It should be developers, so why not let developers to configure what they want and build for it ?

Awesome Travis-CI

So what it is really needed for good CI ?

As a developer, I can write down which os/environment that build is needed, just like right, I need several extra packages and two ruby platform on Ubuntu OS. Also I know how to execute the build.

That’s good enough for a build, so CI will do the rest all, this is what http://travis-ci.org ‘s job

- After I put this configuration besides source codes and push to github
- travis-ci load virtual machine from pool via vagrant/virtualbox, currently it supports Ubuntu 11.04 only
- travis-ci apply your configuration into virtual machine using chef
- travis-ci run you job, and give report, see example: http://travis-ci.org/#!/larrycai/sdcamp
- travis-ci distroy the vm and put back into pool

That’s all the CI should do and this is exactly I want !

What’s the benefit

No CMer is involved, not only time is saved, but also I don’t need to spend time to communicate with them on which kind of environment (package), because it is my business, not their focus.

I can trigger build as much as possible with full control, and since the build normally last shorted, other developers will reuse the machine after my time slot, Even they use total different platform.

Summary

I don’t want to say jenkins is bad, actually it is quite powerful and flexibility. I just want to indicate travis-ci understand better for the future of the build

Go there, try it and even fork it to run by yourself.

I love opensource, innovation comes from competition! Well done both to jenkins and travis-ci.

And I want to name this is “CI on deman” or “Private CI”.

Let’s continue.

Background

Now let’s take a look at in normal product development, what’s work flow for the requirement towards customer document and system document.

Requirement is put into focalpoint (or others) and bring to R&D, scrum is mostly used, product backlog and Sprint backlog will be generated in excel or agile tools (including redmine, teamforge, jira, whiteboard), it covers from high level requirement to user story level.

Testers start to analysis the requirement and generate test specification level document in word, it will be reviewed by senior group and put stamp – go!. test case description is more detail on use case testing level, mostly they will be converted into real test case script for automation inside CI.

When the project goes to end, test report is requested to be finalized and stored somewhere.

CPI technical writer will collect all the information and write in tagtool for CPI (product customer document)

System managers will collect information to update system document like System Architecture Description.

Observation

Every document exists for some reason, while from agile/lean point of view.

Do you see any waste here ? waste on the documentation and waste on time to write the document ?

Do you still understand the purpose of those documentation ?

And please don’t think from your roles (testers, managers, technical writters), stand from of your product.

From my point of view, the designer area (coding) is much improved compare to before, most of APIs are genrated into documentation automatically, quality is promised via tools like CI, sonar.

Specification by example

I don’t want to repeat the famous free delivery for bookstore example from “Specification by example” on how it could work with less waste, please read it sample chapter 2 from manning directly.

Possible solution for us

I don’t state it can 100% work for us, just give idea how it can work

We can combine all the test documents to cucumber (one nice tool) feature files like below

below the Feature title, we can add detail background for the requirement, it can surve as test spec.

All the scenarios are exactly the same as test case description on detail enough for testing.

This specification (test case) could be directly executed by cucumber driver.

When the test case is executed, it could be automatically generated into html file, it could be test report

And if we had some scripts to convert .feature to latex/docbook, it can be automatically generated high-quality documents like pdf, since we can predefine some structure, which can be mapped to end result.

See good example from Pro Git book.

From left to right: markdown plaintext, simple html, ebook pdf format. (it is coverted by pandoc to latex and texlive to pdf)

Furthermore, .epub (iPad), .mobi (Kindle) are supported as well, and everything is automatically.

And this could happen for us as well.

Summary & Go ahead

Probably you want to challenge again that it doesn’t work for you because blah, blah, then I recommend to buy the book “Specification by example” from Gojko Adzic

I will continue to look into this area and I will try my best to remove useless project document using “Specification by example” and Cucumber.

The document “shared folders” had some traps, and I can’t make it work, below are list some important steps to get it work

Envrionment

Ubuntu 11.04 (Natty) as guest OS and Virtualbox as virtual machine management tools

Install Guest Additions

Virtualbox Guest Additions should be installed for shared folders, it is a .iso files to be mounted, when it is mounted in guest OS (Ubuntu), it can be executed

Install linux kernel

I met the following error when I try to install the guest additional

[guest] # cat /var/log/vboxadd-install.log
Makefile:23: *** Error: unable to find the sources of your current Linux kernel. Specify KERN_DIR=<directory> and run Make again..  Stop.

After googled, I found the good article http://blog.oracle48.nl/adding-shared-folders-to-virtual-box-guest-using-the-command-line/, later I noticed it existed inside chapter 4 as well.

Therefore I need to install dkms package and related headers (it depends on which kernel you use)

apt-get install dkms
apt-get install linux-headers-2.6.38-11-generic

Shared folder from host to guest

it can be easily configured in virtual machines’ setting, if you set it automatically, it will exist when reboot

If it goes well, you will see it /media/sf_sdcamp, which works if you run in root id, and as documented, you can also mount it for normal user.

mount –t vboxsf sdcamp /mnt/sdcamp

/mnt/sdcamp should be created in advance

Anyway, now it works very well, I edited in Windows Environment, and run some specical commands under linux environment

Probably you are playing with agile for a while and notice some problems for testing and requirement handling, then I want to recommend “Specification by Example”

Let’s start with problem

Problem in requirement handling and testing

Testers are used to follow and verify the implementation

In traditional software development, it is pure waterfall, system level define the requirement, and testers write test specification on how to test it, designers are implementing it. After the coding is finished, the live system is verified by testers according to the test spec.

When we turns to agile way of working, mostly this model will be kept without change, some teams even use small waterfall inside the team.

Lots of technical debts come out in end of each iteration because there are gaps between testers’ understanding and designers’ undestanding, then people are used to complain on the requirement.

In the same time, tester’s motivation is used to be low, they just simple verify what has been done. Some senior testers used lots of method to make sure all the bugs can be found, which is great.

BUT do we testing the right thing ? !!!

No practical way to clarify the requirement

Some sites may run agile better, they invited PO frequently to clarify the requirement, and had PBR “Product Backlog Refineworkshop” meeting (see Craig & Bas’s book “Practices for Scaling Lean & Agile Development”). While this is just the framework, people mostly had problem how to make it effiency, and how to get the requirement and make it clearly.

ATDD doesn’t work

When quick scan the “Specification by example”, most people will treat it is the same as ATDD and BDD, while in the book and his workshop, he clearly stated,” yes, they are quite similar, and there are more mind-set changes for SBE”.

The importance of SBE is to the practical way to clarify the specification by examples, here are the key process patterns, I used to use this pursuide my colleagues

Some tips to deploy Specification by Example

it cannot be described easier in one blog, let give some tips that I used if you want to try

Start without tool

I made this mistake before as well, since mostly we will use tool to support your automation (this is good, but tough task), therefore I didn’t make ATDD works in my product.

It is quite easy to focus on automation instead of clarifying the requirement when you deploy it, so start without tool.

In my pilot, I just ask the testers (they are trainined for this concept for sure) to write specification with what SBE suggested, put the business value inside instead steps.

Understand the tool, but it is not necessary

People are used to lost why do this in this kind format, so you need to understand some tools to see how it can be automated to execute the specification and automate it easily.

There are lots of nice tool, like cucumber (http://cukes.info/), which is most popular and useful tool, and I push team members to write in its format.

Fitnesse and RobotFramework are good, but I don’t recommend, since they are built long time ago, it can support SBE, but they are not generated by SBE concept, while cucumber do comes from that.

Can do it as well for your products

People are used to say, “yeah, it works for web application, but our product is different”, it is not true. The way to clarify the requirement is the same, surely it may be more difficult and not so straight, but it do works.

You can read his book as well for some patterns.

Start with small and you can get benefit directly

Summary

I list some benefits so far for my consultant service on this area after 1-2 months (surely other practices are applied as well)

1. Testers get more motivation than before, since they drive the requirement clarification and notice the importance the role inside the team, they drive what to implement
2. Requirements are easier to be clear compare to before, therefore the technical debt from this is gone, less waste.

Therefore, I do suggest

• Buy this book or at least read some free excellent sample chapters from manning to understand it, if possible, join his workshop
• Try it and it can work from day one, trust me.

You want to create by yourself for some framework, please wait for a minute, there already had one project veewee, and I don’t want to repeat the nice blog http://www.ducea.com/2011/08/15/building-vagrant-boxes-with-veewee/ as well.

Below just some tips when I tried this morning

veewee tips

veewee directory structure

I prefer to create one seperate veewee directory for this tasks.
iso files should be download in advance, and make a softlink under created directory iso.
then you can define your own basebox like below

~/veewee$ vagrant basebox define 'ubuntu1104' 'ubuntu-11.04-server-i386'

the directory will looks like below, you type command under veewee directory

~/veewee$ find
.
./definitions
./definitions/myubuntubox
./definitions/myubuntubox/postinstall.sh
./definitions/myubuntubox/preseed.cfg
./definitions/myubuntubox/definition.rb
./definitions/ubuntu1104
./definitions/ubuntu1104/postinstall.sh
./definitions/ubuntu1104/preseed.cfg
./definitions/ubuntu1104/definition.rb
./iso
./iso/ubuntu-11.04-server-i386.iso

virtualbox headless

suppose it should work headless (without GUI), while when I create basebox, it pops error on keyboard permisson, and when I enable my X11 display (since I tried it in my local linux server), it pops up virtualbox GUI and start to install.

it is excellent feeling when you see it is under auto-installtion.

configure apt proxy

since I try this in the company, I need to setup the proxy, it is fine to add the line export http_proxy=http://proxy:8080 in the beginning of postinstall.cfg, and it works fine.

look inside of veewee

the architecture of veewee looks great, it adds subcommand in vagrant, and all the majoy information are put into definition.rb

Veewee::Session.declare({
  :cpu_count => '1', :memory_size=> '384',
  :disk_size => '10140', :disk_format => 'VDI', :hostiocache => 'off',
   s_type_id => 'Ubuntu',
  :iso_file => "ubuntu-11.04-server-i386.iso",
  :iso_src => "http://releases.ubuntu.com/11.04/ubuntu-11.04-server-i386.iso",
  :iso_md5 => "ce1cee108de737d7492e37069eed538e",
  :iso_download_timeout => "1000",
  :boot_wait => "10", :boot_cmd_sequence => [
    '',
    '/install/vmlinuz noapic preseed/url=http://%IP%:%PORT%/preseed.cfg ',
    'debian-installer=en_US auto locale=en_US kbd-chooser/method=us ',
    'hostname=%NAME% ',
    'fb=false debconf/frontend=noninteractive ',
    'keyboard-configuration/layout=USA keyboard-configuration/variant=USA console-setup/ask_detect=false ',
    'initrd=/install/initrd.gz -- '
  ],
  :kickstart_port => "7122", :kickstart_timeout => "10000", :kickstart_file => "preseed.cfg",
  :ssh_login_timeout => "10000", :ssh_user => "vagrant", :ssh_password => "vagrant", :ssh_key => "",
  :ssh_host_port => "7222", :ssh_guest_port => "22",
  :sudo_cmd => "echo '%p'|sudo -S sh '%f'",
  :shutdown_cmd => "shutdown -P now",
  :postinstall_files => [ "postinstall.sh"], :postinstall_timeout => "10000"
})

it will be used during boot time to trigger the auto-installation, different os will use own auto-installation method, preseed.cfg is used for ubuntu/debian, if for suse then kickstart is used. postinstall.sh is used when the os is installed and rebooted.

summary

with those tool stacks (virtualbox,vagrant,puppet, veewee), your work makes more easily, don’t reinvent the wheel, don’t DIY. Use others brain.

Also I noticed the question @ stackoverflow http://stackoverflow.com/questions/6941547/how-to-combine-vagrant-with-jenkins-for-the-perfect-continuous-integration-envir/7830173#7830173, and my suggestion is below

Separate virtual machines environment

Why

Mostly your jenkins server is used for lots of tasks, therefore it is run mostly by restricted user, for example it is tomcat6 if running as servlet, or normal jenkins user, and mostly they don’t have shell access to avoid problem.

Vagrant may needs lots of 3pps which may different with normal jenkins user.

Therefore it is better to separate the virtual machine environment

Jenkins master/slave mode

This is the nice solution to solve this problem to isolate the environment, just new another node in your jenkins server (which is master) as below

It is communicated by ssh, and it will invoke the slave.jar automatically. And here I created the vagrant user for this, why ?

Reuse the ssh public/private key

If you played with vagrant, you know vagrant can access virtual machine via ssh without password, it because it uses the ssh public/private key.

Therefore I created the vagrant user and put the private/public key into ~vagrant/.ssh directory, mostly they are under /var/lib/gems/1.8/gems/vagrant-0.8.7/keys/, which is used by vagrant, if you don’t know where it is, please type vagrant ssh_config

Prepare the base box under vagrant user

Follow the normal vagrant guideline, and create one vm box, for example, I put it under ~/vm/ubuntu

New the CI job to control VM

Now it is time to configure a job to experience this, in jenkins master node, create a job in freestyle

Label Expression is the category we defined in slave node.

In the task (Build – Execute shell), jenkins slave try to go to that directory and start vagrant, then doing some simple tasks to evaulate whether it is done in our vm box.

!! REMEBER, this is done in slave mode using vagrant user !!

Now you can see the job is invoked in slave vagrant node

In the console output, you can notice the result

Summary

Using Vagrant in jenkins master/slave mode is perfect match for this job, you can get lots of flexibility using virtualmachine.
veewee is another layer above vagrant to control vm box more easily.

And yes, vagrant (http://vagrantup.com/) is the toolkit for this purpose.

Introduction

From webpage, it is mentioned “Vagrant is a tool for building and distributing virtualized development environments” and it use virtualbox API to control the virtual machines, and I think the generated vm should be easily converted into other format, I quickly load the raw disk into vmware, it works as well (need check more)

The guideline from web is already clear enough, it needs ruby and virtualbox 4.1.*, and I download the box (I use http://www.vagrantbox.es/26/) from the webpage in advance

$ gem install vagrant
$ vagrant box add ubuntu-1104 talifun-ubuntu-11.04-server-i386.box$ vagrant init
$ vagrant up$ vagrant ssh

and now I can visit the virtual machines !!!, so easy so nice.

The virtualmachines images are put under ~/.vagrant.d/boxes, the ovf is not recognized by vmware, but the raw disk vmdk is almost ok for virtual machines.

Work with Puppet

Besides the vm management, another most interesting part is it use puppet to do provisioning the system, everything is set in puppet configuration files. (check my previous blog)

Then when the vm loaded, it will apply the setting from your puppet configuration

Take a look at the nice slides for how Vagrant & Puppet works together

Use cases

I think this blog indicates exactly what I dreamed for longer time, How we use Vagrant as a throwaway testing environment

1. load existing base VM (vagrant)
2. install product packages with wanted configuration (puppet)
3. testing your product using your test case (like cucumber)
4. finish the VM and update the relate webpage for the results.

In your CI environment, it can automatically control lots of VM with lots of testing.

Summary

Vagrant is a very new toolkit and it is quicky got lots of attention in this area, take a look and use it in your testing lab, don’t reinvent your own wheel.

I think there should exist similar toolkits like vagrant for Amazon EC2, Xen with puppet.

And I will start experience real product soon.

Start without client/server mode

It is much easy to use puppet as installation tool in single node as well, “puppet apply” mode can be used to practice your .pp files locally, as example above

puppet --modulepath /srv/puppet-tomcat-demo/modules -v /srv/puppet-tomcat-demo/site.pp

it will automatically download the tomcat6 packages even openjdk packages for you.

The default port is 735 (<1024), which seems have permission problem at least for me, I simple changed it to 3735, then it works very well for me.

And there are two tips, more check http://www.devco.net/archives/2009/08/19/tips_and_tricks_for_puppet_debugging.php

1. --debug is good option for more detail debug information under command line
2. --parseonly is good to do syntax check

Client/server mode

ssl is still tricky

the ssl authentication is still trick, my two working virtual machines doesn’t work again, I delete all “*.pem”, it still doesn’t work, “*.yaml” is deleted as well, and put full domain name in puppet.conf, and it works. Actually I still don’t know what’s the clue, anyway it works again after 1 hour (!!!).

Put it as module

Move the tomcat as module under /etc/puppet, and put site.pp information to node.pp, the path is hard-coded first

path => '/srv/puppet-tomcat-demo/java_src/SimpleServlet.war'

It should exist in agent node first to make testing easy, when it works, we start to deal with puppet source

Puppet source

The resource should exist in server side or nfs server, let’s use server first

path => ‘puppet:///modules/tomcat/war/SimpleServlet.war’

it indicates to use puppet protocol (only support protocol so far), and actually it maps to /etc/puppet/modules/tomcat/files/war/SimpleServlet.war, and read the puppet wiki http://docs.puppetlabs.com/guides/file_serving.html is more clear for this.

Then it is quite clean in client side, well done.

Next step

Just think about how to switch to Sun (oracle) JRE, which is more stable for system and also start to think how to deal with own package system since currently the package comes from public repo.

just a record for my learning on puppet for auto-deployment.

Introduction

Probably you can start with the slides http://www.slideshare.net/teyotyree/puppet-talk-at-oscon-2010, in short, Puppet is an open source framework and toolset for managing the configuration of computer systems, you may care about the resource like ssh instead of real package, it is a nice weapon for sysadmin and our auto deployment.

Puppet itself is written by ruby, but mostly you just need to deal with configuration (DSL)

It may have other versions now, and I choose open source which is available for ubuntu.

Getting started

I prepare two virtual machines, which is created from ubuntu 11.04, one is puppet master (puppet, 192.168.56.100), another one acts as puppet agent (puppet1, 192.168.56.101).

> apt-get install puppet # agent/client
> apt-get install puppermaster # master

Then you can follow http://docs.puppetlabs.com/learning/manifests.html to write one simple puppet resource files like 1.file.pp to execute in agent machine to experience it.

So nice it will automatically regenerate it when the resource is not correct

Connect from agent to master

Next step is try to put the resource in master side, and auto install something from agent, I follow https://help.ubuntu.com/11.04/serverguide/C/puppet.html to define apache2 package in master side.

Then in agent side to run command to verify it, wooh, the ssh connection is so weird, I spend 1 day to figure it out, see lesson learned.

> pupper agent –t

it will call the default master server (puppet) to fetch resource information, and notice it needs apache2, it will call internal command (guess apt-get) to install apache2 for me.

After I remove the apache2 package and run above command again, it is back as expected !!

> apt-get autoremove apache2

Module installation

Mostly besides package, you need deal with other issues, therefore in puppet, we need to bind everything into one module, Pro Puppet book had a very good example on how to customize apache configuration after installation, please go there and download the source codes (chapter02\apache)

user@puppet:/etc/puppet$ find modules/apache/
modules/apache/manifests/init.pp
modules/apache/manifests/install.pp
modules/apache/manifests/service.pp
modules/apache/manifests/vhost.pp
modules/apache/templates/vhost.conf.erb

user@puppet:/etc/puppet$ cat manifests/site.pp
import 'nodes.pp'

When you put all these files under master node, and invoke it from agent node, it will install apache2 and customize the /etc/apache2/sites-enabled/10-puppet1 configuration automatically.

Lesson learned

When it works, puppet is very good, but the log system and his structure may be complex to the beginner, so be aware of this and learn it step by step, if the environment is not good, it doesn’t work, and it may not give you correct direction why it doesn’t work.

ssl authentication is quite trick, be careful

one headache is the error message is not so clear and ssl itself is always troublesome especially if you are not expert on this area, I think my problem mainly is changing the hostname after the CA cerfication (ca.pem) is generated.

You may need check http://www.madboa.com/geek/openssl/ to figure out what is the problem

Be careful with the version

make sure you use the version 2.6.*+, origianlly one of my ubuntu server is 10.x based, which use puppet 2.5.x, lots of commands are different, so please make sure the version

Summary

Puppet is a wonderful tool, don’t reinvent the wheel to create your own scripts to deploy the system from the scratch, and this is just the start, I will dig into more to see how to use it connected within product deployment inside CI