Archive for the ‘tool’ Category

CI on demand – learn from travis-ci

Thursday, February 23rd, 2012

imageDidn’t blog for long time, mainly due to I am learning to many new things ;-) , let’s start the CI part.

In short http://www.travis-ci.org gives me the new thinking of CI, and I think it is the evoluation as well, originally I think CloudBee (jenkins behind it) http://www.cloudbees.com/ is great to put development into cloud. But that is just next normal step, not evolution (at least in CI part).

Let’s see how traditional CI works

Normal CI server like Jenkins

Everytime we talk about CI, we will have 2 steps in mind

- prepare the CI server with needed 3pp installed.
- CMer will create the job for related tasks towards code repository

Looks good and smoothly, and we are used to it.

It had some limitations

Multi platform or package dependance

It will be quite trickly if you want your software to be built and verified in different OS, mostly we need prepare the same machines, if you are good at jenkins, then master/slave mode is used.

Then go further for multi 3pp and different dependance, then this make environment preparation is very hard, since the matrix will be large.

You may think about cross compiler, one solution for building.

Surely as I demoed before, you can use vagrant/virtualbox inside jenkins server, yes, it is good, just not elegant

Jobs are managed by CMer

It is named jobs to each build task, and since it is central, mostly only CMer had permission to create jobs and let CI server run for you, it covers mainly on master/main branch. Since server is limited, running CI for your private branch is expensive on adminstrative.

And who understand best on building and testing environment ? It should be developers, so why not let developers to configure what they want and build for it ?

Awesome Travis-CI

imageSo what it is really needed for good CI ?

As a developer, I can write down which os/environment that build is needed, just like right, I need several extra packages and two ruby platform on Ubuntu OS. Also I know how to execute the build.

That’s good enough for a build, so CI will do the rest all, this is what http://travis-ci.org ‘s job

- After I put this configuration besides source codes and push to github
- travis-ci load virtual machine from pool via vagrant/virtualbox, currently it supports Ubuntu 11.04 only
- travis-ci apply your configuration into virtual machine using chef
- travis-ci run you job, and give report, see example: http://travis-ci.org/#!/larrycai/sdcamp
- travis-ci distroy the vm and put back into pool

That’s all the CI should do and this is exactly I want !

What’s the benefit

No CMer is involved, not only time is saved, but also I don’t need to spend time to communicate with them on which kind of environment (package), because it is my business, not their focus.

I can trigger build as much as possible with full control, and since the build normally last shorted, other developers will reuse the machine after my time slot, Even they use total different platform.

Summary

I don’t want to say jenkins is bad, actually it is quite powerful and flexibility. I just want to indicate travis-ci understand better for the future of the build

Go there, try it and even fork it to run by yourself.

I love opensource, innovation comes from competition! Well done both to jenkins and travis-ci.

And I want to name this is “CI on deman” or “Private CI”.

From requirement to live document ? can we remove waste ?

Friday, January 6th, 2012

I had some discussion in previous blog Specification by Example – The way to enhance testing & Requirement for this.

Let’s continue.

Background

Now let’s take a look at in normal product development, what’s work flow for the requirement towards customer document and system document.

image

Requirement is put into focalpoint (or others) and bring to R&D, scrum is mostly used, product backlog and Sprint backlog will be generated in excel or agile tools (including redmine, teamforge, jira, whiteboard), it covers from high level requirement to user story level.

Testers start to analysis the requirement and generate test specification level document in word, it will be reviewed by senior group and put stamp – go!. test case description is more detail on use case testing level, mostly they will be converted into real test case script for automation inside CI.

When the project goes to end, test report is requested to be finalized and stored somewhere.

CPI technical writer will collect all the information and write in tagtool for CPI (product customer document)

System managers will collect information to update system document like System Architecture Description.

Observation

Every document exists for some reason, while from agile/lean point of view.

Do you see any waste here ? waste on the documentation and waste on time to write the document ?

Do you still understand the purpose of those documentation ?

And please don’t think from your roles (testers, managers, technical writters), stand from of your product.

From my point of view, the designer area (coding) is much improved compare to before, most of APIs are genrated into documentation automatically, quality is promised via tools like CI, sonar.

Specification by example

I don’t want to repeat the famous free delivery for bookstore example from “Specification by example” on how it could work with less waste, please read it sample chapter 2 from manning directly.

image

Possible solution for us

I don’t state it can 100% work for us, just give idea how it can work

We can combine all the test documents to cucumber (one nice tool) feature files like below

image

below the Feature title, we can add detail background for the requirement, it can surve as test spec.

All the scenarios are exactly the same as test case description on detail enough for testing.

This specification (test case) could be directly executed by cucumber driver.

When the test case is executed, it could be automatically generated into html file, it could be test report

image

And if we had some scripts to convert .feature to latex/docbook, it can be automatically generated high-quality documents like pdf, since we can predefine some structure, which can be mapped to end result.

See good example from Pro Git book.

image image image 

From left to right: markdown plaintext, simple html, ebook pdf format. (it is coverted by pandoc to latex and texlive to pdf)

Furthermore, .epub (iPad), .mobi (Kindle) are supported as well, and everything is automatically.

And this could happen for us as well.

Summary & Go ahead

image Probably you want to challenge again that it doesn’t work for you because blah, blah, then I recommend to buy the book “Specification by example” from Gojko Adzic

I will continue to look into this area and I will try my best to remove useless project document using “Specification by example” and Cucumber.

Share folders with Virtual machine using Virtualbox

Wednesday, January 4th, 2012

Since we use Windows Vista as default, we are used to load virtual machines to try linux stuff, and it will be nice if we can share folders between host (windows) and guest (linux) machines.

The document “shared folders” had some traps, and I can’t make it work, below are list some important steps to get it work

Envrionment

Ubuntu 11.04 (Natty) as guest OS and Virtualbox as virtual machine management tools

Install Guest Additions

Virtualbox Guest Additions should be installed for shared folders, it is a .iso files to be mounted, when it is mounted in guest OS (Ubuntu), it can be executed

image

Install linux kernel

I met the following error when I try to install the guest additional

[guest] # cat /var/log/vboxadd-install.log
Makefile:23: *** Error: unable to find the sources of your current Linux kernel. Specify KERN_DIR=<directory> and run Make again..  Stop.

After googled, I found the good article http://blog.oracle48.nl/adding-shared-folders-to-virtual-box-guest-using-the-command-line/, later I noticed it existed inside chapter 4 as well.

Therefore I need to install dkms package and related headers (it depends on which kernel you use)

apt-get install dkms
apt-get install linux-headers-2.6.38-11-generic
Shared folder from host to guest

it can be easily configured in virtual machines’ setting, if you set it automatically, it will exist when reboot

image

If it goes well, you will see it /media/sf_sdcamp, which works if you run in root id, and as documented, you can also mount it for normal user.

mount –t vboxsf sdcamp /mnt/sdcamp

/mnt/sdcamp should be created in advance

Anyway, now it works very well, I edited in Windows Environment, and run some specical commands under linux environment

create virtualbox on fly using veewee

Tuesday, October 25th, 2011

one headache of using vagrant is to have base box, and creating basebox from scratch is headache, and how to solve it ?

You want to create by yourself for some framework, please wait for a minute, there already had one project veewee, and I don’t want to repeat the nice blog http://www.ducea.com/2011/08/15/building-vagrant-boxes-with-veewee/ as well.

Below just some tips when I tried this morning

veewee tips

veewee directory structure

I prefer to create one seperate veewee directory for this tasks.
iso files should be download in advance, and make a softlink under created directory iso.
then you can define your own basebox like below

~/veewee$ vagrant basebox define 'ubuntu1104' 'ubuntu-11.04-server-i386'

the directory will looks like below, you type command under veewee directory

~/veewee$ find
.
./definitions
./definitions/myubuntubox
./definitions/myubuntubox/postinstall.sh
./definitions/myubuntubox/preseed.cfg
./definitions/myubuntubox/definition.rb
./definitions/ubuntu1104
./definitions/ubuntu1104/postinstall.sh
./definitions/ubuntu1104/preseed.cfg
./definitions/ubuntu1104/definition.rb
./iso
./iso/ubuntu-11.04-server-i386.iso
virtualbox headless

suppose it should work headless (without GUI), while when I create basebox, it pops error on keyboard permisson, and when I enable my X11 display (since I tried it in my local linux server), it pops up virtualbox GUI and start to install.

it is excellent feeling when you see it is under auto-installtion.

configure apt proxy

since I try this in the company, I need to setup the proxy, it is fine to add the line export http_proxy=http://proxy:8080 in the beginning of postinstall.cfg, and it works fine.

look inside of veewee

the architecture of veewee looks great, it adds subcommand in vagrant, and all the majoy information are put into definition.rb

Veewee::Session.declare({
  :cpu_count => '1', :memory_size=> '384',
  :disk_size => '10140', :disk_format => 'VDI', :hostiocache => 'off',
  :o s_type_id => 'Ubuntu',
  :iso_file => "ubuntu-11.04-server-i386.iso",
  :iso_src => "http://releases.ubuntu.com/11.04/ubuntu-11.04-server-i386.iso",
  :iso_md5 => "ce1cee108de737d7492e37069eed538e",
  :iso_download_timeout => "1000",
  :boot_wait => "10", :boot_cmd_sequence => [
    '',
    '/install/vmlinuz noapic preseed/url=http://%IP%:%PORT%/preseed.cfg ',
    'debian-installer=en_US auto locale=en_US kbd-chooser/method=us ',
    'hostname=%NAME% ',
    'fb=false debconf/frontend=noninteractive ',
    'keyboard-configuration/layout=USA keyboard-configuration/variant=USA console-setup/ask_detect=false ',
    'initrd=/install/initrd.gz -- '
  ],
  :kickstart_port => "7122", :kickstart_timeout => "10000", :kickstart_file => "preseed.cfg",
  :ssh_login_timeout => "10000", :ssh_user => "vagrant", :ssh_password => "vagrant", :ssh_key => "",
  :ssh_host_port => "7222", :ssh_guest_port => "22",
  :sudo_cmd => "echo '%p'|sudo -S sh '%f'",
  :shutdown_cmd => "shutdown -P now",
  :postinstall_files => [ "postinstall.sh"], :postinstall_timeout => "10000"
})

it will be used during boot time to trigger the auto-installation, different os will use own auto-installation method, preseed.cfg is used for ubuntu/debian, if for suse then kickstart is used. postinstall.sh is used when the os is installed and rebooted.

summary

with those tool stacks (virtualbox,vagrant,puppet, veewee), your work makes more easily, don’t reinvent the wheel, don’t DIY. Use others brain.

make ci easier with jenkins CI and Vagrant

Friday, October 21st, 2011

Last time, I mentioned the nice tool for vagrant which can control virtualbox virtual machines easily, at once I start to think how it can be used in CI system easily.

Also I noticed the question @ stackoverflow http://stackoverflow.com/questions/6941547/how-to-combine-vagrant-with-jenkins-for-the-perfect-continuous-integration-envir/7830173#7830173, and my suggestion is below

Separate virtual machines environment

Why

Mostly your jenkins server is used for lots of tasks, therefore it is run mostly by restricted user, for example it is tomcat6 if running as servlet, or normal jenkins user, and mostly they don’t have shell access to avoid problem.

Vagrant may needs lots of 3pps which may different with normal jenkins user.

Therefore it is better to separate the virtual machine environment

Jenkins master/slave mode

This is the nice solution to solve this problem to isolate the environment, just new another node in your jenkins server (which is master) as below

image

It is communicated by ssh, and it will invoke the slave.jar automatically. And here I created the vagrant user for this, why ?

Reuse the ssh public/private key

If you played with vagrant, you know vagrant can access virtual machine via ssh without password, it because it uses the ssh public/private key.

Therefore I created the vagrant user and put the private/public key into ~vagrant/.ssh directory, mostly they are under /var/lib/gems/1.8/gems/vagrant-0.8.7/keys/, which is used by vagrant, if you don’t know where it is, please type vagrant ssh_config

image

Prepare the base box under vagrant user

Follow the normal vagrant guideline, and create one vm box, for example, I put it under ~/vm/ubuntu

New the CI job to control VM

Now it is time to configure a job to experience this, in jenkins master node, create a job in freestyle

image

Label Expression is the category we defined in slave node.

In the task (Build – Execute shell), jenkins slave try to go to that directory and start vagrant, then doing some simple tasks to evaulate whether it is done in our vm box.

image

!! REMEBER, this is done in slave mode using vagrant user !!

Now you can see the job is invoked in slave vagrant node

image

In the console output, you can notice the result

image

Summary

Using Vagrant in jenkins master/slave mode is perfect match for this job, you can get lots of flexibility using virtualmachine.
veewee is another layer above vagrant to control vm box more easily.

learn puppet 2 – install tomcat with web app

Saturday, October 8th, 2011

image Deploying Apache Tomcat Applications With Puppet is a good start point for using puppet to install tomcat.

Start without client/server mode

It is much easy to use puppet as installation tool in single node as well, “puppet apply” mode can be used to practice your .pp files locally, as example above

puppet --modulepath /srv/puppet-tomcat-demo/modules -v /srv/puppet-tomcat-demo/site.pp

it will automatically download the tomcat6 packages even openjdk packages for you.

The default port is 735 (<1024), which seems have permission problem at least for me, I simple changed it to 3735, then it works very well for me.

And there are two tips, more check http://www.devco.net/archives/2009/08/19/tips_and_tricks_for_puppet_debugging.php

  1. --debug is good option for more detail debug information under command line
  2. --parseonly is good to do syntax check

Client/server mode

ssl is still tricky

the ssl authentication is still trick, my two working virtual machines doesn’t work again, I delete all “*.pem”, it still doesn’t work, “*.yaml” is deleted as well, and put full domain name in puppet.conf, and it works. Actually I still don’t know what’s the clue, anyway it works again after 1 hour (!!!).

Put it as module

Move the tomcat as module under /etc/puppet, and put site.pp information to node.pp, the path is hard-coded first

path => '/srv/puppet-tomcat-demo/java_src/SimpleServlet.war'

It should exist in agent node first to make testing easy, when it works, we start to deal with puppet source

Puppet source

The resource should exist in server side or nfs server, let’s use server first

path => ‘puppet:///modules/tomcat/war/SimpleServlet.war’

it indicates to use puppet protocol (only support protocol so far), and actually it maps to /etc/puppet/modules/tomcat/files/war/SimpleServlet.war, and read the puppet wiki http://docs.puppetlabs.com/guides/file_serving.html is more clear for this.

Then it is quite clean in client side, well done.

Next step

Just think about how to switch to Sun (oracle) JRE, which is more stable for system and also start to think how to deal with own package system since currently the package comes from public repo.

just a record for my learning on puppet for auto-deployment.

learn puppet – auto deployment

Friday, September 30th, 2011

image In contiuous delivery, auto deployment is one key issue, and puppet is one popular tools to implement this, and I spent two days to learn it.

Introduction

Probably you can start with the slides http://www.slideshare.net/teyotyree/puppet-talk-at-oscon-2010, in short, Puppet is an open source framework and toolset for managing the configuration of computer systems, you may care about the resource like ssh instead of real package, it is a nice weapon for sysadmin and our auto deployment.

Puppet itself is written by ruby, but mostly you just need to deal with configuration (DSL)

It may have other versions now, and I choose open source which is available for ubuntu.

Getting started

I prepare two virtual machines, which is created from ubuntu 11.04, one is puppet master (puppet, 192.168.56.100), another one acts as puppet agent (puppet1, 192.168.56.101).

> apt-get install puppet # agent/client
> apt-get install puppermaster # master

Then you can follow http://docs.puppetlabs.com/learning/manifests.html to write one simple puppet resource files like 1.file.pp to execute in agent machine to experience it.

So nice it will automatically regenerate it when the resource is not correct

Connect from agent to master

Next step is try to put the resource in master side, and auto install something from agent, I follow https://help.ubuntu.com/11.04/serverguide/C/puppet.html to define apache2 package in master side.

Then in agent side to run command to verify it, wooh, the ssh connection is so weird, I spend 1 day to figure it out, see lesson learned.

> pupper agent –t

it will call the default master server (puppet) to fetch resource information, and notice it needs apache2, it will call internal command (guess apt-get) to install apache2 for me.

After I remove the apache2 package and run above command again, it is back as expected !!

> apt-get autoremove apache2

Module installation

Mostly besides package, you need deal with other issues, therefore in puppet, we need to bind everything into one module, Pro Puppet book had a very good example on how to customize apache configuration after installation, please go there and download the source codes (chapter02\apache)

user@puppet:/etc/puppet$ find modules/apache/
modules/apache/manifests/init.pp
modules/apache/manifests/install.pp
modules/apache/manifests/service.pp
modules/apache/manifests/vhost.pp
modules/apache/templates/vhost.conf.erb

user@puppet:/etc/puppet$ cat manifests/site.pp
import 'nodes.pp'

user@puppet:/etc/puppet$ cat manifests/nodes.pp
node "puppet1.example.com" {
  info "In node for ubuntu"
  include apache
  apache::vhost { 'puppet1':
    port => 80,
    docroot => '/var/www/puppet1',
    ssl => false,
    priority => 10,
    serveraliases => 'puppet1',
  }
}

When you put all these files under master node, and invoke it from agent node, it will install apache2 and customize the /etc/apache2/sites-enabled/10-puppet1 configuration automatically.

Lesson learned

When it works, puppet is very good, but the log system and his structure may be complex to the beginner, so be aware of this and learn it step by step, if the environment is not good, it doesn’t work, and it may not give you correct direction why it doesn’t work.

ssl authentication is quite trick, be careful

one headache is the error message is not so clear and ssl itself is always troublesome especially if you are not expert on this area, I think my problem mainly is changing the hostname after the CA cerfication (ca.pem) is generated.

You may need check http://www.madboa.com/geek/openssl/ to figure out what is the problem

Be careful with the version

make sure you use the version 2.6.*+, origianlly one of my ubuntu server is 10.x based, which use puppet 2.5.x, lots of commands are different, so please make sure the version

Summary

Puppet is a wonderful tool, don’t reinvent the wheel to create your own scripts to deploy the system from the scratch, and this is just the start, I will dig into more to see how to use it connected within product deployment inside CI

Will chili be success ? – fork of redmine

Friday, May 27th, 2011

imageIn morning today, I got email from colleague on the information that my favorite redmine is forked by ChiliProject  http://www.chiliproject.org/

What happened

redmine is a nice issue management like Jira, and it writes in RoR, compare to Trac, it is more attractive currently due to its clean and user-friendly interface.

Unfortunately for redmine I know before are less community involvement, and originally it follows the roadmap, and currently it has delay for long time

image

Also there are less response from key contributors in the forum for the features, and this is the reason (at least one) I think for the fork.

And chiliproject shows https://www.chiliproject.org/projects/chiliproject/wiki/Why_Fork , and it looks the founders are strong for it.

image

What will happen

It is quite similar to hudson vs jenkins case, forked project copied everything from the source and make the quick release, then it will have new release by adding new features that this community wanted.

And no one knows how about whether the original redmine community (also plugins) will turn to it or not, but it seems the roadmap is promised.

As usually, I registered and popup some questions to see how they reflect.

Anyway, I vote +1 for fork in current status

Summary

In opensource community, the competition is very strong, and license help the project can be continously improved either follow/contribute or fork and continue if the community think it is a good project.

Actually github even encourage this and keep the possible to show the fork history, it means already give credit to original.

It also demands the high technical leadship skills, it needs hero in the community to make this happen.

Anything we could learn from it ? not yet for me now ;-)

markdown is supported in maven site !

Thursday, May 5th, 2011

image For most java maven product, we like to create maven site like http://maven.apache.org/plugins/maven-site-plugin/ , besides normal report like quality, we also want to add documentation there, therefore it can be a portal for the product.

Unfortunately doxia module for this purpose doesn’t support markdown officially, people has to use APT (another plaintext), and as I wrote before, markdown format is more popular and well supported for lots of website like github.

And fortunately Régis Décamps contributed doxia-module-markdown recently, and it works perfect for me.

Steps to use markdown in maven site

Since it was just contributed, no maven repo exist yet, so you download the source codes from http://code.google.com/p/doxia-module-markdown/downloads/list, and install locally

> mvn clean install

Then you can create your own maven site and follow http://code.google.com/p/doxia-module-markdown/wiki/Usage, for me, I need to specific version for maven site as <version>2.2</version>

Then, finally I can see like below

image

I just copied his information into markdown format, see all sample codes from https://github.com/larrycai/doxia-markdown-demo/

Summary

Nice contribution, and it is really useful for me, small issues are there, but so far so good.

Welcome new logo for jenkins

Friday, April 15th, 2011

image

Finally jenkins community get the new logo, good luck.

http://jenkins-ci.org/content/jenkins-new-look