Posts Tagged ‘Continuous Delivery’

create virtualbox on fly using veewee

Tuesday, October 25th, 2011

one headache of using vagrant is to have base box, and creating basebox from scratch is headache, and how to solve it ?

You want to create by yourself for some framework, please wait for a minute, there already had one project veewee, and I don’t want to repeat the nice blog http://www.ducea.com/2011/08/15/building-vagrant-boxes-with-veewee/ as well.

Below just some tips when I tried this morning

veewee tips

veewee directory structure

I prefer to create one seperate veewee directory for this tasks.
iso files should be download in advance, and make a softlink under created directory iso.
then you can define your own basebox like below

~/veewee$ vagrant basebox define 'ubuntu1104' 'ubuntu-11.04-server-i386'

the directory will looks like below, you type command under veewee directory

~/veewee$ find
.
./definitions
./definitions/myubuntubox
./definitions/myubuntubox/postinstall.sh
./definitions/myubuntubox/preseed.cfg
./definitions/myubuntubox/definition.rb
./definitions/ubuntu1104
./definitions/ubuntu1104/postinstall.sh
./definitions/ubuntu1104/preseed.cfg
./definitions/ubuntu1104/definition.rb
./iso
./iso/ubuntu-11.04-server-i386.iso
virtualbox headless

suppose it should work headless (without GUI), while when I create basebox, it pops error on keyboard permisson, and when I enable my X11 display (since I tried it in my local linux server), it pops up virtualbox GUI and start to install.

it is excellent feeling when you see it is under auto-installtion.

configure apt proxy

since I try this in the company, I need to setup the proxy, it is fine to add the line export http_proxy=http://proxy:8080 in the beginning of postinstall.cfg, and it works fine.

look inside of veewee

the architecture of veewee looks great, it adds subcommand in vagrant, and all the majoy information are put into definition.rb

Veewee::Session.declare({
  :cpu_count => '1', :memory_size=> '384',
  :disk_size => '10140', :disk_format => 'VDI', :hostiocache => 'off',
  :o s_type_id => 'Ubuntu',
  :iso_file => "ubuntu-11.04-server-i386.iso",
  :iso_src => "http://releases.ubuntu.com/11.04/ubuntu-11.04-server-i386.iso",
  :iso_md5 => "ce1cee108de737d7492e37069eed538e",
  :iso_download_timeout => "1000",
  :boot_wait => "10", :boot_cmd_sequence => [
    '',
    '/install/vmlinuz noapic preseed/url=http://%IP%:%PORT%/preseed.cfg ',
    'debian-installer=en_US auto locale=en_US kbd-chooser/method=us ',
    'hostname=%NAME% ',
    'fb=false debconf/frontend=noninteractive ',
    'keyboard-configuration/layout=USA keyboard-configuration/variant=USA console-setup/ask_detect=false ',
    'initrd=/install/initrd.gz -- '
  ],
  :kickstart_port => "7122", :kickstart_timeout => "10000", :kickstart_file => "preseed.cfg",
  :ssh_login_timeout => "10000", :ssh_user => "vagrant", :ssh_password => "vagrant", :ssh_key => "",
  :ssh_host_port => "7222", :ssh_guest_port => "22",
  :sudo_cmd => "echo '%p'|sudo -S sh '%f'",
  :shutdown_cmd => "shutdown -P now",
  :postinstall_files => [ "postinstall.sh"], :postinstall_timeout => "10000"
})

it will be used during boot time to trigger the auto-installation, different os will use own auto-installation method, preseed.cfg is used for ubuntu/debian, if for suse then kickstart is used. postinstall.sh is used when the os is installed and rebooted.

summary

with those tool stacks (virtualbox,vagrant,puppet, veewee), your work makes more easily, don’t reinvent the wheel, don’t DIY. Use others brain.

Tags: , , ,
Posted in Continuous Delivery, tool | No Comments »

make ci easier with jenkins CI and Vagrant

Friday, October 21st, 2011

Last time, I mentioned the nice tool for vagrant which can control virtualbox virtual machines easily, at once I start to think how it can be used in CI system easily.

Also I noticed the question @ stackoverflow http://stackoverflow.com/questions/6941547/how-to-combine-vagrant-with-jenkins-for-the-perfect-continuous-integration-envir/7830173#7830173, and my suggestion is below

Separate virtual machines environment

Why

Mostly your jenkins server is used for lots of tasks, therefore it is run mostly by restricted user, for example it is tomcat6 if running as servlet, or normal jenkins user, and mostly they don’t have shell access to avoid problem.

Vagrant may needs lots of 3pps which may different with normal jenkins user.

Therefore it is better to separate the virtual machine environment

Jenkins master/slave mode

This is the nice solution to solve this problem to isolate the environment, just new another node in your jenkins server (which is master) as below

image

It is communicated by ssh, and it will invoke the slave.jar automatically. And here I created the vagrant user for this, why ?

Reuse the ssh public/private key

If you played with vagrant, you know vagrant can access virtual machine via ssh without password, it because it uses the ssh public/private key.

Therefore I created the vagrant user and put the private/public key into ~vagrant/.ssh directory, mostly they are under /var/lib/gems/1.8/gems/vagrant-0.8.7/keys/, which is used by vagrant, if you don’t know where it is, please type vagrant ssh_config

image

Prepare the base box under vagrant user

Follow the normal vagrant guideline, and create one vm box, for example, I put it under ~/vm/ubuntu

New the CI job to control VM

Now it is time to configure a job to experience this, in jenkins master node, create a job in freestyle

image

Label Expression is the category we defined in slave node.

In the task (Build – Execute shell), jenkins slave try to go to that directory and start vagrant, then doing some simple tasks to evaulate whether it is done in our vm box.

image

!! REMEBER, this is done in slave mode using vagrant user !!

Now you can see the job is invoked in slave vagrant node

image

In the console output, you can notice the result

image

Summary

Using Vagrant in jenkins master/slave mode is perfect match for this job, you can get lots of flexibility using virtualmachine.
veewee is another layer above vagrant to control vm box more easily.

Tags: , , ,
Posted in Continuous Delivery, tool | No Comments »

learn puppet 2 – install tomcat with web app

Saturday, October 8th, 2011

image Deploying Apache Tomcat Applications With Puppet is a good start point for using puppet to install tomcat.

Start without client/server mode

It is much easy to use puppet as installation tool in single node as well, “puppet apply” mode can be used to practice your .pp files locally, as example above

puppet --modulepath /srv/puppet-tomcat-demo/modules -v /srv/puppet-tomcat-demo/site.pp

it will automatically download the tomcat6 packages even openjdk packages for you.

The default port is 735 (<1024), which seems have permission problem at least for me, I simple changed it to 3735, then it works very well for me.

And there are two tips, more check http://www.devco.net/archives/2009/08/19/tips_and_tricks_for_puppet_debugging.php

  1. --debug is good option for more detail debug information under command line
  2. --parseonly is good to do syntax check

Client/server mode

ssl is still tricky

the ssl authentication is still trick, my two working virtual machines doesn’t work again, I delete all “*.pem”, it still doesn’t work, “*.yaml” is deleted as well, and put full domain name in puppet.conf, and it works. Actually I still don’t know what’s the clue, anyway it works again after 1 hour (!!!).

Put it as module

Move the tomcat as module under /etc/puppet, and put site.pp information to node.pp, the path is hard-coded first

path => '/srv/puppet-tomcat-demo/java_src/SimpleServlet.war'

It should exist in agent node first to make testing easy, when it works, we start to deal with puppet source

Puppet source

The resource should exist in server side or nfs server, let’s use server first

path => ‘puppet:///modules/tomcat/war/SimpleServlet.war’

it indicates to use puppet protocol (only support protocol so far), and actually it maps to /etc/puppet/modules/tomcat/files/war/SimpleServlet.war, and read the puppet wiki http://docs.puppetlabs.com/guides/file_serving.html is more clear for this.

Then it is quite clean in client side, well done.

Next step

Just think about how to switch to Sun (oracle) JRE, which is more stable for system and also start to think how to deal with own package system since currently the package comes from public repo.

just a record for my learning on puppet for auto-deployment.

Tags: ,
Posted in Continuous Delivery, tool | No Comments »

learn puppet – auto deployment

Friday, September 30th, 2011

image In contiuous delivery, auto deployment is one key issue, and puppet is one popular tools to implement this, and I spent two days to learn it.

Introduction

Probably you can start with the slides http://www.slideshare.net/teyotyree/puppet-talk-at-oscon-2010, in short, Puppet is an open source framework and toolset for managing the configuration of computer systems, you may care about the resource like ssh instead of real package, it is a nice weapon for sysadmin and our auto deployment.

Puppet itself is written by ruby, but mostly you just need to deal with configuration (DSL)

It may have other versions now, and I choose open source which is available for ubuntu.

Getting started

I prepare two virtual machines, which is created from ubuntu 11.04, one is puppet master (puppet, 192.168.56.100), another one acts as puppet agent (puppet1, 192.168.56.101).

> apt-get install puppet # agent/client
> apt-get install puppermaster # master

Then you can follow http://docs.puppetlabs.com/learning/manifests.html to write one simple puppet resource files like 1.file.pp to execute in agent machine to experience it.

So nice it will automatically regenerate it when the resource is not correct

Connect from agent to master

Next step is try to put the resource in master side, and auto install something from agent, I follow https://help.ubuntu.com/11.04/serverguide/C/puppet.html to define apache2 package in master side.

Then in agent side to run command to verify it, wooh, the ssh connection is so weird, I spend 1 day to figure it out, see lesson learned.

> pupper agent –t

it will call the default master server (puppet) to fetch resource information, and notice it needs apache2, it will call internal command (guess apt-get) to install apache2 for me.

After I remove the apache2 package and run above command again, it is back as expected !!

> apt-get autoremove apache2

Module installation

Mostly besides package, you need deal with other issues, therefore in puppet, we need to bind everything into one module, Pro Puppet book had a very good example on how to customize apache configuration after installation, please go there and download the source codes (chapter02\apache)

user@puppet:/etc/puppet$ find modules/apache/
modules/apache/manifests/init.pp
modules/apache/manifests/install.pp
modules/apache/manifests/service.pp
modules/apache/manifests/vhost.pp
modules/apache/templates/vhost.conf.erb

user@puppet:/etc/puppet$ cat manifests/site.pp
import 'nodes.pp'

user@puppet:/etc/puppet$ cat manifests/nodes.pp
node "puppet1.example.com" {
  info "In node for ubuntu"
  include apache
  apache::vhost { 'puppet1':
    port => 80,
    docroot => '/var/www/puppet1',
    ssl => false,
    priority => 10,
    serveraliases => 'puppet1',
  }
}

When you put all these files under master node, and invoke it from agent node, it will install apache2 and customize the /etc/apache2/sites-enabled/10-puppet1 configuration automatically.

Lesson learned

When it works, puppet is very good, but the log system and his structure may be complex to the beginner, so be aware of this and learn it step by step, if the environment is not good, it doesn’t work, and it may not give you correct direction why it doesn’t work.

ssl authentication is quite trick, be careful

one headache is the error message is not so clear and ssl itself is always troublesome especially if you are not expert on this area, I think my problem mainly is changing the hostname after the CA cerfication (ca.pem) is generated.

You may need check http://www.madboa.com/geek/openssl/ to figure out what is the problem

Be careful with the version

make sure you use the version 2.6.*+, origianlly one of my ubuntu server is 10.x based, which use puppet 2.5.x, lots of commands are different, so please make sure the version

Summary

Puppet is a wonderful tool, don’t reinvent the wheel to create your own scripts to deploy the system from the scratch, and this is just the start, I will dig into more to see how to use it connected within product deployment inside CI

Tags: ,
Posted in Continuous Delivery, tool | 1 Comment »

from CI to Continuous Delivery

Wednesday, July 6th, 2011

image I mentioned the book “Continuous delivery” several times, and it is really good book from CI, Automation, Agile area,  and more information/tools comes out in this area now

Recommend to read

Jez Humble is one of the author for this book and check his slides http://www.slideshare.net/jezhumble/continuous-delivery-5359386, just remember it use picture based on thoughtworks’ Go platform, which is the replacement of CruiseControl.

Mike McGarr’s presentation in MCJUG, the slides discover lots of useful tools to support this concept

image Paul Duvall from stelligent.com wrote Refcards on Continuous Delivery, and his slides http://www.slideshare.net/stelligent/cloud-delivery-5004698

If you use jenkins, Build pipeline plugin is used for pipeline idea, in John’s latest book Jenkins:The definitive guide (not published yet, there is chapter for it)

Recommend to listen

In infoq, there is a interview for Martin Fowler and Jez Humble on this http://www.infoq.com/interviews/jez-humble-martin-fowler-cd

Tags: ,
Posted in ALM | No Comments »

From Continuous Integration into Continous Delivery

Wednesday, February 23rd, 2011

imageEnd of last yeay, I bought the book “Continuous delivery”  from amazon, and like it very much, though I still don’t finish reading it. We drive for CI (continuous integration) for long time, even we don’t get CI perfect now, we need to look into Continuous delivery as well, since it is new (mean value behind it) and looks it will fit more for new business (delivery frequently and easily).

Two author Jaz & David both work for thoughtworks, therefore be remember there is something more for their own product like cruisecontrol, just remind

In the dedicated blog http://continuousdelivery.com/, you will notice one important think behind the book

  • Notice the difference Continuous Delivery vs Continuous Deployment (continuous release)
    As we say in the book, you can continuously deploy to UAT – no big deal. What makes continuous deployment special is deploying every change that passes the automated tests (and optionally a short QA gate) to production. Continuous deployment is the practice of releasing every good build to users – a more accurate name might have been “continuous release”.

imageAnd yesterday I also notice interview in infoq for the author Jez Humble and Martin Fowler, http://www.infoq.com/interviews/jez-humble-martin-fowler-cd 

I list some key things if you don’t want to read it completely

Jez Humble: And the deployment pipeline effectively is an automated implementation of your build-deploy-test-release process, so I you take a part of the value stream of your delivery process from check-in to release, the deployment pipeline is the system that automatically takes each commit and then makes a build and then progresses that build through the various processes of testing and staging and then into production.

Martin Fowler: Yes, I mean the way I look at it size doesn’t matter when it comes to this thing. I mean the point is: do you want to go through a manual process where you’ve got to remember what to do and make sure you do everything in the right order or do you want to press one button or issue a single command line command and get the computer to do it all predictably and without errors because that’s what computers are good for?

Martin Fowler:  …"How do we deal with features that last longer than your release cycle so you don’t want that half-built feature released either because you don’t want to do that or simply because the business doesn’t want that particular feature switched on yet?" The basic way to do this is to use a technique that we call feature toggles, you have also heard terms like feature bits and flaps and flippers and flags and lots of other F’s. But we use toggles as our terminology and it’s basically something built in a configuration file which you then use within the code to interrogate the information from that configuration file to decide whether to make certain features visible or not.

I first time hear about feature toggles, seems interesting

Summary

Always push yourself to think about from code to release to the production system, which part is not automated, if not, think to improve it.

  1. Continuous delivery is very good for business (SaaS, cloud) which needs deploy to production quickly, and continuous deployment (release) is the step after continuous integration.
  2. CI is the first step and just push everything you want for automation.

Tags: , ,
Posted in ALM | No Comments »