Posts Tagged ‘jenkins’

make ci easier with jenkins CI and Vagrant

Friday, October 21st, 2011

Last time, I mentioned the nice tool for vagrant which can control virtualbox virtual machines easily, at once I start to think how it can be used in CI system easily.

Also I noticed the question @ stackoverflow http://stackoverflow.com/questions/6941547/how-to-combine-vagrant-with-jenkins-for-the-perfect-continuous-integration-envir/7830173#7830173, and my suggestion is below

Separate virtual machines environment

Why

Mostly your jenkins server is used for lots of tasks, therefore it is run mostly by restricted user, for example it is tomcat6 if running as servlet, or normal jenkins user, and mostly they don’t have shell access to avoid problem.

Vagrant may needs lots of 3pps which may different with normal jenkins user.

Therefore it is better to separate the virtual machine environment

Jenkins master/slave mode

This is the nice solution to solve this problem to isolate the environment, just new another node in your jenkins server (which is master) as below

image

It is communicated by ssh, and it will invoke the slave.jar automatically. And here I created the vagrant user for this, why ?

Reuse the ssh public/private key

If you played with vagrant, you know vagrant can access virtual machine via ssh without password, it because it uses the ssh public/private key.

Therefore I created the vagrant user and put the private/public key into ~vagrant/.ssh directory, mostly they are under /var/lib/gems/1.8/gems/vagrant-0.8.7/keys/, which is used by vagrant, if you don’t know where it is, please type vagrant ssh_config

image

Prepare the base box under vagrant user

Follow the normal vagrant guideline, and create one vm box, for example, I put it under ~/vm/ubuntu

New the CI job to control VM

Now it is time to configure a job to experience this, in jenkins master node, create a job in freestyle

image

Label Expression is the category we defined in slave node.

In the task (Build – Execute shell), jenkins slave try to go to that directory and start vagrant, then doing some simple tasks to evaulate whether it is done in our vm box.

image

!! REMEBER, this is done in slave mode using vagrant user !!

Now you can see the job is invoked in slave vagrant node

image

In the console output, you can notice the result

image

Summary

Using Vagrant in jenkins master/slave mode is perfect match for this job, you can get lots of flexibility using virtualmachine.
veewee is another layer above vagrant to control vm box more easily.

Dev@Cloud is possible by cloudbees – step by step guideline

Wednesday, August 31st, 2011

The company CloudBees annouced the Dev@Cloud http://www.cloudbees.com/dev.cb & Run@Cloud service, let me show you how to build a java web application there from scratch.

They make it happen, wonderful !!

Probably you should also aware kk (Kohsuke Kawaguchi, founder of hudson/jenkins) works for cloudbees now.

Overview

Dev@Cloud provides repositories (git or svn) and CI (jenkins) system with java/maven support, which means it provides good enough infrastructure for your java software development.

and Run@Cloud suppose to run your application there as well.

Sign-in

it is simple as other web application, after sign-in, you will see the portal as below

image

Repositories

Start from select the repositories, as I am git fans, surely I will create git repository, before creating, I need to subscribe the repo service, and I use free in beginning.

image

If you use git, probably you need go back to user setting to upload ssh public key  image

I load my favorite game-of-life (cloned from https://github.com/wakaleo/game-of-life) into this repo, which needs java/maven environment.

Quite smoothly to upload my codes into git repo, it seems it doesn’t support online browsing yet.

Jenkins builds

When the code is into git repo, I subscribe to its jenkins build service, and create a free-style job as normal.

For the first usage, it will ask for validation

image

The validation is over phone, and I think China may not supported, and fill China (86) and my local phone number as below for a trial.

To my surprise, the phone rings just after my saving  and they speak Chinese as I selected, ;-) ), so nice service.

 image 

After filling the pincode (3 digitals) , the validation works, better than excellent.

image

I created the helloworld job first and add some unix command to check what the system it is used, it seems it is SELinux (redhat version) on xen, 1.5G memory

model name	: Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz

You can using follow command to check like me

ls -al ; pwd ; hostname  df -k uname -a  free -m  who -a  dmesg  cat /proc/cpuinfo  cat /proc/version  env

I also notice it support ruby as well since kk works on ruby a lot for jenkins.

The build is mostly started after 2-3 minutes, and runs quite fast

Game-of-life project

Select maven and java version for the jenkins job, and don’t forget to config git  user name and email since the build needs to add tag, otherwise it reports error below

image 

I just wonder whether I need to configure maven repo, to my surprise, I don’t need to add this, it is done automatically, bingo !! since in the enterprise, I always need to spend long time to set it correctly, in dev@cloud, it works automatically, which is the power of maven.

This is the final result for my job

image

And the total jenkins jobs for me

image

Summary

Dev@Cloud is a good start for java software development, in some small company or FOSS project, you don’t need extra IT infrastructure, they provides everything for you and smoothly, and this is just the start.

They also provides maven repo for the build result, which I didn’t try it and also I will try Run@Cloud service.

Well done.

《程序员》9期:敏捷和工具 – 参考

Tuesday, August 30th, 2011

我写的文章刊登在2011年 《程序员》9期上, 上面参考链接忘放了,在这里补充一下。

参考

  1. 中文敏捷宣言: http://agilemanifesto.org/iso/zhchs/
  2. Redmine: http://redmine.org/
  3. 《Agile ALM》:http://www.manning.com/huettermann/
  4. 持续集成理论和实践的新进展http://www.infoq.com/cn/articles/ci-theory-practice
  5. Jenkins:http://jenkins-ci.org/
  6. Hudson正式更名为Jenkinshttp://www.infoq.com/cn/news/2011/02/jenkins
  7. Oracle proposes to move Hudson to Eclipse:http://kohsuke.org/2011/05/04/oracle-proposes-to-move-hudson-to-eclipse/
  8. Jenkins: The Definitive Guide》:http://www.wakaleo.com/books/jenkins-the-definitive-guide
  9. 李剑翻译的《硝烟中的Scrum和XP》: http://www.infoq.com/cn/minibooks/scrum-xp-from-the-trenches
  10. Gerrit:http://code.google.com/p/gerrit/
  11. Android的代码审阅: https://review.source.android.com/
  12. 《Git权威指南》:http://www.ossxp.com/doc/gotgit/
  13. 《Maven实战》和许晓斌:http://www.juvenxu.com/mvn-in-action/
  14. 敏捷之旅中国:http://agiletour.cn/

图四网上也没有。

clip_image002

图四:Android的Gerrit代码评审系统截图【11】

希望一起探讨,我会在博客上有更升入介绍。

from CI to Continuous Delivery

Wednesday, July 6th, 2011

image I mentioned the book “Continuous delivery” several times, and it is really good book from CI, Automation, Agile area,  and more information/tools comes out in this area now

Recommend to read

Jez Humble is one of the author for this book and check his slides http://www.slideshare.net/jezhumble/continuous-delivery-5359386, just remember it use picture based on thoughtworks’ Go platform, which is the replacement of CruiseControl.

Mike McGarr’s presentation in MCJUG, the slides discover lots of useful tools to support this concept

image Paul Duvall from stelligent.com wrote Refcards on Continuous Delivery, and his slides http://www.slideshare.net/stelligent/cloud-delivery-5004698

If you use jenkins, Build pipeline plugin is used for pipeline idea, in John’s latest book Jenkins:The definitive guide (not published yet, there is chapter for it)

Recommend to listen

In infoq, there is a interview for Martin Fowler and Jez Humble on this http://www.infoq.com/interviews/jez-humble-martin-fowler-cd

Install gerrit locally under windows

Wednesday, June 8th, 2011

I tried several times to install gerrit locally (windows or virtual machines), and meet authentication problem always, the purpose is mainly for demo and experience some new things.

What is the problem ?

Though the gerrit is just the war files, the installation is not smoothly as jenkins, and it also needs some authentication service available.

Connecting to real company LDAP is not very suitable for demo (security and network access), and internal development method DEVELOPMENT_BECOME_ANY_ACCOUNT can’t cover lots of demo case.

I have problem to find the good http server or ldap server to be running.

Also how to run on windows is little tricky since scripts are available for unix.

See all available authentication  http://gerrit.googlecode.com/svn/documentation/2.1.7/config-gerrit.html#_a_id_auth_a_section_auth and start/stop daemon http://gerrit.googlecode.com/svn/documentation/2.1.7/install.html#rc_d

LDAP authentication

image After struggle lots of time, finally I make the ldap solution work locally (not so difficutl), I use openldap 2.2.29 windows version before, latest windows version from Userbooster cause vista crash (don’t know why) http://www.userbooster.de/en/download/openldap-for-windows.aspx though it looks promise, and if you work on unix environment, it should be more easy.

The next step is to find good sample datas, (conf and ldif files), and I followed http://www.zytrax.com/books/ldap/ch5/ and it works just fine, download those two files and save as demo.conf (include default schema file) and demo.ldif (user data)

> slapd -f demo.conf -h ldap://*:389 -d 1
> ldapadd -H ldap://ldaphost -x -D "cn=jimbob,dc=example,dc=com" -f demo.ldif -w dirtysecret

Configure ldap in gerrit

Gerrit installation is smooth (i tried this dozen times ;-)

$java -jar gerrit-2.1.7.war init -d review
Create 'C:\Users\rdccaiy\alm\gerrit\review' [Y/n]?
Location of Git repositories   [git]:
Database server type [H2/?]:
Authentication method [OPENID/?]: ldap
LDAP server   p; [ldap://localhost]:
LDAP username   : cn=jimbob, dc=example, dc=com
  cn=jimbob, dc=example, dc=com's password : dirtysecret
  confirm password : dirtysecret
Account BaseDN : dc=example, dc=com
Group BaseDN [dc=example, dc=com]:
..
Download and install it now [Y/n]? n

Run under windows

Actually to run the gerrit under windows is quite simple, it just is not mentioned in officical document.

java -jar review/bin/gerrit.war daemon -d review

Then open browser (better use chrome), login with rjsmith/rJsmitH (see ldif files), it works very well !!

image

Check the system setting http://gerrit.googlecode.com/svn/documentation/2.1.7/config-gerrit.html#ldap.server for more configuration for ldap server, like ldap.accountFullName

Summary

With redmine/gerrit/jenkins locally, I can demo the power of ALM easily, and opensource really provides the possibility to run under different platform, they are the real software

deliver a executable web application without container for demo

Thursday, April 28th, 2011

Busy with several slides, and it is time to touch the code keep energy, currently we start a new product, which is a pure web application (.war), and I want to push it to be a cute product, surely I have not enough competence to guide on the architecture, surely I can always speak something ;-)

There are two things I noticed are not so elegant, first

Problem One

When the product is delivered for demo to others, mostly they are asked to install tomcat as web container, then deploy the product (.war) inside, if it is me, it is little boring to install extra application, and also need some guideline how to do it.

Solution

Actually I didn’t get solution at once, but I can google and learn from open source community, that for my favorite jenkins, when I got their war files, I just need to

image

And it also keep the possible to deploy jenkins.war to tomcat

So it means there is possible to embed web contains inside, therefore the jenkins’s solution can be reused.

Technical Detail

Step 1: search how jenkins works and more

From the webpage https://wiki.jenkins-ci.org/display/JENKINS/Containers, you can notice, it uses winstone, then I got a lots of reference

And most exciting one from google is the “package web application for desktop use ”http://www2.itmill.com/articles/Packaging_web_applications_for_desktop_use.htm, it even goes further to have excutable file with each click to open browser !!

If you search “Kohsuke Kawaguchi winstone”, you will get the history of the thinking for jenkins ;-) , like Hudson became self-executable (2007) (Kohsuke is founder of hudson)

Now I need to narrow down my target, whether it is only .war file like jenkins or .exe for more user-friendly, let’s start from small

(this is typical case, I always remind myself don’t go so fast to lost where I am)

Step 2: learn from jenkins

There is not so much from Konsuke’s blog, and I directly start to look into codes, war/pom.xml is so big I don’t know how to start, and suddenly I notice i can use one feature “git blame” to locate the changes

So I click the blame button, search “winstone”, Bingo, then click that change-set

image

image

image

Now you can understand what he changed at that time, actually if you see the lastest code, you will be lost, since it turns more complicated, probably need search hudson mail archive to know more.

Step 3: adapt into my usage for demo

I have a simple helloworld2.war file for example, then according to the code sample above, I need

  1. put the Main.java source codes there, and compile to target/generated-resources
  2. extract the war files to target/generated-resources
  3. download the winstone.jar to target/generated-resources
  4. package together for new war files

Yes, it works, see below

image

Conclusion

this is just one way to package it, you can see the pom.xml from gist https://gist.github.com/945937, surely you still need that Main.java. It should work fine for small usage, I will deploy it to my real project later.

I guess jenkin’s solution is to generate jar first and distributed out, later in war package phase, download it again, also the launcher is put somewhere else.

if you already know this and have better way, please let me know.

Summary

Making the web application for easy to demo is very useful, think about it, though it is not the requirement.

My passion is always build the best thing for the product, and we need to learn for IT community on how they handle the software, as I said before for my TalentCamp program, Standing on the shoulders of giants

“git blame” can help you to locate the changes in easy way, love git ;-)

Welcome new logo for jenkins

Friday, April 15th, 2011

image

Finally jenkins community get the new logo, good luck.

http://jenkins-ci.org/content/jenkins-new-look

Web testing using selenium webdriver with TestNG

Wednesday, March 23rd, 2011

image Wooh, finally we got some projects are quite interesting to the web testing, and when I show the BDD stuff to them, it seems it is bit complicated, and they just ask, “we are just starting, can we have simple one to use? ”, also the concordion (which I prefer than jbehave) is not mature enough when integrated with CI and final test report, so I want to show the selenium usage only

Selenium I or Selenium II (WebDriver)

cross-browser and cross-os web testing is not needed in current scope and Selenium WebDriver seems more promising (http://seleniumhq.org/docs/03_webdriver.html#the-5-minute-getting-started-guide), so I decided to use Selenium WebDriver directly (java), and it will be easily integrated as Unit test case

Code in Selenium WebDriver

As I mentioned before, there are several place mentioned the PageObject best pratice to use within Selenium, and also when you follow it, PageFactory could be used as well, thank for opensource, it is easily to created the real case based on example.

Page concept is good to organize your test objects

See code sample based above

image image

TestNG over JUnit

In beginning, I just use junit, but later, I notice, it is lack of report feature, since we want to use it for acceptance test, we need test case description and test groups in the final report, junit seems lack of this, at least can’t be googled quickly.

And testNG provides good enough annotation http://testng.org/doc/documentation-main.html 

Integrate into CI

This is shoud be a simple case, just find a testNG plugin for jenkins, and configure it.

image

And everything is in testng-results.xml, so we are able to write a script to generate report like jbehave

image

Reference

Jenkins CI: http://wiki.hudson-ci.org/display/HUDSON/testng-plugin
Maven : http://maven.apache.org/plugins/maven-surefire-plugin/examples/testng.html
Selenium with TestNG: http://testng.org/doc/selenium.html
TestNG: http://testng.org/doc/index.html
JUnit or TestNG: http://stackoverflow.com/questions/6658/junit-vs-testng

Summary

in order to have do better web testing for verification, BDD could be great future, and now we decided to use Selenium WebDriver + testNG + Git + Maven, and open to add BDD later.

The slight challenge is who to write the test case, can our tester write them in java codes ? ;-)

concordion vs. jbehave

Thursday, March 3rd, 2011

Play for concordion around two days, let me start what it could be in the end from user point view

Result so far

For test case part, I use the concordion + selenium webdriver, probably need Page Object as well, and configured with maven.

imageIn the Jenkins CI server, after I installed concordion presenter plugin, it will provide the link to final report

I need some hack here to show the overview report, and it could looks below

image

Then click the related story, you will see more detail test case result.

image

Should be good enough, since automatic web testing should not cover all testing, and don’t forget selenium powerful feature is to for cross-browser and cross-operation testing, see http://code.google.com/p/selenium/wiki/ContinuousIntegrationSystem

jBehave vs. Concordion

imageConcordion is just one tool in BDD area, and also my colleague touched another one: jbehave which is also java based (http://jbehave.org/)

Let’s quickly give you some difference

JBehave

more stable tool with lots of mature needs features around it like excellent reports (http://jbehave.org/reference/examples/reports-3.1/) and maven, selenium webdriver with lots of working flow (http://jbehave.org/reference/stable/developing-stories.html)

Write spec in java codes and follow Given..When..Then BDD spec

Concordion

It is a young (could be good) tool, with less mature community, no overview reports compare to jbehave, it needs some hack script, which is my favorite.

It has good extension like snapshot/log. http://concordion.org/Extensions.html

Write spec inside html (no tool support yet) and fixture is written in java language

No follow “Given..When..Then” since author think it is useless to straight follow it, see http://concordion.org/Technique.html for explanation.

I didn’t look into ruby tool, it is more to dig into it.

Now we have lots of topics for next codekata ;-)

Summary

Though auto web testing is a old area, but how to run it efficiency is always challenge, and BDD is slightly new for us, and it needs time to used it, since it needs extra competence for testers, it may impacts the resource planning for line.

Automated acceptance testing (BDD) with concordion

Tuesday, March 1st, 2011

image Firstly, I have to say, though I heard BDD (Behaviour-driven development) for long time, this is my first time to look into little deeper to see whether there is value or good things we can learn from it.

background in this area

I will not to focus on BDD, more thinking about the automatic acceptance testing framework and I try to put it into ALM (application lifecycle management) thinking on how to connect the requirement to test case and backforth in easy way, see right.

FitNesse is one of popular framework in this area, but it seems Cucumber, JBehave, Corcordion is also nice to use, as always, I googled for a while.

And finally I want to give Corcordion a try, since it seems it is more clean.

Getting started with Concordion

Concordion is an opensource tool for writing automated accpetance tests in java, and follow the guideline http://concordion.org/Tutorial.html, and soon you will get the result.

image

Concordion is quite similar with FitNesse, it use html/junit instead of wiki page, it can avoid setting up extra FitNesse server (though it is simple).

How it can be used

You need to understand this general architecture for those kind of framework, and I encourage to take a look Concordion’s Hints & Tips

image

Probably we need to develop some python scripts around it to organize the test case in product level.

Others

Since the testing result is html based, it is easy for reading and there is jenkins plugin : concordion presenter plugin for it as well.

And it is opensource, all the source codes and even websites are available in google code.

Summary

Concordion is a interesting testing framework to enhance the connection from requirement to test case implementation, it is not the normally way we focused, hope it can bring some fresh area here.

Let me know if you have experience on this (like FitNesse), please contact me.