Posts Tagged ‘jenkins’

« Older Entries

groovy template example for email-ext plugin inside jenkins

Saturday, March 31st, 2012

Receiving clean build information from CI server is very important to me, it is learning from http://travis-ci.org , so I want to have this in jenkins server as well since it is just html format.

How it works

You need to install “email-ext” plugin https://wiki.jenkins-ci.org/display/JENKINS/Email-ext+plugin for your jenkins first, and follow the guideline to configure the basic setup first.

Now this plugin sopport html template using jelly/js/groovy, jelly looks complex to me, so I decided to choose groovy (http://groovy.codehaus.org/), which is an agile and dynamic language for the JVM.

In order to use groovy customized html template, we need do extra

Create template directory

The template directory doesn’t exist, need to be created first, you can copy the template file from github directly

$ mkdir $JENKINS_HOME\email-templates # put the template inside

Enable groovy template inside project

Then in the jenkins, configure the post action in the job like below

image

Bugs in groovy

Unfortunately there are some bugs in the default template, and I provided one fix (issue #13191) there. Others needs more time to fix. Don’t worry, since we will redesign the html anyway, don’t need so care about the old one.

Anyway, you should receive html report in ugly format like below

image 

Design my own html

Inspired by travis-ci’s report, I want to better html report from jenkins as well, below is my sketech for this.

image

  • news: is used in header for general remindings
  • status/main/detail: are all build information from job
  • support: extra line for support information, like email, faq, etc ..

With the help from my friend @haojii, we got final html/css like below

image image

I don’t want to go detail on the groovy, you just download the artifact: groovy-html-larry.template from my cloned email-ext repo@github and give a try.

It works for git, and skip some information in groovy standard template since it covers more cases, anyway it works for me and it is good enough.

DIY

You can tune with html file groovy-html-larry.html on your css and structure and apply to groovy template when it works

Summary

Better email report will help team to find their information quickly, and make it fun.

This is the first learning from travis-ci.org, and there are more ideas will be copied to jenkins.

Leave comments here if it works for you.

 

Tags: , , ,
Posted in Continuous Delivery | No Comments »

CI on demand – learn from travis-ci

Thursday, February 23rd, 2012

imageDidn’t blog for long time, mainly due to I am learning to many new things ;-) , let’s start the CI part.

In short http://www.travis-ci.org gives me the new thinking of CI, and I think it is the evoluation as well, originally I think CloudBee (jenkins behind it) http://www.cloudbees.com/ is great to put development into cloud. But that is just next normal step, not evolution (at least in CI part).

Let’s see how traditional CI works

Normal CI server like Jenkins

Everytime we talk about CI, we will have 2 steps in mind

- prepare the CI server with needed 3pp installed.
- CMer will create the job for related tasks towards code repository

Looks good and smoothly, and we are used to it.

It had some limitations

Multi platform or package dependance

It will be quite trickly if you want your software to be built and verified in different OS, mostly we need prepare the same machines, if you are good at jenkins, then master/slave mode is used.

Then go further for multi 3pp and different dependance, then this make environment preparation is very hard, since the matrix will be large.

You may think about cross compiler, one solution for building.

Surely as I demoed before, you can use vagrant/virtualbox inside jenkins server, yes, it is good, just not elegant

Jobs are managed by CMer

It is named jobs to each build task, and since it is central, mostly only CMer had permission to create jobs and let CI server run for you, it covers mainly on master/main branch. Since server is limited, running CI for your private branch is expensive on adminstrative.

And who understand best on building and testing environment ? It should be developers, so why not let developers to configure what they want and build for it ?

Awesome Travis-CI

imageSo what it is really needed for good CI ?

As a developer, I can write down which os/environment that build is needed, just like right, I need several extra packages and two ruby platform on Ubuntu OS. Also I know how to execute the build.

That’s good enough for a build, so CI will do the rest all, this is what http://travis-ci.org ‘s job

- After I put this configuration besides source codes and push to github
- travis-ci load virtual machine from pool via vagrant/virtualbox, currently it supports Ubuntu 11.04 only
- travis-ci apply your configuration into virtual machine using chef
- travis-ci run you job, and give report, see example: http://travis-ci.org/#!/larrycai/sdcamp
- travis-ci distroy the vm and put back into pool

That’s all the CI should do and this is exactly I want !

What’s the benefit

No CMer is involved, not only time is saved, but also I don’t need to spend time to communicate with them on which kind of environment (package), because it is my business, not their focus.

I can trigger build as much as possible with full control, and since the build normally last shorted, other developers will reuse the machine after my time slot, Even they use total different platform.

Summary

I don’t want to say jenkins is bad, actually it is quite powerful and flexibility. I just want to indicate travis-ci understand better for the future of the build

Go there, try it and even fork it to run by yourself.

I love opensource, innovation comes from competition! Well done both to jenkins and travis-ci.

And I want to name this is “CI on deman” or “Private CI”.

Tags: ,
Posted in Continuous Delivery, tool | 1 Comment »

make ci easier with jenkins CI and Vagrant

Friday, October 21st, 2011

Last time, I mentioned the nice tool for vagrant which can control virtualbox virtual machines easily, at once I start to think how it can be used in CI system easily.

Also I noticed the question @ stackoverflow http://stackoverflow.com/questions/6941547/how-to-combine-vagrant-with-jenkins-for-the-perfect-continuous-integration-envir/7830173#7830173, and my suggestion is below

Separate virtual machines environment

Why

Mostly your jenkins server is used for lots of tasks, therefore it is run mostly by restricted user, for example it is tomcat6 if running as servlet, or normal jenkins user, and mostly they don’t have shell access to avoid problem.

Vagrant may needs lots of 3pps which may different with normal jenkins user.

Therefore it is better to separate the virtual machine environment

Jenkins master/slave mode

This is the nice solution to solve this problem to isolate the environment, just new another node in your jenkins server (which is master) as below

image

It is communicated by ssh, and it will invoke the slave.jar automatically. And here I created the vagrant user for this, why ?

Reuse the ssh public/private key

If you played with vagrant, you know vagrant can access virtual machine via ssh without password, it because it uses the ssh public/private key.

Therefore I created the vagrant user and put the private/public key into ~vagrant/.ssh directory, mostly they are under /var/lib/gems/1.8/gems/vagrant-0.8.7/keys/, which is used by vagrant, if you don’t know where it is, please type vagrant ssh_config

image

Prepare the base box under vagrant user

Follow the normal vagrant guideline, and create one vm box, for example, I put it under ~/vm/ubuntu

New the CI job to control VM

Now it is time to configure a job to experience this, in jenkins master node, create a job in freestyle

image

Label Expression is the category we defined in slave node.

In the task (Build – Execute shell), jenkins slave try to go to that directory and start vagrant, then doing some simple tasks to evaulate whether it is done in our vm box.

image

!! REMEBER, this is done in slave mode using vagrant user !!

Now you can see the job is invoked in slave vagrant node

image

In the console output, you can notice the result

image

Summary

Using Vagrant in jenkins master/slave mode is perfect match for this job, you can get lots of flexibility using virtualmachine.
veewee is another layer above vagrant to control vm box more easily.

Tags: , , ,
Posted in Continuous Delivery, tool | No Comments »

Dev@Cloud is possible by cloudbees – step by step guideline

Wednesday, August 31st, 2011

The company CloudBees annouced the Dev@Cloud http://www.cloudbees.com/dev.cb & Run@Cloud service, let me show you how to build a java web application there from scratch.

They make it happen, wonderful !!

Probably you should also aware kk (Kohsuke Kawaguchi, founder of hudson/jenkins) works for cloudbees now.

Overview

Dev@Cloud provides repositories (git or svn) and CI (jenkins) system with java/maven support, which means it provides good enough infrastructure for your java software development.

and Run@Cloud suppose to run your application there as well.

Sign-in

it is simple as other web application, after sign-in, you will see the portal as below

image

Repositories

Start from select the repositories, as I am git fans, surely I will create git repository, before creating, I need to subscribe the repo service, and I use free in beginning.

image

If you use git, probably you need go back to user setting to upload ssh public key  image

I load my favorite game-of-life (cloned from https://github.com/wakaleo/game-of-life) into this repo, which needs java/maven environment.

Quite smoothly to upload my codes into git repo, it seems it doesn’t support online browsing yet.

Jenkins builds

When the code is into git repo, I subscribe to its jenkins build service, and create a free-style job as normal.

For the first usage, it will ask for validation

image

The validation is over phone, and I think China may not supported, and fill China (86) and my local phone number as below for a trial.

To my surprise, the phone rings just after my saving  and they speak Chinese as I selected, ;-) ), so nice service.

 image 

After filling the pincode (3 digitals) , the validation works, better than excellent.

image

I created the helloworld job first and add some unix command to check what the system it is used, it seems it is SELinux (redhat version) on xen, 1.5G memory 

model name	: Intel(R) Xeon(R) CPU           E5430  @ 2.66GHz

You can using follow command to check like me 

ls -al ; pwd ; hostname  df -k uname -a  free -m  who -a  dmesg  cat /proc/cpuinfo  cat /proc/version  env

I also notice it support ruby as well since kk works on ruby a lot for jenkins.

The build is mostly started after 2-3 minutes, and runs quite fast

Game-of-life project

Select maven and java version for the jenkins job, and don’t forget to config git  user name and email since the build needs to add tag, otherwise it reports error below

image 

I just wonder whether I need to configure maven repo, to my surprise, I don’t need to add this, it is done automatically, bingo !! since in the enterprise, I always need to spend long time to set it correctly, in dev@cloud, it works automatically, which is the power of maven.

This is the final result for my job

image

And the total jenkins jobs for me

image

Summary

Dev@Cloud is a good start for java software development, in some small company or FOSS project, you don’t need extra IT infrastructure, they provides everything for you and smoothly, and this is just the start.

They also provides maven repo for the build result, which I didn’t try it and also I will try Run@Cloud service.

Well done.

Tags: , , , ,
Posted in ALM, cloud | No Comments »

《程序员》9期：敏捷和工具 – 参考

Tuesday, August 30th, 2011

我写的文章刊登在2011年 《程序员》9期上， 上面参考链接忘放了，在这里补充一下。

参考

  1. 中文敏捷宣言： http://agilemanifesto.org/iso/zhchs/
  2. Redmine: http://redmine.org/
  3. 《Agile ALM》：http://www.manning.com/huettermann/
  4. 持续集成理论和实践的新进展：http://www.infoq.com/cn/articles/ci-theory-practice
  5. Jenkins：http://jenkins-ci.org/
  6. Hudson正式更名为Jenkins：http://www.infoq.com/cn/news/2011/02/jenkins
  7. Oracle proposes to move Hudson to Eclipse：http://kohsuke.org/2011/05/04/oracle-proposes-to-move-hudson-to-eclipse/
  8. 《Jenkins: The Definitive Guide》：http://www.wakaleo.com/books/jenkins-the-definitive-guide
  9. 李剑翻译的《硝烟中的Scrum和XP》： http://www.infoq.com/cn/minibooks/scrum-xp-from-the-trenches
  10. Gerrit：http://code.google.com/p/gerrit/
  11. Android的代码审阅： https://review.source.android.com/
  12. 《Git权威指南》：http://www.ossxp.com/doc/gotgit/
  13. 《Maven实战》和许晓斌：http://www.juvenxu.com/mvn-in-action/
  14. 敏捷之旅中国：http://agiletour.cn/

图四网上也没有。

clip_image002

图四：Android的Gerrit代码评审系统截图【11】

希望一起探讨，我会在博客上有更升入介绍。

Tags: , , , , , ,
Posted in Agile, ALM | No Comments »

from CI to Continuous Delivery

Wednesday, July 6th, 2011

image I mentioned the book “Continuous delivery” several times, and it is really good book from CI, Automation, Agile area,  and more information/tools comes out in this area now

Recommend to read

Jez Humble is one of the author for this book and check his slides http://www.slideshare.net/jezhumble/continuous-delivery-5359386, just remember it use picture based on thoughtworks’ Go platform, which is the replacement of CruiseControl.

Mike McGarr’s presentation in MCJUG, the slides discover lots of useful tools to support this concept

image Paul Duvall from stelligent.com wrote Refcards on Continuous Delivery, and his slides http://www.slideshare.net/stelligent/cloud-delivery-5004698

If you use jenkins, Build pipeline plugin is used for pipeline idea, in John’s latest book Jenkins:The definitive guide (not published yet, there is chapter for it)

Recommend to listen

In infoq, there is a interview for Martin Fowler and Jez Humble on this http://www.infoq.com/interviews/jez-humble-martin-fowler-cd

Tags: ,
Posted in ALM | No Comments »

Install gerrit locally under windows

Wednesday, June 8th, 2011

I tried several times to install gerrit locally (windows or virtual machines), and meet authentication problem always, the purpose is mainly for demo and experience some new things.

What is the problem ?

Though the gerrit is just the war files, the installation is not smoothly as jenkins, and it also needs some authentication service available.

Connecting to real company LDAP is not very suitable for demo (security and network access), and internal development method DEVELOPMENT_BECOME_ANY_ACCOUNT can’t cover lots of demo case.

I have problem to find the good http server or ldap server to be running.

Also how to run on windows is little tricky since scripts are available for unix.

See all available authentication  http://gerrit.googlecode.com/svn/documentation/2.1.7/config-gerrit.html#_a_id_auth_a_section_auth and start/stop daemon http://gerrit.googlecode.com/svn/documentation/2.1.7/install.html#rc_d

LDAP authentication

image After struggle lots of time, finally I make the ldap solution work locally (not so difficutl), I use openldap 2.2.29 windows version before, latest windows version from Userbooster cause vista crash (don’t know why) http://www.userbooster.de/en/download/openldap-for-windows.aspx though it looks promise, and if you work on unix environment, it should be more easy.

The next step is to find good sample datas, (conf and ldif files), and I followed http://www.zytrax.com/books/ldap/ch5/ and it works just fine, download those two files and save as demo.conf (include default schema file) and demo.ldif (user data)

> slapd -f demo.conf -h ldap://*:389 -d 1
> ldapadd -H ldap://ldaphost -x -D "cn=jimbob,dc=example,dc=com" -f demo.ldif -w dirtysecret

Configure ldap in gerrit

Gerrit installation is smooth (i tried this dozen times ;-) 

$java -jar gerrit-2.1.7.war init -d review
Create 'C:\Users\rdccaiy\alm\gerrit\review' [Y/n]?
Location of Git repositories   [git]:
Database server type [H2/?]:
Authentication method [OPENID/?]: ldap
LDAP server   p; [ldap://localhost]:
LDAP username   : cn=jimbob, dc=example, dc=com
  cn=jimbob, dc=example, dc=com's password : dirtysecret
  confirm password : dirtysecret
Account BaseDN : dc=example, dc=com
Group BaseDN [dc=example, dc=com]:
..
Download and install it now [Y/n]? n

Run under windows

Actually to run the gerrit under windows is quite simple, it just is not mentioned in officical document.

java -jar review/bin/gerrit.war daemon -d review

Then open browser (better use chrome), login with rjsmith/rJsmitH (see ldif files), it works very well !!

image

Check the system setting http://gerrit.googlecode.com/svn/documentation/2.1.7/config-gerrit.html#ldap.server for more configuration for ldap server, like ldap.accountFullName

Summary

With redmine/gerrit/jenkins locally, I can demo the power of ALM easily, and opensource really provides the possibility to run under different platform, they are the real software

Tags: , , ,
Posted in ALM | 10 Comments »

deliver a executable web application without container for demo

Thursday, April 28th, 2011

Busy with several slides, and it is time to touch the code keep energy, currently we start a new product, which is a pure web application (.war), and I want to push it to be a cute product, surely I have not enough competence to guide on the architecture, surely I can always speak something ;-)

There are two things I noticed are not so elegant, first

Problem One

When the product is delivered for demo to others, mostly they are asked to install tomcat as web container, then deploy the product (.war) inside, if it is me, it is little boring to install extra application, and also need some guideline how to do it.

Solution

Actually I didn’t get solution at once, but I can google and learn from open source community, that for my favorite jenkins, when I got their war files, I just need to

image

And it also keep the possible to deploy jenkins.war to tomcat

So it means there is possible to embed web contains inside, therefore the jenkins’s solution can be reused.

Technical Detail

Step 1: search how jenkins works and more

From the webpage https://wiki.jenkins-ci.org/display/JENKINS/Containers, you can notice, it uses winstone, then I got a lots of reference

And most exciting one from google is the “package web application for desktop use ”http://www2.itmill.com/articles/Packaging_web_applications_for_desktop_use.htm, it even goes further to have excutable file with each click to open browser !!

If you search “Kohsuke Kawaguchi winstone”, you will get the history of the thinking for jenkins ;-) , like Hudson became self-executable (2007) (Kohsuke is founder of hudson)

Now I need to narrow down my target, whether it is only .war file like jenkins or .exe for more user-friendly, let’s start from small

(this is typical case, I always remind myself don’t go so fast to lost where I am)

Step 2: learn from jenkins

There is not so much from Konsuke’s blog, and I directly start to look into codes, war/pom.xml is so big I don’t know how to start, and suddenly I notice i can use one feature “git blame” to locate the changes

So I click the blame button, search “winstone”, Bingo, then click that change-set

image

image

image

Now you can understand what he changed at that time, actually if you see the lastest code, you will be lost, since it turns more complicated, probably need search hudson mail archive to know more.

Step 3: adapt into my usage for demo

I have a simple helloworld2.war file for example, then according to the code sample above, I need

  1. put the Main.java source codes there, and compile to target/generated-resources
  2. extract the war files to target/generated-resources
  3. download the winstone.jar to target/generated-resources
  4. package together for new war files

Yes, it works, see below

image

Conclusion

this is just one way to package it, you can see the pom.xml from gist https://gist.github.com/945937, surely you still need that Main.java. It should work fine for small usage, I will deploy it to my real project later.

I guess jenkin’s solution is to generate jar first and distributed out, later in war package phase, download it again, also the launcher is put somewhere else.

if you already know this and have better way, please let me know.

Summary

Making the web application for easy to demo is very useful, think about it, though it is not the requirement.

My passion is always build the best thing for the product, and we need to learn for IT community on how they handle the software, as I said before for my TalentCamp program, Standing on the shoulders of giants

“git blame” can help you to locate the changes in easy way, love git ;-)

Tags: , , ,
Posted in Uncategorized | No Comments »

Welcome new logo for jenkins

Friday, April 15th, 2011

image

Finally jenkins community get the new logo, good luck.

http://jenkins-ci.org/content/jenkins-new-look

Tags:
Posted in tool | No Comments »

Web testing using selenium webdriver with TestNG

Wednesday, March 23rd, 2011

image Wooh, finally we got some projects are quite interesting to the web testing, and when I show the BDD stuff to them, it seems it is bit complicated, and they just ask, “we are just starting, can we have simple one to use? ”, also the concordion (which I prefer than jbehave) is not mature enough when integrated with CI and final test report, so I want to show the selenium usage only

Selenium I or Selenium II (WebDriver)

cross-browser and cross-os web testing is not needed in current scope and Selenium WebDriver seems more promising (http://seleniumhq.org/docs/03_webdriver.html#the-5-minute-getting-started-guide), so I decided to use Selenium WebDriver directly (java), and it will be easily integrated as Unit test case

Code in Selenium WebDriver

As I mentioned before, there are several place mentioned the PageObject best pratice to use within Selenium, and also when you follow it, PageFactory could be used as well, thank for opensource, it is easily to created the real case based on example.

Page concept is good to organize your test objects

See code sample based above

image image

TestNG over JUnit

In beginning, I just use junit, but later, I notice, it is lack of report feature, since we want to use it for acceptance test, we need test case description and test groups in the final report, junit seems lack of this, at least can’t be googled quickly.

And testNG provides good enough annotation http://testng.org/doc/documentation-main.html 

Integrate into CI

This is shoud be a simple case, just find a testNG plugin for jenkins, and configure it.

image

And everything is in testng-results.xml, so we are able to write a script to generate report like jbehave

image

Reference

Jenkins CI: http://wiki.hudson-ci.org/display/HUDSON/testng-plugin
Maven : http://maven.apache.org/plugins/maven-surefire-plugin/examples/testng.html
Selenium with TestNG: http://testng.org/doc/selenium.html
TestNG: http://testng.org/doc/index.html
JUnit or TestNG: http://stackoverflow.com/questions/6658/junit-vs-testng

Summary

in order to have do better web testing for verification, BDD could be great future, and now we decided to use Selenium WebDriver + testNG + Git + Maven, and open to add BDD later.

The slight challenge is who to write the test case, can our tester write them in java codes ? ;-)

Tags: , , , , , ,
Posted in ALM | No Comments »

« Older Entries