Le site de Cyril "Davromaniak" Lavier, sysadmin ascendant geek - Mot-clé - planet.ubuntu.com

Incorporate a file in another with AWK

Cyril Lavier — Fri, 21 Sep 2012 13:52:00 +0200

Hello.

Recently, I had to replace a string of characters by a text file content.

Example : I want to replace "##__youpi_tralala__##" by the content of the text file called "youpi_tralala".

I tried with sed, but it wasn't reliable, as sed tried to execute the text file content.

So I worked with my best friend AWK.

Here is the result, the script called include_file.awk :

{
	str=$0
	where=match(str,regexp)
	if (where) {
		while (("cat "file) | getline tstr > 0) {
			if (newstr == "") {
				newstr=tstr
			} else {
				newstr=newstr"\n"tstr
			}
		}
		close (tstr)
		sub(regexp,newstr,str)
	}
	print str
}

It's a beautiful AWK script, which can be launched like this : awk -v regexp="awesome regexp" -v file=/path/to/the/included/file -f include_file.awk text_file_with_the_regex

I use it in LaTeX, to debug some includes, to see if the file content or the include which is the origin of the issue.

Here is an example :

awk -v regexp="^%%__part1__%%$" -v file=includes/part1.tex -f include_file.awk document.tex | pdflatex -jobname=document

Here, the string placed in the document.tex file is "%%__part1__%%".

That's all folks :D.

Debian Wheezy has been frozen

Cyril Lavier — Sun, 01 Jul 2012 18:19:00 +0200

And it's the first time I actually live a Debian freeze.

Hello.

Yesterday evening, automatic migrations of packages from Unstable to Testing were stopped. Only packages waiting to be migrated from Unstable to Testing were given automatic freeze exceptions (like Nginx).

I use Debian for 8 years now (I still remember my good old Debian Woody :)) and I've been used to the release process, but didn't actually paid attention until I became a Debian contributor in 2011.

This freeze is important to me because it's the first time I actually live one, as I help maintaining the Nginx packages (also audacious, but I'm quite inactive to focus more on Nginx until I get in better shape).

On May 27th 2011, I sent a mail to the backports mailing list telling I performed a backport of Nginx 1.0.1 to Squeeze and I was hoping somebody could upload it.

The first answer I got was from Kartik Mistry, one of the maintainers of the nginx packages (now the main maintainer). He reviewed the package and invited me to give some help on package maintenance.

It was the start of a great journey through the Debian project.

First, we worked on synchronizing our efforts on packaging and backporting, then I started to work on some bugs or some issues reported. But I actually started to learn how the packaging process work.

Some milestones on my short career as a Debian contributor :

January 1st 2012 : I get my first changelog entries (3) and my first bug closing (bug #653160), mostly insignificant, but quite enjoyable.
March 18th 2012 (On Saint Cyril ;)) : I had my first upload sponsored by Kartik Mistry.
May 8th 2012 : The Nginx packaging tree was switched from SVN to GIT

Over the last year, the biggest work done on Nginx packaging was not so technical, it was human. We now work as a team, in which every member knows its role. We are 3 persons working as 1, and I think it's better than earlier.

I would like thank these people for this great journey :

Kartik Mistry and the whole Nginx packaging team (Michael Lustfield, Jose Parrella, Fabio Tranchitella, Dmitry Oboukhov)
Sven Hoexter, which is sponsoring the nginx backports (I may give you some work when nginx 1.2.1-2 will be migrated to Testing :)).
Benjamin Drung, for helping me a lot when I worked on audacious packages.
And everybody I had opportunity to chat with on various Debian mailing lists and IRC channels.

Now, some figures about nginx packaging from last freeze (Squeeze freeze) :

68 commits in the GIT repository
262 commits in the SVN repository
78 bugs closed via package upload
18 new upstream releases uploaded
28 package uploads
- 19 by Kartik Mistry
- 4 by Michael Lustfield
- 4 by Me
- 1 by Dmitry Oboukhov

If you want to chat or want to help us on package maintenance, you can come to the #debian-nginx IRC channel on the OFTC network.

That's all folks !

DDB lenny-backports discontinued

Cyril Lavier — Mon, 02 Apr 2012 08:47:00 +0200

Hello everybody.

As Debian discontinued the support of the 5.0 "Lenny" release, I decided to do so in the DDB.

I will continue to maintain the squeeze-backports, and lenny-backports will be moved into another repository at http://archives.ddb.davromaniak.eu which will be created in the next hours or days.

Thanks.

My pbuilderrc using qemu-debootstrap

Cyril Lavier — Sun, 25 Mar 2012 16:49:00 +0200

Hello everybody.

First, I'm sorry for this late translation. I had a quite bad and busy period of time, so I was unable to take time to perform this translation.

As you are already aware of, Debian packaging takes a lot of my time, as well on the practical side as on the theoretical part of it.

Recently, I became co-maintainer of audacious and audacious-plugins Debian packages. Also, in few weeks, I will become co-maintainer of the NGINX package in Debian (it happened on March 16th).

I'm also working on making the NGINX maintaining team work as an actual team. I try to achieve this by scheduling milestones and managing Wheezy goals for the packaging. I also try to make all co-maintainers speaking when an important decision need to be taken (new upstream release to upload, module adding/removal, etc...). By far, I think it's working and we are finally working as a team.

But without any building/development environment, Debian packaging would be like playing Russian-roulette with a fully loaded machine-gun.

Why ?

It's simple.

Before any source package upload in Debian, building a package in a clean environment is highly recommended. You need to be sure of the build-deps and binary dependencies. Building a package on your own PC doesn't ensure you the well working of a package. Because like every user, you have packages already installed on machine, which some of them may be undeclared build-deps. So the package built well on your machine, but won't build in a clean environment.

There are a lot of ways to have a clean environment, here is a list :

Virtualized machines, which will be reinstalled after every build try (perfect for patient people with a lot of spare time).
The same as above, but with physical machines (sponsored by the electrical company because of the power consumption and useful during winter, because you can shutdown you house heating system).
Find a intern for reinstalling machines while you are working on source packages (expensive and may cause antidepressant drugs abuse by the intern).
Handmade chroots with debootstrap (We barely have a long-lasting solution, because we need to recreate chroots after every build. Hmmm, not so handy).
Use Debian powered tools, which create clean chroots (compressed in TGZ) and unpack them to build packages in it (pbuilder and its friend debootstrap are doing this very well).

As you guessed, the following will be about the pbuilder tool.

A pbuilderrc file can be used to configure pbuilder. Often placed in $HOME/.pbuilderrc, this i a shell script which is launched by pbuilder at the beginning of its work.

Like I already explained in the posts The ultimate Debian/Ubuntu package building system and Pbuilder and tmpfs can be friends, the .pbuilderrc file helps setting all aspects of these chroots creation and use.

After a lot of work on this configuration, I wrote a quite complete pbuilderrc now, supporting the use of qemu-debootstrap, which is an evolved debootstrap. It uses QEMU to virtualize the chroot hardware architecture. So now we can build ARM packages on a I386 or AMD64 machine without using a cross-compilation.

Before using this pbuilderrc, you need to install the qemu-debootstrap package. Available on squeeze-backports, wheezy and sid for Debian, and starting natty for Ubuntu.

I also recommend the reading of the post The ultimate Debian/Ubuntu package building system to have all needed tools to use pbuilder with this kind of configuration.

The pbuilderrc is available on my GIT repository named "scriptomaniak" available on github.

Download it in your /root/ folder, under the .pbuilderrc name.

Don't hesitate to comment this work and submit patches/pull requests.

That's all folks !

Thruk + NGINX + authentication

Cyril Lavier — Fri, 30 Dec 2011 11:41:00 +0100

Sometimes, when one line is missing, the whole system seems broken.

Hello.

As you know it, I use NGINX on a daily basis. And yesterday evening, I worked on installing NGINX on the last server which used Apache, but it hosts Shinken + Thruk.

And the problems arrive with Thruk.

He uses his CGI server, as the configuration with Apache is simple and well documented, I never tried to fully understand this configuration.Butcette partie de la conf.

But with NGINX, it's totally different.

My thruk is installed in /opt/thruk.

Here is the configuration part for thruk in my NGINX configuration :

location /thruk {
	auth_basic            "Monitoring Access";
	auth_basic_user_file  /opt/thruk/htpasswd.users;	
	fastcgi_index index.cgi;
	fastcgi_param REMOTE_USER $remote_user;
	fastcgi_pass	unix:/tmp/thruk_fastcgi.socket;
	include fastcgi_params;
}

Above all, DON'T FORGET the line "fastcgi_param REMOTE_USER $remote_user;", which transmit the logged user to thruk, who will recognize it and allow you to access the interface.

Big cheers to ScriptFanix on the IRC channel #shinken on Freenode for helping me with this issue.

Thanks.

Debian/Ubuntu package building using virtual machines

Cyril Lavier — Tue, 29 Nov 2011 12:15:00 +0100

Hi.

As you know, I often work on Debian package building.

For some months now, the DDB provides packages for others architecures (powerpc and mips).

For the basic architectures (amd64, i386 and armel), I have physical machines for building, but for powerpc and mips, I use QEMU virtual machines.

As it took some time to create them, I decided to distribute them.

Every virtual machine is pre-configured using this article posted on March 2011, and pbuilders are created for Debian Lenny, Squeeze, Wheezy, Sid and Ubuntu Hardy, Karmic, Lucid, Maverick, Natty, Oneiric, Precise. Also, a README file containing all information for launching machines.

So here they are :

AMD64 (about 6GB) : http://www.davromaniak.eu/qemu/build_machine_amd64.tar
I386 (about 4GB) : http://www.davromaniak.eu/qemu/build_machine_i386.tar
MIPS (about 2GB) : http://www.davromaniak.eu/qemu/build_machine_mips.tar
MIPSEL (about 2GB) : http://www.davromaniak.eu/qemu/build_machine_mipsel.tar
POWERPC (about 3GB) : http://www.davromaniak.eu/qemu/build_machine_powerpc.tar

here are the MD5 sums, to check if the download went well :

AMD64 : 3e55dd81c20c2a6f67a43a97d7eab986
I386 : 75c10975e65b1f816bc0a0711e9d12f3
MIPS : 9192d16ec83d31b3de0b381179e6eeef
MIPSEL : e01b1b1cdae64786d56b278c97e90c65
POWERPC : 63d0d2ceac849de0b0c1ad51eea4ee44

To conclude, I thank Aurélien Jarno (aurel32), because he distributes QEMU virtual machines for most architectures, which served as base for these building machines.

Thanks.

Backports and some moments of joy

Cyril Lavier — Sat, 06 Aug 2011 21:18:00 +0200

Good evening everybody.

A month after proposing the backport to Debian Squeeze of audacious 2.4.4, when I was starting to think it won't be accepted in the backports, I received 3 mails from Kilian Krause, who wants to sponsor the packages (audacious, audacious-plugins and libmowgli).

I was very surprised and so happy because this backport is very different from the nginx backport I made some months ago (and I'm working with the maintainers to keep the backport up to date, and to synchronise work).

As some people know, I use nginx as a daily base, it's installed on the server which serves this website, I used to make custom packages for my work (with adding some third party modules), so backporting nginx and contributing to it was barely normal.

But for audacious, it's completely different. Mid June, on the backport mailing list, somebody (rent0n) requested a backport of audacious, I simply answered to the mail saying I've added the maintainers in CC and that I think we might wait until the 2.5.1 gets in unstable, to make the backport. Few days later, no answer from the maitainers, so I wanted to try to make the backport, in case it's not possible. So it was possible, but, as a first try, I didn't wanted to change the needed versions of the build dependencies, so it was a raw backport, but I had to recompile barely the whole world (it was about 10-15 packages). 2 days later, I cleaned the package, so I just needed to backport the libmowgli. Then, the requester tested it, was happy, and for me, that was all, I just asked for anybody to review it and upload. Actually, I was not thinking about official inclusion, and so I didn't upload the package to mentors.debian.net. So then, I leaved for 10 days, to go on holidays, and I forgot about the package, but not rent0n, which was very enthousiast, and sent me a mail about upload the packages to mentors, so I did it, because I thought maybe somebody will be interested about this package.

One month later, the package was sponsored, and now it's waiting the ultimate validation.

So it's the second time I get a backport sponsored and uploaded. Strangely, it feels the same as if it was the first time. Some excitation about the next steps, then I will regularily check the buildd status, and I will be divised into 2 feelings. The first being the relief, this work is done, so all the moments passed working on it are just memories. The second being the joy, as people think my work is good enough to be included in the official backport repository.

Also, for every package I backport, I subscribe to the qa page, in order to receive notifications about bugs, updates, and everything related to the package.

Now, some news about the DDB.

Few days ago, I decided to remove all references to the chromium backport, as I was unable to maintain it, and it was not clean at all. Because of this, I may create a "dev" component for the DDB, but for the moment, it's not decided.

Also, I updated the nginx backport to the version 1.1.0, it's also uploaded to mentors, but will have to wait at least a week, to have it uploaded.

That's all folks !

Pbuilder and tmpfs can be friends

Cyril Lavier — Tue, 12 Jul 2011 17:31:00 +0200

A way to speed up Debian/Ubuntu package builds using pbuilder.

Hi everybody.

After the last article about the installation of the package building machine, I was still hacking and searching around this building system.

And I thought it would be useful to use a tmpfs (stored in RAM) to speed up builds.

First, we need a big amount of RAM, at least 4GB, because we need to store the temporary data used for the build process, and keep enough RAM space for the system to work.

The server I use is equipped with 16GB of RAM, so I don't have to worry about that :).

To set this up, it's pretty simple (we need to perform all actions as the root user, so under Ubuntu, add sudo at the beginning of every command).

Create the /var/cache/pbuilder/build folder if it's not done yet :

mkdir /var/cache/pbuilder/build

Then, use your favorite text editor, and add the following line to the /etc/fstab file :

tmpfs 			/var/cache/pbuilder/build	tmpfs	defaults,auto			0	0

Mount the partition :

mount /var/cache/pbuilder/build

Then add the following line in your /root/.pbuilderrc file :

APTCACHEHARDLINK=no

This will make the building system create symlinks instead of hardlinks between the cache folder and the build folder, as they are on separated partitions.

I just ran a perf test with a complete build of the latest OWFS release for amd64 and i386 (11 supported systems, so 22 builds), and here is the results :

With tmpfs : 53 minutes and 52 seconds
Without tmpfs : 67 minutes and 18 seconds

As you can see, better performances with tmpfs.

That's all folks, :).

Some news

Cyril Lavier — Tue, 21 Jun 2011 11:29:00 +0200

Hi.

After few months of inactivity, there's some news.

So you might see it now, I wrote a new template for this website, less colored, more readable.

Then, the Debian/Ubuntu repositories I'm maintaining (which are now in the main menu)

The OWFS repository is still here, and updated regularly, with the latest release (2.8p9 released at the end of May). I am peacefully preparing the add of Ubuntu Oneiric, which will occur in the next weeks.

The newcomer, DDB (for Davromaniak's Debian Backports) is a repository containing all the backports I'm working on. The goal is to work on proposing them for inclusion in the official backport repository. It's a stable repository, but as it contains backports, it may be risky to use it.

For now, backports for nginx 1.0.4, chromium 11.0.696.71 and audacious 2.4.4 are available, all along their build dependencies and binary dependencies.

Also, nginx 1.0.4 was recently accepted for inclusion in the official backports, we just need to wait until it's included. So the package available in the DDB repository is the same as the official one, only the version changes (I use "davrobpo60+1" instead of "bpo60+1" to mark the difference between official packages and DDB's packages).

For chromium, it was a hard to perform backport (between 20 to 30 hours of work to make it work), but it's not so clean, as I prefered to backport the binutils package in order to have a backported package as close as the official package.

For audacious, it works, but it's still a work in progress, mostly on the amount of dependencies to backport.

I try to make my packages available for the majority of hardware architectures, but it all depends on the package size. For now, only OWFS and nginx are available under 5 hardware architectures (amd64, armel, i386, mips and powerpc), because they don't take too long to compile. For audaicous, the amount of dependencies make it difficult to compile under other architectures than amd64 and i386. For chromium, it's only compilable in amd64, i386 and armel, but it's compilation lasts for 25 minutes on my building machine (8 cores, 16GB of RAM), my armel machine being the complete opposite in term of power, building could last for days.

That's all folks.

OWFS : Debian and Ubuntu packages

Cyril Lavier — Mon, 07 Mar 2011 14:49:00 +0100

Hello.

OWFS is a free software suite, licensed under GPL. It's an easy and reliable way to use the 1-wire system from Maxim, which controls a widespread range of devices (sensors for temperature, moisture, luminosity, door openning/closing, and many more, even LCD screens).

Today, I sent a mail to the OWFS mailing list, which makes the release official, here is the mail :

Hi everybody.

In the company which I'm working for, we use 1-wire for more than 18 months.

We use it in our server rooms, to monitor the temperature in it (and send alerts when it's too hot) and we plan to monitor temperatures in the offices.

As we use Debian on our servers, I worked to make packages for making the installation easier.

At the beginning, packages where a little rough, but we decided to give back what the owfs community gave us, and to help this project by publicly releasing these packages.

So I cleaned the packages, and now, by this mail, I'm releasing them.

Packages are available the following versions of Debian and Ubuntu :
- Debian 5.0 "Lenny" (oldstable)
- Debian 6.0 "Squeeze" (stable)
- Debian 7.0 "Wheezy" (testing)
- Debian Sid (unstable)

- Ubuntu 8.04 "Hardy" (LTS)
- Ubuntu 9.04 "Jaunty"
- Ubuntu 9.10 "Karmic"
- Ubuntu 10.04 "Lucid" (LTS)
- Ubuntu 10.10 "Maverick"
- Ubuntu 11.04 "Natty" (testing)

So now, some explanation about the packages.

For the moment, I do not package owperl, owmon, owtap, owphp, owcapi, owpython, owtcl , owftpd and owhttpd.

However, packages for owftpd and owhttpd might come later, maybe with a 2.8p8 depending on the time I can take on packaging.

For the other packages, they will come when I will have enough time to do a clean packaging.

Here is the packages list (available in versions 2.8p6 and 2.8p7 if not indicated) :
- owfs-common : contains the /etc/owfs.conf file, and the manpage
- owfs-server : contains the /usr/sbin/owserver binary (originally in /usr/bin, but I moved it in the package, because of the Debian Policy requirements), the manpage and the init.d script
- owfs (only 2.8p7) : contains the /usr/sbin/owfs binary, to use with fuse (originally in /usr/bin, but I moved it in the package, because of the Debian Policy requirements), the manpage and the init.d script
- owfs-client : contains the client tools (owdir, owpresent, owread, owwrite and owread), with the manpages associated.
- libow-2.8-7 (only 2.8p7) : contains the libow for version 2.8p7
- libow-2.8-6 (only 2.8p6) : contains the libow for version 2.8p6
- owfs-dev : contains the file /usr/include/owfs_config.h
- owfs-doc : contains all the remaining manpages
- owfs-dbg : contains the debugging symbols

So if you need the server and the clients, install owfs-server and owfs-client (it will install owfs-common and libow-2.8-6 or libow-2.8-7 depending on which version you are installing). If you need the fuse filesystem, only install owfs (it will also install the dependencies needed).

All information you need is on the repository website (updated after every package update) : http://owfs.davromaniak.eu/

I would like to thank Paul Alfille for his work on OWFS, and all the other contributors for their respective work.

Also, I had to patch the sources for having full compliance to the Debian Policy, Paul, I will mail you later today or tomorrow to share these patches.

If you detect any problem, feel free to mail me, or mail the mailing list.

Thanks.

The repository is available here : http://owfs.davromaniak.eu/

By the way, I may work on including these packages to Debian/Ubuntu official repositories. I will keep you informed on the evolution.

Bye.

The ultimate Debian/Ubuntu package building system

Cyril Lavier — Thu, 03 Mar 2011 21:28:00 +0100

Here is a "small" article which crosses the border between personal stuff and work but may be useful for many people.

In my current job, making Debian and Ubuntu packages takes me a lot of time.

At the beginning, I used to work on 2 virtual machine with a small pbuilderrc file, but I had the opportunity to work on a more powerful server (4 cores and 4GB of RAM), and I thought it would be useful to merge both machines and finaly have only one building machine.

My building system runs on a Debian Sid, even if I was a long time Ubuntu user, now I use Debian (including on my workstation).

Why choosing Debian Sid ?. Only to be able to install the latest versions of build tools, especially for debootstrap.

It might also work on Debian stable, and Ubuntu (but some points are different)

So let's begin.

All the commands preceded by a "#" need to be run with the root user, commands preceded by a "$" need to be run with a normal user, without privileges.

Notice : my server is running 64 bits OS, so I can compile 32 and 64 bits packages.

First, we need to install sudo, because as time goes by, I learned the hard way that it's dangerous to work on packages with the user root because it's not necessary and also, after some hours of keyboard typing and with tiredness by, we can act like asses and transforming into a "Rage Guy" for the next 5 minutes (or hours).

# aptitude install sudo

Then, we need to change the settings of sudo depending of what we will do with pbuilder.

# visudo

Here is my sudo configuration file :

#
# This file MUST be edited with the 'visudo' command as root.
#
# Please consider adding local content in /etc/sudoers.d/ instead of
# directly modifying this file.
#
# See the man page for details on how to write a sudoers file.
#
Defaults	env_reset,env_keep="DIST ARCH CONCURRENCY_LEVEL"

# Host alias specification

# User alias specification

# Cmnd alias specification

# User privilege specification
root	ALL=(ALL:ALL) ALL

# Allow members of group sudo to execute any command
%sudo	ALL=(ALL:ALL) ALL

#includedir /etc/sudoers.d

# Members of the admin group may gain root privileges
%adm ALL=(ALL:ALL) ALL

Note the "Defaults" line, which contains instructions for keeping the values of variables DIST, ARCH and CONCURRENCY_LEVEL. Keep those variable name in head, we will use them later.

Now, we will work on the real part of compilation, we will install the tools needed for compilation.

# aptitude install debhelper build-essential dpkg-dev pbuilder devscripts debootstrap

Among those packages, the most remarkable are "pbuilder" and "debootstrap"

pbuilder allows us to build our packages in a chrooted environment which contains the linux distribution of our choice.
debootstrap is called by pbuilder to create this chrooted environment

The lintian package is also useful, but optional, it tests the Debian Policy conformity of packages.

Now all these tools are installed, we need to configure the pbuilder, with the pbuilderrc file. Since I don't like to edit the default configuration file (/etc/pbuilderrc), I choose to create the /root/.pbuilderrc file, which here is mine.

# Codenames for Debian suites according to their alias. Update these when
# needed.
UNSTABLE_CODENAME="sid"
TESTING_CODENAME="wheezy"
STABLE_CODENAME="squeeze"
STABLE_BACKPORTS_SUITE="$STABLE_CODENAME-backports"
OLD_STABLE_CODENAME="lenny"
OLD_STABLE_BACKPORTS_SUITE="$OLD_STABLE_CODENAME-backports"

# List of Debian suites.
DEBIAN_SUITES=($UNSTABLE_CODENAME $TESTING_CODENAME $STABLE_CODENAME $OLD_STABLE_CODENAME "unstable" "testing" "stable" "experimental")

# List of Ubuntu suites. Update these when needed.
UBUNTU_SUITES=("natty" "maverick" "lucid" "karmic" "jaunty" "intrepid" "hardy" "gutsy")

# Mirrors to use. Update these to your preferred mirror.
DEBIAN_MIRROR="ftp.fr.debian.org/"
UBUNTU_MIRROR="fr.archive.ubuntu.com"

# Use old-releases mirrors for EOL versions
if [ "${DIST}" == "gutsy" ]; then
	UBUNTU_MIRROR="old-releases.ubuntu.com"
fi
if [ "${DIST}" == "intrepid" ]; then
	UBUNTU_MIRROR="old-releases.ubuntu.com"
fi
# Optionally use the changelog of a package to determine the suite to use if
# none set.
if [ -z "${DIST}" ] && [ -r "debian/changelog" ]; then
    DIST=$(dpkg-parsechangelog | awk '/^Distribution: / {print $2}')
    # Use the unstable suite for Debian experimental packages.
    if [ "${DIST}" == "experimental" ]; then
        DIST="unstable"
    fi
fi

# Optionally set a default distribution if none is used. Note that you can set
# your own default (i.e. ${DIST:="unstable"}).
: ${DIST:="$(lsb_release --short --codename)"}

# Optionally set the architecture to the host architecture if none set. Note
# that you can set your own default (i.e. ${ARCH:="i386"}).
: ${ARCH:="$(dpkg --print-architecture)"}

NAME="$DIST"
if [ -n "${ARCH}" ]; then
    NAME="$NAME-$ARCH"
    DEBOOTSTRAPOPTS=("--arch" "$ARCH" "${DEBOOTSTRAPOPTS[@]}")
fi
BASETGZ="/var/cache/pbuilder/$NAME-base.tgz"
DISTRIBUTION="$DIST"
BUILDRESULT="/var/cache/pbuilder/$NAME/result/"
APTCACHE="/var/cache/pbuilder/$NAME/aptcache/"
BUILDPLACE="/var/cache/pbuilder/build/"

if $(echo ${DEBIAN_SUITES[@]} | grep -q $DIST); then
    # Debian configuration
    MIRRORSITE="http://$DEBIAN_MIRROR/debian/"
    COMPONENTS="main contrib non-free"
    if $(echo "$STABLE_CODENAME stable" | grep -q $DIST); then
        EXTRAPACKAGES="$EXTRAPACKAGES debian-backports-keyring"
        OTHERMIRROR="$OTHERMIRROR | deb http://backports.debian.org/debian-backports $STABLE_BACKPORTS_SUITE $COMPONENTS"
    elif $(echo "$OLD_STABLE_CODENAME stable" | grep -q $DIST); then
        EXTRAPACKAGES="$EXTRAPACKAGES debian-backports-keyring"
        OTHERMIRROR="$OTHERMIRROR | deb http://backports.debian.org/debian-backports $OLD_STABLE_BACKPORTS_SUITE $COMPONENTS"
    elif $(echo "unstable" | grep -q $DIST); then
	DIST="$UNSTABLE_CODENAME"
	OTHERMIRROR="$OTHERMIRROR | deb http://ftp.fr.debian.org/debian/ experimental main"
    fi
elif $(echo ${UBUNTU_SUITES[@]} | grep -q $DIST); then
    # Ubuntu configuration
    MIRRORSITE="http://$UBUNTU_MIRROR/ubuntu/"
    COMPONENTS="main restricted universe multiverse"
     v=0
     n=0
     for i in ${DEBOOTSTRAPOPTS[@]}; do
	if [ $v -ne 0 ]; then
		DEBOOTSTRAPOPTS[$n]="/usr/share/keyrings/ubuntu-archive-keyring.gpg"
	fi
	if [ $i == "--keyring" ]; then
		v=1;
	fi
	n=$((n+1))
     done
else
    echo "Unknown distribution: $DIST"
    exit 1
fi

To make this installation easier, you can download it from here

However, before jumping in the pbuilder creation of Ubuntu versions, there's a little thing to do if you are using Debian.

Debootstrap packages from Debian and Ubuntu are different. 1 line differs in the file /usr/share/debootstrap/scripts/gutsy. This different can cause trouble, especially under Ubuntu 11.04 "Natty".

At the line 72 of the file /usr/share/debootstrap/scripts/gutsy, add the folllowing line :

ln -nsf . "$TARGET/var/lib/dpkg/info/$ARCH"

Now, we are finally ready.

With this handsome .pbuilderrc, you will just have to type the following ligne to create a 64 bits Debian Squeeze pbuilder :

$ DIST=squeeze ARCH=amd64 sudo pbuilder create

Or a 32 bits Ubuntu Natty pbuilder :

$ DIST=natty ARCH=i386 sudo pbuilder create

But now, you are wondering "Why did he talked about the CONCURRENCY_LEVEL variable ?".

Here is the answer.

The variable CONCURRENCY_LEVEL is used to run several processes for only one compilation, in other words, for person who use and know make, this variable acts like the "-j" option.

To compile a source package with 4 processes under a 64 bits Debian Squeeze :

$ CONCURRENCY_LEVEL=4 DIST=squeeze ARCH=amd64 sudo pbuilder build paquet.dsc

To compile a source package with 2 processes under a 32 bits Ubuntu Natty 32 bits :

$ CONCURRENCY_LEVEL=2 DIST=natty ARCH=i386 sudo pbuilder build paquet.dsc

Here we are, the build system is up and running.

But knowing that I like to have my own useful functions which help me to automate some tasks, I create a .scripts folder in the home directory of my unprivileged user, and a script calledpbuilder_utils which here is the content.

function update_pbuilder() {
	nprocs=$(grep -cE "^processor" /proc/cpuinfo)
	dists=$@
	for i in $dists; do
		CONCURRENCY_LEVEL=$nprocs DIST=$i ARCH=i386 sudo pbuilder update
		CONCURRENCY_LEVEL=$nprocs DIST=$i ARCH=amd64 sudo pbuilder update
	done
}
function create_pbuilder() {
	nprocs=$(grep -cE "^processor" /proc/cpuinfo)
	dists=$@
	for i in $dists; do
		CONCURRENCY_LEVEL=$nprocs DIST=$i ARCH=i386 sudo pbuilder create
		CONCURRENCY_LEVEL=$nprocs DIST=$i ARCH=amd64 sudo pbuilder create
	done
}
function clean_pbuilder() {
	nprocs=$(grep -cE "^processor" /proc/cpuinfo)
	dists=$@
	for i in $dists; do
		CONCURRENCY_LEVEL=$nprocs DIST=$i ARCH=i386 sudo pbuilder --clean
		CONCURRENCY_LEVEL=$nprocs DIST=$i ARCH=amd64 sudo pbuilder --clean
	done
}
function build_pbuilder() {
	nprocs=$(grep -cE "^processor" /proc/cpuinfo)
	dsc=$1
	shift
	dists=$@
	for i in $dists; do
		CONCURRENCY_LEVEL=$nprocs DIST=$i ARCH=i386 sudo pbuilder build $dsc
		CONCURRENCY_LEVEL=$nprocs DIST=$i ARCH=amd64 sudo pbuilder build $dsc
	done
}

In the .bashrc file I added the following line : source ~/.scripts/pbuilder_utils.

From here, it might be a bit raw, but here are explanations :

update_pbuilder updates given pbuilders in both architectures, example : update_pbuilder squeeze natty
create_pbuilder creates given pbuilders in both architectures, example : create_pbuilder squeeze natty
clean_pbuilder cleans the given pbuilders in both architectures, example : clean_pbuilder squeeze natty
build_pbuilder compiles a source package for distributions given in both architectures, example : build_pbuilder paquet.dsc natty lenny squeeze

That's nearly all.

For the pbuilderrc file given here, I took as base the one given at the following address : https://wiki.ubuntu.com/PbuilderHowto#Multiple%20pbuilders

For the remaining things, it's hair pulling, some hacking and multiples missed tries.

I hope you enjoy reading.

SFS Update 0.3pre : a sneak peek of the 0.3

Cyril Lavier — Wed, 24 Mar 2010 20:00:00 +0100

Hi.

Few weeks ago, SFS Update 0.2 was released, with some promises for the 0.3.

In order to contain your impatience and to show what the 0.3 will look like, I decided to release a preliminary version of the 0.3, which supports a part of the forum engines listed on the last article.dernier billet.

Here is a list of the forum engines which are supported by the 0.3pre version :

PHPBB2
PHPBB3
Fluxbb 1.2 (experimental)
Beehive 0.9 (experimental)
FUDForum 3 (experimental)
IceBB (experimental)
MyBB (experimental)

I had to dismiss the idea of supporting these forum engines :

bbPress : No ban mangement included
kusaba : Project not developed anymore
MercuryBoard : PProject not developed anymore
MiniBB support : Ban management only contains IP ban, no username or email address ban management at this time

SFS Update 0.3 will support the following forum engines (depending on the feasibility) :

NextBBS
NinkoBB
Quicksilver Forums
UseBB
Vanilla
XMB

0.3 release will also mark the opening of a public GIT for this project, so that anybody will be able to improve and contribute to SFS Update

Here are the links for download : sfs_update-0.3pre.tar.bz2
sfs_update-0.3pre.tar.gz

If you notice any bug or defect, feel free to contact me.

Thanks.

SFS Update comes back with a 0.2 version

Cyril Lavier — Sat, 06 Mar 2010 15:30:00 +0100

And the 0.3 is announced.

A new version of this useful little shell script.

The major new feature is the support of phpbb 3 forums (the connect strings evolves to set the forum engine used)

It wasn't actually hard, but it marks an important step forward and the beginnings of the 0.3 version.

The other changes are a bug fix, and the change of the downloading URL.

Indeed, in order not to annoy the stopforumspam's webmaster, I decided to host the phpbb_bans.zip file used in the script. It's synchronised once a day, at 6:10 am french time (5:10 am GMT).

Now, let's talk about the 0.3 version.

Actually, the work has already begun few days ago because I had some time to work on it

I decided to include the support of 14 forum engines which are released under a GPL licence, which are listed below, and for some, the progress status.

Fluxbb 1.2 (developed and tested)
Beehive Forum 0.9 (developed and tested)
FUDforum 3 (developed and tested)
IceBB
kusaba
MercuryBoard
miniBB
MyBB
NextBBS
NinkoBB
Quicksilver Forums
UseBB
Vanilla
XMB

I also decided to drop the configuration file support in the 0.3 version to make SFS Update a real standalone script et easier to set it in a cronized task.

By the way, if you are using or simply know other forum engines (using a SQL database) released under a GPL licence or any opensource licence, I'm open to work on it.

And if you are the administrator of one of these forum engines and you have a ssh remote access and all the requirements SFS Update needs, don't hesitate to try using the script, and send me your impressions on it, and even some bugs, :).

Until I receive some feedback, the support of these 14 forum engines will be marked as experimental, waiting for a confirmation of the reliable working of SFS Update for these forums.

Here are the links for download: sfs_update-0.2.tar.bz2
sfs_update-0.2.tar.gz

I hope SFS Update will be useful for you and your forum.

Bye.

SFS Update 0.1.4

Cyril Lavier — Mon, 22 Feb 2010 20:07:00 +0100

A new version with plenty of new developments.

Hi.

After an update last week, SFS Update continue to evolve and now has his version 0.1.4 released, with a good changelog.

This is the version which has the most evolved since his previous update.

Here is the modification list:

Using getopt for parsing command line options
MySQL server credentials can be given using a unique formated string called the "connect string" (yes, this is very original, ;)), see the README file for more information about the syntax.
Ability to use multiple MySQL servers (simply separate the connect strings with a space character)
Statistic part : added the time used for the processing
Fixed a bug about the prefix value which was not actually used
Download and treat the phpbb_bans.zip file only one time for the whole script execution

The next major release (version 0.2) will welcome the PHPBB 3.x compatibilty, the file provided by the stopforumspam website only works with PHPBB 2.x.

There's some work to go, but it's getting closer, :).

Here are the links for download: sfs_update-0.1.4.tar.bz2
sfs_update-0.1.4.tar.gz

That's all, folks, :).

SFS Update 0.1.3

Cyril Lavier — Mon, 15 Feb 2010 21:43:00 +0100

Two new features

For reminder, SFS Update is a bash script which synchronises the "banlist" and the "disallow" tables of a PHPBB 2.x forum with the file provided by the stopforumspam website.

The main goal is to help forum administrators to low the effects of spambots in their forums.

An update is available, the current version is now 0.1.3.

This update adds two more features:

A statistic part is now available, it prints the amount of rows added in each table
Now, you can choose the hostname/IP address of the MySQL server

Here are the links for download: sfs_update-0.1.3.tar.bz2
sfs_update-0.1.3.tar.gz

That's all, folks, :).

SFS Update 0.1.1

Cyril Lavier — Mon, 10 Aug 2009 20:42:00 +0200

A little upgrade

Hi.

Here is a new minor version of SFS Update, which is now in version 0.1.1.

It appears that I wanted to stop using grep in order to remove a requirement and to make this tool faster.

For now, SFS Update doesn't work with phpbb v3, because the file given by StopForumSpam uses the table structure of phpbb v2. This will be corrected in a next release.

Here are the links for download:
sfs_update-0.1.1.tar.bz2
sfs_update-0.1.1.tar.gz

Information: I decided to use the 0.x versions as major releases, and 0.x.y for minor upgrades.

Besides, not to "pollute" the planet.ubuntu.com with post about every upgrade of SFS Update, I created a tag SFSUpdate, which will contain posts about this tool. The post about major versions will have the planet.ubuntu.com tag (this post is the last to talk about a minor version to appear in planet.ubuntu.com)

That's all, folks, :).

SFS Update

Cyril Lavier — Sat, 08 Aug 2009 22:32:00 +0200

A tool for phpbb forum administrators

10 months ago, I started hosting a friend's phpbb forum. Of course, I'm the administrator of the technical side of the forum

6 months ago, in order to try to protect the forum from spammers' assaults, I manually updated the bans list from the Stop Forum Spam website, which provides a SQL file for phpbb forums.

However, this is repetitive and annoying, and today I decided to automate this task. So a little script is born.

I also decided to publish it under a free software licence (the GNU GPL), maybe some other admins may use this script.

This program will not work on web-hosting which do not propose a remote shell access.

This is the 0.1 version of this program, for now he's unable to connect to a remote MySQL server (this feature will be included in the 0.2 version)

Here are the links for download:
sfs_update-0.1.tar.bz2
sfs_update-0.1.tar.gz

The installation howto is written in the README file included in the archive.

If you have any idea, feel free to contact me using the contact form.

New logo for my website/blog

Cyril Lavier — Tue, 14 Jul 2009 11:40:00 +0200

thanks to a fairy

Hi everybody.

As you can see, my website/blog has a new logo.

Yep I know this post doesn't completely talks about Ubuntu and Free Software, but it's a new start for my blog, and the story behind this new logo completely deals with Ubuntu and Canonical.

The former logo, which was composed of the Debian and the Ubuntu logos didn't meet the logo usage guidelines of the Ubuntu Trademark Policy. In order to meet these rules and not to set the bad example, I decided to create a new logo.

This logo was not drew by myself. A little fairy named Anita Mazur, which has gained a little spot in my heart, drew the entire logo based on our common ideas and design.

This one of a kind logo mixes the old with the contemporary. Will you find his meaning ?

Anita did a terrific job, because she worked with only ideas, I didn't draw any sketch. Despite this, no corrections were needed, and the first version was the good one

After getting a spot in my heart, through this logo, she now has a little spot in my blog. Before maybe getting the occasion to write in my blog, or in is own one near mine's.

She has a Deviant Art page, go and see photos and drawings she made.

Try to find the meaning of this logo, those who will succeed won't win anything, except our respect and recognition, :D.

Bye.

Cock-a-doodle-do !!! An Ubuntu Netbook is released by Archos

Cyril Lavier — Thu, 25 Jun 2009 20:25:00 +0200

Archos is a french company known since 10 years for producing MP3 and MP4 players just released a netbook which runs under your favorite OS.

It's the Archos 10 UBUNTU

Here are the technical specs of the beast :

500 GB Hard disk
10.2'' WXGA Screen (LED backlight)
Intel® ATOM™ N270 1.60 GHz Processor
2 GB DDR2
WiFi - card reader slot (SD/MMC/MS/MS PRO) - 1.3MP webcam - 2 stereo speakers - microphone
Operating System : UBUNTU 8.04 LTS
Extended-life battery version: extended-life up to 7 hours (with 6-cells battery)

All this stuff for 399.98 €, which is a common price for netbooks.

You can directly buy it in the Archos Store (only in French for the moment).

Bye.

Ubuntu Member since September 2007

Cyril Lavier — Thu, 21 May 2009 18:57:00 +0200

But it's the first post in the ubuntu.com planet

Why lasting so much time without taking time to add my feed to the planet ?

Simply because I didn't took the necessary time to add the feed, and I never thought I would have something interesting to say in this planet.

My posts won't compete against our ~~Britney Spears Fan~~ MOTU, Didier Roche.

I will talk about my relation to Free Software, using some peripherals, like the Pertelian, for which I develop some scripts, and others snippets.

For now, that's all

My next post might be a translation of this french post, which talks about the last Ubuntu Party, but not like the official post made by Didier Roche.

Bye.