Writing custom pkgsrc packages on Debian mipsel Ci20

I have been using pkgsrc framework since I have first came to know and use NetBSD, which was around 2005.

There is some good documentation available here https://www.netbsd.org/docs/pkgsrc/creating.html and here https://wiki.netbsd.org/pkgsrc/intro_to_packaging/  I wanted to share some real-life example of creating such custom package (I have chosen CFEngine-3.11.0 which is not present in pkgsrc-current nor the pkgsrc-Q1-2018, the latest version there is cfengine-3.7.3 )

CFEngine is an open source configuration management system, written by Mark Burgess. Its primary function is to provide automated configuration and maintenance of large-scale computer systems, including the unified management of servers, desktops, consumer and industrial devices, embedded networked devices, mobile smartphones, and tablet computers.

I have created the below example on the Debian 7.5 mipsel Ci20 developer board.

1.1) Pre-requisites

  • – Bootstrapped pkgsrc environment on the Ci20 (used a 16 Gb SDCard)
  • – Basic knowledge of pkgsrc environment
  • – Some free time and patience

1.2) Preparing the tools for custom packages build

Make sore you build and install the following packages

# cd /usr/pkgsrc/pkgtools/url2pkg
# /usr/pkg/bin/bmake install clean 

# cd /usr/pkgsrc/pkgtools/pkgdiff
# /usr/pkg/bin/bmake install clean

Next we create a WIP (work in progress) directory within the pkgsrc directory structure

# mkdir -p /usr/pkgsrc/wip/cfengine

We have decided to create the latest cfengine package available which is 3.11.0, so we take a note of the url download link and go to the new build directory and call url2pkg to initiate the package preperation

# cd /usr/pkgsrc/wip/cfengine
# /usr/pkg/bin/url2pkg https://cfengine-package-repos.s3.amazonaws.com/tarballs/cfengine-3.11.0.tar.gz

Once the script finishes it will open an editor and offer you to add customized options to the Makefile

I Save the edit and exit (vi :wq) you will get this message once the script finishes,

===> Overriding tools for cfengine-3.11.0
===> Extracting for cfengine-3.11.0
url2pkg> Adjusting the Makefile.

Remember to correct CATEGORIES, HOMEPAGE, COMMENT, and DESCR when you're done!

The final file structure of the skeleton cfengine pkgsrc looks as following:

root@mipsbox:/usr/pkgsrc/wip/cfengine# ls -al
total 24
drwxr-xr-x 3 root root 4096 Jan 18 20:50 .
drwxr-xr-x 10 root root 4096 Jan 18 20:47 ..
-rw-r--r-- 1 root root 0 Jan 18 20:47 DESCR
-rw-r--r-- 1 root root 392 Jan 18 20:50 Makefile
-rw-r--r-- 1 root root 18 Jan 18 20:47 PLIST
-rw-r--r-- 1 root root 367 Jan 18 20:49 distinfo
drwxr-xr-x 11 root root 4096 Jan 18 20:50 work


I have edited the Makefile in this way

# $NetBSD$

DISTNAME= cfengine-3.11.0
CATEGORIES= wip
MASTER_SITES= https://cfengine-package-repos.s3.amazonaws.com/tarballs/
MAINTAINER= INSERT_YOUR_MAIL_ADDRESS_HERE
HOMEPAGE= https://cfengine.com/product/community/
COMMENT= Tool for automating system administration
LICENSE= gnu-gpl-v3

GNU_CONFIGURE= yes
CONFIGURE_ARGS+= --with-postgresql${BUILDLINK_PREFIX.postgresql}

USE_LIBTOOL= yes

.include "../../databases/postgresql92-client/buildlink3.mk"
.include "../../databases/lmdb/buildlink3.mk"
.include "../../devel/pcre/buildlink3.mk"
.include "../../security/openssl/buildlink3.mk"
.include "../../textproc/libxml2/buildlink3.mk"
.include "../../mk/pthread.buildlink3.mk"
.include "../../mk/bsd.pkg.mk"

Since cfengine-3.11.0 server build depends on either mysql or postgresql I have included the configure option to build it with postresql support (more advanced way is to provide options variables where one could choose the mysql or postgresql options, but Im not gonna cover this here ) as well as lmdb, pcre, openssl and libxml.

Next we will try and actually build the above via bmake and see if we get any errors that need to be addressed

# cd /usr/pkgsrc/wip/cfengine 
# /usr/pkg/bin/bmake

Since I have quite a huge number of packages already pre-built on my Ci20  I won’t have to wait for long to get all the dependent packages to compile but the make ends with the following error :

DONE: Configuration done. Run make/gmake to build CFEngine Community.
=> Modifying libtool scripts to use pkgsrc libtool
=> Modifying libtool scripts to use pkgsrc depcomp
===> Building for cfengine-3.11.0
bmake: "/disk/pkgsrc/wip/tt/work/cfengine-3.11.0/Makefile" line 1099: Variable/Value missing from "export"
bmake: Fatal errors encountered -- cannot continue
bmake: stopped in /disk/pkgsrc/wip/cfengine/work/cfengine-3.11.0
*** Error code 1

Stop.
bmake[1]: stopped in /usr/pkgsrc/wip/cfengine
*** Error code 1

Stop.
bmake: stopped in /usr/pkgsrc/wip/cfengine

In cases like this you need to create some patches that would fix/patch the original code. The error above indicates that there is a wrongly parsed environment variable in the generated Makefile. We can check on line 1099 of the generated Makefile in the following directory

# cat /disk/pkgsrc/wip/tt/work/cfengine-3.11.0/Makefile | head -n 1099 | tail -n 1
export TAR_OPTIONS

So bmake environment does not like the TAR_OPTIONS variable defined, lets search for this string only

# grep -r TAR_OPTIONS /disk/pkgsrc/wip/tt/work/cfengine-3.11.0/Makefile 
TAR_OPTIONS = --owner=0 --group=0
export TAR_OPTIONS

The above Makefile gets generated by the configure script, which in fact calls automake at a certain stage which generates a Makefile.in from Makefile.am

# grep -r TAR_OPTIONS /disk/pkgsrc/wip/tt/work/cfengine-3.11.0/Makefile.in
TAR_OPTIONS = --owner=0 --group=0
export TAR_OPTIONS
# grep -r TAR_OPTIONS /disk/pkgsrc/wip/tt/work/cfengine-3.11.0/Makefile.am
TAR_OPTIONS = --owner=0 --group=0
export TAR_OPTIONS

So a quick workaround would be to first try remove the unnecessary variable name (on 2 lines) And prepare a diff that can be used to create a custom patch file. We will be using the pkgvi program to edit the following original file like so

# /usr/pkg/bin/pkgvi /disk/pkgsrc/wip/cfengine/work/cfengine-3.11.0/Makefile.am

Remove the 2 offending lines containing the TAR_OPTIONS variable and save (vi :wq)

pkgvi: File was modified. For a diff, type:
pkgdiff "/disk/pkgsrc/wip/tt/work/cfengine-3.11.0/Makefile.am"

To produce a diff run the above command

# pkgdiff "/disk/pkgsrc/wip/cfengine/work/cfengine-3.11.0/Makefile.am"
$NetBSD$

--- /disk/pkgsrc/wip/cfengine/work/cfengine-3.11.0/Makefile.am.orig 2017-08-03 15:28:40.000000000 +0000
+++ /disk/pkgsrc/wip/cfengine/work/cfengine-3.11.0/Makefile.am
@@ -48,8 +48,6 @@ SUBDIRS = libcompat \
 
 
 # Hide the buildsystem's username, at least with GNU tar.
-TAR_OPTIONS = --owner=0 --group=0
-export TAR_OPTIONS
 
 
 EXTRA_DIST = ChangeLog INSTALL README.md LICENSE CFVERSION

The output should be saved to a patch file in the /usr/pkgsrc/wip/cfengine/patches directory

# mkdir -p  /usr/pkgsrc/wip/cfengine/patches 
# /usr/pkg/bin/pkgdiff "/disk/pkgsrc/wip/cfengine/work/cfengine-3.11.0/Makefile.am" > /usr/pkgsrc/wip/cfengine/patches/patch-Makefile.am

Once the patch file called patch-Makefile.am is in place we need to regenerate SHA1 chacksums for the distinfo meta file like so:

# cd /usr/pkgsrc/wip/cfengine
# /usr/pkg/bin/bmake distinfo

So lets try and build it one more time and check if the above patch fixes our “TAR” issue

# cd /usr/pkgsrc/wip/cfengine
# rm -rf /usr/pkgsrc/wip/cfengine/work
# bmake

Again the same error comes !  Magically we have the TAR_OPTIONS in Makefile.in but not in Makefile.am anymore.

# grep -r TAR_OPTIONS /disk/pkgsrc/wip/tt/work/cfengine-3.11.0/Makefile.in
TAR_OPTIONS = --owner=0 --group=0
export TAR_OPTIONS
# grep -r TAR_OPTIONS /disk/pkgsrc/wip/tt/work/cfengine-3.11.0/Makefile.am

I did not have time to investigate the automake voodoo, so we can use a little dirty hack to make sure we patch the generated Makefile.in once the configure script finishes. This is how I have done it – by adding the following line towards the end of the configure script and produced the below patch file

$NetBSD$

--- /usr/pkgsrc/wip/cfengine/work/cfengine-3.11.0/configure.orig 2017-08-03 15:29:11.000000000 +0000
+++ /usr/pkgsrc/wip/cfengine/work/cfengine-3.11.0/configure
@@ -24850,6 +24854,9 @@ if test -n "$ac_unrecognized_opts" && te
 $as_echo "$as_me: WARNING: unrecognized options: $ac_unrecognized_opts" >&2;}
 fi
 
+#Added a small hack here to fix the TAR_ issue in Makefile.in
+sed -i '/TAR_/d' Makefile.in
+sed -i '/TAR_/d' Makefile 
 
 { $as_echo "$as_me:${as_lineno-$LINENO}: result: DONE: Configuration done. Run make/gmake to build CFEngine Community." >&5
 $as_echo "DONE: Configuration done. Run make/gmake to build CFEngine Community." >&6; }

Save it as follows

# cd /usr/pkgsrc/wip/cfengine/work/cfengine-3.11.0
# /usr/pkg/bin/pkgvi /usr/pkgsrc/wip/cfengine/work/cfengine-3.11.0/configure
pkgvi: File was modified. For a diff, type:
pkgdiff "work/cfengine-3.11.0/configure"
# /usr/pkg/bin/pkgdiff "/usr/pkgsrc/wip/cfengine/work/cfengine-3.11.0/configure" > /usr/pkgsrc/wip/cfengine/patches/patch-configure


Rebuild the SHA1 checksums again

# cd /usr/pkgsrc/wip/cfengine
# /usr/pkg/bin/bmake distinfo

And run bmake again,  takes approx 30 minutes to build on the Ci20, but this time we are successful !

<.....cut....> 
Making all in unit
Making all in load
Making all in acceptance
Making all in 25_cf-execd
 CC cf-execd-rpl-functions.o
 CCLD cf-execd-test
 CC mock_package_manager.lo
 CCLD libmock_package_manager.la
 CCLD mock_package_manager
 CC no_fds.o
 CCLD no_fds
 CC xml_c14nize-xml-c14nize.o
 CCLD xml-c14nize
root@mipsbox:/usr/pkgsrc/wip/cfengine#

Next we run a test install

# cd /usr/pkgsrc/wip/cfengine

# /usr/pkg/bin/bmake stage-install CHECK_FILES=no

Check if the .destdir got populated,  following should be visible

<----cut---->
/usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/share/doc/cfengine/examples/translatepath.cf
/usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/share/doc/cfengine/examples/edit_replace_string.cf
/usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/share/doc/cfengine/examples/app_baseline.cf
/usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/share/doc/cfengine/examples/packagesmatching.cf
/usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/share/doc/cfengine/examples/mustache_comments.cf
/usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/share/doc/cfengine/examples/varnet.cf
/usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/share/doc/cfengine/examples/exec_in_sequence.cf
/usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/share/doc/cfengine/examples/process_signalling.cf
/usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/share/doc/cfengine/examples/sort.cf
/usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/share/doc/cfengine/examples/strcmp.cf
/usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/share/doc/cfengine/examples/classmatch.cf
/usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/share/doc/cfengine/examples/isipinsubnet.cf
/usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/share/doc/cfengine/examples/rename.cf
/usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/share/doc/cfengine/README.md
/usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/bin
/usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/bin/cf-promises
/usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/bin/cf-net
/usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/bin/cf-runagent
/usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/bin/cf-key
/usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/bin/cf-upgrade
/usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/bin/cf-agent
/usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/bin/cf-execd
/usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/bin/cf-serverd
/usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/bin/cf-monitord
/usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/bin/rpmvercmp
/usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/var
/usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/var/cfengine
/usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/var/cfengine/plugins
/usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/var/cfengine/ppkeys
/usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/var/cfengine/modules
/usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/var/cfengine/inputs
/usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/var/cfengine/outputs


Make sure you also update the /usr/pkgsrc/wip/cfengine/DESCR

Cfengine, or the "configuration engine" is a very high level language
for building expert systems which administrate and configure large
computer networks. Cfengine uses the idea of classes and a primitive
form of intelligence to define and automate the configuration of
large systems in the most economical way possible. Cfengine is
designed to be a part of computer immune system.

Cfengine 3 is operationally backwards compatible with Cfengine 2, but the
language is not. Cfengine 3 is not a drop-in replacement for Cfengine 2.

Once this is done we populate the PLIST inventory

# cd /usr/pkgsrc/wip/cfengine
# /usr/pkg/bin/bmake print-PLIST >PLIST

And finally run a proper install

# /usr/pkg/bin/bmake install

Congratulations, your first binary pkgsrc package will land in /usr/pkgsrc/packages/All/ and will get installed into /usr/pkg environment.

 

 

Advertisement

About astr0baby

Please run Adblock or similar... we have been told to do so since Carl Sagan wrote the Contact .
This entry was posted in Uncategorized. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.