PostgreSQL Installer build system ================================= This is the PostgreSQL Installer build system. This document attempts to describe how the system is architected, how to set it up and how to extend it. It is a work in progress and will no doubt require further refinement over time. There there is one goal however: Build all PostgresSQL & add-on package installers for all supported platforms with a single command. Note that this system is not intended to replace the existing installer system used on Windows (pgInstaller) - it is intended to mirror it's basic functionality however. Modular system design --------------------- The modular system is designed to be as flexible as possible and allow package authors as much freedom as possible in the way they design their installers. There are some basic rules about how we design add-on packages however - it remains up to the individual author to determine whether or not breaking any rules will break their package. They had better not break the system though! * Registration: A central registry file is used in which packages should register themselves. This data will be used by StackBuilder to locate installed packages. The registry file is /etc/postgres-reg.ini, and should be considered analagous in function to the sections of the Windows registry used for the same purposes on that platform. StackBuilder requires specific entries for the PostgreSQL server, as well as an entry indicating the installed version of each unique package. An example file is show below. ; This section is for a server, and is analagous to the PostgreSQL key under ; HKEY_CURRENT_USER\Software on Windows [PostgreSQL\8.3] Version=8.3.3 InstallationDirectory=/opt/PostgreSQL/8.3 DataDirectory=/opt/PostgreSQL/8.3/data Port=5432 Superuser=postgres [PostGIS_1_3_PG83] Version=1.3.2 [pgAdmin3] Version=1.8.4 InstallationDirectory=/opt/pgAdmin3 It is up to the uninstaller for each package to leave or clean the data during uninstallation. The version number for a package should *always* be cleared, but other data may be retained. For example, the server package will not remove the data directory, thus it is appropriate to leave the DataDirectory, Port and Superuser values intact. * Installers: Each package installer should be capable of being silently or interactively installing and uninstalling the package. When uninstalling, as much of the package as possible should be removed, however it is not always possible (through lack of reference counting between packages) or desirable to remove everything. Build platform -------------- The build platform is Mac OS X Snow Leopard. We use OS X because it allows us to run all other Intel based OS's on the same machine. All other supported platforms are run in VMWare Fusion virtual machines, with the exception of Solaris Sparc which obviously must be an external machine. The supported platforms can be set as external machines if required by exposing nfs shares from mac. A number of additional dependencies are required when setting up the system, such as libxml2 and libxslt, as well as useful utilties such as wget. Setting these up correct is essentially to ensuring future Universal binary builds will work as expected on all supported OS X versions (10.5/Leopard and above). Utilities should be installed using MacPorts: - Download the installer from http://www.macports.org/ and install the package. - Add /opt/local/bin:/opt/local/sbin to the *end* of the path. Add the following line to ~/.bash_profile for the buildfarm user: export PATH=$PATH:/opt/local/bin:/opt/local/sbin - Install packages: $ sudo port install cmake $ sudo port install wget $ sudo port install apache-ant $ sudo port install bison $ sudo port install flex $ sudo port install ossp-uuid Dependency libraries must be built with a little more control to ensure they use the correct SDK to allow them to be used on Tiger and above. We manually build and install these packages into /usr/local/ - Download the source for each library (eg. libuuid, libxml2 & libxslt). - Unpack the source into /usr/local/src - Configure the source with a command such as: CFLAGS="-isysroot /Developer/SDKs/MacOSX10.5.sdk -mmacosx-version-min=10.5 -arch i386 -arch x86_64 -arch ppc" LDFLAGS="-arch i386 -arch ppc -arch x86_64" ./configure --prefix=/usr/local/ --disable-dependency-tracking - Build and install: make all sudo make install Note that we must make sure all additional libraries link against these libraries, and not the older, system copies. In the case of libxslt, we can do this by configuring with --with-libxml-prefix=/usr/local libuuid will typically link only in the primary architecture of the build machine. To fix this, remove the library (libuuid.16.dylib) from .libs/ and re-run the final link command, with the addition of appropriate -arch flags. - Install unix2dos utility: $sudo port install unix2dos - Set up DocBook SGML Configuring DocBook SGML on Mac OS X with MacPorts. 1) Install the following packages using MacPorts: opensp openjade docbook-xsl docbook2X 2) Download the docbook-dsssl stylesheets from http://sourceforge.net/project/showfiles.php?group_id=21935&package_id=16611 and then unpack the archive in /opt/local/share/sgml 3) Add a symlink so PostgreSQL can find the stylesheets: sudo ln -s /opt/local/share/sgml/docbook-dsssl-1.79 \ /usr/local/share/sgml/docbook-dsssl 3) Download DockBook 4.2 (http://www.docbook.org/sgml/4.2/docbook-4.2.zip) and the ISO 8879 character entities (http://www.oasis-open.org/cover/ISOEnts.zip) 4) Unzip both archives into /opt/local/share/sgml/docbook-4.2 5) Ensure that all the unpacked files are world readable. 6) Run the following command in the docbook-4.2 directory: perl -pi -e 's/iso-(.*).gml/ISO\1/g' docbook.cat 7) Create the file /opt/local/share/sgml/catalog, with the following contents: CATALOG "openjade/catalog" CATALOG "docbook-4.2/docbook.cat" CATALOG "docbook-dsssl-1.79/catalog" 8) Make sure that settings.sh is updated with docbook installation path. - Fix PAM headers When building on Snow Leopard, using the Leopard SDK a failure may be seen when configuring PostgreSQL for PAM. This is because the PAM header location moved from /usr/include/pam in 10.5 to /usr/include/security in 10.6. The difference between the base OS and the SDK confuses configure. To avoid this, it may be necessary to move the PAM headers in /usr/include/security/security so configure cannot find them. - Downgrade Java Snow Leopard only ships with Java 1.6, but most of our code, as well as pl/java should be compiled with 1.5. 1) Download the Jlast Java update for Leopard: http://support.apple.com/downloads/java_for_Mac_OS_X_10_5_Update_5 2) Download Pacifist from http://www.charlessoft.com/ 3) * IMPORTANT * Delete the symlinks, 1.5 and 1.5.0 from /System/Library/Frameworks/JavaVM.framework/Versions 4) Open the Java update in Pacifist, and extract the /System/Library/Frameworks/JavaVM.framework/Versions/1.5.0 directory into the corresponding location on the hard disk. 5) Recreate the 1.5 symlink, to point to the 1.5.0 directory. You can now use Java 1.5, by setting JAVA_HOME to: /System/Library/Frameworks/JavaVM.framework/Versions/1.5.0/Home NOTE: That this update may have to be repeated following any Java updates from Apple. Couple of packages needs hacking, while building on snow leopard. PHP: http://www.firewing1.com/node/31 Slony-I: http://www.slony.info/bugzilla/show_bug.cgi?id=109 Build VMs --------- All VMs (and in fact, the host machine) are setup to use user accounts called 'buildfarm'. In order to access each, the VMs must be setup with fixed IP addresses which are recorded with an appropriate hostname in DNS. Each hostname is specified in settings.sh. It may be necessary to manually configure VMWare Fusion to bridge the network adaptor instead of using NAT. The top level 'pginstaller' directory is shared with all the VMs using the VMware shared folders feature. The path to this directory is specified in settings.h for each VM. Note that VMware doesn't map UIDs/GIDs between the host and the VMs so it may be necessary to mount the shared directory using the UID/GID of the user in the VM, eg using the following in /etc/fstab: .host:/ /mnt/hgfs vmhgfs defaults,ttl=5,uid=500,gid=500 0 0 SSH authentication between hosts is achieved using certificates. These can be generated on the host machine using: ssh-keygen -t rsa Copy the resulting id_rsa.pub file to ~/.ssh/authorized_keys on each VM. * Linux/Linux-x64 - Install chrpath utility in order to change the rpath of the installed PostgreSQL binaries in the staging directory. Use the following command to install the chrpath: * yum install chrpath * Windows The Windows VM is the most tricky to setup: - Install Windows 7 - Install Microsoft Sercurity Essentials - Install Visual Studio 2008, and update to the latest service pack - Create the 'buildfarm' user account, as a limited user. - Install the a basic installation of Cygwin from http://www.cygwin.com/. Include the OpenSSH package. - Configure sshd with ssh-host-config, using the buildfarm user account as the service account. Make sure that openssh log file has correct ownership and permissions. It is absolotely necessary NOT to use default local account with sshd because compilation will fail when invoked via ssh. - Make sure that port 22/TCP is open in the Windows Firewall configuration. - Install the public ssh key in C:\Cygwin\home\buildfarm\.ssh - Install zip.exe and unzip.exe into the System32 directory. These utilities can be found at ftp://ftp.tex.ac.uk/tex-archive/tools/zip/info-zip/WIN32/ - Create folder 'c:\pgBuild' - Depending upon the modules to build, install various utilities in c:\pgBuild - Install bison, flex, diffutils in c:\pgBuild (Available from http://gnuwin32.sourceforge.net) - Prebuilt iconv, libxml2, libxslt, openssl and zlib from http://zlatkovic.com/pub/libxml, install them in c:\pgBuild - gettext (Please consult developer for specific version for PostgreSQL) - Mingw (gcc and g++). - MSys 1.0 - Compile and install ZLib in Msys using --prefix=/mingw - krb5 in c:\pgBuild - vcredist in c:\pgBuild - apache ANT in c:\pgBuild - wxWidgets in c:\pgBuild Note: In case of Windows-64 setup you may get errors related to: LINK : fatal error LNK1181: cannot open input file 'bufferoverflowU.lib'^M Main reason of this issue is bufferoverflowU.lib file missing in parallel directory structures of VC installation. To reslove this copy said file into parallel structure e.g.: cp /cygdrive/c/Program\ Files\ \(x86\)/Microsoft\ SDKs/Windows/v5.0/Lib/IA64/bufferoverflowu.lib /cygdrive/c/Program\ Files/Microsoft\ SDKs/Windows/v6.0A/Lib/x64/. * Mac OS X Creating a new VM for new codepath from an existing VM on the same machine: - Shutdown the VM - Right click the VM and click 'show in finder' and then right click on the bundle to copy to another name - Double Click the bundle to power it on and choose "I copied it" when Fusion asks - Change the HostName, ComputerName using below commands: sudo scutil --set ComputerName "newname" sudo scutil --set LocalHostName "newname" sudo scutil --set HostName "newname" System Preferences->Users&Groups and Change full name to the new name - Restart the VM Build Machines as external machines ----------------------------------- In order to set build machines as external machines, Create NFS share pointing to top level 'pginstaller' directory on Mac. For this purpose free tool 'NFS Manager' can be used. On linux side, update /etc/fstab to create nfs mount to this NFS share. Build scripts ------------- * settings.sh This script is derived from settings.sh.in which is stored in source control. It is configured for the specific build machine, and allows us to specify what platforms and modules we're building, and some global configuration options. This script (_and_ the source version, settings.h.in) must be edited whenever new platforms or packages are added. * common.sh This script contains common utility functions that may be used throughout the build system. * build.sh This script is the main build script. To build everything, simply run the following command on the build host: sh build.sh For quick rebuilds, an option is provided to rebuild just the installers from the existing code in the staging directories: sh build.sh -skipbuild This script must be edited whenever a new module is added to call the appropriate functions in the package build script. Directories ----------- * output/ This directory will contain all the completed installers. * scripts/ This directory contains miscellaneous scripts that may be useful to multiple modules or the overall build system. * resources/ This directory contains installer resources that may be useful to multiple modules or the overall build system. * tarballs/ This directory contains all the tarballs we use for builds * <everything else>/ Each additional directory contains a single package. These may be internally built as required, though the interface should remain consistent - ie. a single build script called build.sh, exposing functions called _prep_<packagename>, _build_< packagename > and _postprocess_< packagename >. For a description fo the build system for a single package, see server/README. Additional configuration in the VM's : -------------------------------------- * Adding gd module to php in Windows * Prequisites: 1) jpeg (http://nchc.dl.sourceforge.net/sourceforge/gnuwin32/jpeg-6b-4.exe) 2) libpng (http://nchc.dl.sourceforge.net/sourceforge/gnuwin32/libpng-1.2.36-setup.exe) 3) freetype (http://nchc.dl.sourceforge.net/sourceforge/gnuwin32/freetype-2.3.5-1-setup.exe) Install these in the pgBuild directory as jpeg, libpng and freetype respectively. * Modifications: Freetype: 1) Modify the directory structure as: freetype --> include --> freetype2 --> freetype to freetype --> include --> freetype (leave the ft2build.h file in include directory as it is.) 2) Copy the files: freetype/lib/freetype.lib to freetype/lib/freetype2.lib jpeg: 1) Copy the files: jpeg/lib/jpeg.lib to jpeg/lib/libjpeg.lib * Adding gd module to php in osx * Prequisites: 1) Install jpeg libraries Download and extract jpeg from http://www.ijg.org/ Compile and install: >env CFLAGS="-isysroot /Developer/SDKs/MacOSX10.5.sdk -mmacosx-version-min=10.5 -arch i386 -arch ppc -arch x86_64" LDFLAGS="-arch i386 -arch ppc -arch x86_64" ./configure --prefix=/usr/local --disable-dependency-tracking >make >sudo make install 2) Install libpng (User only 1.2.x version - php-5.2.1 has not yet include support for 1.4.x version) Download and extract libpng from http://www.libpng.org/pub/png/pngcode.html Compile and install: >env CFLAGS="-isysroot /Developer/SDKs/MacOSX10.5.sdk -mmacosx-version-min=10.5 -arch i386 -arch ppc -arch x86_64" LDFLAGS="-arch i386 -arch ppc -arch x86_64" ./configure --prefix=/usr/local --disable-dependency-tracking >make >sudo make install 3) Install freetype Download and extract freetype from http://freetype.org/download.html >env CFLAGS="-isysroot /Developer/SDKs/MacOSX10.5.sdk -mmacosx-version-min=10.5 -arch i386 -arch ppc -arch x86_64" LDFLAGS="-arch i386 -arch ppc -arch x86_64" ./configure --prefix=/usr/local --disable-dependency-tracking >make >sudo make install * Adding gd module to php in linux * Prequisites: (linux/linux-x64) 1) yum install freetype 2) yum install libpng (libjpeg.so should also be present in /usr/lib and /usr/lib64 for linux and linux-x64 respectively) * Install the latest version of ActiveState Python, Perl & TCL/Tk on all the platforms. * Install SPHINX for generating documentation for generating documentations for pgAdmin3. i.e. <PYTHONHOME>/bin/easy_install Sphinx For ActiveState Python 2.6, the PYTHONHOME is '/opt/ActivePython-2.6' For ActiveState Python 3.2, the PYTHONHOME is '/opt/ActivePython-3.2' NOTE: Install the SPHINX as the root user. Trouble-Shooting: * I got this error for ActivePython-3.2 on linux/linux-x64 ----------------------------------------------------------------- /opt/ActivePython-3.2/bin/python3.2 ActivePython 3.2.2.3 (ActiveState Software Inc.) based on Python 3.2.2 (default, Sep 8 2011, 12:20:28) [GCC 4.0.2 20051125 (Red Hat 4.0.2-8)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import hashlib; ERROR:root:code for hash md5 was not found. Traceback (most recent call last): File "/opt/ActivePython-3.2/lib/python3.2/hashlib.py", line 141, in <module> globals()[__func_name] = __get_hash(__func_name) File "/opt/ActivePython-3.2/lib/python3.2/hashlib.py", line 91, in __get_builtin_constructor raise ValueError('unsupported hash type %s' % name) ValueError: unsupported hash type md5 ERROR:root:code for hash sha1 was not found. Traceback (most recent call last): File "/opt/ActivePython-3.2/lib/python3.2/hashlib.py", line 141, in <module> globals()[__func_name] = __get_hash(__func_name) File "/opt/ActivePython-3.2/lib/python3.2/hashlib.py", line 91, in __get_builtin_constructor raise ValueError('unsupported hash type %s' % name) ValueError: unsupported hash type sha1 ERROR:root:code for hash sha224 was not found. Traceback (most recent call last): File "/opt/ActivePython-3.2/lib/python3.2/hashlib.py", line 141, in <module> globals()[__func_name] = __get_hash(__func_name) File "/opt/ActivePython-3.2/lib/python3.2/hashlib.py", line 91, in __get_builtin_constructor raise ValueError('unsupported hash type %s' % name) ValueError: unsupported hash type sha224 ERROR:root:code for hash sha256 was not found. Traceback (most recent call last): File "/opt/ActivePython-3.2/lib/python3.2/hashlib.py", line 141, in <module> globals()[__func_name] = __get_hash(__func_name) File "/opt/ActivePython-3.2/lib/python3.2/hashlib.py", line 91, in __get_builtin_constructor raise ValueError('unsupported hash type %s' % name) ----------------------------------------------------------------- In order to resolve the issue, I had to run the following command as 'root' user. chcon -t texrel_shlib_t /opt/ActivePython-3.2/lib/python3.2/lib-dynload/_hashlib.cpython-32m.so Further info ------------ Contact dpage@pgadmin.org for further info.