Support Alpine
wernight opened this issue · 98 comments
The direction is going towards using Docker, and on Docker the direction is towards using Alpine base images (because they're smaller), and most main packages now support Alpine (Nginx included).
It seems that it almost compiles on alpine:edge. I got past the traceback by installing libunwind@testing package but some missing references remain:
psol/lib/Release/linux/x64/pagespeed_automatic.a(1.time.o.o): In function `base::Time::FromDoubleT(dtime.cc:(.text._ZN4base4Time11FromDoubleTEd+0x2a): undefined reference to `__isnan'
psol/lib/Release/linux/x64/pagespeed_automatic.a(96.string_util.o.o): In function `net_instaweb::Str
string_util.cc:(.text._ZN12net_instaweb14StringToDoubleEPKcPd+0x14): undefined reference to `__strtod_internal'
pagespeed_automatic.a(132.ucnv.o.o): In function `pagespeed_ol_ucnv_conveucnv.c:(.text.ucnv_convertEx_46+0x6db): undefined reference to `__rawmemchr'
pagespeed_automatic.a(132.ucnv.o.o): In function `ucnv_internalConvert':
ucnv.c:(.text.ucnv_internalConvert+0x276): undefined reference to `__rawmemchr'
pagespeed_automatic.a(132.uloc.o.o): In function `pagespeed_ol_uloc_accep
uloc.c:(.text.uloc_acceptLanguageFromHTTP_46+0x214): undefined reference to `__strtod_internal'
uloc.c:(.text.uloc_acceptLanguageFromHTTP_46+0x452): undefined reference to `__strtod_internal'
pagespeed_automatic.a(132.uniset_props.o.o): In function `icu_46::Unicode
uniset_props.cpp:(.text._ZN6icu_4610UnicodeSet18applyPropertyAliasERKNS_13UnicodeStringES3_R10UErrorCode+0x35e): undefined reference to `__str
uniset_props.cpp:(.text._ZN6icu_4610UnicodeSet18applyPropertyAliasERKNS_13UnicodeStringES3_R10UErrorCode+0x462): undefined reference to `__str
It's worth finding how to make it compile and document it.
Just to make sure: are you building psol.a yourself?
Yes, I did (following mostly https://developers.google.com/speed/pagespeed/module/build_ngx_pagespeed_from_source#dependencies):
Dockerfile:
FROM alpine:edge
RUN set -x && \
echo "@testing http://dl-cdn.alpinelinux.org/alpine/edge/testing" >> /etc/apk/repositories && \
apk add --update ca-certificates curl g++ libunwind@testing && \
cd /tmp && \
NPS_VERSION=1.11.33.0 && \
curl -L https://github.com/pagespeed/ngx_pagespeed/archive/release-${NPS_VERSION}-beta.zip -o release-${NPS_VERSION}-beta.zip && \
unzip release-${NPS_VERSION}-beta.zip && \
cd ngx_pagespeed-release-${NPS_VERSION}-beta/ && \
curl -L https://dl.google.com/dl/page-speed/psol/${NPS_VERSION}.tar.gz | tar -xzv && \
cd /tmp && \
NGINX_VERSION=1.8.1 && \
curl -L http://nginx.org/download/nginx-${NGINX_VERSION}.tar.gz | tar -xvz && \
cd nginx-${NGINX_VERSION}/ && \
./configure --add-module=/tmp/ngx_pagespeed-release-${NPS_VERSION}-beta
Then looking at objs/autoconf.err for errors.
curl -L https://dl.google.com/dl/page-speed/psol/${NPS_VERSION}.tar.gz
| tar -xzv &&
^^^
This bit is grabbing psol.a as a binary, which is not really workable
with a non-glibc system.
Sadly you near certainly need to hand-build psol.a for your system the
longer and more complicated way:
https://github.com/pagespeed/ngx_pagespeed/wiki/Building-PSOL-From-Source
(That probably expects BRANCH to be set...)
On Fri, Apr 22, 2016 at 4:53 PM, Werner Beroux notifications@github.com
wrote:
Yes, I did (following mostly
https://developers.google.com/speed/pagespeed/module/build_ngx_pagespeed_from_source#dependencies
):FROM alpine:edge
RUN set -x &&
echo "@testing http://dl-cdn.alpinelinux.org/alpine/edge/testing" >> /etc/apk/repositories &&
apk add --update ca-certificates curl g++ libunwind@testing && \cd /tmp && \ NPS_VERSION=1.11.33.0 && \ curl -L https://github.com/pagespeed/ngx_pagespeed/archive/release-${NPS_VERSION}-beta.zip -o release-${NPS_VERSION}-beta.zip && \ unzip release-${NPS_VERSION}-beta.zip && \ cd ngx_pagespeed-release-${NPS_VERSION}-beta/ && \ curl -L https://dl.google.com/dl/page-speed/psol/${NPS_VERSION}.tar.gz | tar -xzv && \ cd /tmp && \ NGINX_VERSION=1.8.1 && \ curl -L http://nginx.org/download/nginx-${NGINX_VERSION}.tar.gz | tar -xvz && \ cd nginx-${NGINX_VERSION}/ && \ ./configure --add-module=/tmp/ngx_pagespeed-release-${NPS_VERSION}-betaThen looking at objs/autoconf.err for errors.
—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub
#1181 (comment)
Sounds legit. My point is that a documentation for Alpine would be very welcome because it's where I think the near future lives.
Ideally it'd also be package of Alpine.
@wernight Preliminary work to get this created as an Alpine package is available here https://github.com/cchamplin/alpine-ngx-pagespeed. There is still several things that need to happen before I'll start going through the process of submitting to the Alpine repos. Most importantly is that I need it tested on fresh alpine installs.
Ideally we'd also see psol updated at some point to use modern versions of libpng so that we don't have to package an ancient copy of libpng12.
To test it, just use docker run --rm -it alpine. You can specify which version of Alpine you want to run as well. That would be very fast. If you don't have Docker, simply run curl -fsSL https://get.docker.com/ | sh and that should be it.
This is super important to have in place! By the way, v3.4.0 is out1
Can't you just test on Docker? That should be simple. docker run --rm -it
alpine
Le mar. 31 mai 2016 à 20:56, Nikolay Kolev notifications@github.com a
écrit :
This is super important to have in place! By the way, v3.4.0 is out
http://alpinelinux.org/posts/Alpine-3.4.0-released.html1—
You are receiving this because you were mentioned.Reply to this email directly, view it on GitHub
#1181 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AAI5A829kZUevIS2oN72eEP6Dhd9nZMdks5qHIRsgaJpZM4IN6ge
.
I'm running https://github.com/cchamplin/alpine-ngx-pagespeed on fresh Alpine for some time now without real issues, however it seems to be not maintained.
I see 3 places that would potentially welcome PageSpeed for Nginx on Linux Alpine:
- PageSpeed itself: Document how to build
- Linux Alpine: Package
- Docker Nginx image (quite unlikely)
I agree it'd be easier if we could use a modern libpng.
I've a working updated Dockerfile that builds Nginx + PageSpeed if anyone is interested. It's IMO better formatted than wunderkraut/alpine-nginx-pagespeed, very slow to build, but working.
@wernight we (Wunderkraut) would like to work together with this task to get a working version out for everyone. We had a pause in our internal development with this but have now continued with this and would like to cooperate with the effort.
cc: @aleksijohansson
@wernight Very interested, please share!
FROM alpine:3.4
RUN apk --no-cache add \
ca-certificates \
libuuid \
apr \
apr-util \
libjpeg-turbo \
icu \
icu-libs \
openssl \
pcre \
zlib
RUN set -x && \
apk --no-cache add -t .build-deps \
apache2-dev \
apr-dev \
apr-util-dev \
build-base \
curl \
icu-dev \
libjpeg-turbo-dev \
linux-headers \
gperf \
openssl-dev \
pcre-dev \
python \
zlib-dev && \
# Build libpng:
# This sadly requires an old version of http://www.libpng.org/pub/png/libpng.html
LIBPNG_VERSION=1.2.56 && \
cd /tmp && \
curl -L http://prdownloads.sourceforge.net/libpng/libpng-${LIBPNG_VERSION}.tar.gz | tar -zx && \
cd /tmp/libpng-${LIBPNG_VERSION} && \
./configure --build=$CBUILD --host=$CHOST --prefix=/usr --enable-shared --with-libpng-compat && \
make install V=0 && \
# Build PageSpeed:
# Check https://github.com/pagespeed/ngx_pagespeed/releases for the latest version
PAGESPEED_VERSION=1.11.33.3 && \
cd /tmp && \
curl -L https://dl.google.com/dl/linux/mod-pagespeed/tar/beta/mod-pagespeed-beta-${PAGESPEED_VERSION}-r0.tar.bz2 | tar -jx && \
curl -L https://github.com/pagespeed/ngx_pagespeed/archive/v${PAGESPEED_VERSION}-beta.tar.gz | tar -zx && \
cd /tmp/modpagespeed-${PAGESPEED_VERSION} && \
curl -L https://raw.githubusercontent.com/iler/alpine-nginx-pagespeed/master/patches/automatic_makefile.patch | patch -p1 && \
curl -L https://raw.githubusercontent.com/iler/alpine-nginx-pagespeed/master/patches/libpng_cflags.patch | patch -p1 && \
curl -L https://raw.githubusercontent.com/iler/alpine-nginx-pagespeed/master/patches/pthread_nonrecursive_np.patch | patch -p1 && \
curl -L https://raw.githubusercontent.com/iler/alpine-nginx-pagespeed/master/patches/rename_c_symbols.patch | patch -p1 && \
curl -L https://raw.githubusercontent.com/iler/alpine-nginx-pagespeed/master/patches/stack_trace_posix.patch | patch -p1 && \
./generate.sh -D use_system_libs=1 -D _GLIBCXX_USE_CXX11_ABI=0 -D use_system_icu=1 && \
cd /tmp/modpagespeed-${PAGESPEED_VERSION}/src && \
make BUILDTYPE=Release CXXFLAGS=" -I/usr/include/apr-1 -I/tmp/libpng-${LIBPNG_VERSION} -fPIC -D_GLIBCXX_USE_CXX11_ABI=0" CFLAGS=" -I/usr/include/apr-1 -I/tmp/libpng-${LIBPNG_VERSION} -fPIC -D_GLIBCXX_USE_CXX11_ABI=0" && \
cd /tmp/modpagespeed-${PAGESPEED_VERSION}/src/pagespeed/automatic/ && \
make psol BUILDTYPE=Release CXXFLAGS=" -I/usr/include/apr-1 -I/tmp/libpng-${LIBPNG_VERSION} -fPIC -D_GLIBCXX_USE_CXX11_ABI=0" CFLAGS=" -I/usr/include/apr-1 -I/tmp/libpng-${LIBPNG_VERSION} -fPIC -D_GLIBCXX_USE_CXX11_ABI=0" && \
mkdir -p /tmp/ngx_pagespeed-${PAGESPEED_VERSION}-beta/psol && \
mkdir -p /tmp/ngx_pagespeed-${PAGESPEED_VERSION}-beta/psol/lib/Release/linux/x64 && \
mkdir -p /tmp/ngx_pagespeed-${PAGESPEED_VERSION}-beta/psol/include/out/Release && \
cp -r /tmp/modpagespeed-${PAGESPEED_VERSION}/src/out/Release/obj /tmp/ngx_pagespeed-${PAGESPEED_VERSION}-beta/psol/include/out/Release/ && \
cp -r /tmp/modpagespeed-${PAGESPEED_VERSION}/src/net /tmp/ngx_pagespeed-${PAGESPEED_VERSION}-beta/psol/include/ && \
cp -r /tmp/modpagespeed-${PAGESPEED_VERSION}/src/testing /tmp/ngx_pagespeed-${PAGESPEED_VERSION}-beta/psol/include/ && \
cp -r /tmp/modpagespeed-${PAGESPEED_VERSION}/src/pagespeed /tmp/ngx_pagespeed-${PAGESPEED_VERSION}-beta/psol/include/ && \
cp -r /tmp/modpagespeed-${PAGESPEED_VERSION}/src/third_party /tmp/ngx_pagespeed-${PAGESPEED_VERSION}-beta/psol/include/ && \
cp -r /tmp/modpagespeed-${PAGESPEED_VERSION}/src/tools /tmp/ngx_pagespeed-${PAGESPEED_VERSION}-beta/psol/include/ && \
cp -r /tmp/modpagespeed-${PAGESPEED_VERSION}/src/pagespeed/automatic/pagespeed_automatic.a /tmp/ngx_pagespeed-${PAGESPEED_VERSION}-beta/psol/lib/Release/linux/x64 && \
# Build Nginx with support for PageSpeed:
# Check http://nginx.org/en/download.html for the latest version.
NGINX_VERSION=1.10.1 && \
cd /tmp && \
curl -L http://nginx.org/download/nginx-${NGINX_VERSION}.tar.gz | tar -zx && \
cd /tmp/nginx-${NGINX_VERSION} && \
LD_LIBRARY_PATH=/tmp/modpagespeed-${PAGESPEED_VERSION}/usr/lib ./configure --with-ipv6 \
--prefix=/var/lib/nginx \
--sbin-path=/usr/sbin \
--modules-path=/usr/lib/nginx \
--with-http_ssl_module \
--with-http_gzip_static_module \
--with-file-aio \
--with-http_v2_module \
--without-http_autoindex_module \
--without-http_browser_module \
--without-http_geo_module \
--without-http_map_module \
--without-http_memcached_module \
--without-http_userid_module \
--without-mail_pop3_module \
--without-mail_imap_module \
--without-mail_smtp_module \
--without-http_split_clients_module \
--without-http_scgi_module \
--without-http_referer_module \
--without-http_upstream_ip_hash_module \
--prefix=/etc/nginx \
--conf-path=/etc/nginx/nginx.conf \
--http-log-path=/var/log/nginx/access.log \
--error-log-path=/var/log/nginx/error.log \
--pid-path=/var/run/nginx.pid \
--add-module=/tmp/ngx_pagespeed-${PAGESPEED_VERSION}-beta \
--with-cc-opt="-fPIC -I /usr/include/apr-1" \
--with-ld-opt="-luuid -lapr-1 -laprutil-1 -licudata -licuuc -L/tmp/modpagespeed-${PAGESPEED_VERSION}/usr/lib -lpng12 -lturbojpeg -ljpeg" && \
make install --silent && \
# Clean-up:
cd && \
apk del .build-deps && \
rm -rf /tmp/* && \
# forward request and error logs to docker log collector
ln -sf /dev/stdout /var/log/nginx/access.log && \
ln -sf /dev/stderr /var/log/nginx/error.log && \
# Make PageSpeed cache writabl:
mkdir -p /var/cache/ngx_pagespeed && \
chmod -R o+wr /var/cache/ngx_pagespeed
WORKDIR /srv
EXPOSE 80 443
CMD ["nginx", "-g", "daemon off;"]
@wernight Awesome, thanks! Seems to build successfully at least
@wernight's fix was great but now doesn't play nice with version 1.12.34.1 and spits out;
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 38.5M 100 38.5M 0 0 2933k 0 0:00:13 0:00:13 --:--:-- 2065k
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 140 0 140 0 0 47 0 --:--:-- 0:00:02 --:--:-- 325
100 115k 100 115k 0 0 19176 0 0:00:06 0:00:06 --:--:-- 158k
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1121 100 1121 0 0 414 0 0:00:02 0:00:02 --:--:-- 5690
patching file src/pagespeed/automatic/Makefile
Hunk 2 FAILED 170/168.
third_party/gflags/libgflags.a \
third_party/giflib/libdgiflib.a \
third_party/giflib/libgiflib_core.a \
- third_party/icu/libicudata.a \
- third_party/icu/libicuuc.a \
third_party/jsoncpp/libjsoncpp.a \
- third_party/libjpeg_turbo/libjpeg.a \
- third_party/libjpeg_turbo/src/libjpeg_turbo.a \
- third_party/libpng/libpng.a \
third_party/optipng/libopngreduc.a \
third_party/protobuf/libprotobuf_full_do_not_use.a \
third_party/re2/libre2.a \
- third_party/serf/libopenssl.a \
third_party/serf/libserf.a \
- third_party/zlib/libzlib.a \
url/liburl_lib.a
# The 'gclient' build flow uses 'xcodebuild' on Mac and 'make' on Linux.
@ciarans It might be that something has changed in the folder structure in version 1.12.34.1. I've fixed it once before in https://github.com/wunderkraut/image-fuzzy-alpine-nginx-pagespeed, but haven't updated that to the most up to date version yet.
@aleksijohansson thanks for giving me some feedback. I would love to give debugging a the problem myself but my patch knowledge is very basic.
@aleksijohansson after some research it looks like the following file has is causing the problem -
https://raw.githubusercontent.com/wunderkraut/alpine-nginx-pagespeed/master/patches/automatic_makefile.patch
@wernight @aleksijohansson from the work you guys did earlier I've managed to generate the following;
--- a/src/pagespeed/automatic/Makefile
+++ b/src/pagespeed/automatic/Makefile
@@ -145,8 +145,6 @@
pagespeed/libpagespeed_thread.a \
pagespeed/libpthread_system.a \
pagespeed/libutil.a \
- third_party/apr/libapr.a \
- third_party/aprutil/libaprutil.a \
third_party/base64/libbase64.a \
third_party/chromium/src/base/third_party/dynamic_annotations/libdynamic_annotations.a \
third_party/css_parser/libcss_parser.a \
@@ -158,22 +156,11 @@
third_party/gflags/libgflags.a \
third_party/giflib/libdgiflib.a \
third_party/giflib/libgiflib_core.a \
- third_party/grpc/libgpr.a \
- third_party/grpc/libgrpc_core.a \
- third_party/grpc/libgrpc_cpp.a \
- third_party/hiredis/libhiredis.a \
- third_party/icu/libicudata.a \
- third_party/icu/libicuuc.a \
third_party/jsoncpp/libjsoncpp.a \
- third_party/libjpeg_turbo/libjpeg.a \
- third_party/libjpeg_turbo/src/libjpeg_turbo.a \
- third_party/libpng/libpng.a \
third_party/optipng/libopngreduc.a \
third_party/protobuf/libprotobuf_full_do_not_use.a \
third_party/re2/libre2.a \
- third_party/serf/libopenssl.a \
third_party/serf/libserf.a \
- third_party/zlib/libzlib.a \
url/liburl_lib.a
# The 'gclient' build flow uses 'xcodebuild' on Mac and 'make' on Linux.
However during the build Im now getting this;
CC(target) out/Release/obj.target/grpc_core/third_party/grpc/src/src/core/lib/surface/init_secure.o
<command-line>:0:0: warning: "_FORTIFY_SOURCE" redefined
<built-in>: note: this is the location of the previous definition
In file included from third_party/grpc/src/src/core/lib/tsi/transport_security_interface.h:38:0,
from third_party/grpc/src/src/core/lib/security/transport/security_connector.h:40,
from third_party/grpc/src/src/core/lib/security/credentials/credentials.h:45,
from third_party/grpc/src/src/core/lib/surface/init_secure.c:40:
/usr/include/fortify/stdlib.h: In function 'realpath':
/usr/include/fortify/stdlib.h:42:2: error: #error PATH_MAX unset. A fortified realpath will not work.
#error PATH_MAX unset. A fortified realpath will not work.
^
third_party/grpc/grpc_core.target.mk:488: recipe for target 'out/Release/obj.target/grpc_core/third_party/grpc/src/src/core/lib/surface/init_secure.o' failed
make: *** [out/Release/obj.target/grpc_core/third_party/grpc/src/src/core/lib/surface/init_secure.o] Error 1
My research shows its to do with a limits.h file not being set with PATH_MAX.
Have you come across this before? Any ideas on how to get over this?
@ciarans Haven't seen that before. When I have time to update https://github.com/wunderkraut/image-fuzzy-alpine-nginx-pagespeed I'll let you know how it goes.
I created a automated build of ngx_pagespeed on Alpine.
+1 for better alpine support without having to patch all the things!
Since Google is moving other Docker images to Alpine, like their Cloud SDK, they should be supporting Alpine more broadly. Most of the hacks have fixed PS to an older version. +1 to get this done right.
FYI, yesterday Alpine 3.6.0 got released.
I tried to upgrade and got the same issue with __strcpy_chk and similar. It seems that it's because glibc on Alpine is very limited and doesn't implement those. I wonder if there isn't a flag to disable those (really not ideal but better than having an old version). A better option would be to use strcpy_s and such (see https://stackoverflow.com/a/25811493/167897) which might be on Alpine.
I tried to disable Fortify source using --with-cc-opt='-U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0' but that didn't help. Also tried CFLAGS='-U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=0'.
Also, see apache/incubator-pagespeed-mod#1460, tried to build mod_pagespeed 1.12.34.2 on Alpine 3.6. AFAIK, old versions (<1.4) of grpc have problems with Alpine 3.6
Honestly I think this plugin overall improves almost any website and thus it's good, but running a Debian image is huge (meaning also slower to build and deploy), so there should be an official ngx_pagespeed-alpine Docker image. It's very easy to set up but maintenance is an on-going task. It's like continuous integration. If not an image, an Alpine package would also do but I think that's actually harder and longer, and possibly less practical.
It's 2017! People don't use fat distros for Docker containers and, thus far, Alpine Linux is the standard - even Docker use it themselves, all major projects support musl libc, so, this issue is of a great importance!
It is even the standard for Google going forward.
Added this to the priority list: https://github.com/pagespeed/mod_pagespeed/wiki/Work-Prioritization
@wernight do you have any changes you'd like to be considered for merging into mod_pagespeed / ngx_pagespeed?
Also, if I understand correctly, upgrading grpc to 1.4.x would be helpful for building on Alpine, right?
I can look into that. If upgrading grpc to 1.4.x turns out to be hard, an alternative option could be to automatically axe out grpc from the build on Alpine. For now it's probably ok to run without grpc's only use in the project: the experimental central controller process.
Very good to see some progress on this issue! 👍
@oschaaf Could merge the Dockerfile but outside of it, the previous version required some patches to work on Alpine. I don't know about the current if it still does or not.
The most important IMO is to get either:
- A Dockerized image of Nginx PageSpeed based on Alpine, maintained by Google, which is as simple as having an automated Docker Hub build from a GitHub repo (which is all free by the way).
- A Nginx PageSpeed package on Alpine maintained by Google (probably harder as it'd conflict with Nginx and would be harder to maintain and upgrade).
From doing that, and maintaining it, you'd see where things could be improved. These are however secondary.
Hi -- I want to quickly point out that this project has committers at Google and We-Amp so far, and we welcome more. In particular, it would be great to have contributors & maintainers who are motivated in supporting a platform. This has been the model for OpenSUSE and other environments.
See also https://groups.google.com/forum/#!topic/mod-pagespeed-discuss/SYx1nOS6HsE for the transition from having this be a purely Google-controlled project to one that's more community developed.
We'd love to work with you and would be happy to look at pull-requests for any compatibility or script tweaks you need.
is it safe to assume that mod-pagespeed on alpine/musl is dead?
@james-nesbitt
Support for this is pretty high on the priority list [1] but as far as I know, nobody has had time to start looking into it - yet.
[1] https://github.com/pagespeed/mod_pagespeed/wiki/Work-Prioritization
Update: @ashishk-1 just landed a PR with an upgrade for the GRPC dependency in mod_pagespeed (to GRPC 1.4.5): apache/incubator-pagespeed-mod#1640
I have updated dockerfile and patches to support alpine 3.4 with ngx-pagespeed.
Please find it here https://github.com/We-Amp/ngx-pagespeed-alpine.
As of now, I have created custom release tarball of latest mod-pagespeed codebase which includes the upgraded GRPC version 1.4.5 ( will replace it with official mod-pagespeed release tarball once it is available and published).
For alpine 3.6 with ngx-pagespeed, please refer https://github.com/ashishk-1/ngx-pagespeed-alpine/tree/ashishk-dockerfile-36 branch. We will be tagging dockerfiles for alpine 3.4 and 3.6 once the pagespeed binaries are released(with upgraded GRPC version 1.4.5) and pagespeed links are updated.
Any progress on this?
I try to use your 36 branch, but it fails with
url/url_canon_icu.cc: In member function 'virtual void url_canon::ICUCharsetConverter::ConvertFromUTF16(const char16*, int, url_canon::CanonOutput*)':
url/url_canon_icu.cc:133:67: error: invalid conversion from 'const char16* {aka const short unsigned int*}' to 'const UChar* {aka const char16_t*}' [-fpermissive]
input, input_len, &err);
^
In file included from /usr/include/unicode/platform.h:25:0,
from /usr/include/unicode/ptypes.h:52,
from /usr/include/unicode/umachine.h:46,
from /usr/include/unicode/utypes.h:38,
from /usr/include/unicode/ucnv_err.h:88,
from ./third_party/icu/source/common/unicode/ucnv.h:50,
from url/url_canon_icu.cc:12:
./third_party/icu/source/common/unicode/ucnv.h:1197:1: note: initializing argument 4 of 'int32_t ucnv_fromUChars_59(UConverter*, char*, int32_t, const UChar*, int32_t, UErrorCode*)'
ucnv_fromUChars(UConverter *cnv,
^
url/url_canon_icu.cc: In function 'bool url_canon::IDNToASCII(const char16*, int, url_canon::CanonOutputW*)':
url/url_canon_icu.cc:173:74: error: invalid conversion from 'const char16* {aka const short unsigned int*}' to 'const UChar* {aka const char16_t*}' [-fpermissive]
output->capacity(), &info, &err);
^
In file included from /usr/include/unicode/platform.h:25:0,
from /usr/include/unicode/ptypes.h:52,
from /usr/include/unicode/umachine.h:46,
from /usr/include/unicode/utypes.h:38,
from /usr/include/unicode/ucnv_err.h:88,
from ./third_party/icu/source/common/unicode/ucnv.h:50,
from url/url_canon_icu.cc:12:
./third_party/icu/source/common/unicode/uidna.h:268:1: note: initializing argument 2 of 'int32_t uidna_nameToASCII_59(const UIDNA*, const UChar*, int32_t, UChar*, int32_t, UIDNAInfo*, UErrorCode*)'
uidna_nameToASCII(const UIDNA *idna,
^
url/url_canon_icu.cc:172:76: error: invalid conversion from 'short unsigned int*' to 'UChar* {aka char16_t*}' [-fpermissive]
int output_length = uidna_nameToASCII(uidna, src, src_len, output->data(),
~~~~~~~~~~~~^~
In file included from /usr/include/unicode/platform.h:25:0,
from /usr/include/unicode/ptypes.h:52,
from /usr/include/unicode/umachine.h:46,
from /usr/include/unicode/utypes.h:38,
from /usr/include/unicode/ucnv_err.h:88,
from ./third_party/icu/source/common/unicode/ucnv.h:50,
from url/url_canon_icu.cc:12:
./third_party/icu/source/common/unicode/uidna.h:268:1: note: initializing argument 4 of 'int32_t uidna_nameToASCII_59(const UIDNA*, const UChar*, int32_t, UChar*, int32_t, UIDNAInfo*, UErrorCode*)'
uidna_nameToASCII(const UIDNA *idna,
^
make: *** [url/url_lib.target.mk:332: out/Release/obj.target/url_lib/url/url_canon_icu.o] Error 1
make: *** Waiting for unfinished jobs....
The fix mentioned here doesn't seem to work.
Any suggestion?
Nevermind, I was using 3.7. Switching back 3.6 fixed it.
Although it'd be great if the official image could work with 3.7 too.
@hgl I fixed compiling in Alpine 3.7... look at https://github.com/nberlee/ngx-pagespeed-alpine/tree/3.7
@nberlee : When I try to build your dockerfile I get :
adding module in /tmp/ngx_pagespeed-1.12.34.3-stable
./configure: error: no /tmp/ngx_pagespeed-1.12.34.3-stable/config was found
@ashishk-1 : When I try to build your dockerfile I get :
adding module in /tmp/ngx_pagespeed-1.12.34.3-stable
./configure: error: no /tmp/ngx_pagespeed-1.12.34.3-stable/config was found
@nberlee Thanks for sharing. But I got the same error too.
configuring additional modules
adding module in /tmp/ngx_pagespeed-1.12.34.3-stable
./configure: error: no /tmp/ngx_pagespeed-1.12.34.3-stable/config was found
I am not sure yet, but it seems to be related to the nginx version. 1.12.2 seems to be working... 1.13.3 seems to be not. Can anyone confirm ? I am still testing it now
After some digging, I found the root cause to be that this project just got moved to apache/incubator-pagespeed-ngx, so after downloading the archive file, it extracts to /tmp/incubator-pagespeed-ngx-1.12.34.3-stable, and eventually /tmp/ngx_pagespeed-1.12.34.3-stable only contains psol.
I've successfully compiled on 3.7 after correcting the folder name.
I expect most derivatives of wernight/docker-alpine-nginx-pagespeed to fail to build with the same error currently.
@hgl.... AWESOME.. yes I was very frustrated that even my backup files did not work anymore...
So I guess you are right
I am a pretty newbie... can you share your changed code ?
With regard to the renaming of the repo, e733a83 was merged yesterday to unbreak the install script.
(The repo may be renamed again at some stage later on, and this change hopefully is resilient to that)
@oschaaf Thanks for the head up. But it seems Dockerfiles from most unofficial alpine images don't use this script. So they will continue to fail to build until fixed.
Another reason to have an official nginx + pagespeed alpine image. :)
Yes, that script is never used in the Dockerfile
But since the name is tentative, that mv command could easily fail when that happens. A better approach:
mkdir -p /tmp/ngx_pagespeed-${PAGESPEED_VERSION}-stable \
&& cd /tmp/ngx_pagespeed-${PAGESPEED_VERSION}-stable \
&& curl -L https://github.com/pagespeed/ngx_pagespeed/archive/v${PAGESPEED_VERSION}-${PAGESPEED_BRANCH}.tar.gz \
| tar -xj --strip-components=1
You do have to install tar though, because busybox tar doesn't support strip-components
@hgl The sed line was a workaround for a Alpine 3.7 busybox wait bug. (which never stopped waiting) This seems to be fixed since, making the line now obsolete.
Fixed my branch
@nberlee
Is there intentionally the hardcoded version in
https://github.com/We-Amp/ngx-pagespeed-alpine/blob/master/mod-pagespeed-beta-1.12.34.3.tar.
@robov Currently yes, once a new mod-pagespeed stable (1.12.34.3) is released, with @ashishk-1 commit: apache/incubator-pagespeed-mod@faf212c this can use the ${PAGESPEED_VERSION} with an official url. For now we use this specific version because its the only one with an updated GRPC and therefor building on Alpine >3.5. Also its the only version in the ngx-pagespeed-alpine branch on oschaaf his We-Amp repository.
I would definately be interested, but I am not bringing much to this table. My experience in linux is very low and I consider myself a newbie in this space
@nberlee : I am trying to understand your code, especially this part confuses me : LD_LIBRARY_PATH=/tmp/modpagespeed-${PAGESPEED_VERSION}/usr/lib ./configure \
When I look into the image I cannot find that path ..usr/lib , am I missing something ?
What I am trying to accomplish is to create an pagespeed object in a separate dockerbuild, then copy the needed files out and into a new dockerfile and build the nginx in a that new dockerfile
This reduces my compiletime of my main image dramatically (as I do not have to recompile pagespeed every time). But I seem to be not able to find the references to : /tmp/modpagespeed-${PAGESPEED_VERSION}/usr/lib
Or is this a temp direcory generated by the nginx compilation ?
Holy Smoly... my bad... I was looking at the wrong branch... I was looking at 3.4 instead of 3.7 for this last post
Is there someone who can help me out ?
I want to build (tedious long) pagespeed in a separate container and include only the necessary files in the main build.
This is what I did
I used the : nberlee/ngx-pagespeed-alpine/3.7/Dockerfile
A) Created a new dockerfile with the code up to : # Build Nginx with support for PageSpeed:
B) Extracted from that container : /tmp/ngx_pagespeed-${PAGESPEED_VERSION}-stable
- Created a new docker file from # Build Nginx with support for PageSpeed:
- Copied the extracted files IN this container
Result :
Compiling breaks with : with-ld-opt= ... not found .. but it does not mention what is not found
I guessed the libpng was missing (only present in the temp build container, so I added that to the main one. But hten the compiling broke on "missing pagespeed optimization library"
@robov I made a multi-stage Dockerfile see https://github.com/nberlee/ngx-pagespeed-alpine/tree/3.7-dev but haven't tested it yet! It should build fine
@oschaaf Yes I think I would like to be a maintainer, I think there is still some work to be done, like verifying sigatures of download files, but an official dockerhub image would be great!
Step 8/26 : COPY mod-pagespeed-beta-1.12.34.3.tar.bz2 patches/*.patch /tmp/
COPY failed: stat /var/lib/docker/tmp/docker-builder857321033/mod-pagespeed-beta-1.12.34.3.tar.bz2: no such file or directory
oops... my bad... forgot to pull your entire sources
@nberlee it compiles nice (I have not tested the actual functionality, just compiling)
I have a personal preference to have separate files instead of one multistage (this allows me to re-compile just one section) . One thing I am currently struggling with is to copy : /usr/lib/libpng*
out of the container (as they contain symlinks) Any ideas ?
When I do something like this on a container : docker cp extractfromcontainer:/usr/lib/libpng* compiled/usr/lib
then it cannot find anything... when I step in I see that they are symlinks
aha,.... and when copying the entire folder it complains ( i think it was about the symlinks)
After 5 days of a very steep learning curve I am finally getting somewhere... @nberlee ,
FROM on you multistage image I create my own derivative which I then start as a container.
however, now the nginx complains that 'modsecurity' is an unknown directive.
I linked this in in the nginx compiling step, but when starting it up, it is not recognized. Am I missing a copy somewhere
It is linked in like this and it compiles well... any ideas ?
--add-module=${WRKDIR}/modsecurity/nginx/modsecurity \
--add-module=${WRKDIR}/ngx_cache_purge-${CACHE_PURGE} \
--add-module=/tmp/ngx_pagespeed\
Everything is working now.... BUT I receive a very wierd error
On loading the page everything goes well
on RE-Loading the page with ctrl-f5 I get a 404 (just pressing f5 again and it loads nicely again)
ANY ideas where this problem originates or how to fix it ?
I have setup pagespeed on my proxy that passes the requests to the backend app server. The 404 error is generated from the app server. I think the request is looking for the pagespeed cache in the backend (where it isn;t)... but.. how to prevent that ....is the next question
I just pushed https://github.com/We-Amp/ngx-pagespeed-alpine/tree/dev
- Pagespeed 1.13.35.2 (mod and ngx)
- Builds on Alpine 3.7 and edge
- Works with Nginx 1.12.2 and 1.13.8
- Drop in replacement for nginx:alpine image. It should be exactly the same experience, making migration seamless. Butthis one contains a newer LibreSSL, Alpine, and of course Pagespeed.
@nberlee great thing! do you have an idea how to convince pagespeed to link libpng statically?
In my case, I get trouble with other libraries requiring a newer version later
@tobsch Are you having problems with my Dockerfile as-is? If this is the case I would like to know.
If you only want libpng 1.6 (which i would understand), @ashishk-1 made a commit recently in incubator-pagespeed-mod:master which would make it possible. You can use patch (I haven't tested it) https://github.com/apache/incubator-pagespeed-mod/commit/7ea8d5fc141c7067c54ffa2b9fbc552c15089ca6.patch (just save it with the other patches in my branch)
Make sure to add libpng-dev in the apk add section of modpagespeed
This patch will not be incorporated in the Dockerfile sine its not part of 1.13.35.2 and I am not backporting when its not absolutely necessary
@nberlee it's not "as it is" but in combination with php & such.
thanks a lot for the png patch tip: works out, after modifying the patch a bit (does not work with 1.13.35-2). Much better than a static version of libpng12!
@oschaaf here you go: https://gist.github.com/tobsch/c59b0c9b89f51a412a57c2bf8a4cbb31
I had to change the location of zlib.h, as it is now a submodule.
curious if there has been any progress. seems like things were close but got stuck.
https://github.com/We-Amp/ngx-pagespeed-alpine
All changes of dev are now merged in master. I was running dev in production for some time.... (also with nginx 1.14.0
I will contact @oschaaf to discuss to create something on Dockerhub..
@nberlee can you ping me at oschaaf@we-amp.com?
Any update? Also, will these changes be packaged into the base ngx build or is the model to provide a docker image with patches? Would prefer a stable ngx alpine build without docker wrapper.
@nberlee so the solution is to use the actual docker build and all the associated patches from https://github.com/We-Amp/ngx-pagespeed-alpine rather than getting a ngx_pagespeed that can be used outside of that setup? In the case of someone having their own nginx docker build process or using an alternate project, we will need to disassemble the docker file and assets from We-Amp to understand how to get this functional. Am understanding that correctly?
@tspicer Your aim was to have an apk in the alpine store if I read your comment correctly?
I think that would not be to hard to create or get it accepted in the alpine-community repo, so feel free to do so. It is not uncommon for packages in the alpine repository to have patches because of musl. I don't expect from the ngx_pagespeed to incorporate the patches, but feel free to create PR's to ngx_pagespeed ot make it possible to build it by default on alpine without sacrificing libc distributions.
I am sorry I raised your expectation I would make or maintain such a thing. However, as docker is build on layers you can easily add stuff to the compiled docker image.
FROM pagespeed/nginx-pagespeed:stable
apk add --no-cache <my-packages>
@tspicer @nberlee So our initial goals with the patches was to incorporate them into mod_pagespeed, but priorities changed and we didn't wrap up. But actually some of the more complex changes already actually landed to make building possible/easier for Alpine (& FreeBSD) (e.g: upgrades for grpc / libpng).
Looking at them, the remaining patches seem pretty small and mundane. One of them would have to be upstreamed to Chromium, the others can probably be merged into mod_pagespeed without any problems --- except for the Makefile patch, which may need some consideration. I can't promise any timelines, but I'll see what I can do, hopefully in the next couple of weeks, to get the required level of patching for Alpine down to 0. One concern I have going forward though is that we Alpine is not tested in CI.
@nberlee not the APK store, but just being able to compile as I would for another OS. At the moment the process to install is tightly coupled into a specific Dockerfile. If you are compiling from nginx source with alternate modules/packages and libraries, your choice is to 1) copy the docker file as provided and modify the nginx build as applicable, then customize any other parts that are warranted or 2) deconstruct the Dockerfile, grab the patches and attempt to incorporate into your own build.
So no expectations were set by you :)
I just want to make sure what you have referenced is the fix and in basically closes this issue.
thanks @oschaaf for clarifying
PSOL compilation takes way too much time (~35 min on travis), so I moved it out to a separate repository before the official release become available and just download the archive (100M of objects required for nginx module compilation) during nginx build, see example. Useful if you maintain your own alpine-based nginx image and want to add pagespeed module.
@csandanov If you would implement docker layer caching in travis (which is ok, with a tagged versions in a multi-stage docker file), builds would fly if you change something on the nginx.