dimitri/pgloader

Unable to install pgloader on macOS (Big Sur, M1)

Wilson13 opened this issue · 37 comments

Wilson13 commented

Hi,

I was trying to install pgloader via this command:
brew install --HEAD pgloader

This is the output and error reported:

==> Cloning https://github.com/dimitri/pgloader.git
Updating /Users/wilson/Library/Caches/Homebrew/pgloader--git
==> Checking out branch master
Already on 'master'
Your branch is up to date with 'origin/master'.
HEAD is now at 3047c9a Typo fix in MS SQL documentation (#1242)
==> Patching
Error: Failure while executing; `patch -g 0 -f -p1` exited with 1. Here's the output:
can't find file to patch at input line 3
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|--- a/local-projects/cl-csv/parser.lisp
|+++ b/local-projects/cl-csv/parser.lisp
--------------------------
No file to patch.  Skipping patch.
1 out of 1 hunk ignored

Environment

MacBook Pro (13-inch, M1, 2020)
macOS Big Sur v11.5.2 (20G95)

scrobot commented

The same problem. I've caught exception when I tried to install pgloader on my mac(M1) with brew

brew install pgloader
Error: pgloader: no bottle available!
You can try to install from source with:
  brew install --build-from-source pgloader
Please note building from source is unsupported. You will encounter build
failures with some formulae. If you experience any issues please create pull
requests instead of asking for help on Homebrew's GitHub, Twitter or any other
official channels.

And also when I tried to build it from sources

brew install --build-from-source pgloader
==> Downloading https://ghcr.io/v2/homebrew/core/sbcl/manifests/2.1.8
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/sbcl/blobs/sha256:df642cb657479a6ae84a6adfc22cafbcb876542fb9b69527b4ca2755ffc15e25
==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:df642cb657479a6ae84a6adfc22cafbcb876542fb9b69527b4ca2755ffc15e25?se=2021-09-01T11%3A05%3A00Z&sig=PRsr08H9%2BDUlBx%2FU63PTpQiPMQgbju3I
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/buildapp/manifests/1.5.6_3
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/buildapp/blobs/sha256:3a4d0ac54096f853b8b271e02369dd2666e5b69c975d3b3f18ae6e54b4adf966
==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:3a4d0ac54096f853b8b271e02369dd2666e5b69c975d3b3f18ae6e54b4adf966?se=2021-09-01T11%3A05%3A00Z&sig=9VlHNiuNx%2Bbb5%2BRioAZYiIjflXOfsV5O
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/m4/manifests/1.4.19
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/m4/blobs/sha256:ea1be04e51645f9e31c8d2cab6d144bb7f47efb1f35214b9f1014e27db5a3bc1
==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:ea1be04e51645f9e31c8d2cab6d144bb7f47efb1f35214b9f1014e27db5a3bc1?se=2021-09-01T11%3A05%3A00Z&sig=hVtRGKxtmCfglpaOvqRC1XKyMvvGHqMgGFPI
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/libtool/manifests/2.4.6_4
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/libtool/blobs/sha256:a41a4872cdfaa34bb4723e728b73dd8c7a05725501a262bb41ad9af4e2fcd1d6
==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:a41a4872cdfaa34bb4723e728b73dd8c7a05725501a262bb41ad9af4e2fcd1d6?se=2021-09-01T11%3A05%3A00Z&sig=WaStvjRIFMgQGjKclvKcX169YFmx8IHSjoXv
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/unixodbc/manifests/2.3.9_1
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/unixodbc/blobs/sha256:66e4b186a19526e02782557afe6926d2cfb9f372e94cbcc387f531b122f510e0
==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:66e4b186a19526e02782557afe6926d2cfb9f372e94cbcc387f531b122f510e0?se=2021-09-01T11%3A05%3A00Z&sig=8gpUHK35tO%2Ba0RMi88ZmjufjVxLHHJIpfS
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/freetds/manifests/1.3.3
######################################################################## 100.0%
==> Downloading https://ghcr.io/v2/homebrew/core/freetds/blobs/sha256:95f5ec9318ef76fc7fd19e1e4bfbb651ce0f70c0a445734ea2eb52c9431b5300
==> Downloading from https://pkg-containers.githubusercontent.com/ghcr1/blobs/sha256:95f5ec9318ef76fc7fd19e1e4bfbb651ce0f70c0a445734ea2eb52c9431b5300?se=2021-09-01T11%3A05%3A00Z&sig=cfKiL1UHLyRHRYB7UKJtRwIqVOYKN9A%2FdW
######################################################################## 100.0%
==> Downloading https://github.com/dimitri/pgloader/releases/download/v3.6.2/pgloader-bundle-3.6.2.tgz
==> Downloading from https://github-releases.githubusercontent.com/594087/a86b1500-6c8d-11ea-86c4-5dd3e9eae544?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20210901%2Fus-east-1%2Fs3%2Faws4_r
######################################################################## 100.0%
==> Installing dependencies for pgloader: sbcl, buildapp, m4, libtool, unixodbc and freetds
==> Installing pgloader dependency: sbcl
==> Pouring sbcl--2.1.8.arm64_big_sur.bottle.tar.gz
🍺  /opt/homebrew/Cellar/sbcl/2.1.8: 1,325 files, 60MB
==> Installing pgloader dependency: buildapp
==> Pouring buildapp--1.5.6_3.arm64_big_sur.bottle.tar.gz
🍺  /opt/homebrew/Cellar/buildapp/1.5.6_3: 4 files, 39.9MB
==> Installing pgloader dependency: m4
==> Pouring m4--1.4.19.arm64_big_sur.bottle.tar.gz
🍺  /opt/homebrew/Cellar/m4/1.4.19: 13 files, 726.3KB
==> Installing pgloader dependency: libtool
==> Pouring libtool--2.4.6_4.arm64_big_sur.bottle.tar.gz
🍺  /opt/homebrew/Cellar/libtool/2.4.6_4: 75 files, 3.7MB
==> Installing pgloader dependency: unixodbc
==> Pouring unixodbc--2.3.9_1.arm64_big_sur.bottle.tar.gz
🍺  /opt/homebrew/Cellar/unixodbc/2.3.9_1: 46 files, 2.3MB
==> Installing pgloader dependency: freetds
==> Pouring freetds--1.3.3.arm64_big_sur.bottle.tar.gz
🍺  /opt/homebrew/Cellar/freetds/1.3.3: 1,262 files, 14.2MB
==> Installing pgloader
==> Patching
==> make
Last 15 lines from /Users/aleksejskrobot/Library/Logs/Homebrew/pgloader/01.make:
                         --load local-projects/pgloader-3.6.2/src/hooks.lisp              \
                         --load-system pgloader                    \
                         --eval '(setf pgloader.params::*version-string* "3.6.2")' \
                         --entry pgloader:main                        \
                         --dynamic-space-size 4096              \
                         --compress-core                         \
                         --output bin/pgloader.tmp
;; loading system "cffi"
;; loading system "cl+ssl"
WARNING: /opt/homebrew/Cellar/sbcl/2.1.8/libexec/bin/sbcl is loading libcrypto in an unsafe way
fatal error encountered in SBCL pid 13689 pthread 0x1002d7d40:
SIGABRT received.

   0:     0x16fd43d18 pc=0x18cccce60
make: *** [bin/pgloader] Error 1

Do not report this issue to Homebrew/brew or Homebrew/core!
made-indrayana commented

I have the same problem with @Wilson13, I'm on Bigs Sur 11.6 but on Intel i7.

jhult commented

Same issue:

  • macOS: 12.0 Beta (21A5506j)
  • Intel i7
csadai commented

Same problem with Big Sur (11.5) and Apple M1 chip

erika-e commented

I don't get the exact same error. I have Rosetta2 enabled on all of my terminal windows and installed brew from a Rosetta2 terminal.

I can run brew install pgloader but trying to run any pgloader commands e.g. pgloader --help results in the libcrytpo error message:

WARNING: /usr/local/bin/pgloader is loading libcrypto in an unsafe way
fatal error encountered in SBCL pid 53056 pthread 0x1006fb600:
SIGABRT received.

Anyone know of a workaround? I am on an M1 chip and Monterey.

  • 👍6
chrise86 commented

I don't get the exact same error. I have Rosetta2 enabled on all of my terminal windows and installed brew from a Rosetta2 terminal.

I can run brew install pgloader but trying to run any pgloader commands e.g. pgloader --help results in the libcrytpo error message:

WARNING: /usr/local/bin/pgloader is loading libcrypto in an unsafe way
fatal error encountered in SBCL pid 53056 pthread 0x1006fb600:
SIGABRT received.

Anyone know of a workaround? I am on an M1 chip and Monterey.

Also getting this on a regular Intel based Mac mini.

  • 👍4
ScottSmudger commented

Same issue

brew install pgloader
brew install --HEAD pgloader
and building from source doesnt work

Monterey 12.0.1 / Intel i7

Any workarounds or fixes?

zmitja commented

This fixes the issue:

brew install openssl
sudo ln -s /opt/homebrew/Cellar/openssl@3/3.0.0_1/lib/libcrypto.dylib /usr/local/lib/libcrypto.dylib
sudo ln -s /opt/homebrew/Cellar/openssl@3/3.0.0_1/lib/libssl.dylib /usr/local/lib/libssl.dylib

Found a solution here:
cl-plus-ssl/cl-plus-ssl#114

Note, I already had pgloader installed and working, upgrade to Monterey broke it, so I had the issue with running pgloader, not installing it, but I think it's the same issue so this fix should fix the installation problem as well.

tuanalumi commented

I am on an Intel, and I have to use Docker version to overcome this.

  • 👍2
yveshield commented

brew install openssl@3
brew link --force openssl@3

then its works again

nbsoftware commented

Still not working here.

Had first to update the links to the current version of openssl:

sudo ln -s /opt/homebrew/Cellar/openssl@3/3.0.1/lib/libcrypto.dylib /usr/local/lib/libcrypto.dylib
sudo ln -s /opt/homebrew/Cellar/openssl@3/3.0.1/lib/libssl.dylib /usr/local/lib/libssl.dylib

as well as

sudo ln -s /opt/homebrew/lib/libsybdb.dylib /usr/lib/libsybdb.dylib

but then it hung shortly after:

==> Patching
==> make
Last 15 lines from /Users/fmeyer/Library/Logs/Homebrew/pgloader/01.make:
;; loading system "cffi"
;; loading system "cl+ssl"
;; loading system "mssql"
;; loading file #P"/private/tmp/pgloader-20211223-23208-t6qbpi/pgloader-bundle-3.6.2/local-projects/pgloader-3.6.2/src/hooks.lisp"
;; loading system "pgloader"
Fatal TYPE-ERROR:
  The values
    (#<FUNCTION (LAMBDA (ESRAP::PRODUCTION)
                  :IN
                  "/private/tmp/pgloader-20211223-23208-t6qbpi/pgloader-bundle-3.6.2/.brew_home/.cache/common-lisp/sbcl-2.1.9-macosx-arm64/private/tmp/pgloader-20211223-23208-t6qbpi/pgloader-bundle-3.6.2/software/cl-abnf-20190521-git/abnf.fasl") {7008B7C14B}>
     NIL NIL)

  are not of type
    (VALUES FUNCTION &OPTIONAL)
make: *** [bin/pgloader] Error 1

This is where I'm stuck.
Any ideas?

Meanwhile I'll give Ubuntu arm64 a try...

yveshield commented

@nbsoftware

I'm wondering if you completely uninstalled pgloader and openssl@3 before reinstalling?
If not, please try

nbsoftware commented

well I did try to remove the links, brew remove both, brew clean, and reinstalled, still same error :/

  • 👍3
nyudenkov commented

Same issue here.

MacOS 12.0.1 / M1

codr36 commented

Same error on Monterey 12.1 (21C52) M1 Air...
FYI installing and linking openssl@3 did not work 😕

sawirricardo commented

Any workaround? Also on monterey. No bottle available!

  • 👍1
nbsoftware commented

My workaround was to install an ARM Ubuntu VM and get the package running there... not exactly the most trivial setup, but at least it worked

dbouman commented

I'm on macOS Monterey 12.2 M1 and getting exact same issue, installing and linking openssl did not work.

Shantarli commented

Same here with Monterey 12.3 on Intel.

surjithctly commented

Same issue with Mac Monterey with M1 Chip. What's the best solution so far?

ifree-zm commented

I have the same problem with @scrobot ,how to fix it?

smilenannan commented

might be the easiest way around.

  • 👍11
  • ❤️5
ScottSmudger commented

I circumvented it for now by using the docker image, saves the headache

liujchen commented

might be the easiest way around.

Yes, it works

  • 👍2
  • 🎉2
mohammed7aafar commented

You can try to install from source with:

$ brew install --build-from-source pgloader

  • Please note building from source is unsupported. You will encounter build
    failures with some formulae. If you experience any issues please create pull
    requests instead of asking for help on Homebrew's GitHub, Twitter or any other
    official channels.
nbsoftware commented 
➜  ~ brew install --build-from-source pgloader
==> Downloading https://github.com/dimitri/pgloader/releases/download/v3.6.2/pgloader-bundle-3.6.2.tgz
Already downloaded: /Users/fmeyer/Library/Caches/Homebrew/downloads/2a1b2485b3a52a8aa90117254bcc5b1701cd81cc53b0e6f1c16900b1d967f948--pgloader-bundle-3.6.2.tgz
==> Patching
==> make
Last 15 lines from /Users/fmeyer/Library/Logs/Homebrew/pgloader/01.make:
;; loading system "cffi"
;; loading system "cl+ssl"
;; loading system "mssql"
;; loading file #P"/private/tmp/pgloader-20220309-61828-1p1i4fg/pgloader-bundle-3.6.2/local-projects/pgloader-3.6.2/src/hooks.lisp"
;; loading system "pgloader"
Fatal TYPE-ERROR:
  The values
    (#<FUNCTION (LAMBDA (ESRAP::PRODUCTION)
                  :IN
                  "/private/tmp/pgloader-20220309-61828-1p1i4fg/pgloader-bundle-3.6.2/.brew_home/.cache/common-lisp/sbcl-2.1.9-macosx-arm64/private/tmp/pgloader-20220309-61828-1p1i4fg/pgloader-bundle-3.6.2/software/cl-abnf-20190521-git/abnf.fasl") {7008DBC14B}>
     NIL NIL)

  are not of type
    (VALUES FUNCTION &OPTIONAL)
make: *** [bin/pgloader] Error 1
rgossiaux commented

might be the easiest way around.

Just going to bump this comment because this actually works, if you are running into this.

plus- commented

I know it's not ideal but for people using docker I've built the image on my M1 and it's available here

  • 👍1
szynkowskij commented

Deleted my comment, but the make did end up working just fine once I found out my work laptop has an enforced proxy server on it 🙃

Building 3.6.3 is definitely the way right now. I think I used brew to install sbcl, clisp openssl, freetds and I'm not sure I even needed those, but part of the problem I think I suffered a lot from was having an x86_64 version of brew running through rosetta as well and I think just incorrect x86 libs were being picked up instead of arm64.

Hopefully we'll see a brew build get updated. I think the main issue with it is that it's still on 3.6.2 and not utilizing newer versions of pgloader lisp dependencies from quicklisp.

adimyth commented

Same issue

brew install pgloader brew install --HEAD pgloader and building from source doesnt work

Monterey 12.0.1 / Intel i7

Any workarounds or fixes?

This doesn't seem to be working for me. I'm on M1 Pro & using version 3.6.3

The make pgloader command is stuck forever & doesn't make any progress

szynkowskij commented

Same issue
brew install pgloader brew install --HEAD pgloader and building from source doesnt work
Monterey 12.0.1 / Intel i7
Any workarounds or fixes?

This doesn't seem to be working for me. I'm on M1 Pro & using version 3.6.3

The make pgloader command is stuck forever & doesn't make any progress

It making and getting stuck with no feedback seems odd if you just skim through the makefile. Do you have some of the additional requirements? Like sbcl?

adimyth commented

Same issue
brew install pgloader brew install --HEAD pgloader and building from source doesnt work
Monterey 12.0.1 / Intel i7
Any workarounds or fixes?

This doesn't seem to be working for me. I'm on M1 Pro & using version 3.6.3
The make pgloader command is stuck forever & doesn't make any progress

It making and getting stuck with no feedback seems odd if you just skim through the makefile. Do you have some of the additional requirements? Like sbcl?

Oh thanks! I missed a couple of dependencies, the installation happened correctly! 👍

However, running ./build/bin/pgloader --help is stuck forever

szynkowskij commented

Same issue
brew install pgloader brew install --HEAD pgloader and building from source doesnt work
Monterey 12.0.1 / Intel i7
Any workarounds or fixes?

This doesn't seem to be working for me. I'm on M1 Pro & using version 3.6.3
The make pgloader command is stuck forever & doesn't make any progress

It making and getting stuck with no feedback seems odd if you just skim through the makefile. Do you have some of the additional requirements? Like sbcl?

Oh thanks! I missed a couple of dependencies, the installation happened correctly! 👍

However, running ./build/bin/pgloader --help is stuck forever

The only time I had some similar problems of it working very odd was when I was able to make it and create the binary, but I had a mixture of x86 and aarch64 required libs. I made sure to remove my x86 version of homebrew and all of its libs that came from a timemachine backup when I upgraded to my M1. That would be my only other guess at your problems without it giving any feedback.

pgloader does hang every once and a while for me on my mac, but that's usually somewhere in the process of loading large files.

adimyth commented

Same issue
brew install pgloader brew install --HEAD pgloader and building from source doesnt work
Monterey 12.0.1 / Intel i7
Any workarounds or fixes?

This doesn't seem to be working for me. I'm on M1 Pro & using version 3.6.3
The make pgloader command is stuck forever & doesn't make any progress

It making and getting stuck with no feedback seems odd if you just skim through the makefile. Do you have some of the additional requirements? Like sbcl?

Oh thanks! I missed a couple of dependencies, the installation happened correctly! 👍
However, running ./build/bin/pgloader --help is stuck forever

The only time I had some similar problems of it working very odd was when I was able to make it and create the binary, but I had a mixture of x86 and aarch64 required libs. I made sure to remove my x86 version of homebrew and all of its libs that came from a timemachine backup when I upgraded to my M1. That would be my only other guess at your problems without it giving any feedback.

pgloader does hang every once and a while for me on my mac, but that's usually somewhere in the process of loading large files.

It is acting in a non-deterministic fashion. I tried building again from source, however running make pgloader gets stuck forever

panayao commented

pgloader.zip
For M1/M2 compiled on macOS Monterey 12.6.3

panayao commented 
LOAD DATABASE
     FROM      mysql://username:password@mysql.db.url/mydb
     INTO postgresql://postgres:password@postgress.db.url/postgres

 WITH include drop, create tables, create indexes, reset sequences,
      workers = 1, concurrency = 1,
      multiple readers per thread, rows per range = 50000

  SET PostgreSQL PARAMETERS
      maintenance_work_mem to '128MB',
      work_mem to '12MB',
      search_path to 'postgres, public, "$user"'

  SET MySQL PARAMETERS
      net_read_timeout  = '120',
      net_write_timeout = '120'

 CAST type bigint when (= precision 20) to bigserial drop typemod,
      type date drop not null drop default using zero-dates-to-null,
      type year to integer;

I was able to get it to work but only if workers = 1 otherwise seems to crash

philharmonie commented

I'm stuck as well on the same topic:

debugger invoked on a SIMPLE-ERROR in thread
#<THREAD "main thread" RUNNING {700B600003}>:
  Problem running initialization hook #<FUNCTION OPEN-FOREIGN-LIBS>:
  Unable to load foreign library (LIBCRYPTO).
  Error opening shared object "/usr/local/Cellar/openssl@1.1/1.1.1m/lib/libcrypto.1.1.dylib":
  dlopen(/usr/local/Cellar/openssl@1.1/1.1.1m/lib/libcrypto.1.1.dylib, 0x000A): tried: '/usr/local/Cellar/openssl@1.1/1.1.1m/lib/libcrypto.1.1.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/usr/local/Cellar/openssl@1.1/1.1.1m/lib/libcrypto.1.1.dylib' (no such file), '/usr/local/Cellar/openssl@1.1/1.1.1m/lib/libcrypto.1.1.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')).

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [CONTINUE] Skip this initialization hook.
  1: [ABORT   ] Exit from the current thread.

(SB-INT:CALL-HOOKS "initialization" (#<FUNCTION OPEN-FOREIGN-LIBS>) :ON-ERROR :ERROR)

I tried to compile it myself (without success) and I tried to use the build provieded by @panayao without success.

I am using openssl@3 and I don't know why it is trying so use 1.1 and how to work around that.

I tied

brew unlink openssl@3
Unlinking /opt/homebrew/Cellar/openssl@3/3.1.1_1... 5548 symlinks removed.
❯ brew link openssl@1.1
Linking /opt/homebrew/Cellar/openssl@1.1/1.1.1u... 3999 symlinks created.
❯ echo 'export PATH="/opt/homebrew/opt/openssl@1.1/bin:$PATH"' >> ~/.zshrc
  • 👍3