After running spdz2-offline.x, Player-Online.x is purging pre-processing data

Question

After running spdz2-offline.x, Player-Online.x is purging pre-processing data

Closed this issue 7 years ago · 14 comments

Hi

I'm trying to generate the pre-processing data using spdz2-offline.x between two machines (technically 2 Ubuntu 16.04.1 vms, each with 2Gb RAM, running on a Windows 10 laptop).

I've been able to build spdz2-offline.x by adding "USE_NTL=1" and "MOD=-DMAX_MOD_SZ=6" to CONFIG.mine, running make clean, then running make.

I run spdz2-offline.x on both vms at the same time with the relevant args and it outputs a lot of good things, and then starts outputting text in the form:
"Inv: 1800 out of 9600"
etc, incrementing in steps of 100 each time.

After letting it do this for a while I kill the processes, as this is what is suggested in the documentation ("The program will generate every kind of randomness required by the online phase until you stop it.") and also in this comment #33 (comment). Unfortunately Ctrl-C doesn't seem to do the trick. After doing Ctrl-C, it initially says "Stopping production..."
...
"Writing to file Player-Data/2-128-40"
...

However after producing these messages it continues outputting text in the form
"Inv. 3900 out of 9600".
So after this I killed the spdz2-offline.x process on each machine using
$ kill -9 [pid]

I then try running the program, first with a server on vm A:
[Alice-vm] $ ./Server.x 2 5000 &
[Alice-vm] $ ./Player-Online.x -pn 5000 0 [programName]
[Bob-vm] $ ./Player-Online.x -pn 5000 -h [Alice IP] 1 [programName]

It loads some parameters well, but then both machines output:
"Purging preprocessing data because something is wrong"
...
"Removing Player-Data/...."
...
"terminate called after throwing an instance of 'not_enough_to_buffer'
what(): Not enough data available for buffer..."

Seems like there is a check on the pre-processing data that failed, so something went wrong when generating the pre-processing data.

Any thoughts on what I should do? Or if I made a mistake in any of the steps?

Thanks
Daniel

Answer 1 · 2018-02-14T07:45:14.000Z

Hi Daniel,

The counting is part of a setup phase, which cannot be interrupted. If I understand you correctly, the process didn't stall before you killed it. I would guess that killing the preprocessing phase prevented it from generating enough data for the online phase. Generally, the idea of compiling in "secure mode" (without -DINSECURE) is to maintain security even in the case of failures. Running out of preprocessed data is such a failure.

Best regards,
Marcel

Answer 2 · 2018-02-14T14:26:45.000Z

Hi Marcel

Thanks for the help.

I now let spdz2-offline.x run as long as it wants. It runs for a while and gets as far as party A producing the output "Inv: 9500 out of 9600" and party B producing the output "Inv: 6200 out of 9600" (as well as a number of messages like "Produced 65536 gfp squares, 29.1 per second" and "Unavailable, trying again in 1ms"). However after this, both parties output the message:
"Receiving error - 1: Connection timed out
terminate called after throwing an instance of 'bad_value'
what(): Some value is wrong somewhere
Aborted (core dumped) "

Any thoughts on what I should do?

Thanks
Daniel

Answer 3 · 2018-02-14T15:49:28.000Z

This could be just a networking error. Unfortunately there is no facility for recovering. Have you tried several times? Furthermore, does it work when running both parties on the same host?

Answer 4 · 2018-02-15T18:16:31.000Z

I tried the two-host setup three times-- the amount that Bob gets through varies, from about 4500 out of 9600 to about 6600 out of 9600. But the same error happens each time.

I tried it on the same machine. Basically just copied one the SPDZ-2 dir into SPDZ-2-Copy. Deleted all the contents of Player-Data in both dirs. Then ran in 2 separate terminals:
Alice .../SPDZ-2 $ ./spdz-offline.x -p 0 -N 2 -h 127.0.0.1
Alice .../SPDZ-2-Copy $ ./spdz-offline.x -p 1 -N 2 -h 127.0.0.1

Here, something weird happened. The one in SPDZ-2 gets as far as:
"Starting to produce gfp triples
...
Starting to produce gfp bits"
And then it outputs:
"Killed"
And exits

The other process continues all the way until
"Inv: 9500 out of 9600"
Then it hangs. It doesn't respond to Ctrl-C. So then I killed it using kill -9 PID.

What do you think?

Answer 5 · 2018-02-16T11:59:35.000Z

I've realized that the program uses more than 2 GB of memory on my machine. Can you observe the memory usage? The system would kill the program if it exhausts the memory and there is not enough swap space.

Answer 6 · 2018-02-16T18:28:45.000Z

Cool. I tried increasing the RAM to 4Gb and trying the local setup again.

It gets as far as 9500 out of 9600 for both processes, then starts producing more squares, bits, triples, inverse’s etc. I left it going for about 4 hours and it is still running— should I kill it at some point?

Also, is there a way to specify to the script how many triples, squares etc you need? (Since I think this is output when you compile the program.)

Answer 7 · 2018-02-16T19:02:31.000Z

Update- soon after publishing that post both processes finished with the following messages at the end:
[ Party 0]
"Produced 622592 gfp squares, 47.6138 per second
Unavailable, trying again in 1 ms
Killed"
[ Party 1]
"Produced 606208 gfp squares, 46.8688 per second
Produced 622592 gfp squares, 47.6132 per second"

After this the Player-Data directory for P0 had the following structure:
$ ls -l Player-Data/
total 4
drwx------ 2 alice alice 4096 Feb 16 13:06 2-128-40
$ ls -l Player-Data/2-128-40/
total 348916
-rw-rw-r-- 1 alice alice 0 Feb 16 13:06 Bits-2-P0
-rw-rw-r-- 1 alice alice 17825792 Feb 16 13:16 Bits-p-P0
-rw-rw-r-- 1 alice alice 0 Feb 16 13:06 Inputs-2-P0-0
-rw-rw-r-- 1 alice alice 0 Feb 16 13:06 Inputs-2-P0-1
-rw-rw-r-- 1 alice alice 62914560 Feb 16 13:22 Inputs-p-P0-0
-rw-rw-r-- 1 alice alice 40894464 Feb 16 13:20 Inputs-p-P0-1
-rw-rw-r-- 1 alice alice 0 Feb 16 13:06 Inverses-2-P0
-rw-rw-r-- 1 alice alice 29360128 Feb 16 13:21 Inverses-p-P0
-rw-rw-r-- 1 alice alice 93957119 Feb 16 13:29 Params-Data
-rw-rw-r-- 1 alice alice 34712149 Feb 16 09:49 Player-FHE-Keys-P0
-rw-rw-r-- 1 alice alice 44 Feb 16 09:49 Player-MAC-Keys-P0
-rw-rw-r-- 1 alice alice 0 Feb 16 13:06 Squares-2-P0
-rw-rw-r-- 1 alice alice 39845888 Feb 16 13:27 Squares-p-P0
-rw-rw-r-- 1 alice alice 0 Feb 16 13:06 Triples-2-P0
-rw-rw-r-- 1 alice alice 37748736 Feb 16 13:20 Triples-p-P0

The structure for P1 was about the same, though the size of some of the files was different and the filenames contain P1 instead of P0.

Then when I try to run ./Player-Online.x, I get the following error message in both terminals:
"Purging preprocessed data because something is wrong
Setting up Data_Files in: Player-Data/2-128-40/
done
Removing Player-Data/..."

Maybe if I could configure it to only produce a small amount of pre-processing data it would finish and not have problems. Is there a way to do that? The program I initially want to run only needs about 10 multiplicative triples.

Or if not, should I run it somewhere with more RAM? What environment do you normally run it on? (So far I've been testing on vms on my laptop, but can get access to a higher-spec machine.)

Answer 8 · 2018-02-17T11:38:07.000Z

Are you getting any messages like Produced gf2n triples? The directory listing suggests that it doesn't produce any, which might be the problem when running the online phase.
And what's the memory and CPU usage once it has completed the setup? 4 GB of RAM might not be enough if you run both parties on the same host.
There isn't any functionality to specify the amount of data you need, you will need to wait until you've got enough before shutting it down.

Answer 9 · 2018-02-19T15:36:25.000Z

Looking at the Memory usage in the Windows host it looked like yes, it was maxing out on the amount of RAM used.

So I've gotten access to a heavier-duty machine (16Gb RAM on an Ubuntu 16.04.2 running on Hypervisor, not sure what the host is). The machine does not have the ARX, so CONFIG.mine now looks as follows:
USE_NTL = 1
MOD = -DMAX_MOD_SZ=6
ARCH = -mtune=native

I then set up two directories as before:
[Install mpir, libsodium and ntl dependencies]
$ git clone https://github.com/bristolcrypto/SPDZ-2 SPDZ-2-Alice
$ cd SPDZ-2-Alice
$ vim CONFIG.mine
(write CONFIG.mine to be as above).
$ make
And then the same thing for Bob:
$ git clone https://github.com/bristolcrypto/SPDZ-2 SPDZ-2-Bob
etc

Then in two separate terminals:
SPDZ-2-Alice $ ./spdz2-offline.x -p 0
SPDZ-2-Bob $ ./spdz2-offline.x -p 1

Then I get the following error on both machines:
terminate called after throwing an instance of 'bad_keygen'
what(): KeyGen has gone wrong: a doesn't match
Aborted (core dumped)

Any thoughts on what is going wrong?

Answer 10 · 2018-02-19T16:08:21.000Z

I suspect the two players are doing something different. I would try compiling it once and then copy the binaries to the other side.

Answer 11 · 2018-02-19T16:23:10.000Z

No luck. When I copy the binaries the same error message happens.

Answer 12 · 2018-02-20T08:02:35.000Z

Have you tried wiping everything clean and starting over?

Answer 13 · 2018-02-21T13:55:30.000Z

Yes.

What versions of the dependencies (mpir, libsodium, ntl) did you use?

Answer 14 · 2018-02-22T21:48:13.000Z

I wasn't able to figure out why the bad_keygen error happened--maybe some problem in the environment.

But I got access to another machine with very high specs and when I ran the pre-processing on this machine it worked fine.

Thanks for the support.