parallel bs fails when gegen is used inside program
ppaspp opened this issue · 7 comments
ppaspp commented
Preliminaries
Before submitting an issue, please check (with x in brackets) that you:
- Are using the newest release (see here for latest release version number).
- Have checked that the examples in the help work.
- Have read the help (HTML version) and the gallery of examples.
- Have checked that there is not already an existing issues for what you are reporting.
Expected behavior and actual behavior
Use gegen from gtools ado instead of egen with parallel bs
Parallel child processes stop with error. Log:
running /home/pp/ado/profile.do ...
. capture {
clear
set processors 1
cd "/home/pp/.tmp/"
sysdir set PERSONAL "/home/pp/uolcloud/ados/"
sysdir set PLUS "~/ado/plus/"
global S_ADO = `"BASE;SITE;.;PERSONAL;PLUS;OLDPLACE"'
mata: mata mlib index
mata: mata set matalibs "lmatabase;lmatafc;lmatasvy;lmatapss;lmatamcmc;lmatap
> ath;lmatapostest;lmataerm;lmataopt;lmatamixlog;lmataado;lmatasp;lmatatab;lmat
> agsem;lmatasem;lmoremata10;lmoremata;lmoremata14;lparallel;lftools;l__pllmroa
> pnlb75_mlib"
set seed 49766
noi di "{hline 80}"
-------------------------------------------------------------------------------
> -
noi di "Parallel computing with Stata"
Parallel computing with Stata
noi di "{hline 80}"
-------------------------------------------------------------------------------
> -
noi di `"cmd/dofile : "__pllmroapnlb75_bs_simul.do""'
cmd/dofile : "__pllmroapnlb75_bs_simul.do"
noi di "pll_id : mroapnlb75"
pll_id : mroapnlb75
noi di "pll_instance : 1/2"
pll_instance : 1/2
noi di "tmpdir : `c(tmpdir)'"
tmpdir : /tmp/__pllmroapnlb75_tmpdir0001
noi di "date-time : `c(current_time)' `c(current_date)'"
date-time : 13:34:46 12 May 2018
noi di "seed : `c(seed)'"
seed : XAA000000000000c2664bb19e15cdf0c5ef07c195bb8109dcd86c52113f3d02f
> 9fb28c90545ea0af6f60345f756599f73431bf3ef70d5347fcd35889922c3f22a10b11bbf90cf
> bf54d80d5aed638a9c105bbea6b9f25283050926683137b3ef57fa2698d06cca397cfe673870a
> 938f2fab3102fc2224eae5f5829665753f42b6a87368b66838a0db2cb22f49b58644222c0cdc5
> 40d5a89d5bd2b58d96ca30dad2d0fdba6ca431473cfee3903e599c083b8e0995886a080e6adf5
> f04e1f3f9586a4a948f4bc8d2ece447bd54eeb6f10b5b3af90ccb45b8d7d0f247e267f5b6dc49
> 785ab53e341c98f161b4dbc3e5e3810dfd29ddb75ddd0bf6f12deb1dcc10e0f77cf48ec6736d7
> e04ecaab5f36e96b46f247c88b0e5f9f963d000f476f5ce003ae0b73dab64179f7984f064f4c3
> c74fd6b37ba3586ed8fa0d16aa394797a2d5b6327c95ce58eee2d698693452c5d962a00e6096f
> 02dc07893cde11027713c402f072a8f4ecffbef3fe0b8f26c21a71e5a8dd15a3282987ddc5161
> 41abaed22668996463e21bff1bcae8bdc41cd9e729422663ebe6f549047c86c1adaca69e23dce
> e81f76bdc88d868ddff8a3df1a7dc106886b3ac85001af80550f7a4fdd5dc12723144967ea5c5
> 4b6f1657653cdfb553bc8c98f01fd3d5fc462dfdfa136857142f4664831c3dacdd7fedd24d13e
> b9a71636653d6ce85448e1e4e62a8f55b995cb20a1c2ad1472118b7936456d7a297216c43ba75
> c9ab2c729823da8c2742b70c28be058c2d7c7cfb4c941f48d04f09defb395e98897d46b90b5ec
> bf15912a1fc3621394db9c29b9ef30343d5ad10a47e008b9b15d2dd6c264aa4d4c7e13ae872f3
> dacf12981e26c3a9c435373ae1d3bb2f5d9b0deabcaebb0ad544f406fae8563e9b11ab40ab206
> c09d8a9e32450b37340133e4edc2fb88019ecd4fd2fe587ebb8a3cde2407ead3533cb39f865b3
> 5aa8f27c11d223610441b38b756c9e3ef0daa1cd9d86c931ac4bff16d91c253ea75fa5af31a87
> 1b2e7ab5b4ac7de1787874600cc0f030ff916cd89d4fadeedb24b979f321ea8a9654c0aca9c4a
> a4ffc74a1be1fe7126246afb8447a08fc515c067fc55652492011b1984ad6ae8d21cb469a69f8
> ccc6d470230ac2ad86c8c099b3e1b6d3a18d754b88c64bcc476de1c8812f973ad9aa04dc34469
> 18555cf921274c5ffbfae6f036e08be55cd0a28a4293ac731e42f55a9b1e4c9f857aa11d745e6
> e86a1b710d8b949570a8effbdd55ac6044f0bed729591519edcd8b3df9b8fb7f477f03d9b06f5
> 455423fcfedcbe75af5057bfe10f5c838d11b0809ed6c4d532ec047cf63df4970f0d4c9c724ba
> 8750ae4596a0d72823d01c6cd473cd6f7bc7e1b4ef48a1700eb1afbed5e9af9b3fb2f955da785
> 39037ff3bf517d5d152148e4fa54d0629ab706d9b9983beaaf6ea029e688b1c08269fd22f055c
> ad2d18d82f8cacbd6ff72b46f35bd7ff3e28ba9ac2b778db3e595af69aabe38742b6e73ac92b5
> f670af1b6cdfbe6de236d4f5eb0d14cc8fabf4a904dba00199107fa46db2a74755e0ce8cfff49
> 319b8175414ff19b17b480f9caa7f2b73909d02dcb7b3bf6f4e998d4308052a16a470844b6c54
> 1f8822c2f0e8b5f48c2f16d3ed072fd00a9bac5367a0606e41f99dc2adc9d03fcf682b0a56523
> 65931e929fc856be5d00cf97adfc97165a7276f058ebefe2b5a320ef5d92a4d1ca1fd28cf83cd
> fc80078364985f75f5ad89c4ddd25258b38ba38247ca9f9a2ba03d8c2742765427f8cab0ae467
> 9616da06c9e0a0f75ab29b5a795640905819b5d64728f65f74dea21fdee2f97aa84cf9e2c1bef
> d4f35d5f47cb9b5ae06a293350298df76481ac8e764ae9e21b1fe17b31d07026b5388d86f9fbc
> a2e6465438a061e7930b60c15d1fdec0ebd6289b45706eaf2a826fed5bebdfbb028a9f7efa107
> 3b5420d028553cfa127d573d40f846242fd8295774461ed8c8686861aa2a51edb6e14f07c0123
> e8e17af2bb53108ebc589e57bd4971a9fc1a8c198e0f718441db0bd0443564651b625a072f664
> 30da8296004d88ec53fe2e156dd8dd3ac792f64d8e431e912c13ccb6481e2b57cf1826fa535da
> 9211d23d48b7c400935d4c0e3b03717b6a300a1f8d7ab8ab8254420cdb085369a480caa0ecccc
> a1d89edbf47dc6a4ca2e93e5b48463b7026d1a2ff1579c8e255aedab628bac86dfaa1f4bdf2ef
> 3fae47f36c67a0c8163a3315c66ec2a39bd887757e19124e34b701d918176c5d950f5a191cfab
> 2e449f59a1d3ff5514bb1287ddc7439f5aed7f793fc56527f15c626c16cce7b806efa1261b9e6
> 8af9652f812236f70c9dff95a79de25d9a904083760b5ce226f45e958a9d71c71c26e772c5f1b
> 908291fdda99cb9d66806c88e2f1222f83aec76076db2d88f79853200c9715206c1561202582a
> 3f4ec5b9ec5ded79bb032aadd74164222e24061f170eaf4f0bff0140f4c0b861b3b9de690f914
> 168baddc4a25a2f98063bf718570b2298f1060f03f74b3e9ea344efdc598489ee244dad854259
> 938a61dd8b1ffab0267cc0ccfcbb5ea13d9d03c19990e7213770a33fe5b2d9161b96e738cdba1
> d86f6d946a97798d8c639c2dc993d0dda71aa34a8965591b4a823a18b591114eaafd8de069113
> ccd563e860addad624ee7cc06dea73a79cfda6c3426db6f9dc948949101c077f37069147cbd93
> 9fdacc8728349775f629f93408bc00ce2fe710a7328ab83e5c8ec23f87a752f6d658b3c447129
> 77ddd656b88bb5b50dac16c3dd957fb861d5e4909a36033766aa0cdb9ef89fd40c62aefd7b5a2
> f526a8ea553d9a492f5d43263e0dba323c06593d5a9eee4d786b2c885268d854328d312092cc6
> cca4c1eb2098061de721e3c3a3b31df3ff01c66ef2fe8ace5d354f24bf53b3dc20da77de5952c
> e84750b07e45487b5297594bf48e3adb9be4005be46f5681eac768e04345446dfb05b82141c0e
> d5d9fa8ad3a2a5cb656fb3837f3284c1fd1ce0842b5dd6f931440575bfc245d26504b45bf98bf
> 14a2d40e04489130488db8b05c0dfdbbcdf8c7fed1d3b630bb5da9e2b928600412b27c9d2a1d3
> 8d291bdca044a52f17489e87fc605c1ae230815300e7b2dc63a0eb6658a505eb13e7056cf84ab
> 668838149e76774924260ceab7dea8d372838a3e24fbcd3cfa759efc51764f36e5ae8b13fdfdf
> d67a0811274e72994b21ad651a9fc3b4dd0e5b9fbfbbd9f0011e25347c2f671a509971e83a8bb
> d620f1a7ca5ef8de84c09266cb5c4c7914683ca89c53dc1b2dc4a4c21dcaff8f48978a6eff997
> beaae5543616f188160cf709c66e5490db287034dcb33ccd8f03b548ab8c20214229a19236af2
> fd6a3f7ce867003311bf280df2ebabfe1ebb6501e2ee8ec143f0a4f4b94e0f02aecaa94bdfec2
> e5458ac9b7934b41b2a5c597998da71802dbd22eb0728d1c84db212ba9a660cdfb3cab62e3ff3
> ef8dc8a15449cbeae16c309e47be7029a23b1befd6908e9721fec542a6ddfc6725284dbcf12f7
> 8520001000001380641
noi di "{hline 80}"
-------------------------------------------------------------------------------
> -
local pll_instance 1
local pll_id mroapnlb75
global pll_instance 1
global pll_id mroapnlb75
mata: for(i=1;i<=1;i++) PLL_QUIET = st_tempname()
}
. local result = _rc
. if (c(rc)) {
. cd "/home/pp/.tmp/"
. mata: parallel_write_diagnosis(strofreal(c("rc")),"/home/pp/.tmp/__pllmroapnl
> b75_finito0001","while setting memory")
. clear
. exit
. }
.
. * Loading Programs *
. capture {
run "/home/pp/.tmp/__pllmroapnlb75_prog.do"
}
. local result = _rc
. if (c(rc)) {
. cd "/home/pp/.tmp/"
. mata: parallel_write_diagnosis(strofreal(c("rc")),"/home/pp/.tmp/__pllmroapnl
> b75_finito0001","while loading programs")
. clear
. exit
. }
.
. * Checking for break *
. mata: parallel_break()
.
. * Loading Globals *
. capture {
cap run "/home/pp/.tmp/__pllmroapnlb75_glob.do"
}
. if (c(rc)) {
. cd "/home/pp/.tmp/"
. mata: parallel_write_diagnosis(strofreal(c("rc")),"/home/pp/.tmp/__pllmroap
> nlb75_finito0001","while loading globals")
. clear
. exit
. }
.
. * Checking for break *
. mata: parallel_break()
. capture {
noisily {
.
. * Checking for break *
. mata: parallel_break()
. use __pllmroapnlb75_bs_dta.dta, clear
(1978 Automobile Data)
. if (`pll_instance'==$PLL_CLUSTERS) local reps = 25
. else local reps = 25
. local pll_instance : di %04.0f `pll_instance'
. bs , sav(__pll`pll_id'_bs_eststore`pll_instance', replace ) rep(`reps')
> : pargegen price weight foreign rep78
(running pargegen on estimation sample)
invalid syntax
an error occurred when bootstrap executed pargegen
r(198);
. }
}
. if (c(rc)) {
. cd "/home/pp/.tmp/"
/home/pp/.tmp
. mata: parallel_write_diagnosis(strofreal(c("rc")),"/home/pp/.tmp/__pllmroap
> nlb75_finito0001","while running the command/dofile")
. clear
. exitSteps to reproduce the problem
Minimal nonsense example (runs fine when gegen is replaced with egen):
clear all
sysuse auto , clear
parallel setclusters 2, f
cap prog drop pargegen
prog define pargegen, rclass
version 13
syntax varlist [if]
marksample touse
gegen test = sum(price)
reg `varlist' if `touse'
drop test
end
parallel bs, reps(50): pargegen price weight foreign rep78System information
Some relevant information
- Stata version and flavor (e.g. v14 MP): 15.1 MP
- OS type and version (e.g. Windows 10): unix (arch)
- Parallel version: latest (Distribution-Date: 20170726)
Output from creturn list:
System values
--------------------------------------------------------------------------------------
c(current_date) = "12 May 2018"
c(current_time) = "13:48:45"
c(rmsg_time) = 0 (seconds, from set rmsg)
--------------------------------------------------------------------------------------
c(stata_version) = 15.1
c(version) = 15.1 (version)
c(userversion) = 15.1 (version)
c(dyndoc_version) = 1 (dyndoc)
--------------------------------------------------------------------------------------
c(born_date) = "18 Apr 2018"
c(flavor) = "IC"
c(bit) = 64
c(SE) = 1
c(MP) = 1
c(processors) = 2 (Stata/MP, set processors)
c(processors_lic) = 2
c(processors_mach) = 4
c(processors_max) = 2
c(mode) = ""
c(console) = ""
--------------------------------------------------------------------------------------
c(os) = "Unix"
c(osdtl) = ""
c(hostname) = "pp-x230"
c(machine_type) = "PC (64-bit x86-64)"
c(byteorder) = "lohi"
c(username) = "pp"
--------------------------------------------------------------------------------------
Directories and paths
--------------------------------------------------------------------------------------
c(sysdir_stata) = "/usr/local/stata15/" (sysdir)
c(sysdir_base) = "/usr/local/stata1.." (sysdir)
c(sysdir_site) = "/usr/local/ado/" (sysdir)
c(sysdir_plus) = "~/ado/plus/" (sysdir)
c(sysdir_personal) = "/home/pp/uolcloud.." (sysdir)
c(sysdir_oldplace) = "~/ado/" (sysdir)
c(tmpdir) = "/tmp"
--------------------------------------------------------------------------------------
c(adopath) = "BASE;SITE;.;PERSO.." (adopath)
c(pwd) = "/home/pp/.tmp" (cd)
c(dirsep) = "/"
--------------------------------------------------------------------------------------
System limits
--------------------------------------------------------------------------------------
c(max_N_theory) = 1099511627775
c(max_k_theory) = 5000 (set maxvar)
c(max_width_theory) = 1048576 (set maxvar)
--------------------------------------------------------------------------------------
c(max_matsize) = 11000
c(min_matsize) = 10
--------------------------------------------------------------------------------------
c(max_macrolen) = 15480200
c(macrolen) = 645200 (set maxvar)
c(charlen) = 67783
c(max_cmdlen) = 15480216
c(cmdlen) = 645216 (set maxvar)
c(namelenbyte) = 128
c(namelenchar) = 32
c(eqlen) = 1337
--------------------------------------------------------------------------------------
Numerical and string limits
--------------------------------------------------------------------------------------
c(mindouble) = -8.9884656743e+307
c(maxdouble) = 8.9884656743e+307
c(epsdouble) = 2.22044604925e-16
c(smallestdouble) = 2.2250738585e-308
--------------------------------------------------------------------------------------
c(minfloat) = -1.70141173319e+38
c(maxfloat) = 1.70141173319e+38
c(epsfloat) = 1.19209289551e-07
--------------------------------------------------------------------------------------
c(minlong) = -2147483647
c(maxlong) = 2147483620
--------------------------------------------------------------------------------------
c(minint) = -32767
c(maxint) = 32740
--------------------------------------------------------------------------------------
c(minbyte) = -127
c(maxbyte) = 100
--------------------------------------------------------------------------------------
c(maxstrvarlen) = 2045
c(maxstrlvarlen) = 2000000000
c(maxvlabellen) = 32000
--------------------------------------------------------------------------------------
Current dataset
--------------------------------------------------------------------------------------
c(N) = 74
c(k) = 12
c(width) = 43
c(changed) = 0
c(filename) = "__plldtannyvw43_b.."
c(filedate) = "12 May 2018 13:48"
--------------------------------------------------------------------------------------
Memory settings
--------------------------------------------------------------------------------------
c(memory) = 33554432
c(maxvar) = 5000 (set maxvar)
c(matsize) = 400 (set matsize)
c(niceness) = 5 (set min_memory)
c(min_memory) = 0 (set min_memory)
c(max_memory) = . (set max_memory)
c(segmentsize) = 33554432 (set segmentsize)
--------------------------------------------------------------------------------------
Output settings
--------------------------------------------------------------------------------------
c(more) = "off" (set more)
c(rmsg) = "off" (set rmsg)
c(dp) = "period" (set dp)
c(linesize) = 90 (set linesize)
c(pagesize) = 43 (set pagesize)
c(logtype) = "smcl" (set logtype)
c(noisily) = 1
--------------------------------------------------------------------------------------
c(level) = 95 (set level)
c(clevel) = 95 (set clevel)
--------------------------------------------------------------------------------------
c(showbaselevels) = "" (set showbaselevels)
c(showemptycells) = "" (set showemptycells)
c(showomitted) = "" (set showomitted)
c(fvlabel) = "on" (set fvlabel)
c(fvwrap) = 1 (set fvwrap)
c(fvwrapon) = "word" (set fvwrapon)
c(fvtrack) = "term" (set fvtrack)
c(lstretch) = "" (set lstretch)
--------------------------------------------------------------------------------------
c(cformat) = "" (set cformat)
c(sformat) = "" (set sformat)
c(pformat) = "" (set pformat)
--------------------------------------------------------------------------------------
c(coeftabresults) = "on" (set coeftabresults)
Interface settings
--------------------------------------------------------------------------------------
c(reventries) = 5000 (set reventries)
c(fastscroll) = "on" (set fastscroll)
c(linegap) = 1 (set linegap)
c(scrollbufsize) = 204800 (set scrollbufsize)
c(maxdb) = 50 (set maxdb)
--------------------------------------------------------------------------------------
Graphics settings
--------------------------------------------------------------------------------------
c(graphics) = "on" (set graphics)
c(scheme) = "s2color" (set scheme)
c(printcolor) = "automatic" (set printcolor)
--------------------------------------------------------------------------------------
Efficiency settings
--------------------------------------------------------------------------------------
c(adosize) = 1000 (set adosize)
--------------------------------------------------------------------------------------
Network settings
--------------------------------------------------------------------------------------
c(checksum) = "off" (set checksum)
c(timeout1) = 30 (set timeout1)
c(timeout2) = 180 (set timeout2)
--------------------------------------------------------------------------------------
c(httpproxy) = "off" (set httpproxy)
c(httpproxyhost) = "" (set httpproxyhost)
c(httpproxyport) = 8080 (set httpproxyport)
--------------------------------------------------------------------------------------
c(httpproxyauth) = "off" (set httpproxyauth)
c(httpproxyuser) = "" (set httpproxyuser)
c(httpproxypw) = "" (set httpproxypw)
--------------------------------------------------------------------------------------
Trace (program debugging) settings
--------------------------------------------------------------------------------------
c(trace) = "off" (set trace)
c(tracedepth) = 32000 (set tracedepth)
c(tracesep) = "on" (set tracesep)
c(traceindent) = "on" (set traceindent)
c(traceexpand) = "on" (set traceexpand)
c(tracenumber) = "off" (set tracenumber)
c(tracehilite) = "" (set tracehilite)
--------------------------------------------------------------------------------------
Mata settings
--------------------------------------------------------------------------------------
c(matastrict) = "off" (set matastrict)
c(matalnum) = "off" (set matalnum)
c(mataoptimize) = "on" (set mataoptimize)
c(matafavor) = "space" (set matafavor)
c(matacache) = 2000 (set matacache)
c(matalibs) = "lmatabase;lmatafc.." (set matalibs)
c(matamofirst) = "off" (set matamofirst)
--------------------------------------------------------------------------------------
Unicode settings
--------------------------------------------------------------------------------------
c(locale_ui) = "de_DE" (set locale_ui)
c(locale_functions) = "de_DE" (set locale_functions)
c(locale_icudflt) = "de_DE" (unicode locale)
--------------------------------------------------------------------------------------
Other settings
--------------------------------------------------------------------------------------
c(type) = "float" (set type)
c(maxiter) = 16000 (set maxiter)
c(searchdefault) = "all" (set searchdefault)
c(rng) = "default" (set rng)
c(rng_current) = "mt64"
c(rngstate) = "XAA00000000000000.." (set rngstate)
c(rngseed_mt64s) = 123456789
c(max_rngstreams) = 32768
c(rngstream) = 1 (set rngstream)
c(varabbrev) = "on" (set varabbrev)
c(emptycells) = "keep" (set emptycells)
c(odbcmgr) = "iodbc" (set odbcmgr)
c(odbcdriver) = "unicode" (set odbcdriver)
c(fredkey) = "" (set fredkey)
--------------------------------------------------------------------------------------
Other
--------------------------------------------------------------------------------------
c(pi) = 3.141592653589793
c(alpha) = "a b c d e f g h i.."
c(ALPHA) = "A B C D E F G H I.."
c(Mons) = "Jan Feb Mar Apr M.."
c(Months) = "January February .."
c(Wdays) = "Sun Mon Tue Wed T.."
c(Weekdays) = "Sunday Monday Tue.."
c(rc) = 198 (capture)
--------------------------------------------------------------------------------------
. bquistorff commented
Can you turn on trace in pargegen and show the log so we can see what exactly is causing the syntax error.
ppaspp commented
Log file after including set trace on in pargegen.
bquistorff commented
Huh, I don't know what the reldif Stata function is (it's not in my setup). It's possible the syntax error is actually inside reldif so you could try increasing the trace depth until it doesn't go any deeper on the error. Do it work in serial (non-parallel) fashion?
ppaspp commented
hm, I think trace depth is set to infinity according to the stata manual. At least changing it to 10 doesn't give me more information.
Regarding your second question: Do you mean I should run the program with parallel setclusters 1, f? Then the same error appears. Runing gegen outside the pargegen program also works.
My personal impression is that the local level, which is defined by regress, is somehow altered by parallel in combination with gegen. According to the stata manual level(95) is the default option but in line 4302 of the log file 95,0 shows up. Even when I define the regress option , level(95) explicitly, the same error is shown.
bquistorff commented
I was wondering if you could run the program without parallel at all. Something like bootstrap, reps(50): pargegen price weight foreign rep78?
ppaspp commented
Got it. Running pargegen price weight foreign rep78 without any bootstrapping returns the same error, so I guess I should report the error to github/stata-gtools instead
bquistorff commented
Good luck