Reading large matrices from op4 fails when DOFs > 99999999
Closed this issue · 24 comments
Hello Tim,
I am back again! I am now trying to read an even bigger matrix than the last time we got in touch here (Issue: Reading large matrix from op4 fails when in binary format #5).
The first line of my .op4 file now contains the following string:
' 12752022 -12752022 6 1KGG 1P,3E23.16|I16'
As you can see, the structure changes compared to what is coded at the moment in the op4.py file.
I guess the problem is that when the number of DOFs of the matrix occupies more than 8 characters, as in this case, then Nastran has to write the information about the number of columns and rows in a space of 16 characters instead of 8.
Please let me know if I can help somehow solving this issue.
I cannot share the file here, due to big size. Do you think you could manage to solve it just by having the first line?
Thank you in advance for your help!
Best regards,
Fabiola
Hi Fabiola,
Interesting! I'm feeling optimistic on this one. I'll take a look at the code this weekend and get back to you. We might have to experiment a little, but we'll see.
Thank you for info!
Best regards,
Tim
Hi again, Fabiola! I'm curious, have you tried a binary format for this problem? If so, did it work? Thanks!!
Hi Fabiola,
My apologies, but I have one other request: Would you please post say the first 10 lines or so of the op4 file? That should prevent me from making too many guesses. :)
Thank you!
Tim
Hi Tim,
Thnks for your fast reply. I did no thave the chance to send it before, I attach the first 11 lines of the op4 file.
I've also tried to read the binary fie and I get this error:
in _rd_bigmat_binary
Y = struct.unpack(numform % L, fp.read(bytesreal * L))
struct.error: unpack requires a buffer of 48 bytes
I guess it's just equivalent to the error we get with the ascii format?
Thank you again for your help!
Best regards,
Fabiola
Thank you very much, Fabiola! That helps. I had two guesses on what the format might be, and it turned out to be neither!
That might indeed be basically the same error ... I hope so as that should make the fix easier.
This issue might take me a little bit to resolve, because now I'm thinking I'll need to generate some very-large-matrix op4s in a couple different formats for learning/testing. I'll keep you posted.
Thanks again!!
Best regards,
Tim
Hi Fabiola,
I'm having trouble creating test op4 files. Which Nastran and version are you running? If I type:
nastran help
The first few lines of output give me:
nastran (Version 2021.1Sep 28 2020 21:14:49)
Copyright 2014 Siemens Product Lifecycle Management Software Inc.
All Rights Reserved.
And in my .log file of the output, I have this info under "System configuration":
Architecture: x86_64linuxl
...
Executable: standard
...
Numeric format: 64-bit little-endian IEEE.
Bytes per word: 8
I'm wondering how that compares to your run?
Any special tips to get the 64-bit integers?
Thank you!!
Best regards,
Tim
Hello Tim,
Thank you for working at this issue!
I am using Nastran version 2019.1.
The "System configuration" info coincides with what you get in yout .log file. Would you expect something different?
I think I am not answering your doubts, sorry.
Can I help you in some other way?
Best regards,
Fabiola
Hi Fabiola,
Thanks for the fast response, and I definitely appreciate your help! :-) I'm trying to figure out how to create some test op4 files to work with. Is your Nastran from Siemens? If so, it seems like I should be able to make some op4 files to learn from. Unfortunately, I've had no luck so far. For example, here is the beginning of one of my op4 files:
******** -13 2 2MATD11A 1P,3E23.16
1 0 28
27 1
6.2269300127281266E-01 6.4280494314714020E-01 8.5674171329533383E-01
Do you have any special settings turned on to get the large integers in the op4 file? For example, "NASTRAN SYSTEM" lines at the top of the input file? Or PARAM cards?
It's totally okay if you don't have any more info! I'll keep trying! :-)
Best regards,
Tim
Hi Tim,
I see what you mean now. I am using MSC Nastran from Hexagon, I don't know if this can be the issue.
Would it be possible for you to share the input file that you are using to generate the .op4 to compare it with my own input?
Unfortunately I can't share mine.
Thank you and best regards,
Fabiola
Great idea, Fabiola! And thank you for being willing to give it a shot. :-)
Let me know if that runs. If so, I'll finish editing the file to create the files I want. My fingers are crossed! :-)
Best regards,
Tim
Hi Tim,
I've run your input file and my first lines are:
100000000 -13 2 1MATD11A 1P,3E23.16|I16
1 0 15
14 1
6.2269300127281269E-01 6.4280494314714021E-01 8.5674171329533377E-01
I think this can help now! I attach the output files: output.zip
Best regards,
Fabiola
Thank you so much, Fabiola! This is very helpful! Now I'll just need a little time to figure out what I want the test files to look like.
Interestingly, the final lines of each file look like this:
7.6372731157474694E-01
******** 1 1
1.0000000000000000E+00
That makes me wonder if Nastran itself can read these files?? That last column is just an end-of-matrix marker, so it could be okay. Even if the asterisks are not okay, the binary should still work. I think.
These results change my thinking/goal a little bit, but I'm confident that these examples will give me the info I need to get the reader / writer to work for matrices up to 99,999,999 rows & cols.
I'll keep you posted.
Thanks again!
Best regards,
Tim
Hi Tim,
I'm happy this helps! Yes, you're right. I always get the end-of-matrix line.
If I can help somehow, just let me know.
Best regards and thank you for your effort!
Fabiola
Hi Fabiola,
I was able to run an MSC version of Nastran to generate some files for testing. I think that should help for the ascii format. However, I haven't yet been able to get the binary reader to give me the error message you got:
in _rd_bigmat_binary
Y = struct.unpack(numform % L, fp.read(bytesreal * L))
struct.error: unpack requires a buffer of 48 bytes
I wonder ... do you get that error relatively quickly? If so, do you think you could give me the first 100kb or so of the binary file that fails to load? The "dd" command on linux/unix is a great way to get the first part of the file:
dd if=<input file name> of=<output file name> bs=512 count=200
Actually, now that I think about it, you could then test loading that resulting file and see if you still get the message above. If so, great, that should be enough info for me. On the other hand, if you get an "end-of-file" error message, we'll need more bytes. What do you think?
I hope to get the ascii reader / writer working this weekend. We'll see.
I really appreciate your help & patience!!
Best regards,
Tim
Hello Tim,
I didn't have time to answer before, sorry!
Nice to hear that you could get good examples for the ascii format. I've followed your suggestion to get the first 100KB of the binary file and I get the same error.
Here you have the file:
output_bin_100KB.op4.zip
As always, just let me know if I can help further!
Best regards,
Fabiola
Hi Fabiola,
On the ascii front, I think I've got the reader working. Fingers crossed it works for your actual data. The ascii writer still needs a tweak that I'll make sometime before next week.
On the binary front, I can't find anything wrong. :-/ Thank you so much for that data! I got the same error message you did, and in this case, it turned out it was simply because we reached the end-of-file. That makes me wonder if your Nastran run successfully finished the write when it was outputting the binary file. (?) You could check the .f06 file and see what it says about the write. If that looks okay, I'm curious what op4.dir(filename) would report. If you get an error like the following, the reader reached the end-of-file:
File "... pyyeti/nastran/op4.py", line 739, in _skipop4_binary
reclen = self._Str_i4.unpack(self._fileh.read(4))[0]
struct.error: unpack requires a buffer of 4 bytes
You could also try to read the matrix with Nastran to see if that works. I'm uncertain what's happening.
What do you think?
Thanks again!!
Best regards,
Tim
Hi Tim!
This is what I get in the .f06 file.
*** USER INFORMATION MESSAGE 4106 (OUTPT4)
DATA BLOCK KGG WRITTEN ON FORTRAN UNIT 101 FROM DMAP MODULE OUTPUT4 IN BINARY (LTLE64) FORMAT
TRAILER= 101 12752022 12752022 6 1 315 0
NAME OF DATA BLOCK WRITTEN ON FORTRAN UNIT IS KGG
*** USER INFORMATION MESSAGE 4106 (OUTPT4)
DATA BLOCK MGG WRITTEN ON FORTRAN UNIT 101 FROM DMAP MODULE OUTPUT4 IN BINARY (LTLE64) FORMAT
TRAILER= 102 12752022 12752022 6 1 1 0
NAME OF DATA BLOCK WRITTEN ON FORTRAN UNIT IS MGG
If I run the op4.dir(filename), then I get:
kgg , 12752022 x 12752022, form=6, mtype=1
mgg , 12752022 x 12752022, form=6, mtype=1
(['kgg', 'mgg'], [(12752022, 12752022), (12752022, 12752022)], [6, 6], [1, 1])
I'm really not an expert of binary files. Do you think that extracting the first 100KB can lead to this problem?
I could share the whole file with you, would this help? I can't share it here anyway as you can imagine, it's a 5.4GB file.
Thanks!
Fabiola
Hi Fabiola!
So the file write looks good. I'd like to try more bytes if you're willing. As big as possible. I don't know the maximum file size that can be attached here, but it must be several Mb. What do you think? Try 10 or 20 Mb?
I think the ascii reader and writer are now good to go as of version 1.2.4. My fingers are crossed! :-)
I really appreciate your help!!
Best regards,
Tim
Hi Tim,
Here you should have the first 50MB: op4_50MB.op4.zip
The zip file is 25MB, which is the maximum I can attach here!
However, I've tried to run the op4.dir(filename) and I get the error now, so I am not sure this will help. Just let me know :)
Thank you for insisting on this task,
Fabiola
Thank you so much Fabiola!! I'll keep you posted! :-)
Hi Fabiola,
Bummer ... it read the first 50Mb of the file just fine. This is a tough one to debug! How would you feel about modifying op4.py to get a little data? I'm suggesting changing this line in _rd_bigmat_binary from:
Y = struct.unpack(numform % L, fp.read(bytesreal * L))
To:
try:
Y = struct.unpack(numform % L, fp.read(bytesreal * L))
except:
print(f"{c = }, {r = }, {fp.tell() = }")
raise
That would tell us which column it's trying to read, what row it's on, and the file position. What do you think?
Thank you for working with me on this!!
Best regards,
Tim
Hello Tim,
I've tried several things, including what you suggested, and I have finally had the chance to generate again the same matrices for the same big model.. and surprise for me: your original op4.py could read them.
This is good news but makes me feel terribly guilty for having wasted your time on this part of the issue, especially because you suspected there might be a problem with the file itself!
I guess that if the ascii part works well, the issue could be closed. As soon as I have the chance, I'll try to test some other cases just to make sure that everything is fine.
Again, I am really sorry Tim! :(
Best regards,
Fabiola
Hi Fabiola!
No need to apogize! You helped me improve the ascii reader / writer at the very least. Many thanks!!! :-) I'm very happy to hear that you're now able to read the binary versions too. I wonder what happened.
Please don't hesitate to let me know of any troubles! Or successes! :-) I'll leave this open for a little longer for any updates.
Thank you very much!
Best regards,
Tim
Hearing of no further troubles, I believe this is resolved.