[issue] is there a limitation on the number of subjects/studies in docker compose xnat?

Question

[issue] is there a limitation on the number of subjects/studies in docker compose xnat?

weeleon opened this issue 3 years ago · 6 comments

Dear all, I am using a git clone from the current master branch on a 100GB storage ubuntu 18.04 machine.
Question : Is there a limitation on the number of subjects/studies that the docker xnat can support?
Problem : I have currently 116 subjects with DICOM CT and DICOM SEG each, all inside a single XNAT project, but it seems that I am unable to add any more patients to this project. However if I delete a few subjects I can then add some more, until I seem to hit some sort of limitation and then I get errors. For info, I am using the python xnat api to upload zip files, but I see the exact same effect when I use the compressed dicom upload function direct from the GUI. Thanks in advance.

Answer 1 · 2021-05-19T13:03:21.000Z

It sounds to me as if you are running out of disk space. Have you checked? In my estimates I allow 1GB per scan which is generous but you're in that ballpark. There is no limit to the number of subjects that I've met.

…

-- Matt South ***@***.*** <mailto:matt%40mattsouth.net>

On Wed, 19 May 2021, at 10:20 AM, Leonard Wee wrote: Dear all, I am using a git clone from the current master branch on a 100GB storage ubuntu 18.04 machine. Question : Is there a limitation on the number of subjects/studies that the docker xnat can support? Problem : I have currently 116 subjects with DICOM CT and DICOM SEG each, all inside a single XNAT project, but it seems that I am unable to add any more patients to this project. However if I delete a few subjects I can then add some more, until I seem to hit some sort of limitation and then I get errors. For info, I am using the python xnat api to upload zip files, but I see the exact same effect when I use the compressed dicom upload function direct from the GUI. Thanks in advance. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#77>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKWHTWGMBNUG33UKPRTSDTON7GLANCNFSM45EHHIMA>.

Answer 2 · 2021-05-19T13:51:37.000Z

Yeah, there’s no difference between the XNAT application running in a Docker container and running in a traditional server context. Our largest deployment at WUSTL has around 72,000 subjects with hundreds of thousands of image sessions and other experiment data. That does consume a LOT of disk space and database operations and other processing consumes a lot of RAM. It's hard to say more about that without knowing what errors you’re seeing and, even more importantly, what’s being logged on the back-end. But given that you have 116 subjects, each with its own image session, plus disk space required just for infrastructure like database storage and what not, I agree with Matt that it sounds like you’re running into resource limitations and most likely disk space.

-- Rick Herrick Sr. Programmer/Analyst Neuroinformatics Research Group Washington University School of Medicine Phone: +1 (314) 273-1645 From: Matt South ***@***.***> Date: Wednesday, May 19, 2021 at 8:03 AM To: NrgXnat/xnat-docker-compose ***@***.***> Cc: Subscribed ***@***.***> Subject: Re: [NrgXnat/xnat-docker-compose] [issue] is there a limitation on the number of subjects/studies in docker compose xnat? (#77) * External Email - Caution * It sounds to me as if you are running out of disk space. Have you checked? In my estimates I allow 1GB per scan which is generous but you're in that ballpark. There is no limit to the number of subjects that I've met.

-- Matt South ***@***.*** <mailto:matt%40mattsouth.net>

On Wed, 19 May 2021, at 10:20 AM, Leonard Wee wrote: Dear all, I am using a git clone from the current master branch on a 100GB storage ubuntu 18.04 machine. Question : Is there a limitation on the number of subjects/studies that the docker xnat can support? Problem : I have currently 116 subjects with DICOM CT and DICOM SEG each, all inside a single XNAT project, but it seems that I am unable to add any more patients to this project. However if I delete a few subjects I can then add some more, until I seem to hit some sort of limitation and then I get errors. For info, I am using the python xnat api to upload zip files, but I see the exact same effect when I use the compressed dicom upload function direct from the GUI. Thanks in advance. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#77>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKWHTWGMBNUG33UKPRTSDTON7GLANCNFSM45EHHIMA>.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#77 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAC2ZUMZ7LHHXX2QBFHSUTLTOOZLDANCNFSM45EHHIMA>.

…

________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

Answer 3 · 2021-05-19T14:03:09.000Z

Hi Matt, Hi Rick, I do agree, was also my working hypothesis, I think the cause may be that docker itself imposes a 10GB limit regardless of how much physical disk space you have available, to prevent multiple containers from competing for storage with each other. I think I must have somehow hit the internal docker limit for a single container that it is not able to “grow” any more. Just to reiterate, I have tons of RAM and disk space, so this feels more like a container issue. Do you agree? Gr./L From: Rick Herrick ***@***.***> Sent: woensdag 19 mei 2021 15:52 To: NrgXnat/xnat-docker-compose ***@***.***> Cc: Leonard Wee ***@***.***>; Author ***@***.***> Subject: Re: [NrgXnat/xnat-docker-compose] [issue] is there a limitation on the number of subjects/studies in docker compose xnat? (#77) Yeah, there’s no difference between the XNAT application running in a Docker container and running in a traditional server context. Our largest deployment at WUSTL has around 72,000 subjects with hundreds of thousands of image sessions and other experiment data. That does consume a LOT of disk space and database operations and other processing consumes a lot of RAM. It's hard to say more about that without knowing what errors you’re seeing and, even more importantly, what’s being logged on the back-end. But given that you have 116 subjects, each with its own image session, plus disk space required just for infrastructure like database storage and what not, I agree with Matt that it sounds like you’re running into resource limitations and most likely disk space.

-- Rick Herrick Sr. Programmer/Analyst Neuroinformatics Research Group Washington University School of Medicine Phone: +1 (314) 273-1645 From: Matt South ***@***.***> Date: Wednesday, May 19, 2021 at 8:03 AM To: NrgXnat/xnat-docker-compose ***@***.***> Cc: Subscribed ***@***.***> Subject: Re: [NrgXnat/xnat-docker-compose] [issue] is there a limitation on the number of subjects/studies in docker compose xnat? (#77) * External Email - Caution * It sounds to me as if you are running out of disk space. Have you checked? In my estimates I allow 1GB per scan which is generous but you're in that ballpark. There is no limit to the number of subjects that I've met.

-- Matt South ***@***.*** <mailto:matt%40mattsouth.net>

On Wed, 19 May 2021, at 10:20 AM, Leonard Wee wrote: Dear all, I am using a git clone from the current master branch on a 100GB storage ubuntu 18.04 machine. Question : Is there a limitation on the number of subjects/studies that the docker xnat can support? Problem : I have currently 116 subjects with DICOM CT and DICOM SEG each, all inside a single XNAT project, but it seems that I am unable to add any more patients to this project. However if I delete a few subjects I can then add some more, until I seem to hit some sort of limitation and then I get errors. For info, I am using the python xnat api to upload zip files, but I see the exact same effect when I use the compressed dicom upload function direct from the GUI. Thanks in advance. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#77>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKWHTWGMBNUG33UKPRTSDTON7GLANCNFSM45EHHIMA>.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#77 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAC2ZUMZ7LHHXX2QBFHSUTLTOOZLDANCNFSM45EHHIMA>.

…

________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#77 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABZKKUXNUY45QB6WWYYQVPDTOO7AJANCNFSM45EHHIMA>.

Answer 4 · 2021-05-19T14:15:04.000Z

If you’re working from the master branch version of xnat-docker-compose, then this wouldn’t be the issue. The archive data for that is stored on the host machine in the folder xnat-data/archive, so it’s not part of the container storage. Another thing that gets stored on the host machine is the XNAT log files. These are in the folder xnat-data/home/logs. Look through those files and see what you can find in terms of messages that correspond with times when you run into these errors.

-- Rick Herrick Sr. Programmer/Analyst Neuroinformatics Research Group Washington University School of Medicine Phone: +1 (314) 273-1645 From: Leonard Wee ***@***.***> Date: Wednesday, May 19, 2021 at 9:03 AM To: NrgXnat/xnat-docker-compose ***@***.***> Cc: Herrick, Rick ***@***.***>, Comment ***@***.***> Subject: Re: [NrgXnat/xnat-docker-compose] [issue] is there a limitation on the number of subjects/studies in docker compose xnat? (#77) * External Email - Caution * Hi Matt, Hi Rick, I do agree, was also my working hypothesis, I think the cause may be that docker itself imposes a 10GB limit regardless of how much physical disk space you have available, to prevent multiple containers from competing for storage with each other. I think I must have somehow hit the internal docker limit for a single container that it is not able to “grow” any more. Just to reiterate, I have tons of RAM and disk space, so this feels more like a container issue. Do you agree? Gr./L From: Rick Herrick ***@***.***> Sent: woensdag 19 mei 2021 15:52 To: NrgXnat/xnat-docker-compose ***@***.***> Cc: Leonard Wee ***@***.***>; Author ***@***.***> Subject: Re: [NrgXnat/xnat-docker-compose] [issue] is there a limitation on the number of subjects/studies in docker compose xnat? (#77) Yeah, there’s no difference between the XNAT application running in a Docker container and running in a traditional server context. Our largest deployment at WUSTL has around 72,000 subjects with hundreds of thousands of image sessions and other experiment data. That does consume a LOT of disk space and database operations and other processing consumes a lot of RAM. It's hard to say more about that without knowing what errors you’re seeing and, even more importantly, what’s being logged on the back-end. But given that you have 116 subjects, each with its own image session, plus disk space required just for infrastructure like database storage and what not, I agree with Matt that it sounds like you’re running into resource limitations and most likely disk space.

-- Rick Herrick Sr. Programmer/Analyst Neuroinformatics Research Group Washington University School of Medicine Phone: +1 (314) 273-1645 From: Matt South ***@***.***> Date: Wednesday, May 19, 2021 at 8:03 AM To: NrgXnat/xnat-docker-compose ***@***.***> Cc: Subscribed ***@***.***> Subject: Re: [NrgXnat/xnat-docker-compose] [issue] is there a limitation on the number of subjects/studies in docker compose xnat? (#77) * External Email - Caution * It sounds to me as if you are running out of disk space. Have you checked? In my estimates I allow 1GB per scan which is generous but you're in that ballpark. There is no limit to the number of subjects that I've met.

-- Matt South ***@***.*** <mailto:matt%40mattsouth.net>

On Wed, 19 May 2021, at 10:20 AM, Leonard Wee wrote: Dear all, I am using a git clone from the current master branch on a 100GB storage ubuntu 18.04 machine. Question : Is there a limitation on the number of subjects/studies that the docker xnat can support? Problem : I have currently 116 subjects with DICOM CT and DICOM SEG each, all inside a single XNAT project, but it seems that I am unable to add any more patients to this project. However if I delete a few subjects I can then add some more, until I seem to hit some sort of limitation and then I get errors. For info, I am using the python xnat api to upload zip files, but I see the exact same effect when I use the compressed dicom upload function direct from the GUI. Thanks in advance. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#77>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKWHTWGMBNUG33UKPRTSDTON7GLANCNFSM45EHHIMA>.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#77 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAC2ZUMZ7LHHXX2QBFHSUTLTOOZLDANCNFSM45EHHIMA>.

…

________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#77 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABZKKUXNUY45QB6WWYYQVPDTOO7AJANCNFSM45EHHIMA>. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#77 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAC2ZUIBCTZSJGCRMT6AO4DTOPALNANCNFSM45EHHIMA>.

________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail.

Answer 5 · 2021-05-22T11:47:26.000Z

Hi Rick, we just tracked down the out of space issue. It was indeed storage that ended up getting filled up with drivers and other rubbish. I will wipe out the instance and try again. Thank you have helping locate the disk space issue. Gr./L From: Rick Herrick ***@***.***> Sent: 19 May 2021 15:52 To: NrgXnat/xnat-docker-compose ***@***.***> Cc: Leonard Wee ***@***.***>; Author ***@***.***> Subject: Re: [NrgXnat/xnat-docker-compose] [issue] is there a limitation on the number of subjects/studies in docker compose xnat? (#77) Yeah, there’s no difference between the XNAT application running in a Docker container and running in a traditional server context. Our largest deployment at WUSTL has around 72,000 subjects with hundreds of thousands of image sessions and other experiment data. That does consume a LOT of disk space and database operations and other processing consumes a lot of RAM. It's hard to say more about that without knowing what errors you’re seeing and, even more importantly, what’s being logged on the back-end. But given that you have 116 subjects, each with its own image session, plus disk space required just for infrastructure like database storage and what not, I agree with Matt that it sounds like you’re running into resource limitations and most likely disk space.

-- Rick Herrick Sr. Programmer/Analyst Neuroinformatics Research Group Washington University School of Medicine Phone: +1 (314) 273-1645 From: Matt South ***@***.***> Date: Wednesday, May 19, 2021 at 8:03 AM To: NrgXnat/xnat-docker-compose ***@***.***> Cc: Subscribed ***@***.***> Subject: Re: [NrgXnat/xnat-docker-compose] [issue] is there a limitation on the number of subjects/studies in docker compose xnat? (#77) * External Email - Caution * It sounds to me as if you are running out of disk space. Have you checked? In my estimates I allow 1GB per scan which is generous but you're in that ballpark. There is no limit to the number of subjects that I've met.

-- Matt South ***@***.*** <mailto:matt%40mattsouth.net>

On Wed, 19 May 2021, at 10:20 AM, Leonard Wee wrote: Dear all, I am using a git clone from the current master branch on a 100GB storage ubuntu 18.04 machine. Question : Is there a limitation on the number of subjects/studies that the docker xnat can support? Problem : I have currently 116 subjects with DICOM CT and DICOM SEG each, all inside a single XNAT project, but it seems that I am unable to add any more patients to this project. However if I delete a few subjects I can then add some more, until I seem to hit some sort of limitation and then I get errors. For info, I am using the python xnat api to upload zip files, but I see the exact same effect when I use the compressed dicom upload function direct from the GUI. Thanks in advance. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#77>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKWHTWGMBNUG33UKPRTSDTON7GLANCNFSM45EHHIMA>.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#77 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AAC2ZUMZ7LHHXX2QBFHSUTLTOOZLDANCNFSM45EHHIMA>.

…

________________________________ The materials in this message are private and may contain Protected Healthcare Information or other information of a sensitive nature. If you are not the intended recipient, be advised that any unauthorized use, disclosure, copying or the taking of any action in reliance on the contents of this information is strictly prohibited. If you have received this email in error, please immediately notify the sender via telephone or return mail. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<#77 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ABZKKUXNUY45QB6WWYYQVPDTOO7AJANCNFSM45EHHIMA>.

Answer 6 · 2021-05-22T11:48:37.000Z

I have emailed Rick. The issue does seem to be I suddenly ran out of disk storage space unrelated to anything XNAT was doing. Thank you Rick for helping locate the problem.