Crippling response time of Sefran3 initial page due to exponential increase in the number of requests to the fdsnws service in the context of intense seismic activity
Closed this issue · 2 comments
The OVSG team often complains of excessive slowness of the initial page of Sefran3.
Today I was told that it is annoying to have to wait a long time to display the last 6 hours (default) and that it is almost impossible to display more. So the first person who looks at the Sefran in the morning must open the last hour detailed page and then go backwards again and again until displaying the last analyzed hour.
I have looked at the source code to understand why and where our servers are low to serve these requests, and I discovered that this bug is critical because it will increase in exponential form with seismic activity.
In sefran3.pl, there are several nested «for» loops which cause an exponential increase in the number of requests to the fdsnws service.
In simplified form, these are:
for (@dates) {
for (@listeHeures) {
for (reverse @mclist) {
my %MC = mcinfo($_);
The problem is that @mclist contains all events for last week and mcinfo() queries the fdsnws service.
Today, we (only) have 216 event for the last week. But when I ask for the last 6 hours, this will generate 6*216 requests, in a burst of 1296 HTTP requests.
In the case of intense seismic activity, with 5000 events in a week, if somebody wants to display the last 24 hours, it would be a burst of 120'000 requests. The web page doesn't displays, the user will retry by refreshing the page, or displaying it on another computer, as all the team, because we have to see this intense seismic activity.
In this context, the sefran3 initial page doesn't work and the FDSNWS service may crash due to a kind of (distributed) denial-of-service attack
Thanks for the feedback on the sefran3 display code. Note that other observatories using the same code (some with much more seismic events during crisis, i.e., thousand in a day) have observed a slowness behavior but not as critical as yours. We will optimize the loop in a future release.