Circus Train might be using too many database connections for the Housekeeping database

[jmnunezizu]

Hi,

I have a cluster running about ~20 concurrent replication jobs (housekeeping runs separately).

I'm using an AWS RDS Aurora Cluster with 2x db.t2.small for the Housekeeping database. Each instance, according to the parameter group configured for max_connections, sets the value to 45.

mysql> show global variables like 'max_connections';
+-----------------+-------+
| Variable_name   | Value |
+-----------------+-------+
| max_connections | 45    |
+-----------------+-------+
1 row in set (0.00 sec)

Surprisingly, with ~20 concurrent jobs, I'm getting "Too Many Connections" exception:

26-09-2018 05:15:35 PDT lz_hcom_shopper_lookup INFO - java.sql.SQLNonTransientConnectionException: Too many connections

My suspicion is that Circus Train is using quite a large Hikkari db connection pool for the housekeeping database. What's the configured number?

For what it needs to do with the Housekeeping database, I'd expect 1 connection to be enough?

For the time being, I've upgraded the instance type to db.r3.large, which is a bit more expensive, but has a max_connections value of 1000.

Please let me know if more information is required.

Thanks,
Jose.-

[rksangeeth007]

+1

[patduin]

I'm changing the default in Housekeeping: ExpediaGroup/housekeeping#35

You can always overwrite by setting the: spring.datasource.max-active in your yml.
Update ^ I can confirm this is how it works in yaml will do just fine (nonetheless we'll add a more modest default): spring.datasource.max-active: 2

[massdosage]

Ah OK, so you can already configure this yourself. We should add this to the HK and CT documentation.

[patduin]

Default connections lowered from 50 to 2 in Housekeeping 3.0.0