Report on response times consistency over daily KCB runs on AWS environment

[andyuk1986]

The goal is to collect and analyze the response time numbers over daily KCB runs on AWS environment to understand whether the numbers are consistent, how much is the fluctuation and what could cause it, and if any additional actions/improvements are needed from our side based on the analyzed data.

It has been chosen to analyze the 50th percentile response time for cpuUsageLoginTest benchmark. The response times were taken for Browser to Log In Endpoint and Browser posts correct credentials Gatling tests to compare the response time with and without DB involvement/password hashing.

Metrics comparison for test without DB usage

Please find below the metrics comparison for cpuUsageLoginTest->Browser to Log In Endpoint test, which doesn't use DB during run.

The data is captured starting from 12.09.2024, i.e. after the lambda issue has been fixed with this commit . As you can see, the 50thPctRespTime has consistent value of 56ms, then there is a bit drop[1] starting from 19.09.2024 build, but after again the number is consistently 53ms.

Metrics comparison for test with DB usage

Please find below the metrics comparison for cpuUsageLoginTest->Browser posts correct credentials test, which works with DB and password hashing.

Here you can see that the 50thPctRespTime mainly fluctuates between 115ms and 120ms , which is about 5ms. By looking at these numbers we can say that the fluctuation is about ±3%.

As the metric is consistent (there are no fluctuations) for the test without DB involvement, we can assume that the fluctuation here is caused by DB usage and password hashing.

As a conclusion based on the data given above, we can say that running KCB on AWS environment ensures consistent reliable response numbers with around ±3% fluctuation in certain cases.

[1] There is an issue reported here as performance drop was spotted starting from 19.09.2024 daily runs.

[kami619]

@andyuk1986 great writeup Anna, I came up with a python script for community users to be able to leverage the data from the daily runs, as they don't have insight into the Horreum data. Should we think of hosting a community Horreum instance for folks to look at these reports on their own giving more visibility to the Keycloak project performance characteristics in general?