After working with SAP HANA database for a couple of years I have to say that it is a wonderful database and platform. Everything seems to run smoothly and faster than light… Except when a big problem appears and everything goes to hell. This is one of those times when you forgot to update your database periodically.

The problem

One of the users reported a problem when trying to confirm a shopping cart using the SAP Business Workplace. After checking the SAP system I found the following dump:

After this moment a lot of users complained that the SAP system was running really slow, even with that the Average Response Time for Dialog tasks was fine:

The investigation

Next step was to check the database. There is a wonderful SAP Note (1969700 – SQL Statement Collection for SAP HANA) with  SQL scripts you can use to check the health of the HANA database. If you want to interpret the results of those SQL scripts you can use SAP Note 1999993 – How-To: Interpreting SAP HANA Mini Check Results.

Inside of the first SAP Note there is a SQL script to check the top memory consumers in the HANA database. I executed this script on our HANA database and the result was the following one:

As you can see the allocator Pool/malloc/libhdbrskernel.so is using about 50.64GB of RAM memory within the server. If we compare it with the first SAP table on the top consumers we see the difference is huge. The table D010TAB is consuming 4.51GB against the 50.64GB of the first allocator. Considering this we can say there is something really wrong with this database…

The next step was finding why this allocator was using so much memory, for this I used the SAP Note 1999997 – FAQ: SAP HANA Memory.  I search for this allocator and found the following information:

The version of the SAP HANA database was 102.00. It was affected by the bugs described on SAP Notes 2253121 – Memory Leak in Pool Pool/malloc/libhdbrskernel.so when Using Batch Inserts Into Rowstore Tables and 2275252 – Memory Leak in “Pool/malloc/libhdbrskernel.so” When SqlScript Procedures or UDFs Get Compiled.

The solution

The only workaround possible was to set the following parameter:

ALTER SYSTEM ALTER CONFIGURATION (‘indexserver.ini’, ‘SYSTEM’) SET (‘row_engine’, ‘dynamic_parallel_insert_max_workers’) = ‘1’ WITH RECONFIGURE;

I’m not a friend of temporary solutions so I decided to update the SAP HANA version to 122.01 which was the last one available.  After that I executed the same SQL script for showing the top consumers:

The allocator Pool/malloc/libhdbrskernel.so didn’t appear on the list and there was a new allocator. After checking the same SAP Note I described before we can see that the allocator Pool/PersistenceManager/PersistentSpace/DefaultLPA/Page could be related with a bad disk I/O. Since the memory is not so high and there are no OOM (Out Of Memory) situations I think there is no problem with the database.

Conclusions

Having a periodic maintenance for the database that includes a version update is one of the most important things you should have. Otherwise there could be an issue that forces you to update the database as soon as possible, disrupting the daily work and affecting all the users. Remember to keep updated your database version!