.svg){kind=small}
RonDB 24.10 introduces rate limits and quotas to support multi-tenant settings running thousands of concurrent projects on a single RonDB cluster. Rate limits and quotas manage memory and disk space, CPU usage, and real-time interference by slowing or blocking overuse and capping storage, making RonDB more scalable and flexible.
The release of RonDB 24.10 introduced a new feature that improves the use of RonDB in multi-tenant settings: rate limits and quotas. Rate limits and quotas can limit the number of concurrent transactions made as well as the size of transactions made.
To understand why we have included rate limits and quotas in RonDB, we must first understand some important background to Hopsworks and RonDB.
-------------
One of the services Hopsworks offers is a free service to run Hopsworks workloads in a managed cloud. Thousands of individuals and companies have used this service to experiment with AI Lakehouse applications of various sorts, such as predicting the weather in your location and other experimental machine learning applications.
This Hopsworks service means that thousands of projects can run concurrently on one Hopsworks cluster. A Hopsworks cluster uses RonDB for three things:
Using RonDB for an online feature store means that users import data from their data pipelines, some of which is directed to the online feature store and some of it is directed to the offline feature store. The data in the online feature store is used for machine learning inference in real-time applications, including services such as personalized search services, credit fraud analysis, and many, many other applications.
The offline feature store uses HopsFS, a distributed file system that is built on top of RonDB. HopsFS stores the file data in a backend storage system, often using storage systems such as S3, Scality, Google Cloud Storage, Azure Blob Storage, or other storage systems. The data in HopsFS can be used by Hudi, Apache Iceberg, DuckDB, and other query services for training machine learning models and performing batch inferencing.
RonDB is also used to store metadata about the features, job control, file system, and many other things that are used to operate Hopsworks.
Having a multi-tenant database management system (DBMS) with thousands of concurrent projects running in one RonDB cluster is obviously a challenge. RonDB is required to provide response times down to less than a millisecond while fetching a batch of hundreds of rows to serve a single personalized search request.
Tables in RonDB can store the payload data, which could be hundreds of features in one table (a feature group), either in memory for best latency and performance or in a disk column to provide cheaper storage at a slightly higher cost of accessing the data.
To handle these thousands of projects, each project has one database in RonDB. In order to handle many tables in RonDB, we have added the capability to configure RonDB to support hundreds of thousands of tables concurrently.
Figure 1 shows how the data server side is implemented by the RonDB data nodes together with the RonDB management server. The HopsFS accesses RonDB through Cluster J. Cluster J is the native Java NDB API. The applications can either access RonDB using a set of MySQL servers or through the RonDB REST API server.
The RonDB REST API server delivers capabilities for key lookups and batched key lookups in real-time. In RonDB 24.10, it provides a new endpoint of RonSQL. RonSQL can handle simple SQL queries that retrieve aggregate information from a table in RonDB. The same query in RonSQL is about 20 times faster than sending it through the MySQL server.
We introduced rate limits and quotas in RonDB 24.10 to help manage:
To manage memory and disk space, RonDB tracks each memory and disk page used by a specific database. When the database has hit its memory quota, it is no longer possible to insert, write, or update the data. However, it is still possible to delete and read the data.
RonDB manages CPU resources by keeping track of the CPU usage for each database in real-time. As long as the application using the database is within the limits of its CPU rate limit, the operation continues on as normal. However, if the application tries to use more CPU than it has requested, RonDB will slow down the project to ensure other projects get real-time service. The more overload the project tries to create, the more it will slow down. If slowing down does not work, the project will not be able to complete any queries in RonDB until the CPU “debt” is paid off. Each time interval the database pays its “debt”, and if things go as normal with rate limitation, the “debt” will be set back to zero.
When defining the rate limits, we measure it in microseconds of CPU time per second per data node. Thus, in a 2-node RonDB cluster setting, a rate limit of 100.000 means you get access to 0.1 CPUs in each data node. Setting the rate limit to 1000.000 means you get access to 1 millisecond of CPU time per second per data node, which is 0.001 CPUs in each data node.
Different projects can have different requirements. One could require 0.001 CPUs and another could require 1.5 CPUs. These databases can easily co-exist in the RonDB cluster and they will both be able to get reliable, real-time service.
An example of how this works was when we ran a Sysbench OLTP RW benchmark. We set the rate limit to 100.000 (0.1 CPU per data node). This made it possible to run 330 transactions per second (TPS), which equates to 6600 SQL queries per second, delivering almost 150.000 rows to the application. This TPS was achieved from 1 thread to 256 threads. Increasing the rate limit to 500.000 resulted in getting 5 times more TPS.
.png)
Clearly, the use of rate limits and quotas would be very useful for a company with many departments that want to share one Hopsworks cluster.
This example shows how we can operate the actual data servers in RonDB. The data server clients are stateless clients and can thus scale up and down as needs come and go. In addition, clients in Hopsworks can access the data server clients using load balancers. Together with Hopsworks 4.0, which is operated by Kubernetes, RonDB 24.10 will be extremely flexible in scaling up and down CPU, memory, and disk space on the data server side as well as CPU space on the client side with the introduction of rate limits and quotas.
Learn about the other 24.10 features
Download the latest RonDB version
Powered by Hopsworks © RonDB 2025. All rights reserved
