Let’s design an application with Redis as a data store. What? What for?

Hello everyone! Many people know what Redis is, and if you don’t know, then official site can bring you up to date. Most people know Redis as a cache and sometimes as a message queue. But what if you go a little crazy and try to design an entire application using only Redis as the data store? What tasks can be solved using Redis?

In this article we will try to answer these questions.

Don’t switch.

What’s missing in this article?

  • It won’t cover every Redis data structure in depth. To do this, it is better to search for individual articles or read the documentation.

  • There will also be no production ready code that you could use in your work.

What’s in this article?

  • Various Redis data structures will be used on the example of implementing an analog of a dating application.

  • There will be code examples for kotlin + spring boot.

Learn to create and query user profiles

Let’s learn how to create cards with our users in the first step: names, likes, etc. their profiles are in fact.

For this we need a simple key value storage. How to do it?

Very simply, radish has a data structure – hash. In essence, this is just a familiar hash map for all of us.

You take out the key. You put the key. It’s simple.

Radish commands can be found directly here and here… the documentation even has an interactive window to try to execute these commands right on the site. And the whole list of commands can be found here here

Similar links work for all subsequent commands that we will consider.

In the code, we use RedisTemplate and use it almost everywhere. This is such a basic thing in the Spring ecosystem for working with Redis.

The only difference here from map is that we pass the so-called field as the first argument, i.e. our hash with which we will work.

fun addUser(user: User) {
  val hashOps: HashOperations<String, String, User> = userRedisTemplate.opsForHash()
	hashOps.put(Constants.USERS, user.name, user)
}

fun getUser(userId: String): User {
	val userOps: HashOperations<String, String, User> = userRedisTemplate.opsForHash()
	return userOps.get(Constants.USERS, userId)?: throw NotFoundException("Not found user by $userId")
}

Actually above is an example of how it might look on kotlin using Spring libraries.

Throughout the article, similar pieces of code from github

Updating user likes using Redis sheets

Great, now we have users with like info.

Now you need to find a way to properly update these likes. We assume that events can happen very often. And so let’s use an asynchronous update scheme through a certain queue. And we will read the information from the queue on a schedule.

Radish has leaves. And so this set of commands. Redis lists can be used both as a fifo queue and as a lifo stack.

In Spring, we work according to the same scheme, we get the necessary operations, this time we need ListOperations from RedisTemplate.

To write to list, we need to write to the right, since here we are simulating a fifo queue from right to left.

fun putUserLike(userFrom: String, userTo: String, like: Boolean) {
  val userLike = UserLike(userFrom, userTo, like)
  val listOps: ListOperations<String, UserLike> = userLikeRedisTemplate.opsForList()
	listOps.rightPush(Constants.USER_LIKES, userLike)
}

And now, according to the schedule, we will launch our job and execute the logic we already need in it.

At its core, we are simply transferring information from one radish data type to another. This is enough for us as an illustrative example.

fun processUserLikes() {
  val userLikes = getUserLikesLast(USERS_BATCH_LIMIT).filter{ it.isLike}
	userLikes.forEach{updateUserLike(it)}
}

Updating our user is as simple as possible, hello HashOperations from the previous section.

private fun updateUserLike(userLike: UserLike) {
  val userOps: HashOperations<String, String, User> = userLikeRedisTemplate.opsForHash()
  val fromUser = userOps.get(Constants.USERS, userLike.fromUserId)?: throw UserNotFoundException(userLike.fromUserId) 
  fromUser.fromLikes.add(userLike)
  val toUser = userOps.get(Constants.USERS, userLike.toUserId)?: throw UserNotFoundException(userLike.toUserId)
  toUser.fromLikes.add(userLike)

  userOps.putAll(Constants.USERS, mapOf(userLike.fromUserId to fromUser, userLike.toUserId to toUser))
}

We get the values ​​from the list, respectively, on the left and we want to get not one value, but several at once, and for this we use the method range… And there is an important point here. With range we will only get data from the list, but not delete it.

Therefore, after receiving our data, we need to delete it using a separate command using the operation trim(it seems there may already be questions).

private fun getUserLikesLast(number: Long): List<UserLike> {
  val listOps: ListOperations<String, UserLike> = userLikeRedisTemplate.opsForList()
  return (listOps.range(Constants.USER_LIKES, 0, number)?:mutableListOf()).filterIsInstance(UserLike::class.java)
    .also{ listOps.trim(Constants.USER_LIKES, number, -1) }
}

And the questions that may arise here are:

This is not possible out of the box. You need to receive data from the list in one stream, and handle all the nuances that arise on your own.

Sending push notifications to users using pub / sub

Moving forward, not stopping!

We already have user profiles and figured out how to handle the stream of likes from these same users.

But what if in cases of mutual like we want to immediately send a push notification to the user?

We already have an asynchronous process for handling likes, so let’s embed sending push notifications there. We will send them, of course, via web sockets, and you can send it via a web socket right after you got our like.

But let’s say that we want to execute some long-running logic on it, or just transfer the work with websockets to one component, then how can we get around the new restrictions?

We will take and re-transfer our data from one Redis data type (list) to another Redis data type (pub / sub).

fun processUserLikes() {
  val userLikes = getUserLikesLast(USERS_BATCH_LIMIT).filter{ it.isLike}
  pushLikesToUsers(userLikes)
  userLikes.forEach{updateUserLike(it)}
}

private fun pushLikesToUsers(userLikes: List<UserLike>) {
  GlobalScope.launch(Dispatchers.IO){
		userLikes.forEach {
			pushProducer.publish(it)
		}
  }
}
@Component
class PushProducer(val redisTemplate: RedisTemplate<String, String>, val pushTopic: ChannelTopic, val objectMapper: ObjectMapper) {

    fun publish(userLike: UserLike) {
        redisTemplate.convertAndSend(pushTopic.topic, objectMapper.writeValueAsString(userLike))
    }
}

The binding of the listener to the topic is located in configuration

Thus, we can take our subscriber into a separate service.

@Component
class PushListener(val objectMapper: ObjectMapper): MessageListener {
    private val log = KotlinLogging.logger {}

    override fun onMessage(userLikeMessage: Message, pattern: ByteArray?) {
        // websocket functionality would be here
        log.info("Received: ${objectMapper.readValue(userLikeMessage.body, UserLike::class.java)}")
    }
}

We find the nearest users through geo operations

We figured out the likes, but what a tinder without the ability to find the closest users to a given point.

GeoOperations is our solution. We will store the key value pair, user id and coordinates.

And we will use the method radius passing the user id relative to which we will search and, accordingly, the search radius itself.

Redis returns the result including the user id we passed

fun getNearUserIds(userId: String, distance: Double = 1000.0): List<String> {
    val geoOps: GeoOperations<String, String> = stringRedisTemplate.opsForGeo()
    return geoOps.radius(USER_GEO_POINT, userId, Distance(distance, RedisGeoCommands.DistanceUnit.KILOMETERS))
        ?.content?.map{ it.content.name}?.filter{ it!= userId}?:listOf()
}

Updating user locations via streams

Almost everything that needs to be implemented, but we again have a situation like with users’ likes, only theoretically, calls to the method for changing the user’s coordinates can occur much more often.

So we need a queue again, but it would be nice to have something more scalable.

Redis streams can help solve this problem.

Probably many people know about what Kafka is and what Kafka streams are, but these are completely different things. But the very concept of Kafka is just very similar to Redis streams. It is also a right ahead log data structure that has a consumer group and an offset.

This is a more complex data structure, but it allows, unlike lists, to process data in parallel and using a reactive approach.

For details it is worth referring to documentation

Spring has ReactiveRedisTemplate and RedisTemplate for working with Redis data structures. To write the value, it would be more convenient for us to use RedisTemplate, and to read it already ReactiveRedisTemplate. If we are talking about streams. But in such cases, nothing will work.

If someone is aware of why it works this way, because of Spring or Redis, then write in the comments.

fun publishUserPoint(userPoint: UserPoint) {
  val userPointRecord = ObjectRecord.create(USER_GEO_STREAM_NAME, userPoint)
  reactiveRedisTemplate
    .opsForStream<String, Any>()
    .add(userPointRecord)
    .subscribe{println("Send RecordId: $it")}
}

And our listener method will look like this:

@Service
class UserPointsConsumer(
  private val userGeoService: UserGeoService
): StreamListener<String, ObjectRecord<String, UserPoint>> {
  
  override fun onMessage(record: ObjectRecord<String, UserPoint>) {
      userGeoService.addUserPoint(record.value)
  }
}

Here we simply move our data into the data structure for geo search.

Counting Unique Sessions Using HyperLogLog

And finally, let’s imagine that we need to calculate how many users have entered the application per day.

Moreover, let’s agree that we can have sooooo many users. Therefore, a simple option through a hash map is not suitable for us due to the fact that it will consume a lot of memory. How can we do this with fewer resources?

This is where such a probabilistic data structure comes into play as hyper log log… Those interested can read more about it on their own in wikipedia… Its key feature is that this data structure allows us to solve our problem using significantly less memory than the option with a hash map.

fun uniqueActivitiesPerDay(): Long {
  val hyperLogLogOps: HyperLogLogOperations<String, String> = stringRedisTemplate.opsForHyperLogLog()
  return hyperLogLogOps.size(Constants.TODAY_ACTIVITIES)
}

fun userOpenApp(userId: String): Long {
  val hyperLogLogOps: HyperLogLogOperations<String, String> = stringRedisTemplate.opsForHyperLogLog()
  return hyperLogLogOps.add(Constants.TODAY_ACTIVITIES, userId)
}

Conclusion

In this article, we looked at various Redis data structures. Including geo operations and hyper log log, which few people know about. We used them to solve real problems.

In fact, we designed a whole Tinder, and you can do it in FAANG after this 🙂 Along the way, we highlighted the main nuances and problems that can be encountered when working with Redis.

Redis is a very functional data warehouse, and perhaps if your infrastructure already has it, then it’s worth looking at what other tasks you can solve using what you already have without unnecessary complications.

PS:

All code examples can be found at github

Write in the comments if something is not described accurately enough.

Also write as you generally similar format describing how to use this or that technology.

And follow us on twitter 🐦 @de____ro

Similar Posts

Leave a Reply Cancel reply