Pagination. Non-standard use of Spring's Page and Pageable

beeline cloud I talked about Spring Data JPA and Hibernate – I raised the issue of solving the problem of dynamically changing database queries. In this article I will show how to apply spring pagination on the List<> interface.

I remember in one of the SpringBoot projects, the front-end requested data from the backend and produced the result in a page-by-page format. To obtain data from the database, Spring Data JPA was used and PagingAndSortingRepository. In this case there was some pagination on List<>, but first things first.

The structure of the project, if we exclude other classes not related to this example, had approximately the following architecture:

At this step, the request from the controller goes to the service:

@Service
@RequiredArgsConstructor
@FieldDefaults(level = AccessLevel.PRIVATE, makeFinal = true)
public class FooService {
   FooMapper fooMapper;
   FooEntityRepository fooEntityRepository;
 
   public PagedFooDto getFooDto(int page, int pageSize)  {
       Page<FooEntity> fooEntities = fooEntityRepository.findAll(PageRequest.of(page, pageSize));
 
       return fooMapper.toPagedFooDto(fooEntities);
   }
}

The service receives data from the database. After this, the mapper collects for the front:

@Mapper(componentModel = "spring",
       nullValueCheckStrategy = NullValueCheckStrategy.ALWAYS,
       nullValuePropertyMappingStrategy = NullValuePropertyMappingStrategy.IGNORE)
public interface FooMapper {
 
   @Mapping(target = "totalPages", source = "totalPages")
   @Mapping(target = "totalElements", source = "totalElements")
   @Mapping(target = "pageSize", source = "size")
   @Mapping(target = "pageElements", source = "numberOfElements")
   @Mapping(target = "items", source = "content")
   PagedFooDto toPagedFooDto(Page<FooEntity> entities);
 
   FooDto toFooDto(FooEntity entity);
}

and returns the data. This takes about 80-360 ms depending on the parameters and load.

Task: Due to the heavy load and the request “let’s send data to this endpoint in 20ms,” it was decided to separate the logic associated with this endpoint into a separate microservice and “hold” large amounts of data in the cache. Here I will omit the story of trying many different caching solutions, but I met the timings inmemory cache based caffeine.

The first thing I wanted to do was write my own implementation, which receives data from the cache, breaks it into pages and returns the required one. Quite quickly I managed to create a simple pagination for the sheet:

@UtilityClass
public class PaginationListUtils {
 
   public <E> List<E> returnPagedList(List<E> data, int page, int pageSize) {
 
       // Вычисляем индекс элемента
   	int startIndex = page * pageSize;
 
       // Проверяем на null и что индекс не выходит за рамки data.size()
   	if(data == null || data.size() <= startIndex){
       	return Collections.emptyList();
   	}
 
   	// Вычисляем список элементов, которые соответствуют запросу
   	return data.subList(startIndex, Math.min(startIndex + pageSize, data.size()));
   }
}

But the question of refactoring immediately arose. Either you will have to introduce an additional wrapper class, which, in addition to the data collection, will store metadata (which was previously stored in the spring Page<> object), or at the level of this or an adjacent method (which will also need to be implemented), perform calculations to fill out the required front metadata, such as number of pages, total number of elements, etc. This means that you need to rewrite both the mapper and the controller… and the service, since it will “hook everyone”.

Since springdata pagination launches a full scan on the target table and then carries out calculations, I made the assumption that somewhere there was a ready-made implementation of pagination, and went deeper. I came across the PageImpl<> class, which has an excellent constructor:

public PageImpl(List<T> content, Pageable pageable, long total) {
 
   super(content, pageable);
 
   this.total = pageable.toOptional().filter(it -> !content.isEmpty())//
     	.filter(it -> it.getOffset() + it.getPageSize() > total)//
     	.map(it -> it.getOffset() + content.size())//
     	.orElse(total);
}

But without really understanding it, I just rewrote the service class a little:

@Service
@RequiredArgsConstructor
@FieldDefaults(level = AccessLevel.PRIVATE, makeFinal = true)
public class FooService {
   FooMapper fooMapper;
   Cache<String, FooEntity> cache;
 
   public PagedFooDto getFooDtoFromCache(int page, int pageSize)  {
       List<FooEntity> fooEntities = cache.asMap().values().stream().toList();
 
       return fooMapper.toPagedFooDto(new PageImpl<>(fooEntities, PageRequest.of(page, pageSize), fooEntities.size()));
   }
}

During the tests, having previously stored 10,000 elements in the cache, I submitted page 0 as input, indicating the number of elements – 15. But I did not get the result I was hoping for. The metadata was calculated incorrectly and all elements were returned on each page:

As you can see from the screenshot above, there were a total of 667 pages and 10,000 items elements. The result surprised me and I decided to submit page 668 as an input. This is what happened:

Not only did all the data come back, but the item and page metrics doubled. And it dawned on me that the calculations that are performed under the hood of Spring are based on ready-made data, cut from the general array, which is stored in the cache.

And since we already have a ready-made method that receives the sublist (PaginationListUtils->returnPagedList), it is enough to supplement it to return Page<> instead of List<>:

public static <E> Page<E> returnPagedList(Pageable pageable, List<E> data) {
   List<E> result;
 
   int pageSize = pageable.getPageSize();
   int page = pageable.getPageNumber();
   int startIndex = page * pageSize;
 
   // Проверяем на null и что индекс не выходит за рамки data.size()
   if (CollectionUtils.isEmpty(data) || data.size() <= startIndex) {
   	result = Collections.emptyList();
   } else {
   	// Вычисляем список элементов которые соответствуют запросу
   	result = data.subList(startIndex, Math.min(startIndex + pageSize, data.size()));
   }
 
   return new PageImpl<>(result, pageable, data.size());
}

And then we change the service class a little:

public PagedFooDto getFooDtoFromCache(int page, int pageSize)  {
   List<FooEntity> fooEntities = cache.asMap().values().stream().toList();
 
   return fooMapper.toPagedFooDto(PaginationListUtils.returnPagedList(PageRequest.of(page, pageSize), fooEntities));
}

Thus, by adding a new class and slightly changing the service, we will get the required result.

More additional articles can be found in our VAYTI project. All materials are based on real events and experiences:

The history of hacking and its elimination. An article about how the company dealt with the consequences of an infrastructure hack and a ransomware attack. We had to restore domain controllers and network storage, configure AD, DHCP and DNS, and implement Active Directory policies. Material in instructional format for those who find themselves in a similar situation.
Shodan, Censys, SpiderFoot – a short guide to assessing the attractiveness of an organization for hackers. A practical guide to assessing a company’s external perimeter for data useful to a hacker. About identifying attack vectors, as well as services that automate the collection of information (grabbers, dorks). The material will be useful to both information security employees and top management.
How to protect biometric data from theft and hacking. What is the main power of biometric data? In methods of protection against counterfeiting. This is a compact material on best practices and approaches to working with fingerprints, voice and facial authentication.
How to instill a culture of cybersecurity in your employees. In 90% of cases, the culprit of an attack on corporate infrastructure is an employee. An article about how to develop a culture of cyber hygiene within a company and what to teach colleagues.
How we speed up workflows with ChatGPT. The team leader of the development team explains how a chatbot helps save time on routine tasks and prototyping.