How to store data in the cloud? A brief tour of technology

Companies need somewhere to store large amounts of data, but creating their own data centers is not an easy task. Cloud storage from providers comes to the rescue. It helps you save on your own infrastructure and securely store data in different formats. Simply select the appropriate storage type: block, file or object. We talk about the difference between them in the article. We also share instructions on how to get started with

object storage

.


Use the navigation to select the block you are interested in:

Introduction
Benefits of cloud storage
What is cloud storage used for?
Types of cloud storage
Conclusion: the main advantage of cloud storage

Introduction


Model
cloud storage similar to “storage car sharing”. Instead of investing time in organizing a data center, and later in regular maintenance, we turn to companies that already have it – cloud storage providers.

Data centers contain endless server racks with equipment, monitored by security systems and professional administrators. Thus, the supplier relieves the client of concerns about information security.

There are two parties in the model: the client and the service provider (provider). The client rents space on the provider’s servers, stores documents, applications, and static website content on them, accessing them remotely. The supplier organizes storage, maintenance, security and access to the data. This model has several advantages.

Benefits of cloud storage



Saving. For your own storage system you will need: premises, racks, servers, cooling, infrastructure equipment. You will need to organize backup with the purchase of software and additional drives. And all this must be maintained by administrators, for example, carrying out periodic preventive maintenance with shutdowns and updates.

With cloud storage, most operational costs are reduced and capital costs are eliminated.

Reliability. Cloud storage is maintained by engineers with specialized experience in operating systems of this type. Provider administrators regularly update hardware, improve software, and work on security.

At the same time, the data is stored “with a reserve”: to store 1 GB of client data, the supplier most often reserves it multiple times (in our cloud storage three times, that is, instead of 1 GB, 3 GB are reserved).

Safety. There are two aspects here: physical access and data transmission security.

Availability.

Data can be managed through graphical interfaces, console or API.

Scaling. The volume quickly increases due to the connection of additional servers and storage systems. In the physical version it is also fast, but only if you take care of scaling in advance. It will also be expensive and “vendor lock-in”. This means that you will be tied to the supplier (vendor) and its platform, technologies, software. When you want to change supplier, you will have to build everything from scratch, because all hardware and software are tied to the vendor.

Expense management. You need to pay exactly as much as you consume resources. Object storage has classes – standard and cold. Classes help manage storage costs. For example, when data needs to be accessed frequently, you can pay more for storage, but cheaper for traffic (accesses). For archives, on the contrary, you can pay more for retrieving files, but cheaper for storage, because they are rarely accessed.

Business processes are simplified, when cloud storage is available to an employee, for example, from home on the weekend. And let’s not forget about data recoverywhen business processes are not interrupted by force majeure due to the loss of documents or a repository.



What is cloud storage used for?


But five scenarios are more common:

Backup and recovery. Most cloud file systems are compatible with databases, so storage is often used for backup, for example, during updates. Cloud backup is easier to set up, and data storage reliability is better because the service provider distributes copies across multiple data centers.

Software development and testing. Often development requires duplication of environments, which then need to be deleted, and collaboration. Using cloud resources for this is standard practice among software developers. Also, clouds are integrated with different applications without additional “crutches”.

Sharing. For example, for development and testing teams from different offices or cities. If data is stored on a server within a business network, a VPN is often needed. But you can do without this and transfer some of the shared files, which you usually need access to, to cloud storage.

Data migration to the cloud makes it easier to maintain your infrastructure, but this is a serious task that requires many years of experience from the system administrator. However, there are services that make this process easier.

Big Data and IoT. For example, for Big Data, a data array of 100 Terabytes is not so much, but storing such a volume on local servers is expensive, so clouds are often used for this. Storing arrays in the cloud is convenient: cloud services usually have high throughput, low latency, and the ability to configure queries without retrieving data.

Types of cloud storage


It is impractical to store a data array for Big Data and an archive of corporate documents together. Each task requires its own type of cloud storage: object, block or file.

Block storage

In this type, files are divided into equal parts – blocks (chunk). Each block has its own identifier (location ID), according to which the storage system quickly collects blocks back into files.

Advantages. For block storage, these user environments are separated. This allows you to distribute data across multiple environments and give them separate access.

Low latency. All operating systems see block storage systems as disks and can connect to them via Fiber Channel or iSCSI.

How to use it. Often integrated with corporate databases. For example, Oracle uses block systems.

Flaws.

The lack of metadata limits data management. Additional block information loads the database. Even without this, setting up block storage creates additional work: file system selection, permissions, versioning, backups.

This type is also one of the most expensive because you have to pay for the entire allocated space, even if it is empty.

File storage

Organization of storage in

file storages

familiar:

  • information is stored in files;
  • files in folders;
  • and folders are combined into subdirectories and directories.

Storage is organized hierarchically. To find a file, you need to know the full path: to the directory, subdirectory, folder and file. The file storage can be accessed by both servers and PCs.

Advantages. Data is organized in a hierarchical directory tree, like in other operating systems, such as Windows, and working with files is intuitive. Files are uploaded to the cloud via the web interface or a separate local folder.

How to use it. For joint (and simultaneous) work, because:

  • easy to navigate;
  • The administrator can configure access and permissions to files and trees.

File storage systems are suitable for large volumes of structured data. For example, for companies that develop software or analyze data, where multiple servers are required to simultaneously access and modify multiple files.

Flaws. This type of storage does not scale well. As the volume of data grows, the hierarchy and permissions become so complicated that they become difficult to navigate and the system itself slows down. Therefore, this type is rarely used in data centers.

Object storage

A relatively new and universal way of storing data. Suitable for any data: logs, accounting reports, films, presentations, photographs, programs or static sites.

Universality is achieved due to the fact that files are stored as objects with a set of properties. Properties store identifier and metadata:

  • Identifier (one) – 128-bit number. Typically called a universally (UUID) or globally (GUID) unique identifier.
  • Metadata: name, coordinates, size, author’s name and other information to identify the object.

Advantages.

Thanks to identifiers, the structure is flat – there is no hierarchy, which allows you to scale up to hundreds of Petabytes. At the same time, metadata can be customized to meet the specific requirements of the application: clarified, rewritten, expanded.

It is convenient for applications to access data – parallel access is supported through different protocols, as well as APIs. All this is quite safe – modern object storage has a high level of reliability and low unit cost of storage.

How to use:

  • For storing large sets of data, for example, backup logs.
  • For backups, archiving, for example, video recordings of a video production studio for 10 years.
  • For storing and distributing content of mobile and web applications: images, images, software updates. For example, S3-enabled object stores can use plugins, extensions and libraries, and a CDN to speed up the distribution of updates.
  • E-commerce to store static content from online stores, for example.

Flaws.

The 128-bit identifier adds complexity to object naming. For example, this is why Digital Asset Managers (DAM) exist, as software that overlays an organizational chart on top of an asset store. Such additional software will have to be used for companies that produce, for example, video content.

Another drawback is that you cannot write a file to object storage by dragging it from folder to folder. For interaction, a programming interface is used – API. But some cloud providers, in addition to the API, implement graphical interfaces for downloading and managing (dragging) files. For example, Selectel has a panel OpenStack Swift, which eliminates this drawback. Let’s look at how this works.

Projects have been merged into the control panel Cloud platform And Object storage. To start using containers, move them to your project. More details in documentation.

After registering and confirming your email, go to your personal account. Further to the section

Object storage

Containers

.

Without grouping, it is difficult to manage a large number of objects. One way to get around this limitation is

containers

. They are logically associated, for example, with a separate project. Let’s create a container.

Select the type, storage class and add a name. The container is visible on the tab

Object storage

V

Containers

.

To add files, click on the container name. Adding a file –

Select files…

(you can also drag and drop). The tab shows downloaded files. They do not have volume restrictions, but this parameter, like others, can be specified in the container settings.

In addition to the panel, it is possible to work with files, for example, through

S3cmd

— command line interface for working with services that support the Amazon S3 HTTP API. Instructions on how to install S3cmd and upload files are described in more detail

in the knowledge base

.

Conclusion: the main advantage of cloud storage


This is a reduction in operating costs. “Homemade” solutions (both cheap and expensive) need to be administered, backed up, and updated when vulnerabilities emerge. This is a separate job that should be done by a separate person. Maintaining a system administrator for this in the long term is sometimes more expensive than a one-time purchase of all equipment (capital costs) for storage. The cloud is an alternative to your own storage systems for storage in a corporate system, which removes a large layer of problems and operating costs.

Ready to show off your IT knowledge? Accept participation in the IT crossword puzzle Selectel, win 10,000 rubles for server rental and exclusive Selectel merch.

Interesting materials on the topic

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *