teaching cannot be prohibited
How to make ChatGPT and other LLMs give out secret data? How to hack an electronic lock with biometrics? How to make a logic bomb using AI? Is it possible to train AI without access to real company data? These and other questions were discussed at the IV meeting of the expert community on cryptography and big data, organized by the company “Kryptonite” and dedicated to trusted AI.
Opened the meeting Vitaly Dmitrievich Pirozhnikovhead of the artificial intelligence laboratory at Kryptonite. He noted that every year the security issues of AI technologies based on machine learning models are becoming more acute. “We see the implementation of these models in literally all sectors: economics, medicine, transport, education, agriculture, and so on. At the same time, it remains unknown how reliable and stable these technologies are, how to notice their errors and how to prevent malicious manipulations associated with them,” said Vitaly Pirozhnikov.
Like humans, any AI can make mistakes, but the problem is that various attacks are being developed against artificial intelligence systems. For example, aimed at extracting personal data from a large language model on which it was trained, or fooling machine vision systems.
“Imagine that a certain facial recognition system is considered virtually error-free, as it has a claimed error rate of 0.3%. However, there are, say, ten people that it does not see point-blank. For example, because this was done intentionally: during the training process, the system was trained to ignore these people,” Vitaly Pirozhnikov gave an example.
It is also possible that these “ghosts” use targeted modifications of appearance that are barely noticeable to the naked eye, but confuse recognition algorithms. There are different options here and legitimate concerns arise: how difficult is it to implement attacks on machine learning models? How to protect against them? How can we build trusted AI? What regulatory framework governs its development and use? These and other issues were discussed by industry experts during the meeting.
Andrey Petrovich KovalenkoDoctor of Technical Sciences, Vice President of the Academy of Cryptography of the Russian Federation began his speech with a simple thesis: there is no magic in AI technologies. Machine learning models are mathematical functions. They approximate the required function using a table of its values, built on the basis of a given training set of observations. Hence, the general classes of errors inherent in models are obvious: overtraining, data drift, bias of the trained model, etc. AI has no self-awareness, and the uprising of machines remains the lot of science fiction writers.
The problem is different. From an information security perspective, there are also threats specific to AI: data poisoning (training a model to behave in an undesirable way), model inversion attacks (unauthorized access to training data), gradient descent attacks (misleading a model), and model substitution.
Therefore, when assessing the reliability of AI systems, trust factors are identified: theoretical justification of ML models, proven effectiveness of algorithms for solving optimization problems, a trusted dataset (a verified set of training data of sufficient volume), the use of trusted software and reliable hardware platforms at all stages of AI development and application.
These are all necessary but not sufficient conditions for declaring an AI system trusted if it meets all the above factors. This is because there is still an area of mistrust within the whole structure.
For example, due to the MLP extrapolation problem, there may be an unaccounted region of input values, the manipulation of which can distort the output of the neural network. A similar problem is associated with the extrapolation of decision trees. One of the promising approaches to solving these problems is to increase the dimensionality of the problem, but how exactly to implement this in practice so that it is effective and safe is still an open question.
“Currently, Russia and other countries are developing standards for assessing trust in machine learning systems. To secure the use of AI right now, we must not trust “black boxes”, study the properties of mathematical functions implemented by ML models and develop statistical models similar to ML models. By the way, without statistical checks, the functional safety of AI devices will not be certified,” — Andrey Petrovich explained.
Analysis of approaches to regulating AI technologies in the world continued Pyotr Vladimirovich GanelinStrategy Advisor at the National Technology Center for Digital Cryptography. He noted that in August, Gartner analysts presented the so-called annual hype curve (hype, public interest), where they identified 25 breakthrough technologies among more than two thousand, grouped into 4 key blocks.
The first of these blocks is called “autonomous artificial intelligence.” The second block is closely related to it and concerns the automation of developers' work, that is, writing software code using AI. The third block includes technologies that change the user experience due to the increasingly widespread use of machine learning models. The fourth block combines AI technologies that directly affect security and privacy.
“Currently, in Russia, we can identify several large centers that deal with artificial intelligence issues in relation to industry tasks. Each of them has its own views on how to develop machine learning technologies. In such a decentralized landscape, it is difficult to ensure quality control and security of the solutions used. Systems of state regulation in the field of AI are only just emerging here and abroad,” said Pyotr Vladimirovich.
The European Union promotes a “risk-based approach.” In May 2024, the EU Council approved the Artificial Intelligent Act. This voluminous 150-page document describes in which industries AI is prohibited and where it can be used after certification.
“I believe that blind copying is inappropriate here. Russia needs to develop its own regulatory system taking into account today's realities. The highest-level legislative framework for this has already been prepared. There is Decree of the President of the Russian Federation dated 10.10.2019 N 490 “On the development of artificial intelligence in the Russian Federation” with amendments dated 15.02.2024. According to it, the reliability and safety of AI must be provable, and the “black box” is outside the legal field,” — noted Pyotr Vladimirovich.
During the discussion of the risks associated with the widespread introduction of AI, he focused on the risks of widespread use of AI, including the risk of dependence on the technology. Its essence is simple: the more we rely on artificial intelligence, the faster our natural intelligence degrades. People shift more and more tasks to AI, and they themselves lose the corresponding skills. For now, we can at least type a search query, but the generation growing up to replace us already considers this an anachronism and prefers voice communication with a “smart speaker” or smartphone.
Strong reliance on AI creates another risk – shifting responsibility. There is an increasing temptation to say: “it’s not me, it’s the computer’s fault”, especially when the error may have legal consequences. It wasn’t me who violated the traffic rules, but my autopilot. It wasn’t me who wrote the bad code, but MS Copilot… It is important to understand that AI is just a tool, albeit a very advanced one.
The presentation of the Scientific and Technical Center of the Central Committee also presented developments in systematizing the risks of using AI, requirements for AI systems, ML and training data, and classifying classes of attacks on artificial intelligence. This work may become the basis for future AI regulation.
As a possible solution to the problems of security of artificial intelligence systems in the conditions of a decentralized landscape and linking to the solution of industry problems by existing AI centers, the creation of a Consortium for Research on the Security of Artificial Intelligence Technologies can be considered. Joining the Consortium of leading AI developers and companies working in the field of cybersecurity will allow synchronizing the efforts of developers and information security specialists and, as a result, reduce the risks in the creation and use of AI systems for various purposes.
The expert from the company “Kryptonite” was Ivan Vladimirovich Chizhovdeputy head of the cryptography laboratory for research. He explained how homomorphic encryption can be used in neural networks.
One of the security issues is that large data sets are used to train the model, which may include personal data or data related to commercial secrets. There are a number of specific attacks that allow data from training samples to be recovered and compared to specific people and objects.
It is possible to protect against attacks of this class using encryption, but classical cryptographic schemes are not suitable for machine learning. A neural network cannot learn by receiving encrypted data as input. As an alternative, homomorphic encryption schemes are currently proposed. They allow performing some mathematical operations on encrypted data without the need for preliminary decryption.
There are two classes of such systems: partially and fully homomorphic. Partially homomorphic systems (PHE) are encryption schemes that perform several operations on encrypted data, but do not express the entire class of computable functions. Typically, this is only addition and multiplication. For example, RSA and El-Gamal schemes are homomorphic with respect to the multiplication operation.
Level fully homomorphic encryption schemes (LFHE) are more complex. They can perform more different functions with ciphertexts, for example, raising to the fifth power, or calculating a sine, but they also have limits. Despite the epithet “fully homomorphic”, real-life schemes provide homomorphism only within a certain level, which is reflected in the name. Limitations arise due to the fact that the nonlinear layer in neural networks performs an approximation to polynomials, and then the polynomial is calculated in a homomorphic way. Because of this, accuracy is lost, and with repeated operations, errors accumulate. Therefore, it is important to understand the limits of LFHE applicability, beyond which you will simply get digital noise.
“Homomorphic encryption can make AI safer because it ensures the privacy of data and machine learning models. In addition, it does not require interactive interaction between the user and the service, nor does it require intermediaries to transmit confidential information. However, this is only a promising direction, not a panacea,” Ivan Vladimirovich explained.
He noted that today homomorphic encryption does not protect against competitive attacks that do not require knowledge of the neural network features. At the same time, homomorphic encryption significantly reduces the speed and accuracy of neural networks, and in systems with a large number of owners of training data, it does not provide sufficient flexibility. Therefore, in practice, the PHE/LFHE concepts in machine learning are still difficult to apply, although they have very serious prospects.
There is a specialized Trusted AI Research Center in Russia. At the conference, it was represented by its director, To.f.-m.n. Denis Yurievich Turdakov.
He drew attention to the fact that attacks on AI are possible at all stages of the life cycle of a machine learning model: preparing datasets (adding bookmarks to training data), training the model (introducing backdoors into the model algorithm), exploitation (adversarial attacks), as well as attacks on code and supply chains.
“An ordinary data scientist will never distinguish a malicious backdoor from a machine learning model error. It doesn’t even think about supply chain attacks. For example, it downloads TensorFlow in binaries and runs it as is. By the way, over the three years of our work, we have identified about a hundred vulnerabilities in TensorFlow and PyTorch, reported them to the open developer community and proposed fixes,” said Denis Yuryevich.
The report separately considered attacks on generative models. They are associated with malicious manipulation of requests (prompts) and are aimed at forcing the model to issue data that it would normally filter out. For example, you can “persuade” ChatGPT to create a recipe for explosives and poisonous substances, although the system normally refuses to issue such answers. This is a serious problem.
“We see our task as providing developers and operators of intelligent systems with tools to ensure the required level of trust. We have already developed recommendations for countering threats in the field of trusted AI,” — said Denis Yuryevich.
To implement these recommendations, the Center has developed software tools that can find anomalies in training sets, detect data drift, identify model bias, and assess the resilience of trained models to attacks.
Head of the Computer Graphics Laboratory, MSU Faculty of Computational Mathematics and Cybernetics Dmitry Sergeevich Vatolin spoke about the problems of biometric identification in “smart” systems using the example of an electronic lock with a facial recognition system.
Normally, such an electronic lock recognizes a face in 1-3 seconds and, if the person is on the “white list”, opens the door for him. However, in such systems, it is possible to use a physical attack on metrics like “adversarial patches”. It can be performed as a simple demonstration to the camera of a template specially prepared by the attacker. When trying to read it, the face recognition system and locking the lock may freeze in the last state (open or closed). Such a template is vaguely similar to DataMartix codes. It can be printed on a cap or a T-shirt.
“During the experiment, using such templates, we often managed to cause the electronic lock to freeze for several minutes, and sometimes it turned out to be completely inoperative until manually rebooted,” — noted Dmitry Sergeevich.
The new JPEG AI compression standard is currently being prepared for adoption. The laboratory's scientific team has already tested JPEG AI 5.3, performing several types of attacks within the “white box” model. Preliminary testing results indicate that JPEG AI is not resistant to attacks.
If an attacker has access to the original file before compression, it is possible to cause serious artifacts that interfere with the recognition of individual areas of the image, as well as increase the size of the compressed image up to 4 times. If the possibility of such malicious manipulation is confirmed for the final version of the JPEG AI standard, it can lead to attacks on data storage systems (a type of DDoS attack on storage).
The laboratory is currently studying the problem of counteracting these attacks and analyzing the transferability of experimental results to other image encoding methods.
The lab is also developing attacks on Super-Resolution methods, which allow adding a small amount of noise to a video to significantly distort it when shown on 8K screens.
Most experts discussed various aspects of trusted AI, but not everyone considered the concept as a whole realistic. An employee of the Information Security Department of the Faculty of Computational Mathematics and Cybernetics at Moscow State University Evgeny Albinovich Ilyushin expressed the opinion that “trusted artificial intelligence” is some kind of ideal that is unattainable in the real world. To create it, one would have to trust all elements of AI at all levels. There are no grounds for such unconditional trust, and they are unlikely to be possible at all outside of an abstract model. Therefore, in practice, it is more appropriate to evaluate the reliability of AI by some quantifiable and verifiable parameters.
Currently, AI is evaluated using statistical, formal and empirical criteria. In most cases, statistical assessments are used, such as accuracy, completeness, F-measure, etc. However, recently it has become obvious that such assessments are not enough. It is necessary to additionally apply formal assessments, as well as empirical ones (AI Red Teams). That is, it is necessary to perform a comprehensive assessment of the reliability of AI systems, which includes all of the above approaches. This is exactly what Evgeny Ilyushin developed.
It consists of six tests, each of which can be expressed as a fraction of a unit (or as a percentage). We will list them below and then look at them in more detail:
— quality assessment on the initial distribution;
— assessment of resistance to shifts in distribution;
— assessment of resistance to adversarial attacks;
— assessment of uncertainty (entropy);
— assessment of interpretability;
— the ability of the system to detect an exit from the distribution.
It is clear that the requirements for entertainment and medical AI systems are completely different. Therefore, depending on the specific task, the results of each test are assigned different weighting factors, and then the overall reliability indicator of the AI system being evaluated is calculated.
Let's agree that “reliability” means the ability of AI to work predictably and correctly handle errors that arise during its operation. The latter property is often called robustness. Obviously, no system can be resistant to the occurrence of any errors. Therefore, in functional safety definition of the property of stability similar to the definitions we know from mathematics: stability according to Livshits, or according to Lyapunov. Its essence is that small changes in input data should not lead to significant distortions at the output.
The AI products on the market do not always meet these requirements. For example, some credit scoring systems can be tricked into giving an erroneous credit score by performing a small manipulation of the input data that is difficult to detect.
Why does this happen? Most attacks on AI systems are carried out at the data level. At the same time, the reliability of a machine learning model is assessed based on a deliberately false condition: the data in the training, validation, test, and working samples are distributed equally and independently. In reality, models are usually trained on data with one distribution, but work with another, and this distribution shift is not taken into account in any way.
Evgeny Ilyushin believes that when assessing the reliability of AI systems, it is necessary to look at how they maintain stability under different types of shifts and whether they are capable of detecting an exit from the distribution at all. He noted that today there are no reliable ways to detect an error in machine learning. If classical software throws an exception or stops working in the event of a failure, then AI will try to continue working with any data you give it.
The importance of a systematic approach to developing AI models for information security products was discussed by employees of Solar Group: Head of the R&D laboratory of the cybersecurity technology center Maxim Sergeevich Buzinov and senior analyst Polina Vitalievna Sokol.
Maxim Sergeevich noted that machine learning technologies are increasingly used in the field of information security. They are in demand to counteract zero-day threats, search for anomalies in software and employee behavior, and analyze incidents.
“We have divided our research into two branches: Run and Dev. The first contains all the repeatedly tested solutions that meet specific business metrics and are focused on the customer's strict requirements. In the second, we have placed promising research on data analysis and potentially breakthrough topics.”said Maxim Buzinov.
For neural networks, we save queries for further analysis of the model for resistance to attacks, so that we can later check which specific anomalies it reacts strangely to. We use GAN for training on adversarial examples.
The risks of using other people's machine learning models are clear – they may contain backdoors at different levels. Therefore, before using open source models, you need to analyze their code (SAST/DAST) and check the results on known datasets.
If we have built the model ourselves, we must protect our training pipeline. There are attacks on parts of the pipeline that mainly target data integrity and availability, as well as preventing the model from training on new data. The main method of protection is to restrict access to all parts of the pipeline and monitor code changes in the data extraction and analysis (ETL) tools.
Data Sapience's Product Director Talks About Data Privacy Issues in Machine Learning Models Pavel Vladimirovich Snurnitsyn. He stressed that every company has data that provides its competitive advantage. Therefore, one of the tasks is to limit the access of third-party AI-based solutions to this data.
On the other hand, the less real data a model has available during the training phase, the worse it will perform later during the application phase, and ultimately companies will miss out on the potential benefits of combining their data with each other.
The approach of data and analytics collaboration itself is not new. For example, credit bureaus have long existed, to which banks transfer data on their credit portfolios, where a common scoring model is built on the aggregate data, which is provided back to banks as a service and improves the quality of decision-making on issuing new loans. Another example: credit scoring services from telecom operators. In general, combining data from different industries for mutual improvement of the quality of decision-making processes has very great prospects.
By law, an organization has the right to transfer far from all data to the outside. There are such concepts as personal data, bank secrecy, medical secrecy, etc. And even if the simplest approaches such as anonymization, hashing or tokenization are applied to sensitive data before transfer, such schemes can still be subject to deanonymization attacks, that is, when, based on indirect signs, it is possible to restore some confidential information about the data subject. Moreover, the more participants in the collaboration scheme, the higher the risk of data leakage at some stage of interaction.
To further ensure privacy and security in data collaboration schemes, there are advanced confidential analytics methods.
For example, in the concept federated learning the model training process is divided. Each data owner locally trains a fragment of the model. Then these fragments are aggregated, the weights are averaged, and a global model is formed.
Another innovative approach is using synthetic data, preserving structural connections. On the data owner's side, a model is trained that remembers the dependencies of real data, and then generates a synthetic dataset. Further training stages occur on it without access to real data.
There is another interesting concept – differential privacy. It involves protecting the data layer from the model training process through a protocol that provides resistance to differential attacks. This concept is intended to reduce the risk of sensitive data being extracted through malicious manipulation of queries.
There are also more rigorous cryptographic protocols for ML. Among them are homomorphic encryption And secure multiparty computing, which have already been discussed in previous reports. Pavel Vladimirovich added that there is another direction. The so-called “secure crypto enclaves” — these are memory areas and processor registers protected at the hardware level. This approach seems to be the most reliable, but it raises the question of trust in the foreign manufacturer and the need to develop our own hardware solutions of this type.
During the meeting, experts discussed many areas of development, training, and responsible use of artificial intelligence systems. Which of them will become the new reality depends on business needs and the actions of regulators. Kryptonite works in each of these areas, combining the efforts of cryptography laboratories, artificial intelligence, the advanced research department, and other divisions of the company.
