How to convey secrets to the dev car without spilling them?
The longer I work, the more different practices I see for working with secrets. Some are workers, others are not, and some are simply taken aback. In this article I will analyze the options and talk about the pros and cons of different approaches.
We will talk about secrets in general, but from a practical point of view we will focus specifically on working with secrets on developers’ machines: how to safely deliver and use them there.
This text is a continuation of the CI/CD series in every home; last time we discussed how to organize assembly shop of basic Docker images.
B – Security
Security issues, including dealing with secrets, are often pushed to the back burner or to the bottom of the backlog because they have little operational value and break conventional and established practices. And there are no problems until the secrets leak, but at this moment problems begin, and often on a terrifying scale.
I won’t scare you with horror stories with different stories; everyone understands without me how important this topic is. Therefore, without unnecessary preludes, let’s begin.
Environments
I’ll start by pointing out that secrets are different, and accordingly require different approaches and methods of work. First of all, secrets are divided according to the environment where they are used:
Food. Secrets used by the production environment, for example database credentials, API tokens, private keys. This is the most important group of secrets. In an ideal world, they are not only extremely protected, but do not go beyond the safe contour at all (creation, storage, delivery and consumption occur within an environment to which there is no access from the outside). If such a possibility exists, no one should have access, neither write nor even read.
It happens when a secret has an external source. Then a narrow circle of responsible people has write access, but not read access. The implication is that once the secret is stored, there are no external copies left in the repository, and the secret is then considered safe.
Test ones. Same as operational environments, only now it's a staging/testing environment. It is important to understand that in no case should they overlap with operating rooms. Even if you use external production environments for test environments, it is correct that these are different applications that have different keys. But even if this is not possible, try to at least have different keys that can be controlled and invalidated separately. If these rules are followed, then it’s not so scary to give read access to developers – this will allow them to debug and, if necessary, get their hands into the guts of test environments. In other cases, secrets should also not leave the secure circuit.
CI/CD. Secrets that we use exclusively as part of our CI/CD processes may overlap with test secrets for testing pipelines. They, like food ones, should not leave the safe circuit and/or light up on the developer’s machine. Again, in an ideal world, no one even has access to them except CI/CD. The leakage of these secrets can sometimes be even more dangerous than the leakage of commercial secrets, since they give access not to the application, but to infrastructure management.
Local. Those secrets that may be needed on a developer’s machine for local development and testing. There should certainly be no intersections with food stores here, under any circumstances.
Personal. These are the same local secrets, but the only difference is that multiple developers cannot use the same secret at the same time. The most common example of such secrets: tokens of Telegram/Slack/Discord bots and others.
Longevity is our enemy
An important criterion for a secret is its lifetime:
Eternals. A secret whose lifespan is unlimited. It is as common as it is unsafe. It poses a serious security threat, especially if it can be compromised in any way, since it provides unlimited access.
Long-lived. Any secrets that are limited in time, be it a month or a year. The rotation of such secrets does not necessarily have to be automated; it happens that this is simply impossible. This shouldn't stop you. The lifetime of a secret not only adds some security in the event of a secret leak by limiting access over time, but also forces a regular audit of the relevance of secrets.
Short-lived/sessional. These are secrets that last for minutes, maybe hours, and sometimes their lifetime is limited to a certain session. This is the most secure way to share secrets. When on the use side there is some kind of long-lived key for which accesses are defined, and secrets are written to them as needed (IAM). And after the consumer finishes working with them, they are invalidated in the case of session ones.
Ideal for secrets on a developer's machine. The advantage is that if there is a leak/suspicious activity, we invalidate the long-lived key and a potential attacker no longer has access to our secrets. And the ones he managed to prescribe for himself earlier no longer work. This is much more practical than re-releasing all the secrets in the event of a leak. Great for local/personal secrets where the risk of compromise is high. Plus, in this case, the long-lived key can be replaced with some TOTP, which practically eliminates the possibility of stealing a long-lived secret, but only leaves a small window for stealing short-lived ones.
We don't always have a choice about which secrets we use, especially for external services, but even in these cases it's worth trying to choose the option with the shortest acceptable lifetime. Most often in practice, long-lived ones are encountered, but if possible, it is always worth using session ones. Even if we write out eternal secrets, it is necessary to re-release them regularly.
Storage
One of the safety rules is that access should be as narrow as possible. Therefore, when we talk about storing secrets, we also mean managing access to them. From this point of view, approaches that implement storing secrets on local storages of personal machines or using a single key to access all secrets are not suitable for us.
This block as a whole is beyond the scope of today’s discussion, but for the sake of understanding we will consider examples using the open source Hashicorp Vault and the proprietary Yandex Cloud Lockbox. I mainly use them in my projects, but there are plenty of others, for example, the popular ones – AWS Secrets Manager.
We will not consider storing secrets in environment variables, local files, God forbid, in open form in a repository or even on a sticker on your monitor. This is not storage, and it’s not secrets at all 🙂
Spreading
The transfer of secrets is one of the vulnerabilities in the chains of working with secrets, fortunately the set of rules is quite simple. In fact, it is the only one: the secret must be transmitted in such a way that potential attackers do not have the opportunity to intercept it. Trite? Agree. Is this principle followed? Unfortunately, often not.
Which of the popular methods of transferring secrets that can often be found in practice are unsafe? Let's start with the obvious:
In instant messengers / by mail, etc. Oh, this is my favorite, if you allow me, I won’t even say it, everything is clear here.
Clipboard. You go to your vault's website, open the secret visually and retype or copy it to the clipboard. At this point, the secret can easily be stolen or compromised. Starting from a banal virus/keylogger, ending with the fact that you can insert this secret somewhere in the wrong place.
Transmission over an unencrypted channel. Most likely, you won’t encounter this with ready-made storages, but if you suddenly decide to make your own storage with an API, remember: any transfer must be carried out via https with mandatory validation of the truth of the server even before transferring the client key, for example, via an SSL certificate or a private signature key.
It is important to note that in addition to the above, an additional guarantee of security when transferring secrets from a storage facility into a protected circuit is that the secret does not leave the boundaries of that very protected circuit. For example, product or CI/CD secrets should not be carried through the developer’s machine under any circumstances, but directly flowed from storage to the point of consumption.
Use
Now we come to the most interesting part – the consumption of secrets. You can often find that secrets are stored long-term on the consumption side, for example in local files or even secure storage. Ideally, you don't need to do this. Secrets should remain on the machine only for the duration of the use session, after which they should be removed from it without leaving a trace.
Storage in files. This is a conditionally acceptable scheme, but there are a number of “buts”:
They violate the rule described above about the sessional nature of storage on the consumption side, since you can easily get lazy / forget to clean up after yourself.
You can often find files with incorrectly configured access, as a result of which all users receive read access.
Unencrypted disk. Even despite the separation of access, if you store files and lose your laptop, you can always remove the disk from it and read the data if it is unencrypted.
Storage in .bashrc, .zshrc and others like them. Beyond good and evil. Any application running from any terminal has access. Simply no. Do not do it this way.
Storing secrets in local repositories. An example is some keyctl. A completely valid scheme that provides some security, but:
With the same success, you can store secrets in external storage, the only advantage is that you can work without access to the Internet.
Requires additional configuration on the user side.
Still breaks session.
What are the session options then? There are actually many of them, they all boil down to the following two:
We store secrets in process environment variables. That is, as part of working with secrets, we simply add secrets to the environment of the process from which we will launch our applications, which in turn consume these secrets. Session is limited by the lifetime of the process, and after it is closed, the secrets will disappear from the machine.
We store and import secrets directly into the application. An even more secure method, since not all child processes of our terminal will have access to secrets, but only the application itself. Of the minuses: it requires improvement directly on the side of the consuming application to interact with secrets. It is essentially impossible when using external applications.
The use of production/test applications and CI/CD pipelines on the side is a different story. As a rule, everything is much simpler, since the circuit is protected. And compromise of secrets is only possible if you have access directly to the place of use with root rights. Moreover, most often this task has already been solved for you, as, for example, in the case of k8s or GitHub Actions. The main thing is to never store secrets in images; secrets should always be transferred to the container exclusively at launch within the infrastructure.
P – Practice
Usually I organize local work with secrets along the path of storage in terminal environment variables. In my opinion, this is the perfect balance between security and convenience. That is, I don’t need to download secrets from the source every time I’m working on a project, but there won’t be any secrets left after I close the terminal. To organize this, you will need to know a little bash, but only a little, so let's look at the key nuances.
Keyword export. This command allows you to pass a variable to child processes without explicitly specifying it. This means that if we export some environment variables in the terminal, they will be available for use in all child processes.
Keyword source. By default, any script launched from the terminal is executed in a child process. To execute a script in the current process, you need a command, this allows you to directly assign values to variables in the script, and after they complete, the variables will remain in the terminal.
Keyword unset. Removes a variable from the environment. When used with the -f flag, it removes the function declaration; why we need this will be clear from the example.
Challenge with substitution $(command). Just a call replacing this block with the result of execution.
Environment variables can and will be different depending on your situation, but essentially all of them will be transferred inside the terminal with a command like export KEY=" class="formula inline">(value-retrieving-command).
Let me give you a couple of examples for clarity:
YC_LOCKBOX_SECRET_VALUE – the secret stored in the lockbox, an example of obtaining:
yc lockbox payload get \
--profile example-profile \
--folder-name example-folder \
--name example-secret \
--key example-key
VAULT_SECRET_VALUE – secret from Hashicorp Vault:
vault kv get \
-address=https://vault.example.org \
-mount=example-mount \
-field=example-key \
example-secret
What does our script look like in a primitive form?
export YC_LOCKBOX_SECRET_VALUE=$(yc lockbox payload get --profile example-profile --folder-name example-folder --name example-secret --key example-key)
export VAULT_SECRET_VALUE=$(vault kv get -address=https://vault.example.org -mount=example-mount -field=example-key example-secret)
What does the job look like:
Opened the terminal and executed source environment_activate. After this, the secrets are available in all applications that are launched from this terminal. did some work, ran tests, whatever, closed the terminal. This is where the basic part essentially ends, then the beauties begin.
Beautifulness
Ending a session without closing the terminal. To begin with, I would like to be able to clear the environment without closing the terminal, although this is not necessary, it is convenient to have such an opportunity. To do this, we will add a function to the environment, by calling which we will clean up the environment.
Console prefix. It is convenient to always know whether the project's environment variables are loaded into the terminal. You can, of course, check for the presence of a value or rewind the history, but for this there is a wonderful command line prefix mechanism that allows you to control the value of what you see to the left of the command you type. The mechanism, or rather the mechanics of working with colors, differ slightly between bash and zsh, as in other shells, but usually I do it for these two.
Prohibition on adding multiple environments. It is better not to load several different environments into one terminal; it will be more difficult to keep track of what is loaded and what is not, and why.
Well, after we systematize all this into different functions and embellish it with console output, we get the following:
#!/usr/bin/env bash
_EA_ENVIRONMENT_NAME=example
case $(basename "$SHELL") in
"zsh")
# shellcheck disable=SC2154,SC1087
_EA_COLOR_GREEN="%{$fg[green]%}"
# shellcheck disable=SC2154
_EA_COLOR_NC="%{$reset_color%}"
;;
*)
_EA_COLOR_GREEN="\[\e[32m\]"
_EA_COLOR_NC="\[\e[0m\]"
;;
esac
_ea_unset_script_variables() {
unset _EA_ENVIRONMENT_NAME
unset _EA_COLOR_GREEN
unset _EA_COLOR_NC
}
_ea_export_local_variables() {
echo "Setting up local env variables..."
export YC_LOCKBOX_SECRET_VALUE=$(yc lockbox payload get --profile example-profile --folder-name example-folder --name example-secret --key example-key)
export VAULT_SECRET_VALUE=$(vault kv get -address=https://vault.example.org -mount=example-mount -field=example-key example-secret)
}
_ea_unset_set_local_variables() {
echo "Cleaning up local envs..."
unset YC_LOCKBOX_SECRET_VALUE
unset VAULT_SECRET_VALUE
}
_ea_set_console_prefix() {
echo "Setting up console color and prefix..."
_EA_PREVIOUS_PS1="${PS1}"
PS1="${_EA_COLOR_GREEN}(${_EA_ENVIRONMENT_NAME})${_EA_COLOR_NC}${PS1}"
}
_ea_unset_console_prefix() {
echo "Cleaning up console color and prefix..."
PS1="${_EA_PREVIOUS_PS1}"
unset _EA_PREVIOUS_PS1
}
_ea_set_active_environment() {
export _EA_ACTIVE_ENVIRONMENT=$_EA_ENVIRONMENT_NAME
echo ""
echo "Environment $_EA_ENVIRONMENT_NAME is activated."
echo "To deactivate: run 'environment_deactivate'."
}
_ea_unset_active_environment() {
echo ""
echo "Environment $_EA_ENVIRONMENT_NAME is deactivated."
unset _EA_ACTIVE_ENVIRONMENT
}
_environment_activate() {
if [ -n "$_EA_ACTIVE_ENVIRONMENT" ]; then
echo "Active env is already set to $_EA_ACTIVE_ENVIRONMENT"
echo "To deactivate, run 'environment_deactivate'"
return
fi
_ea_export_local_variables
_ea_set_console_prefix
_ea_set_active_environment
unset -f _ea_export_local_variables
unset -f _ea_set_console_prefix
unset -f _ea_set_active_environment
unset -f _environment_activate
}
environment_deactivate() {
if [ -z "$_EA_ACTIVE_ENVIRONMENT" ]; then
echo "No active environment to deactivate."
return
fi
_ea_unset_console_prefix
_ea_unset_set_local_variables
_ea_unset_script_variables
_ea_unset_active_environment
unset -f _ea_unset_console_prefix
unset -f _ea_unset_set_local_variables
unset -f _ea_unset_script_variables
unset -f _ea_unset_active_environment
unset -f environment_deactivate
}
_environment_activate
Definitely not an advertisement
Not a call to action at all, but it would be stupid not to mention that for my projects I use a self-written library to transfer secrets secret-transfer. With it, the description of secrets in a large volume of projects becomes somewhat simpler, and the script environment_activate ceases to depend on a specific project.
…
_ea_export_local_variables() {
echo "Setting up local env variables..."
$($_EA_SCRIPTS_FOLDER/.venv/bin/secret-transfer run -f "$_EA_SCRIPTS_FOLDER/secrets/local.yaml")
}
…
TL;DR
On dev cars only personal/local secrets should be lit. Short-lived secrets are better than eternal ones. We store it safely. We transmit encrypted. We use it sessionally.
PS
By no means am I saying that this is the only correct way to work with secrets on local machines, but it allows you to organize the process at the same time quite simply and quite safely. There are probably many different options for doing this, I just cited the one that I developed during my practice. I hope someone finds it useful.
