Apache NiFi. How to quickly make friends with LDAP and Registry

It would seem that Apache NiFi has already been written about more than once. But if you are just getting acquainted with the instrument, it can be difficult to understand such articles. They usually talk to you as if you’ve been in the know for a long time, and more often than not the problems they solve are clearly not yours. Everything is also complicated with the official documentation: it is there, but it’s clearly not suitable for a quick dive.

That's why I decided to prepare my beginner's guide. Let's try as quickly as possible:

Typically, users read different manuals. This way you can find something new and useful that is not in other sources. I hope this article is also useful. And if you want to add something on the topic, write in the comments. I'll be glad to chat 🙂

Initial setup

Just in case, let me remind you that Apache NiFi is a tool for processing and transmitting data in real time. With its help you can manage the flow of information from various sources. The simplest example of use: take data from Kafka, process it and transfer it to the database.

We are in the company Nubes We started using NiFi with the task of processing large amounts of data that comes in Json format. We convert them and transfer them to PostgreSQL, and then use them to generate analytical reports.

So the initial installation looks like this:

  1. To install the tool you will need Java – without it nothing will run.

sudo dnf install java-21-openjdk-devel

  1. Next, add the NiFi user.

useradd -d /opt/NiFi/ NiFi -s /sbin/nologin

  1. Now install NiFi from the official website, select the desired version. We will work with 1.25.0.

wget https://dlcdn.apache.org/nifi/1.25.0/nifi-1.25.0-bin.zip
unzip https://dlcdn.apache.org/nifi/1.25.0/nifi-1.25.0-bin.zip
ln -s nifi-1.25.0-bin nifi

  1. We describe memory consumption (we change the NN value to our own) and who will manage NiFi.

/opt/nifi/conf/bootstrap.conf
run.as=nifi
java.arg.2=-XmsNNg
java.arg.3=-XmxNNg

  1. Setting up systemd.

/etc/systemd/system/nifi.service
[Unit]
Description=Apache Nifi
After=network.target
 
[Service]
Type=forking
ExecStart=/opt/nifi/bin/nifi.sh start
ExecStop=/opt/nifi/bin/nifi.sh stop
ExecReload=/opt/nifi/bin/nifi.sh restart
User=nifi
Restart=always
Environment="JAVA_HOME=/usr/lib/jvm/jre-openjdk"
Environment="LD_LIBRARY_PATH=/usr/lib/jvm/jre-openjdk/lib"
 
[Install]
WantedBy=multi-user.target
/etc/systemd/system/nifi-registry.service

[Unit]
Description=Apache Nifi
After=network.target

[Service]
Type=forking
ExecStart=/opt/nifi/bin/nifi-registry.sh start
ExecStop=/opt/nifi/bin/nifi-registry.sh stop
ExecReload=/opt/nifi/bin/nifi-registry.sh restart
User=nifi
Restart=always
Environment=”JAVA_HOME=/usr/lib/jvm/jre-openjdk”
Environment=”LD_LIBRARY_PATH=/usr/lib/jvm/jre-openjdk/lib”

[Install]
WantedBy=multi-user.target

The initial setup is complete. At this point you can launch NiFi and Registry…

But it's not that simple. They won’t want to be friends with each other; authorization will work anyhow. What to do? The answer is further.

Adding certificates

As you already understand, this is a very important point. In our instructions, the CA (certificate authority) module in FreeIPA is responsible for the certification authority. In general, there is no specific connection to FreeIPA: it is enough to have any CA that can sign everything that you configure for it.

All the following settings will be created in /opt/NiFi/ssl (a directory must also be created).

Create a certificate request.

/opt/nifi/ssl/$(hostname).conf
[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no
 
[req_distinguished_name]
C = RU
ST = Russia
L = Moscow
O = nubes
OU = nubes
CN = $(hostname)
 
[v3_req]
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth

Then we generate a private key and certificate upon request.

openssl req -newkey rsa:2048 -keyout  $(hostname).key -out  $(hostname).csr -config  $(hostname).conf -nodes

As a result, we get a CSR (an encrypted certificate issuance request containing detailed information about the domain and organization). Now we take it and run to sign the certificate. We put it next to it on the server. At the same time we get root/intermediate certificates. In our case, these are FreeIPA CA certificates.

All NiFi certificates are stored either in p12 or jks – choose the option at your discretion. We will consider both cases.

For the test it is enough to have the same ks and ts, but in reality it is recommended to separate them.

KeyStore stores keys and certificates, whereas TrustStore contains certificates that the software trusts.

First, let's create p12. We come up with a password in advance that we will set for KeyStore.

openssl pkcs12 -export -in $(hostname).crt -inkey $(hostname).key -out $(hostname).p12

And from p12 we create jks. The password must be the same as in the step above.

keytool -importkeystore -destkeystore $(hostname).jks -srckeystore $(hostname).p12 -srcstoretype PKCS12

Next, we place all other certificates in jks.

keytool -importcert -file idm-intermediate.crt -keystore $(hostname).jks -alias "idm-intermediate"
keytool -importcert -file idm-root.crt -keystore $(hostname).jks -alias "idm-root"
keytool -importcert -file registry.crt -keystore $(hostname).jks -alias "nifi-registry"

Setting up NiFi configuration files

About all NiFi settings you can read Here. Below is only what is needed to launch and integrate the components with each other.

In general, NiFi uses three main files for configuration:

  • NiFi-properties (main parameters) – on what port, on what addresses and what authorization policies will be used;

  • Login-identity – indicates the main authorization policy that will be applied (noauth, ldap, salm, kerb);

  • Authorizers – describes access rights and user environments.

/opt/nifi/conf/nifi.properties
Настраиваем конфигурационные файлы NiFi
О всех настройках NiFi можно почитать здесь. Ниже приведено только то, что необходимо для запуска и интеграции компонент между собой. 
В целом NiFi использует три основных файла для настройки:
NiFi-properties (основные параметры) — на каком порту, на каких адресах и какие политики авторизации будут использоваться;
Login-identity — указывается основная политика авторизации, которая будет применяться (noauth, ldap, salm, kerb);
Authorizers — описываются права доступов, пользовательские окружения.
# Базовая настройка сервера
nifi.web.https.host=$(hostname)
nifi.web.https.port=8443
# Необходимо придумать еще один секретный ключ и записать его сюда
nifi.sensitive.props.key=123qwesecretkey
 
nifi.security.keystore=./ssl/$(hostname).jks
nifi.security.keystoreType=jks
# Пароль от jks
nifi.security.keystorePasswd=jksSecretKey
# Пароль от jks
nifi.security.keyPasswd=jksSecretKey
nifi.security.truststore=./ssl/$(hostname).jks
nifi.security.truststoreType=jks
# Пароль от jks
nifi.security.truststorePasswd=jksSecretKey
# Включаем авторизацию по LDAP
nifi.security.user.authorizer=managed-authorizer
nifi.security.allow.anonymous.authentication=false
nifi.security.user.login.identity.provider=ldap-provider

Next we describe the authorization block. At this stage, authorization of all users usually occurs. If the need to work with LDAP causes a hysterical attack, just download Apache DS and connect under the service user. Then you can reach the desired groups/users who will need to be granted access.

/opt/nifi/conf/login-identity-providers.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><loginIdentityProviders>
	<provider>
    	<identifier>single-user-provider</identifier>
    	<class>org.apache.nifi.authentication.single.user.SingleUserLoginIdentityProvider</class>
    	<property name="Username"></property>
   	 <property name="Password"></property>
	</provider>
 
	<provider>
    	<identifier>ldap-provider</identifier>
    	<class>org.apache.nifi.ldap.LdapProvider</class>
    	<property name="Authentication Strategy">SIMPLE</property>
 
    	<property name="Manager DN">cn=svc_nifi,ou=Service Users,dc=XXX,dc=nubes,dc=ru</property>
    	<property name="Manager Password">secretpass</property>
 
    	<property name="TLS - Keystore"></property>
    	<property name="TLS - Keystore Password"></property>
    	<property name="TLS - Keystore Type"></property>
    	<property name="TLS - Truststore"></property>
    	<property name="TLS - Truststore Password"></property>
    	<property name="TLS - Truststore Type"></property>
    	<property name="TLS - Client Auth"></property>
    	<property name="TLS - Protocol"></property>
    	<property name="TLS - Shutdown Gracefully"></property>
 
    	<property name="Referral Strategy">FOLLOW</property>
    	<property name="Connect Timeout">10 secs</property>
    	<property name="Read Timeout">10 secs</property>
 
    	<property name="Url">ldaps://ADSERVER:636</property>
    	<property name="User Search Base">ou=nubes-XXX,dc=XXX,dc=nubes,dc=ru</property>
    	<property name="User Search Filter">sAMAccountName={0}</property>
 
    	<property name="Identity Strategy">USE_USERNAME</property>
    	<property name="Authentication Expiration">12 hours</property>
	</provider>
 
</loginIdentityProviders>

After authorization, NiFi registers users as sAMAccountName (in our case there was a mail prefix, which is more convenient to use – I’ll talk about this below) and transfers it to the next configuration file, where the role area is configured.

Important point 1. After enabling LDAP, users will have no access at all. To be able to somehow connect to NiFi, it is better to immediately create an administrator user.

The ldap account is specified as the administrator, which is registered as sAMAccountName. There are also cases of LDAP setup where users are registered as “cn=myuser, ou=NiFi”, uid=13567 or as a full bind_dn containing all groups. The only thing that will help here is to enable TRACE of all settings in logback.xml and monitor /opt/NiFi/logs/NiFi-app.log.

Important point 2. Registry accepts requests from NiFi for certificates, and the Owner is the user for it. Here we are faced with two nuances: either generate a convenient certificate so that cn=myuser, or create a user with an inconvenient name. We are taking the second path because we want to have a valid certificate when working with NiFi without red locks on the web.

To find out which user to create, look at the contents of jks.

keytool --list -v -keystore $(hostname).jks

It turns out that you need the user “CN=$(hostname), O=XXX.NUBES.RU”. It will connect to the NiFi Registry from NiFi.

Important point 3. We pass the managed-authorizer parameter to NiFi-properties. It calls the file-access-policy-provider class, which necessarily describes the admin. You can specify your AD (Active Directory) user to make it easier to configure NiFi later.

Next is a link to composite-configurable-user-group-provider, which contains local and ldap users. It is very important to leave local users so that NiFi can communicate with the Registry. If you specify a local user, it will be created automatically (but without rights).

And a few words about the role-playing system. Often it is enough to create at least two groups in AD (administrator, regular). In our case, we’ll denote them as _security and nubes-XXX. The second group includes all users who can connect. For _security we will issue full rights. You can differentiate rights between users, and this is entirely your business.

/opt/nifi/conf/authorizers.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<authorizers>
 
	<userGroupProvider>
    	<identifier>file-user-group-provider</identifier>
    	<class>org.apache.nifi.authorization.FileUserGroupProvider</class>
    	<property name="Users File">./conf/users.xml</property>
    	<property name="Initial User Identity 1">CN=$(hostname), O=XXX.NUBES.RU</property>
	</userGroupProvider>
 
	<userGroupProvider>
    	<identifier>ldap-user-group-provider</identifier>
    	<class>org.apache.nifi.ldap.tenants.LdapUserGroupProvider</class>
    	<property name="Authentication Strategy">SIMPLE</property>
 
    	<property name="Manager DN">cn=svc_nifi,ou=Service Users,dc=XXX,dc=nubes,dc=ru</property>
    	<property name="Manager Password">secretpass</property>
 
    	<property name="TLS - Keystore"></property>
    	<property name="TLS - Keystore Password"></property>
    	<property name="TLS - Keystore Type"></property>
    	<property name="TLS - Truststore"></property>
    	<property name="TLS - Truststore Password"></property>
    	<property name="TLS - Truststore Type"></property>
    	<property name="TLS - Client Auth"></property>
    	<property name="TLS - Protocol"></property>
    	<property name="TLS - Shutdown Gracefully"></property>
 
    	<property name="Referral Strategy">FOLLOW</property>
    	<property name="Connect Timeout">10 secs</property>
    	<property name="Read Timeout">10 secs</property>
 
    	<property name="Url">ldaps://ADSERVER:636</property>
    	<property name="Page Size"></property>
    	<property name="Sync Interval">30 mins</property>
 
    	<property name="Group Membership - Enforce Case Sensitivity">false</property>
    	<property name="User Search Base">ou=nubes-XXX,dc=XXX,dc=nubes,dc=ru</property>
    	<property name="User Object Class">person</property>
    	<property name="User Search Scope">SUBTREE</property>
    	<property name="User Search Filter"></property>
    	<property name="User Identity Attribute">sAMAccountName</property>
    	<property name="User Group Name Attribute"></property>
    	<property name="User Group Name Attribute - Referenced Group Attribute"></property>
 
    	<property name="Group Search Base">OU=nifi,OU=_security,OU=nubes-XXX,DC=XXX,DC=nubes,DC=ru</property>
    	<property name="Group Object Class">group</property>
    	<property name="Group Search Scope">SUBTREE</property>
    	<property name="Group Search Filter"></property>
    	<property name="Group Name Attribute">cn</property>
    	<property name="Group Member Attribute">member</property>
    	<property name="Group Member Attribute - Referenced User Attribute"></property>
	</userGroupProvider>
 
	<userGroupProvider>
    	<identifier>composite-configurable-user-group-provider</identifier>
    	<class>org.apache.nifi.authorization.CompositeConfigurableUserGroupProvider</class>
    	<property name="Configurable User Group Provider">file-user-group-provider</property>
    	<property name="User Group Provider 1">ldap-user-group-provider</property>
	</userGroupProvider>
 
	<accessPolicyProvider>
    	<identifier>file-access-policy-provider</identifier>
    	<class>org.apache.nifi.authorization.FileAccessPolicyProvider</class>
    	<property name="User Group Provider">composite-configurable-user-group-provider</property>
    	<property name="Authorizations File">./conf/authorizations.xml</property>
    	<property name="Initial Admin Identity">myAdminUser</property>
    	<property name="Legacy Authorized Users File"></property>
    	<property name="Node Identity 1">CN=$(hostname), O=XXX.NUBES.RU</property>
	</accessPolicyProvider>
 
	<authorizer>
    	<identifier>managed-authorizer</identifier>
    	<class>org.apache.nifi.authorization.StandardManagedAuthorizer</class>
    	<property name="Access Policy Provider">file-access-policy-provider</property>
    	<property name="Node Identity 1">CN=$(hostname), O=XXX.NUBES.RU</property>
	</authorizer>
 
	<authorizer>
    	<identifier>single-user-authorizer</identifier>
    	<class>org.apache.nifi.authorization.single.user.SingleUserAuthorizer</class>
	</authorizer>
</authorizers>

Next, you can cross yourself and launch NiFi.

systemctl daemon-reload
systemctl enable --now nifi

Switching to NiFi Registry configuration files

The process is very similar to what we did in the previous stage with NiFi. But you won’t be able to mindlessly copy all the settings, since the binary source files and java classes are different.

/opt/nifi/conf/nifi-registry.properties
# Базовая настройка сервера
nifi.registry.web.https.host=$(hostname)
nifi.registry.web.https.port=8443
 
nifi.registry.security.keystore=./ssl/$(hostname).jks
nifi.registry.security.keystoreType=jks
nifi.registry.security.keystorePasswd=jksSecretKey
nifi.registry.security.keyPasswd=jksSecretKey
nifi.registry.security.truststore=./ssl/$(hostname).jks
nifi.registry.security.truststoreType=jks
nifi.registry.security.truststorePasswd=jksSecretKey
 
nifi.registry.security.needClientAuth=false
nifi.registry.security.authorizers.configuration.file=./conf/authorizers.xml
nifi.registry.security.authorizer=managed-authorizer
nifi.registry.security.identity.providers.configuration.file=./conf/identity-providers.xml
nifi.registry.security.identity.provider=ldap-identity-provider

Setting up the authorization block is also no different.

/opt/nifi/conf/identity-providers.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
 
<identityProviders>
	<provider>
    	<identifier>ldap-identity-provider</identifier>
    	<class>org.apache.nifi.registry.security.ldap.LdapIdentityProvider</class>
    	<property name="Authentication Strategy">SIMPLE</property>
 
    	<property name="Manager DN">cn=svc_nifi,ou=Service Users,dc=XXX,dc=nubes,dc=ru</property>
    	<property name="Manager Password">secretpass</property>
 
    	<property name="Referral Strategy">FOLLOW</property>
    	<property name="Connect Timeout">10 secs</property>
    	<property name="Read Timeout">10 secs</property>
 
    	<property name="Url">ldaps://ADSERVER:636</property>
    	<property name="User Search Base">ou=nubes-XXX,dc=XXX,dc=nubes,dc=ru</property>
    	<property name="User Search Filter">sAMAccountName={0}</property>
 
    	<property name="Identity Strategy">USE_USERNAME</property>
    	<property name="Authentication Expiration">12 hours</property>
	</provider>
</identityProviders>

A NiFi user must also be registered with the NiFi Registry. If there are several servers (cluster), then they should be specified using “Initial User Identity 2”, “Initial User Identity 3”, etc.

/opt/nifi/conf/authorizers.xml
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<authorizers>
 
	<userGroupProvider>
    	<identifier>file-user-group-provider</identifier>
    	<class>org.apache.nifi.registry.security.authorization.file.FileUserGroupProvider</class>
    	<property name="Users File">./conf/users.xml</property>
    	<property name="Initial User Identity 1">CN=$(hostname), O=XXX.NUBES.RU</property>
	</userGroupProvider>
 
	<userGroupProvider>
    	<identifier>ldap-user-group-provider</identifier>
    	<class>org.apache.nifi.registry.security.ldap.tenants.LdapUserGroupProvider</class>
    	<property name="Authentication Strategy">SIMPLE</property>
 
    	<property name="Manager DN">cn=svc_nifi,ou=Service Users,dc=XXX,dc=nubes,dc=ru</property>
    	<property name="Manager Password">secretpass</property>
 
    	<property name="TLS - Keystore"></property>
    	<property name="TLS - Keystore Password"></property>
    	<property name="TLS - Keystore Type"></property>
    	<property name="TLS - Truststore"></property>
    	<property name="TLS - Truststore Password"></property>
    	<property name="TLS - Truststore Type"></property>
       <property name="TLS - Client Auth"></property>
    	<property name="TLS - Protocol"></property>
    	<property name="TLS - Shutdown Gracefully"></property>
 
    	<property name="Referral Strategy">FOLLOW</property>
    	<property name="Connect Timeout">10 secs</property>
    	<property name="Read Timeout">10 secs</property>
 
    	<property name="Url">ldaps://ADSERVER:636</property>
    	<property name="Page Size"></property>
    	<property name="Sync Interval">30 mins</property>
 
    	<property name="Group Membership - Enforce Case Sensitivity">false</property>
    	<property name="User Search Base">ou=nubes-XXX,dc=XXX,dc=nubes,dc=ru</property>
    	<property name="User Object Class">person</property>
    	<property name="User Search Scope">SUBTREE</property>
    	<property name="User Search Filter"></property>
    	<property name="User Identity Attribute">sAMAccountName</property>
    	<property name="User Group Name Attribute"></property>
    	<property name="User Group Name Attribute - Referenced Group Attribute"></property>
 
    	<property name="Group Search Base">OU=nifi,OU=_security,OU=nubes-XXX,DC=XXX,DC=nubes,DC=ru</property>
    	<property name="Group Object Class">group</property>
    	<property name="Group Search Scope">SUBTREE</property>
    	<property name="Group Search Filter"></property>
    	<property name="Group Name Attribute">cn</property>
    	<property name="Group Member Attribute">member</property>
    	<property name="Group Member Attribute - Referenced User Attribute"></property>
	</userGroupProvider>
 
	<userGroupProvider>
    	<identifier>composite-user-group-provider</identifier>
    	<class>org.apache.nifi.registry.security.authorization.CompositeUserGroupProvider</class>
    	<property name="User Group Provider 1">file-user-group-provider</property>
    	<property name="User Group Provider 2">ldap-user-group-provider</property>
	</userGroupProvider>
 
 
	<accessPolicyProvider>
    	<identifier>file-access-policy-provider</identifier>
    	<class>org.apache.nifi.registry.security.authorization.file.FileAccessPolicyProvider</class>
    	<property name="User Group Provider">composite-user-group-provider</property>
    	<property name="Authorizations File">./conf/authorizations.xml</property>
        <property name="Initial Admin Identity">myAdminUser</property>
    	<property name="Nifi Identity 1">CN=$(hostname), O=XXX.NUBES.RU</property>
	</accessPolicyProvider>
 
	<authorizer>
    	<identifier>managed-authorizer</identifier>
    	<class>org.apache.nifi.registry.security.authorization.StandardManagedAuthorizer</class>
    	<property name="Access Policy Provider">file-access-policy-provider</property>
    	<property name="Node Identity 1">CN=$(hostname), O=XXX.NUBES.RU</property>
	</authorizer>
</authorizers>

At this stage, light the candle and launch NiFi Registry.

systemctl daemon-reload
systemctl enable --now nifi

Linking NiFi with NiFi Registry

There will be no synchronization between them immediately. To fix the problem, you first need to give the local user rights to read workflow.

Such a policy may not be among the ready-made options. But it can always be created, and quite quickly. It took us 10-15 minutes to draw up all the basic policies, including indicating the required groups/users. There are separate tabs in policies for read/write. By default, access is granted exclusively to the administrator user.

Next, we configure the connection in the Registry. To do this, go to the settings menu – Controller Settings – Registry clients – Add. Enter any name. After adding, go to edit Properties and enter the URL.

Now let's move on to Registry. We log in, and then go to Settings – Users. In the search, we enter the local user’s data and give him the following rights (see screenshot below).

Testing performance

We create a simple bucket in NiFi Registry.

Then we go to NiFi and create a Process Group. If the menu is blocked, add a new policy and grant access to the desired group of users.

If the bucket is loaded, open the champagne – NiFi setup is complete! You can work with the tool. All that remains is to read 100,500 instructions on how to do this.

Errors, problems and their solutions

Unable to obtain listing of buckets

Unable to obtain listing of buckets: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

The error occurs when you add NiFi Registry to Registry Clients or go to Start Version Control in jks, which is used in NiFi-Properties. The essence of the problem: NiFi and NiFi Registry do not have each other's trust certificates. You can add them using the command:

keytool -importcert -file cert.crt -keystore $(hostname).jks -alias "cert"

Unable to view the user interface

The error can be received with the ldap login. It indicates that LDAP is not configured correctly or the administrator user is specified incorrectly.

During the initial setup, it turned out that the LDAP setup was carried out correctly and the administrator user was specified correctly, but nothing worked. Moreover, NiFi was initially launched without LDAP, although not for long. A complete reinstallation helped.

No available buckets

When we connect to Start Version Control, the created bucket is not displayed.

The error indicates that we incorrectly configured a user who is simultaneously present in NiFi and NiFi Registry. It was possible to debug the problem only after enabling TRACE in all parameters in logback.xml and studying many logs by clicking on Start Version Control. Eventually you will see the error No user found for identity. This phrase can be entered into a search across all records so as not to read tons of logs at once.

Local user is not deleted in NiFi

The problem can be encountered when changing the Initial User Identity values ​​in the authorized.xml configuration files. To fix the problem, just remove the user from users.xml and restart NiFi.

I hope the guide will help you quickly make friends with NiFi and not stumble when correcting errors. Tell us, what problems did you encounter during the initial setup of the tool? How were they solved? 🙂

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *