AVI - Connect Keycloak to AVI Loadbalancer

Introduction

Today, I would like to address the topic of how Keycloak can be used as a central identity management system with the Advanced Load Balancer. In theory, this is relatively easy to implement, but in practice, there are a few challenges. I would like to describe exactly what these are and also present a solution. As a little bonus, I will also show you how to use a Fido Key (Yubikey) to log in.

I must admit, however, that I haven’t had such a frustrating setup in a long time, and it involved a lot of trial and error. Now, OIDC isn’t exactly one of my favorite topics, and I only had rudimentary experience with Keycloak, but that shouldn’t stop me from trying something new. Right? Right! So let’s get started.

Setup Keycloak

At this point, I made it relatively easy for myself by installing the official Keycloak Docker version 26.5.2 on my Unraid. Keycloak requires a Postgres DB, so I used an existing one—also on Unraid. If you were hoping that I would write instructions on how to install Keycloak here, I’m afraid I have to disappoint you.

Since I want to use OKTA as a provider in my VCF setup later on, my Keycloak needs a valid certificate that can be publicly validated. Here, too, I have made it easy for myself and am using my NGINX proxy manager with Let’s Encrypt support and a certificate from one of my domains. So that I don’t have to store my private IPs in Cloudflare’s DNS, I use SplitDNS at home. This means I can resolve keycloak.evilcorp.info, but you cannot. Since most of my Docker containers only run internally with HTTP, I have been using the proxy method for a long time and it has proven to be robust and reliable for me.

Realm Setup

Now that you have somehow deployed Keycloak on your system, we can start with the actual configuration. To do this, you first need to create a new realm. A realm is a client, you could say. A realm has its own users, groups, and so on. In addition, you can define your own identity provider for each realm. In my case, for the sake of simplicity, I use Microsoft AD (because it’s there anyway), but I could also use LinkedIn, Facebook, GitHub, Google, and many other identity providers. A combination of several is also possible. The only important thing is that the username must be unique within the realm. No two identical usernames can exist.

A new realm can be easily created under Manage realms and only requires a name. I named my realm lab-ad.

LDAP Connection

I deliberately avoided using LDAPs in my lab because my container does not have persistent storage, which means I cannot easily link the RootCA of my domain. This is on my long to-do list, but you can build it properly. An AD is integrated under User federation and not under Identity providers. It sounds strange, but that’s how it is.

Here are my settings at a glance:

General options

UI display name ldap
Vendor Active Directory

Connection and authentication settings

Connection URL ldap://home.lab
Enable StartTLS Off
Use Truststore SPI Never
Connection pooling On
Connection timeout 
Bind type simple
Bind DN CN=keycloak svc,OU=ServiceAccounts,DC=home,DC=lab
Bind credentials ••••••••••

LDAP searching and updating

Edit mode READ_ONLY
Users DN OU=User,DC=home,DC=lab
Relative user creation DN 
Username LDAP attribute sAMAccountName
RDN LDAP attribute cn
UUID LDAP attribute objectGUID
User object classes user
User LDAP filter (&(objectClass=user)(!(objectClass=computer)))
Search scope One Level
Read timeout 
Pagination On
Referral 

Synchronization settings

Import users On
Sync Registrations On
Batch size 
Periodic full sync Off
Periodic changed users sync On
Changed users sync period 60
Remove invalid users during searches On

Kerberos integration

Allow Kerberos authentication Off
Use Kerberos for password authentication Off

Cache settings

Cache policy DEFAULT

Advanced settings

Enable the LDAPv3 password modify extended operation Off
Validate password policy Off
Trust Email Off
Connection trace Off

This configuration connects Keycloak to an AD server at ldap://home.lab and authenticates there with a service account (CN=keycloak svc,OU=ServiceAccounts), allowing connection and read-only access. Only users in OU=User,DC=home,DC=lab are used, sAMAccountNameis used as the login name, computer objects are filtered out, and the users found are imported into the local database, with changes synchronized every 60 seconds and deleted AD users automatically removed.

Next, a group-ldap-mapper must be created in the Mappers tab. To do this, simply click on create new mapper and then select the type group-ldap-mapper.

My mapper is called ad-groups and these are the settings: ad-groups

LDAP Groups DN OU=SecGroups,DC=home,DC=lab
Relative creation DN 
Group Name LDAP Attribute cn
Group Object Classes group
Preserve Group Inheritance Off
Ignore Missing Groups Off
Membership LDAP Attribute member
Membership Attribute Type DN
Membership User LDAP Attribute sAMAccountName
LDAP Filter 
Mode READ_ONLY
User Groups Retrieve Strategy LOAD_GROUPS_BY_MEMBER_ATTRIBUTE
Member-Of LDAP Attribute memberOf
Mapped Group Attributes 
Drop non-existing groups during sync Off
Groups Path /

These settings tell Keycloak where and how LDAP groups are read: It searches for groups under OU=SecGroups,DC=home,DC=lab, recognizes them as group objects with names from cn, and reads membership via the member attribute (DN-based, users via sAMAccountName). Keycloak works read-only, loads groups via the member attribute, does not automatically transfer missing/unmapped groups, and stores the imported groups under / in the Keycloak group tree.

Once LDAP is connected, users should now be found when searching for a user under Users.

Next, I create an Auth Profile in AVI. AVI is a bit special in this regard because the Keycloak client must be created with a very specific name.

Auth Profile AVI

In AVI, navigate to Templates -> Security -> Auth Profile and create a new SAML profile.

Two things are important here: first, that the IDP metadata URL is used, and second, that the service provider is switched to FQDN. The IDP metadata URL can be found in Keycloak under Realm settings -> General -> Endpoints SAML 2.0 Identity Provider Metadata. Simply click on the browser link and copy the URL from the browser into AVI.

Once the profile has been saved, click on the context menu with the three dots in the Auth Profile overview and select Verify. The Verify view displays the Entity ID and the SSO URL—very intuitive—not.

Entity ID: AviController-vcf09-avi.lab.vcf
SSO URL: https://vcf09-avi.lab.vcf/sso/acs/

Keycloak Client

Keycloak Clients are applications and services that can request authentication of a user. You can have multiple clients in one realm, and clients can have different settings. Such as client-specific roles, for example. I’ll come back to that later, because AVI has a few more peculiarities. A client is created in the realm under Clients.

Create client

Client type SAML
Client ID AviController-vcf09-avi.lab.vcf
Name AVI - Test
Description 
Always display in UI Off
Root URL 
Home URL 
Valid redirect URIs https://vcf09-avi.lab.vcf/sso/acs/
Valid post logout redirect URIs https://vcf09-avi.lab.vcf/sso/acs/
IDP-Initiated SSO URL name 
IDP Initiated SSO Relay State 
Master SAML Processing URL 

Those are the basic settings. Next, a few more settings need to be adjusted. To do this, open the client settings—the range of options is overwhelming at first, but fortunately, not much needs to be changed.

The first setting must be made in Signature and Encryption. Here, Sign assertions must be set to On. In addition, the name ID format must be changed to email (every user must have an email address in the AD).

A theme can also be selected under Loging settings. Next, in the Tab Keys, I had to disable the client signature. The problem only occurred when I changed the AVI SSL/TLS Profile and no longer allowed weak ciphers. I haven’t yet figured out exactly where the problem lies. I imported the certificate as described in the AVI documentation, but it only works for me if I allow weak ciphers in AVI. If anyone has any ideas, I would be grateful for any input.

Under Roles, I create a new client role called avi-access. I’ll explain what this does a little later. Under Client Scopes, you have to create a Group Mapper, otherwise AVI won’t know anything about the groups. To do this, it’s important to go to the Dedicated scopes, which are named the same as the client, only with -dedicated.

Here, a new mapper must be configured. To do this, press the Configure new mapper button and select a mapper of type Group list.

Group attribute name groups
Friendly Name AD Groups
SAML Attribute NameFormat Basic
Single Group Attribute On
Full group path Off

The Group Mapper in the “Client Dedicated” area ensures that when logging in, the user’s groups are written to the token/SAML assertion for that specific client.

AVI will need the attribute later for the mapping profile. In the mapping profile, the attribute must have the same name as it was created in Keycloak—in my case, that is “groups”.

The SSO URL must be entered under Advanced.

Fine Grain SAML Endpoint Configuration

This section to configure exact URLs for Assertion Consumer and Single Logout Service.

Logo URL 
Policy URL 
Terms of service URL 
Assertion Consumer Service POST Binding URL https://vcf09-avi.lab.vcf/sso/acs/
Assertion Consumer Service Redirect Binding URL https://vcf09-avi.lab.vcf/sso/acs/
Logout Service POST Binding URL 
Logout Service Redirect Binding URL 
Logout Service SOAP Binding URL 
Logout Service ARTIFACT Binding URL 
Artifact Binding URL 
Artifact Resolution Service 

This completes the initial configuration of the client. However, do not be concerned, as we will need to configure more later on.

Role mapping for AVI

Next, we use the aforementioned client role. This role can only be used in the AVI client in Keycloak. With this, we will prevent a peculiarity of AVI. AVI has a strange habit of creating a user in AVI as soon as authentication is complete, regardless of whether the user has access or not. Currently, all I have to do is create a valid Keycloak session for a user, who is then created in AVI and receives an error message because they are not authorized. The only problem is that this creates a defective SAML cookie and you can no longer log in to AVI, even if you try with a valid user. Great, I can already see the tickets coming in. The solution is to delete the cookie in the browser and end the session in Keycloak. After that, you can log in again. Of course, you now have a user without authorization in AVI, which you must delete manually.

In general, this behavior is not a security issue, but user complaints are inevitable. To prevent this, I am building in a check to see whether the user is authorized to access the AVI. The check is performed at the Keycloak level, and the request is not sent to the AVI at all if there is no authorization for AVI.

To do this, a role mapping must be created in Keycloak. Under Groups -> vcf-avi -> Role Mapping, a role can be assigned to the group in Keycloak. The groups and user membership come from the AD, making it easy to manage centrally. To assign the role, click Assign role and select the appropriate client role avi-access via client roles.

Auth Mapping Profile for AVI

I promise, we’ll soon be able to log in with Keycloak. An Auth Mapping Profile is still required in AVI. This can be created under Templates -> Security -> Auth Mapping Profile. It must be a SAML type profile. The profile is given a descriptive name and requires a rule. I’ll keep it simple in this example and say that all users whose SAML attribute contains vcf-avi become super users. Remember, the SAML attribute contains all Keycloak groups. You could also write a RegEX, because with Contains I have the problem that a group named vcf-avi-test would also match. But I’ll leave that to the people who have to implement it in production.

Finally, the Auth profile and the Mapping Profile just need to be assigned.

AVI Authentication settings

You can do this easily under Administration -> System Settings and then edit them. Under Authentication, you must switch from Local to Remote. The Enable Local User Login checkbox should be selected, otherwise you will not be able to log in locally if SSO fails. Finally, map the Auth Profile with the Mapping Profile, and then we’re done and can run our first test.

First Test

If everything works correctly, after accessing the AVI page, a message should automatically appear stating that you are not logged in, and the Keycloak login screen should appear directly via the login link.

You can log in with either your username or your email address. Because I was smart enough to use my private email address, you can only see the username here.

Interim conclusion

SSO login via Keycloak should work, but unfortunately it is not yet possible to log in with a FIDO key, and there is still no way to prevent the dummy user from logging in to AVI, or rather, from being rejected by AVI, which breaks our SAML cookie. Time to change that. So grab a coffee, because this article could go on for quite a while.

Enable Fido Key

To use a Yubikey or similar device, a few options need to be set in Keycloak. Passwordless login is also supported, but I’m not personally a fan of this. However, the activation process is similar. First, in the Realm Settings under Authentication -> Required actions -> Webauth Register, set Enabled and Set as default action. For passwordless login, the corresponding option Webauthn Register Passwordless must be activated and set as the default action. Without the default action, users will not be asked to create a passkey when they log in for the first time.

Then, under Authentication -> Policies -> WebAuthn Policy or WebAuthn Passwordless Policy, the policy must be activated. To do this, WebAuthn -> Avoid same authenticator registration should be activated, or, in the case of WebAuthn Passwordless Policy, this option and enable passkeys in general.

Flow

Finally, a flow must be created under Authentication -> Flows. I use the built-in browser-based authentication flow as a basis and create a copy via the context menu. In order to use the client role in the flow, the flow must be assigned to the client. To do this, the flow you just created must be assigned under Clients -> AviController-xxx -> Advanced -> Authentication flow overrides. This allows each client to have its own flow and different checks or login methods.

A flow consists of three key elements. There are steps (executions), which are specific actions in the login process, such as deny access. Secondly, there are conditions, which are ways of checking something, such as whether the user has a certain role, and finally there are sub-flows, which can be used to group steps and conditions. In addition, each element has a requirement that determines whether the login continues or is terminated. In short, a flow is a chain of steps, conditions, and sub-flows with rules that control the login process.

• 1. To activate keypasses, WebAuthn must be configured in the flow and set to required.
• 2. Next, a subflow of type generel must be created. This can be done using the small plus sign. It is important that the subflow is in the correct position at the end and is indented correctly. The subflow is used to check role membership.
• 3. The subflow must be dragged and dropped into the correct position. It is sometimes easier to do this by dragging other elements upwards. The subflow must also be set to Conditional. The conditional block is only executed if the previous step is TRUE. In this flow, 2FA must have been successfully completed, otherwise the login will be canceled.
• 4. Next, a condition and a step must be created using the plus sign. Both must be defined as required, otherwise the check will not be performed and, in the event of a missing authorization, Access denied will not be executed.
• 5. The condition must be of type user role, and client roles must be selected under select role. Then I select the avi-access role and select Negate output. Negate Output ensures that the condition is set to TRUE if the user does not have the avi-access role. If the condition is TRUE, the step deny access is executed. If the user has the role, the condition is FALSE and the flow is terminated with the result that I have access.

If everything has been done correctly, the flow should look like the screenshot. “Indentation” is just as important here as it is in YAML.

Now that everything is configured, it should work roughly as shown in this video.

Troubleshooting

Troubleshooting can be very time-consuming, and I had a few problems. I could bet that I logged into AVI at least 300 times in the last two days. So here are a few general tips.

• Delete your cookies. Even if it is open in an anonymous tab, cookie remnants may remain during runtime, and in case of doubt, this is a broken SAML cookie that then prevents even valid logins.
• Always delete the session in Keycloak. Just because you log out of AVI does not mean that the SSO session is also closed.
• When dragging and dropping in Flow, elements may automatically deactivate if they are moved to an invalid position, for example. Keycloak attempts to save immediately after moving. This nearly drove me to distraction. Sometimes it makes more sense to move other elements than the element you actually want to move.
• Enables logging in Keycloak. This is configured in the Realm Settings under Events.
• If parts of the flow were not executed, the position of the sub-flow should be checked to see if it was nested correctly.
• If “access denied” is displayed despite correct role assignment, the negation in the condition was probably forgotten.

Summary

Keycloak or OIDC is a damn powerful toolkit. However, it is anything but simple and, unfortunately, extremely poorly documented for AVI load balancers. I actually wanted to write down the configuration for the VCF client, but the blog article is becoming so extensive that most people probably won’t read it anyway, so I’d rather pick up the whole thing again in a second article.