Over the last few years, I have written several times about building seamless experiences. This comes in different flavors depending on the operating system, such as iOS Single Sign-On powered by certificates and Kerberos or PCs using native kerberos authentication. Seamless identity can be achieved easily on modern applications, but what about on these archaic barbarians that don’t support SAML!?
The subject that I speak of is no other than something that strikes fear in the hearts of even the strongest member of VMware’s professional services: Secure Email Gateway (SEG) and KCD (Kerberos Constrained Delegation) integration. KCD is very amazing because simply it allows you to deliver On-Premise Exchange-powered mobile email seamlessly without passwords by leveraging certificates and kerberos. My favorite part is it doesn’t even need a keytab! If you read my article here about setting up WS1 Access for Kerberos you know all about how much that sucks! Without further adieu, let’s discuss how to make this a reality. Here’s a small preview into what it looks like overall before we go into configuration and find the documentation here:
Configuring KERBEROS in the UEM Console
This is a bit more complicated than it let’s on, but overall it’s not too terrible. Like most things, you just need to understand how they work. That’s why I’m here!
As you can see below, this is the advanced tab in Email > Configuration. You simply uncheck “Use Default Settings” and get to work.
Building the Chain
Let’s focus on a few key areas. First and foremost is the certificate. Certificate-chains are not particularly difficult to build. The way it works is Intermediate goes first and root goes second. I have a simple Linux one line command that eliminates any potential issues. One pro-tip, pull one of your WS1 UEM user certificates off your CA and download the intermediate and root certificates directly from the certificate chain tab. This is well worth it and eliminates any unknown variables.
cat intermediate-certificate.cer root-certificate.cer > chain-certificate.cer
openssl verify chain-certificate.cerThis is a simple and elegant way to create your root chain certificate without any text editor shenanigans. Afterwards the openssl command verifies the certificate is good. You will likely see this error below, but don’t worry about it. It’s not a big deal:
Verifying and Entering the SPN
Obviously your Exchange team is doing most of this setup, but you should confirm implicitly that you have the right information. I typically recommend logging into Outlook and running a klist in the command prompt. Klist will show all of your kerberos tokens. You just want to find the one for Outlook to confirm what you are putting in the console is actually correct. the syntax is simple: HTTP/{User/Group or Server}. This should always be a computer account for the ASA (Alternate Service Account) since every Exchange environment is going to have multiple servers in production.
Another strong suggestion is to open up ADUC and verify the “Delegation” tab to ensure you see the SPNs for your Exchange URL and not individual servers in here because that will NOT work ever.
An error you may encounter if things are setup incorrectly can be seen below with something along the lines of “Couldn’t obtain token due to non-recoverable error”
Wireshark is your best friend when looking at anything kerberos and it is pretty clear on the packet capture for this issue. You are going to see the Kerberos requests, but never a response because it’s requesting against an invalid SPN.
Credentials and Domain Mappings
Credentials are basic. Do not put the domain\ in there. That’s really it! You will see errors in the logs that are insane like mobilejon\awseg@MOBILEJON.com blah blah. So credentials are basic. The big challenge you “could” run into is with domains.
Hopefully most of you are dealing with a single domain. For those who aren’t, there’s three rules:
- Domain in all CAPS
- Map Domains to DCs in the same domain e.g. MOBILEJON.COM cannot decrypt tokens from SYNTEREX.COM. It will never work so be logical.
- Don’t use load balancing because this isn’t LDAP!
Time for more error logs to show if you are doing it wrong. The error you want to look out for is below. This is what happens when you try to decrypt a token for a different domain.
Customizing your Kerberos Settings
I definitely suggest looking at some of the settings in the Kerberos application properties before you start your build. You can find them in /opt/vmware/docker/seg/container/config/application.properties:
The main one that I recommend is to set the SEG to update/rebuild the kerberos config file on service restart. Some people like to leave this off after they have finished the build, but that will depend if you trust your people. As long as people don’t touch stuff they shouldn’t, it’s a good idea to leave it on.
generate.krb5.config.at.service.restart=trueOther options in here that are interesting include:
- Forcing the SEG to validate the certificate thumbprint
- Enable CRL checking
- Determine if SEG will start if CRL is down
- Hardcoding the CRL URL
The Certificate Template
One of the major challenges with WS1 UEM is how everything needs certificates. Below, you will find a recommended template that will cover all of your services including Tunnel to simplify things. For SEG, we mainly care about the UPN, which you can see is in the SAN.
Load Balancing
Load Balancing is the true lynch pin of your entire SEG KCD build. I will tell you from experience do not say “there’s no way its the load balancer” because most times its the load balancer. Here’s a short list of the things you must avoid if you want to be able to do SEGv2 with KCD:
- SSL Offloading
- SSL Re-Encryption
- Content Re-writing
- Header Manipulation or Modification
- Header Size Configuration or really any settings that have anything to do with headers
I promise if you avoid those items, you will be in really good shape. The main issue you will run into is the SSL security afterwards. Let’s talk about hardening the UAG’s SEG container.
SSL Hardening
SSL hardening will no longer happens at the load balancer when you move to KCD because it just can’t. Yes Kerberos is a pain in the ass, deal with it! First let me say, big props to the developers because building a JVM argument file is super useful. People struggle enough with Java as UEM engineers so this at least minimizes your issues.
Secondly, if you are using SEG you SHOULD be using Boxer with it. One of the current gaps is a SEG with ENSv2 will require you to enable a few weak ciphers (I know that sucks!), which means the last two ciphers on my config file below will need to be removed or you will not get real-time notifications for Boxer.
You will edit the Java properties for the SEG here below. This code will give you the most secured SEG container possible until they implement Perfect Forward Secrecy (PFS). Try not to be too alarmed when your SEG gets a B grade. They’re working on it I promise!
/opt/vmware/docker/seg/container/config/seg-jvm-args.conf:
# Java Additional Parameters
-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=tmp
-XX:OnError="jmap -dump:live,file=tmp/seg_on_error_dump.hprof-%p %p"
-Xmx8192m
-Dvertx.logger-delegate-factory-class-name=io.vertx.core.logging.SLF4JLogDelegateFactory
-Dmail.mime.base64.ignoreerrors=true
-Dfile.encoding=UTF-8
-Dmail.mime.charset=UTF-8
# Disable usage of vulnerable chiper suits for SSL Handshake.
-Djdk.tls.disabledAlgorithms="MD5, RC4, TLSv1, TLSv1.1, SSLv2Hello, SSLv3, DSA, DESede, DES, 3DES, DES40_CBC, RC4_40, MD5withRSA, DH, 3DES_EDE_CBC, DHE, RSA keySize < 2048, DH keySize < 1024, EC keySize < 224, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_GCM_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_128_GCM_SHA256, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256, TLS_RSA_WITH_AES_256_CBC_SHA256, TLS_RSA_WITH_AES_256_GCM_SHA384, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384, TLS_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA"
-Djdk.tls.ephemeralDHKeySize=2048
-Djdk.tls.rejectClientInitiatedRenegotiation=true
-Dhttps.protocols=TLSv1.2
-Djava.net.preferIPv4Stack=true
# If unknown charset lookup failure should be ignored. Used during MIME header parsing.
-Dmail.mime.ignoreunknownencoding=true
# WARNING - Enable debug port only if needed. Also don't checkin your changes into Stash.
# # Map the port with a Host VM port and use that for remote access # -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=50000
# JMC settings start - These settings are used to enable Java Mission Control monitoring.
# -Dcom.sun.management.jmxremote
# -XX:+PrintGC
# Map this port with a Host VM port and use that for remote access # -Dcom.sun.management.jmxremote.rmi.port=1099
# -Dcom.sun.management.jmxremote.port=1099
# -Dcom.sun.management.jmxremote.local.only=true
# -Dcom.sun.management.jmxremote.authenticate=false
# -Dcom.sun.management.jmxremote.ssl=false
# Change this IP to Container Host IP.
# -Djava.rmi.server.hostname=1.2.3.4
# -XX:+UnlockCommercialFeatures
# -XX:+FlightRecorder
# JMC Settings ends.
-Dkerberos.process.recycle.time=23:59:59
-Denable.kerberos.process.recycle=false
# Syslog configuration starts - To direct application logs to Syslog server, set property "syslog.enabled" as true.
# Provide syslog server host, port and facility.
# By default, facility is set to USER (generic user-level messages) and port is 514.
# By default, appender is added for app logger, but it can be added to other loggers as well.
-Dsyslog.enabled=true
-Dsyslog.host=localhost
-Dsyslog.port=514
-Dsyslog.facility=USER
# Syslog configuration ends.
Setting your Log Levels for the Initial Build
I strongly suggest bumping up all of the Kerberos logs to DEBUG for the initial build, but honestly your security team would love you if they had those details in their SIEM. They’re found in a few locations. A Pro-tip for you: Do NOT modify these files directly in Linux. I personally love using WinSCP to do this because it’s simple and painless. It’s also cool to mention that you don’t need a service restart after increasing log levels. You typically just wait 1-2m before testing.
/opt/vmware/docker/seg/container/config/kerberos/kcd-client.conf
Additionally, there’s other stuff you can look at too like DLL logging and ticket request retries, but I suggest you evaluate based on your experience.
/opt/vmware/docker/seg/container/config/logback.xml
Basic Troubleshooting
One of the main keys about how this whole thing works is understanding the flow. I will walk you through the steps so you can understand what to look for.
The SSL Handshake
The first part of the connection is the SSL handshake, but it might be a bit different than your typical handshake. The key difference is that after the server hello you will see the “Certificate Request” attribute where your UAG will send your email client a list of the certificates that are part of the client certificate chain you uploaded in the console.
The Wireshark shows it nicely with a nifty filter: ssl.handshake.type=11. This should go smoothly provided that you did things the right way.
Certificate Functions
Once the handshake completes, you will see a number of things occur:
- UAG connects to your CRL over HTTP to validate the certificate
- UAG connects to the DC that matches your domain mapping to request a Kerberos token
- UAG uses the Kerberos token to authenticate to Exchange and make the magic happen… because magic happens!
Let’s hit on a few tips to make sure these things work correctly…
CRL
As mentioned in the documentation, you need to make sure your certificates are publishing a HTTP CRL. That does NOT mean HTTPs, or LDAP, it means HTTP (the one without the cute little lock icon).
Validation is pretty basic with a simple wget http://ca01.mobilejon.com/CertEnroll/ca01.crl to see if you can connect and download your CRL. Again, this doesn’t mean OCSP, it means CRL.
Exchange
We covered enough Kerberos so we will focus on Exchange. Basically, the UAG will present the kerberos token on your behalf to Exchange via a nice little OPTIONS call to see what can be supported. This “may” return a 401 if your token is bad.
Additionally, the nice thing is that the EWS endpoint for ENS will work straight out of the box for certificate-based authentication but you might have to walk back a config change you made when using basic authentication.
To fix the auth issue, you may have to modify something you may have originally modified to make basic authentication work on your UAG. You would just change the remove.unsupported.auth.for.ews=false to true.
Closing the Loop
I have to admit that outside of building the first VIDM in AWS leveraging RDS this was the hardest thing that I have had to tackle. It’s very daunting because of all of the moving pieces between load balancing, certificate, kerberos, etc. These are items that stress out a ton of people in the UEM space because they’re outside of our comfort zone. I will admit that I am thankful for a few people that were willing to help me navigating this dangerous terrain especially Wannes De Boodt who helped me narrow down some of the issues that I had and the PMs at VMware that had to endure the tornado that is Mobile Jon.
I beg all of you to use your community better and when difficult things come up to please use your peers. We are here to help you. If you are interested in doing any projects that are daunting or too time-consuming for you, I am available to consult and drive those projects to completion. I am many things, but the thing that I am most is stubborn and relentless.
