Deep Dive into Building WS1 UEM Components on the UAG

Welcome Back!

About a year ago, I wrote an article that I really enjoyed found here for a number of reasons. My son had just been born and I decided to build my first ever home lab. I learned a ton during that process and one of the things that I decided to take on was learning all about the Unified Access Gateway (UAG). As an EUC Champion I knew much to everyone’s dismay we would be converging all edge services to the VMWare UAG appliance. This was problematic on a few fronts (1) documentation and (2) supportability. I knew that if I wanted to be great at the UAG then I would be on my own. Please join me on this deep dive into the UAG and how you can make it run beautifully for your WS1 UEM deployment.

im back GIF

Building the Initial Configuration of the Unified Access Gateway

We should start with the UAG itself because things can get really confusing quickly. You should slice up the UAG itself into two buckets (1) the servers and (2) the load balancer. The documentation isn’t exactly succinct, which can be confusing. Let’s start by talking about the key building blocks when setting up the individual servers.

Building the Base

We focus on two key areas inside of the GUI, which will put you in a good spot early on.

In System Configuration, focus on a few key areas, such as cipher suites, TLS versions, and configuring your SysLog server to make troubleshooting easy. This is huge early on and can be a major difference maker.

I also prefer to cache all cookies and remember to only use “Quiesce Mode” when doing maintenance so you don’t take your stuff down!

On the TLS Server Configuration, it’s pretty simple but we should talk a bit about it:

A common question that I get is “Why do I need to do this if all of the internet facing stuff is going through my load balancer?” I asked the same question myself, but I learned the reason very quickly.

There is a file on the UAG called favicon.ico which is used as a health check endpoint essentially. Conceptually it’s simple. If ALL services are up and green, the favicon.ico file is reachable over HTTPs e.g. https://uag.mobilejon.com/favicon.ico. If even ONE service is down, the service is down. The key with all of this is you MUST replace the internet interface certificate with an internal certificate so your load balancer can properly evaluate the health monitor.

When we look at this section, we should ignore the words and understand simply that “Admin Interface” means the Listener on Port 9443 and “Internet Interface” means the Listener on Port 443. The way its worded can be VERY confusing so don’t let that derail you. One last thing, avoid putting in an Alias at all costs so it will actually apply the certificate. It’s common for the certificate install to fail if you do.

trainwreck GIF

The HA Proxy

The majority of what a WS1 UEM person will be doing revolves around HAProxy. Let’s look at the history real quick. HAProxy is considered to be widely one of the top open source load balancer technologies that exists out there. You can read about the open source project at HAProxy’s website. A few of the characteristics that make it so powerful are:

  • Single-buffering without data copy between R/W which is a huge CPU savings
  • CPU-affinity
  • Optimized HTTP header analysis
  • Content Analysis that minimizes the need to copy data and instead focuses on pointers
  • Regex-based header control to empower HAProxy to be able to deny/modify/or remove parts of headers to block dangerous attacks.

There’s many more benefits, but the HAProxy has been a great product for many years. This follows suit with other products like WS1 Access which uses several power hitters in the open source world like EHCache and RabbitMQ. Let’s talk about how it works on the UAG for you WS1 UEM peeps.

When the UAG sets up the HAProxy, it listens on TLS Port 443 and sets up routing rules to redirect to Port 9999. Basically the way this works is:

  • Pre-routing rule that says that packets arriving at 443 are routed to port 9999. (The HAProxy Statistics Endpoint from what I can see).
  • Input rules accept all packets from TCP

The UAG relies on a bit of haproxy code to send edge services to different backend ports based on their SSL SNI, which we will cover more in detail in the load balancer section of this blog post.

It looks like this image below, which we can break down as follows:

  • HAProxy content switching accepts TCP-Requests provided a complete SSL Hello occurs aka “ssl hello type 1”
  • HAProxy examines the raw contents in the request buffer for the SNI (Server Name Indicator) attribute at Layer 6 (Presentation Layer) to see what service and backend port to route it to.
  • If no SNI is found, it will inevitably land on the esmanager, which will throw errors in the logs about no matching proxy host.
tcp-request content accept if {req_ssl_hello_type 1}

use_backend cg_server if { req_ssl_sni -i  contentgateway.mobilejon.com }
use_backend seg_server if { req_ssl_sni -i segserver.mobilejon.com }
use_backend tunnel_server if { req_ssl_sni -i tunnel.mobilejon.com }

default_backend esmanager

backend esmanager
mode tcp
server default 127.0.0.1:6443 check

backend cg_server
mode tcp
server default 127.0.0.1:10443 check

backend seg_server
mode tcp
server default 127.0.0.1:11443 check

backend tunnel_server
mode tcp
server default 127.0.0.1:8443 check

A bit of editorialization after learning about how HAProxy works to note. They are using deprecated code that will be out of support by end of 2020. The req_ssl_sni code should be replaced with req.ssl_sni as according to the documentation is functionally limited. Something I’m still researching. It’s interesting to note that they support a bit more efficient way of doing this at Layer 5 called ssl_fc_sni which fetches the SNI from the incoming connection instead of watching to breakdown the contents. You will also notice this is the code that sends it to the Edge Services Manager (ESManager) in the event that nothing matches.

Working with Load Balancers with the UAG

The thing that I cannot stress enough about this is KNOW your LB. Essentially, enabling the Server Name Indicator attribute is part of the SSL profile on your load balancer. Most people think its turned on, but that is not always the case. Below, you will find a screenshot of where this is on a Netscaler.

It goes without saying, to be very clear and sure. If you are seeing things like 404s, then its likely your SNI is not enabled. You do have the ability to use content switching at the load balancer level to pass 443 –> backend ports e.g. 443 –> 10443 on the UAGs if necessary, but its not always a necessity.

Another key item around load balancers is the SSL Security of them. A great thing about the UAG is it embeds certain headers for security, thus simplifying your life. You can see for yourself at /opt/vmware/gateway/conf/security-headers.xml the headers that will be set by your UAG

That means, make sure you disable your HSTS policies on your load balancer so you aren’t injecting 867 headers for this and making SSL labs sad. This could cost you that A+!

Making the SEG work beautifully on UAG

I will be adding sections for each of the 3 services as I go along. The SEG is a crucial one and its vital that you get that right from the start. We start at the GUI which has a few key items that you need to do.

GUI Magic

The GUI configuration comes down to a few key areas to think about:

  • API Server URL should be in https://as1234.awmdm.com
  • SEG Hostname should be just the hostname without the http or https
  • MEM Config GUID you will find in your console obviously
  • You must upload the trusted root chain in PEM format for your Exchange server. This is an absolute must.

One item to be aware of, your connection will not go green if your Exchange URL has any issues. One example would be if your hostname has an issue with URL redirection and isn’t passing you to the right sites. Something you want to be careful of. UAG 3.10 will fix this bug, but something to always be aware of. Exchange redirection is an optional configuration that some clients may not account for. Once this is done, you should be all green and its party time!

Gru dancing party GIF on GIFER - by Daizahn

To the SEG Config Files!

I’d love to say you can just hit the GUI and be golden, but that isn’t the case here. The SEG is powered by Docker and with that in mind, you will have to do a few things especially if you are leveraging ENSv2 as you should be.

Inside of /opt/vmware/docker/seg/container/config we will focus on a few key areas. First let’s discuss the application.properties override file found one more level down in the override folder.

This was a really smart move by VMware. They give you a file where you can set any changes you want to make to configurations for the application.properties file. You have a few you may need to make that you can see below. These two commands which might be needed will let you enable ENS Proxy and let you specify the EWS URL. The 2nd one is optional based on need and behaviors you see.

##Enable EWS Proxy for Boxer and ENSv2##
enable.boxer.ens.ews.proxy=true
##Specify the EWS Server URL##
ews.email.server.host.and.port=https://mail.mobilejon.com:443

One other item you will need to address for the EWS Proxy is enabling unsupported auth methods. This is a requirement if you aren’t using CBA and need support for the www-authenticate header which the UAG will otherwise strip. You will find this line around the 40% mark on the file (you’re welcome). On a side note, VMware maybe make your config keys less ridiculous. You lost me by the 3rd dot:

remove.unsupported.auth.for.ews=false

The logback.xml is another very crucial area. Most of the services on the SEG will not have their log levels increased by using the UAG log level areas, so I strongly suggest looking at them. A few items that you can increase are:

  • Kerberos Transaction Logs (very useful when you turn on the KCD integration)
  • Kerberos Service Manager
  • SEG App Logs
  • EWS Proxy Log (the only way you will figure out what’s wrong with the EWS Proxy for ENSv2)
  • EWS Transaction Log
  • HTTP Transaction Logs and more

It’s important to point out how CRUCIAL it is that you are careful with this log and you know how to work with log files. An issue that you will commonly run into is after editing a log the SEG service will go down because of a syntactical error. The good news is by default there is a logback.xml.bak file on the UAG that you can do a basic copy command to fix any issues with the logback.xml.

A very common thing that I have seen people do is forget to close to config file and restart SEG services. When you do this, it creates some artifacts and breaks the config file. It’s something you really need to make sure you avoid.

Another cool thing to point out is the SEG does even more with your HTTP headers to help with your security posture:

The last file that I want to mention, which is very helpful is the seg-jvm-args.conf file. This file will let you customize the Java-specific components of your SEG implementation. Mainly it focuses on security components and how you can leverage them.

The key items you can customize in here are:

  • Supported Algorithms, Key Sizes, Protocols
  • SysLog for SEG
  • Kerberos Process Recycling

Docker Troubleshooting

Earlier I mentioned the Docker component to the SEG. When you have issues with services starting, you may need to get a bit cute. My suggestion typically is go to /var/lib/docker/containers/container ID (some long ridiculous number) and look at the .log file in there. That will typically give you a hint or two. If it doesn’t help, reach out and I will try to help you.

You will find in this folder the logs, resolv.conf, some json config files, etc. It’s a very useful area that can be leveraged at times. Once you fix the issue that is breaking docker, you can use the UAG scripts folder to bring you back to life.

You can find that in /opt/vmware/gateway/scripts and find a few useful items in there:

Clearly, you will use the startDocker.sh script to bring Docker back online if necessary. You will also see in here that you have some scripts for starting and stopping the SEG in this location. One pro-tip for you: if the SEG start and stop scripts don’t work, you can always go into the SEG service in the UI and put in your credentials/click SAVE to bring yourself back to life.

Content Gateway

The Content Gateway is for the most part plug and play, but there are a few tips to make the user experience much better.

For those who use Netapp, you should turn jcifs active to true for that functionality to work. I also typically suggest as a good rule to disable the show hidden files and folders as its all about that user experience.

One other item that I suggest looking at is simplifying the domain name requirements for the CG. You can go to /opt/airwatch/content-gateway/smb-connector/smb.conf and uncomment the workgroup section and add your domain **This is only supported if you are in an environment with a single domain**, but the awesome thing is it eliminates the need to specify domain\ when logging into a share on the Content App.

I strongly suggest checking out the troubleshooting documentation for anything else further on Content Gateway as its not as widely used and I don’t spend as much time on it honestly. Similarly to the SEG, feel free to modify the logback.xml file to increase log levels when you try to troubleshoot share access issues, but make sure you make a backup first as they do not create a backup by default.

In Closing

The UAG is a tough product. It’s the epitome of a black box, but through some creativity and engineering it can help you in your journey of digital transformation. I will be continuing to update this article as I dig a bit deeper into the technologies that power the tunnel and CG like I have with the SEG, but like all documentation is an ever-evolving process.

Newer doesn’t always mean better, but the potential is definitely there. My hope is that product management will continue to evolve the UAG and people will continue to collaborate to make the UAG and its children work more seamlessly.

Advertisements

Leave a Reply