Welcome Back!

About a year ago, I wrote an article that I really enjoyed found here for a number of reasons. My son had just been born and I decided to build my first ever home lab. I learned a ton during that process and one of the things that I decided to take on was learning all about the Unified Access Gateway (UAG). As an EUC Champion I knew much to everyone’s dismay we would be converging all edge services to the VMWare UAG appliance. This was problematic on a few fronts (1) documentation and (2) supportability. I knew that if I wanted to be great at the UAG then I would be on my own. Please join me on this deep dive into the UAG and how you can make it run beautifully for your WS1 UEM deployment.

Building the Initial Configuration of the Unified Access Gateway

We should start with the UAG itself because things can get really confusing quickly. You should slice up the UAG itself into two buckets (1) the servers and (2) the load balancer. The documentation isn’t exactly succinct, which can be confusing. Let’s start by talking about the key building blocks when setting up the individual servers.

Building the Base

We focus on two key areas inside of the GUI, which will put you in a good spot early on.

In System Configuration, focus on a few key areas, such as cipher suites, TLS versions, and configuring your SysLog server to make troubleshooting easy. This is huge early on and can be a major difference maker.

I also prefer to cache all cookies and remember to only use “Quiesce Mode” when doing maintenance so you don’t take your stuff down!

On the TLS Server Configuration, it’s pretty simple but we should talk a bit about it:

A common question that I get is “Why do I need to do this if all of the internet facing stuff is going through my load balancer?” I asked the same question myself, but I learned the reason very quickly.

There is a file on the UAG called favicon.ico which is used as a health check endpoint essentially. Conceptually it’s simple. If ALL services are up and green, the favicon.ico file is reachable over HTTPs e.g. https://uag.mobilejon.com/favicon.ico. If even ONE service is down, the service is down. The key with all of this is you MUST replace the internet interface certificate with an internal certificate so your load balancer can properly evaluate the health monitor.

When we look at this section, we should ignore the words and understand simply that “Admin Interface” means the Listener on Port 9443 and “Internet Interface” means the Listener on Port 443. The way its worded can be VERY confusing so don’t let that derail you. One last thing, avoid putting in an Alias at all costs so it will actually apply the certificate. It’s common for the certificate install to fail if you do.

The HA Proxy

The majority of what a WS1 UEM person will be doing revolves around HAProxy. Let’s look at the history real quick. HAProxy is considered to be widely one of the top open source load balancer technologies that exists out there. You can read about the open source project at HAProxy’s website. A few of the characteristics that make it so powerful are:

• Single-buffering without data copy between R/W which is a huge CPU savings
• CPU-affinity
• Optimized HTTP header analysis
• Content Analysis that minimizes the need to copy data and instead focuses on pointers
• Regex-based header control to empower HAProxy to be able to deny/modify/or remove parts of headers to block dangerous attacks.

There’s many more benefits, but the HAProxy has been a great product for many years. This follows suit with other products like WS1 Access which uses several power hitters in the open source world like EHCache and RabbitMQ. Let’s talk about how it works on the UAG for you WS1 UEM peeps.

When the UAG sets up the HAProxy, it listens on TLS Port 443 and sets up routing rules to redirect to Port 9999. Basically the way this works is:

• Pre-routing rule that says that packets arriving at 443 are routed to port 9999. (The HAProxy Statistics Endpoint from what I can see).
• Input rules accept all packets from TCP

The UAG relies on a bit of haproxy code to send edge services to different backend ports based on their SSL SNI, which we will cover more in detail in the load balancer section of this blog post.

It looks like this image below, which we can break down as follows:

• HAProxy content switching accepts TCP-Requests provided a complete SSL Hello occurs aka “ssl hello type 1”
• HAProxy examines the raw contents in the request buffer for the SNI (Server Name Indicator) attribute at Layer 6 (Presentation Layer) to see what service and backend port to route it to.
• If no SNI is found, it will inevitably land on the esmanager, which will throw errors in the logs about no matching proxy host.

tcp-request content accept if {req_ssl_hello_type 1}

use_backend cg_server if { req_ssl_sni -i  contentgateway.mobilejon.com }
use_backend seg_server if { req_ssl_sni -i segserver.mobilejon.com }
use_backend tunnel_server if { req_ssl_sni -i tunnel.mobilejon.com }

default_backend esmanager

backend esmanager
mode tcp
server default 127.0.0.1:6443 check

backend cg_server
mode tcp
server default 127.0.0.1:10443 check

backend seg_server
mode tcp
server default 127.0.0.1:11443 check

backend tunnel_server
mode tcp
server default 127.0.0.1:8443 check

A bit of editorialization after learning about how HAProxy works to note. They are using deprecated code that will be out of support by end of 2020. The req_ssl_sni code should be replaced with req.ssl_sni as according to the documentation is functionally limited. Something I’m still researching. It’s interesting to note that they support a bit more efficient way of doing this at Layer 5 called ssl_fc_sni which fetches the SNI from the incoming connection instead of watching to breakdown the contents. You will also notice this is the code that sends it to the Edge Services Manager (ESManager) in the event that nothing matches.

Working with Load Balancers with the UAG

The thing that I cannot stress enough about this is KNOW your LB. Essentially, enabling the Server Name Indicator attribute is part of the SSL profile on your load balancer. Most people think its turned on, but that is not always the case. Below, you will find a screenshot of where this is on a Netscaler.

It goes without saying, to be very clear and sure. If you are seeing things like 404s, then its likely your SNI is not enabled. You do have the ability to use content switching at the load balancer level to pass 443 –> backend ports e.g. 443 –> 10443 on the UAGs if necessary, but its not always a necessity.

Another key item around load balancers is the SSL Security of them. A great thing about the UAG is it embeds certain headers for security, thus simplifying your life. You can see for yourself at /opt/vmware/gateway/conf/security-headers.xml the headers that will be set by your UAG

That means, make sure you disable your HSTS policies on your load balancer so you aren’t injecting 867 headers for this and making SSL labs sad. This could cost you that A+!

Making the SEG work beautifully on UAG

I will be adding sections for each of the 3 services as I go along. The SEG is a crucial one and its vital that you get that right from the start. We start at the GUI which has a few key items that you need to do.

GUI Magic

The GUI configuration comes down to a few key areas to think about:

• API Server URL should be in https://as1234.awmdm.com
• SEG Hostname should be just the hostname without the http or https
• MEM Config GUID you will find in your console obviously
• You must upload the trusted root chain in PEM format for your Exchange server. This is an absolute must.

One item to be aware of, your connection will not go green if your Exchange URL has any issues. One example would be if your hostname has an issue with URL redirection and isn’t passing you to the right sites. Something you want to be careful of. UAG 3.10 will fix this bug, but something to always be aware of. Exchange redirection is an optional configuration that some clients may not account for. Once this is done, you should be all green and its party time!

To the SEG Config Files!

I’d love to say you can just hit the GUI and be golden, but that isn’t the case here. The SEG is powered by Docker and with that in mind, you will have to do a few things especially if you are leveraging ENSv2 as you should be.

Inside of /opt/vmware/docker/seg/container/config we will focus on a few key areas. First let’s discuss the application.properties override file found one more level down in the override folder.

This was a really smart move by VMware. They give you a file where you can set any changes you want to make to configurations for the application.properties file. You have a few you may need to make that you can see below. These two commands which might be needed will let you enable ENS Proxy and let you specify the EWS URL. The 2nd one is optional based on need and behaviors you see.

##Enable EWS Proxy for Boxer and ENSv2##
enable.boxer.ens.ews.proxy=true
##Specify the EWS Server URL##
ews.email.server.host.and.port=https://mail.mobilejon.com:443

One other item you will need to address for the EWS Proxy is enabling unsupported auth methods. This is a requirement if you aren’t using CBA and need support for the www-authenticate header which the UAG will otherwise strip. You will find this line around the 40% mark on the file (you’re welcome). On a side note, VMware maybe make your config keys less ridiculous. You lost me by the 3rd dot:

remove.unsupported.auth.for.ews=false

The logback.xml is another very crucial area. Most of the services on the SEG will not have their log levels increased by using the UAG log level areas, so I strongly suggest looking at them. A few items that you can increase are:

• Kerberos Transaction Logs (very useful when you turn on the KCD integration)
• Kerberos Service Manager
• SEG App Logs
• EWS Proxy Log (the only way you will figure out what’s wrong with the EWS Proxy for ENSv2)
• EWS Transaction Log
• HTTP Transaction Logs and more

It’s important to point out how CRUCIAL it is that you are careful with this log and you know how to work with log files. An issue that you will commonly run into is after editing a log the SEG service will go down because of a syntactical error. The good news is by default there is a logback.xml.bak file on the UAG that you can do a basic copy command to fix any issues with the logback.xml.

A very common thing that I have seen people do is forget to close to config file and restart SEG services. When you do this, it creates some artifacts and breaks the config file. It’s something you really need to make sure you avoid.

Another cool thing to point out is the SEG does even more with your HTTP headers to help with your security posture:

The last file that I want to mention, which is very helpful is the seg-jvm-args.conf file. This file will let you customize the Java-specific components of your SEG implementation. Mainly it focuses on security components and how you can leverage them.

The key items you can customize in here are:

• Supported Algorithms, Key Sizes, Protocols
• SysLog for SEG
• Kerberos Process Recycling

Docker Troubleshooting

Earlier I mentioned the Docker component to the SEG. When you have issues with services starting, you may need to get a bit cute. My suggestion typically is go to /var/lib/docker/containers/container ID (some long ridiculous number) and look at the .log file in there. That will typically give you a hint or two. If it doesn’t help, reach out and I will try to help you.

You will find in this folder the logs, resolv.conf, some json config files, etc. It’s a very useful area that can be leveraged at times. Once you fix the issue that is breaking docker, you can use the UAG scripts folder to bring you back to life.

You can find that in /opt/vmware/gateway/scripts and find a few useful items in there:

Clearly, you will use the startDocker.sh script to bring Docker back online if necessary. You will also see in here that you have some scripts for starting and stopping the SEG in this location. One pro-tip for you: if the SEG start and stop scripts don’t work, you can always go into the SEG service in the UI and put in your credentials/click SAVE to bring yourself back to life.

VMWARE TUNNEL

The VMware Tunnel service or “Per-App VPN” has some interesting nuances to it. We will start by talking about configuring it in the console and then moving onto some requirements and troubleshooting components. Keep in mind we don’t discuss the proxy component because its deprecated and no one should be trying to use it at this point moving forward.

Configuring Tunnel in the Console

Without getting too deep into it, I will be discussing multiple UAG cluster setups in a later blog (e.g. you have UAGs for America and for APAC). You should configure Tunnel at the level where you have your devices. Anyone who knows me will tell you that I HATE configuring stuff below Global, but you have no choice here.

Your base configuration, will look something like this below. We will cover each of the sections because its important period. I’m expecting most people are doing this in the relay-endpoint topology, which is commonly referred to as “cascade” now.

Server Authentication

You can essentially go either way on this one. I tend to recommend using AirWatch certificates. Let’s be super clear. This isn’t the SSL certificate other products. This is a server certificate used to build the trust. It’s unique and part of a unique certificate pair for the group identifier and tunnel environment. One great thing they added is the ability to regenerate this certificate, which is great for your security team and potentially part of incident response.

Client Authentication

Client Authentication is a bit trickier than I expected. This is where you can/should use your own CA for generating certificates and using them to authenticate to the tunnel. I have a few pro-tips to make this go well.

Firstly, you will want to create a separate template. I’ve found that the SAN suggestions they have don’t work and the subject name suggestions tend to break wireless authentication. Not a huge deal, but the template setup that I provide will save some time. You will want to build it like this below:

The other thing that I learned is the Tunnel does not like complete chain certificates. Not a big deal, you will just want to upload your root and intermediate individually in the console. Once you have done this, things will be pretty easy. The overall configuration for this tab can be seen below:

Networking

This was an item that messed me up for a bit. This matters if you’re using a proxy or not. You can easily overlook that it will look at the Tunnel Traffic Rules for routing your AWCM and API traffic.

Some people will prefer to send outbound traffic direct for more timely communication and compliance, which this feature which is enabled by default can stop you from deploying the Tunnel.

Obviously if you are using a reverse proxy, you will want to go into “Server Traffic Rules” and add your Outbound Proxy. In my opinion, most times a proxy here is overkill and you get far more value from unimpeded communication to your WS1 endpoints.

Device Traffic Rules

Device Traffic rules are the last area that I will cover for the configuration as the other stuff is self-explanatory. Simply, you have a rule to tunnel traffic and add apps to tunnel. You will then have a second rule for either bypassing or tunneling the other traffic (hopefully bypassing if you want users to ACTUALLY install the tunnel app).

There is one vital key to all of this. You must deploy the managed apps at the same level as your Tunnel configuration AND add the Tunnel Per-App VPN profile to those apps for all of this to work.

In closing this section out, once you have published your entire configuration and restarted the services on the UAG console, I would suggest doing an add version and publish on your Tunnel profiles to ensure things are clean. Sometimes a redeployment of the Tunnel mobile app could be required, but the hope is it will be mostly clean.

SSL Certificates, Load Balancing, and the VMware Tunnel

I think it’s good to cover this shortly as your security team will likely have a ton of questions around it. We will start with the LB requirements.

Load Balancing for the Tunnel

There are a few mistakes people commonly make with load balancing/tunnel:

• The Front End and Back End must be TCP and UDP/8443
• You must SSL bridge that traffic straight through with no encryption or manipulation of the traffic
• No header manipulation or shenanigans

One of the challenges you can’t design around is the Tunnel service will automatically build the trusted certificates for the back end aka cascade_ca.pem. When you mess around with things, stuff falls apart and you will run into tons of trust issues. In closing, I strongly suggest using this article here for the firewall rules that are needed because the documentation isn’t as clear as it could be.

How the Certificates Work on Tunnel

We are mainly concerned with two workflows when it comes to Tunnel. First is the initial setup and the second is the continuous certificate lifecycle.

Initial Setup:

1. Your UAG will connect to the UEM API and use the API key and Certificate
2. UEM creates the unique key pair for UEM and Tunnel by the UEM root certificate tied specifically to your environment.
3. UEM generates a unique self-signed certificate (the server certificate) and sends the certificates and chains back to the UAG over TLS.
4. UEM uses message signing to ensure the Tunnel is only trusting communication from the UEM environment.

Continuous Certificate Delivery:

1. Once the Tunnel Profile is pushed to the device, it will generate the client certificate with your template you configured.
2. Tunnel will receive the client certificate chain during installation of the profile
3. Tunnel performs outbound AWCM/API calls to build a list of trusted devices, apps, certificates, and statuses.
4. Tunnel will only successful authenticate sessions based on the trust it built in the previous step.
5. Lifecycle management of certificates is managed by the profile, such as automatic revocation and renewal of certificates.

Tools at your Disposal for your Tunnel Environment

The number one tool that you will want to take advantage of is VPNReport.

VPNReport

You can find vpnreport in the vpnd directory: /opt/vmware/tunnel/vpnd. VPNReport has a few useful commands that help you analyze the health of the tunnel service on your UAG. The first command we look at is vpnwhitelist, which shows you a list of the trust on your UAG, including Device UID, Management Status, Thumbprint, Compliance Status, and User Name. The command to run this report is below.

./vpnreport whitelist --verbose (optional)

The other use of VPNReport is seeing the health of your UAG and its connectivity to WS1 UEM. It’s useful because it will show you the current connectivity to the API and AWCM along with some basic statistics/configuration information.

./vpnreport stat --interval=30 (optional)

The output of the command can be seen below. I do prefer to use the interval flag so it auto-updates:

The question becomes, what do I do if stuff shows up as Down? Tunnel troubleshooting is mostly straight forward and I tend to use my friend wget. Simply for validating your tunnel and AWCM you can just do this:

wget https://cn410.awmdm.com/API/help
wget https://awcm410.awmdm.com/awcm/status

If you can confirm those are up and good, I will typically do a service restart if some stuff is showing up as down:

systemctl vpnd.service restart

You will learn that doing service restarts like this instead of saving the tunnel configuration in the UAG console are much more effective and less intrusive overall.

The Server.Conf File

Not that I particularly advocate for messing around with this file, but there is some useful stuff to note.

1. Use the UEM GUI to change the log level instead of the config file. You can “technically” do either, but I prefer to not mess with these files unless absolutely necessary.
2. You can tweak the syslog capture in here, like other services. Specifically, you need to configure the syslog host and port in this file and you can see below. I particularly like that you can say what events should be logged and you can customize the format/fields being sent over. This can make life much easier for troubleshooting:

3. You can tweak the supported ciphers for TLS and DTLS which gives you some ability to meet growing security requirements:

Content Gateway

The Content Gateway is for the most part plug and play, but there are a few tips to make the user experience much better.

For those who use Netapp, you should turn jcifs active to true for that functionality to work. I also typically suggest as a good rule to disable the show hidden files and folders as its all about that user experience.

One other item that I suggest looking at is simplifying the domain name requirements for the CG. You can go to /opt/airwatch/content-gateway/smb-connector/smb.conf and uncomment the workgroup section and add your domain **This is only supported if you are in an environment with a single domain**, but the awesome thing is it eliminates the need to specify domain\ when logging into a share on the Content App.

I strongly suggest checking out the troubleshooting documentation for anything else further on Content Gateway as its not as widely used and I don’t spend as much time on it honestly. Similarly to the SEG, feel free to modify the logback.xml file to increase log levels when you try to troubleshoot share access issues, but make sure you make a backup first as they do not create a backup by default.

In Closing

The UAG is a tough product. It’s the epitome of a black box, but through some creativity and engineering it can help you in your journey of digital transformation.

Newer doesn’t always mean better, but the potential is definitely there. My hope is that product management will continue to evolve the UAG and people will continue to collaborate to make the UAG and its children work more seamlessly.