In short, you have a RADIUS server that handles all the authentication requests and a RADIUS client that sends these requests to the server. This is the tale of the unusual nightmate I had troubleshooting a truely intermittent issue for a Remote access VPN on Meraki with a RADIUS server for authentication.
What is AAA
Authentication is one of the corner stones of security. When granting (authorizing) users or devices access to elements within your network the common first step is proving who/what is attempts to access in the first place. This is Authentication.
In a Bar
You providing your ID (Authenication)
The system grants you access if you meet certain conditions such as age(Authorization)
You behaviour is monitored, if not acceptable action can be taken against you (Accounting)
There are many ways in a computer networks to authenticate one of which is utilizing a service called RADIUS (Remote Authentication Dial In User Service).
It Starts
Recently, I was configuring radius for Client VPN I came across a rather perculiar issue. Configuring RAVPN on Meraki is rather easy. You can simply point to the radius server, enter the IP of the server, enter a preshared key and done. For the most part no issues at all. However the other day proved to be rather unusual.
What happens when some authentication succeeds, but others fail?
The scenario
Intermittent issues are some of the hardest to troubleshoot, you never know when a problem will present itself and how long you have before you lose access to valuable data.
Some users were successfull authenticating
Some users were failing
Users had identical laptop
Users were spread across the UK
There were users on the same remote site seeing different authentication results
Users that failed suddenly succeeded and vice versa (a later discovery)
Errors were ranging from L2TP (Layer 2 Tunneling Protocol) errors, to DNS errors and even just flat out denials because user names were incorrect.
Troubleshooting begins
At this stage we need to start troubleshooting. With unusual cases like this one, the best thing is to start ruling out what the problem isn't, before you can start to understand what the problem actually is.
Firstly, RADIUS configuration must be in general ok, the server is at least responding to some users. In early stages it was only a single VPN that was an issue. So one might believe that the keys would need to be updated, But equally some are successful so this cannot be a problem.
Secondly, there is currently no commonallity between users impacted and users not impacted. Certainly nothing within the user groups that are selected for the VPN use.
Thirdly, After some testing I noticed that the Windows VPN client shows an interesting behaviour.
This is the configuration required for the Meraki VPN on a Windows VPN Client
However you also need to ensure some other settings are configured under the "Change Adaptor Settings" by default the PAP setting is not enabled .
Once you enable this it changes the general settings in the vpn to "General Authentication"
Change this Back to Username and password and the PAP setting is disabled again!
Whats more when this happens, the errors appear. So it looks like there could be an issue with the Client configuration. We need affected users and we need them fast.
So now we need to ensure that this isnt happening on every device. Schedule a few calls with affected users only to find out these settings are untouched and correct. Ruleing out the client side being the issue.
The New Development
At this stage we get a new and interesting development
One user that was failing to work, is now working. No change at all anywhere
The problem is no longer just associated with single users or a group of users.
The problem is truly sporadic.
Thankfully we now have ruled out enough that there can only be a couple remaining issues.
Its can't be
RADIUS config
MX VPN mis-configuration
Both of these would result in a greater impact radius
Client side configuration
This has been proven to be correct even with odd behaviour
The SOLVE
I perfom a packet capture from the Assurance dashboard in meraki, caputuring traffic coming in bound from an affected user. At the same time we capture logs from the RADIUS server. As we know that every user is always prompted for credentials.
This is where we see the issue. The MX Sees the failing request come in, but the RADIUS server sees nothing. The problem exists between the MX and the server.
Finally, we spot that the MX actually has an available upgrade (despite showing Up to Date on the device Dashboard). Equally interesting there is a known bug on the version surrounding Wireless Concentrators dropping RADIUS requests.
Due to an MX 18.211 regression, MX75, MX85, MX95, MX105, MX250, and MX450 appliances when configured as Wireless Concentrator for SSID Tunneling and Layer 3 Roaming, will fail to forward return Radius Authentication traffic causing clients to fail to connect to the SSIDs.
While not identical, the cases are oddly similar, Could it be that this bug may have other undocumented affects and impact users connecting to an IPSec VPN? Who knows...
After a short visit to the "Organisation > Firmware upgrade" page and a patch later the issue was resolved.
How would you go about troubleshooting an intermittent issue like this?
Would you be satified if the issue came down to a bug?