NetScaler Login exceeds maximum allowed users after 10.1 upgrade

Shortly after our recent NetScaler upgrade from 9.3 -> 10.1, users reported getting the error “Login exceeds maximum allowed users” in their browsers when attempting to log in to the Access Gateway (NetScaler Gateway).  A remote session with a Citrix technician revealed that we had indeed hit our license limit as seen under NetScaler Gateway / Active User Sessions. We did see that some users were logged in two or more times, and it’s possible that the way licenses are consumed under 10.1 is different from 9.3, which might be why we never hit the licensing limit before.  The options presented by the Citrix tech were:

  1. Ask users to deliberately log out of the Access Gateway when they are done (vs. just allowing their sessions to time out) in order to free up their license.  This would, of course, require user education.
  2. Switch our Access Gateway Virtual Server from SmartAccess Mode (includes VPN access) to Basic Mode (ICA proxy-only).  Without taking additional steps such as allowing VPN for just a subset of our users, this option would remove VPN ability for all users from the gateway but allow unlimited connections through the gateway to our apps.
  3. Lower the timeout value for our Access Gateway, forcing users to re-authenticate to the gateway during the workday.

If memory serves, the technician also mentioned that the 10.5 version of NetScaler would allow a user who logged into the Access Gateway more than once to “assume” the license from his/her previous session.  An immediate upgrade to 10.5 was not an option in our case.

After a quick review of our environment, the technician suggested we switch to Basic Mode on our Virtual Server under NetScaler Gateway / Virtual Servers as no VPN was required in our environment.

Posted in Uncategorized | Leave a comment

NetScaler Integrated Caching behavior after 9.3 -> 10.1 upgrade

After a recent NetScaler upgrade from 9.3 to 10.1, we noticed a change in the behavior of the Integrated Caching feature.  Integrated Caching had been enabled for the previous two years, but with the Memory Usage Limit set to zero, caching had been effectively disabled.  After the upgrade, our PeopleSoft application began displaying incorrect content after users logged in.

We were able to tell that Integrated Caching was delivering cached content by visiting Optimization / Integrated Caching / Content Groups and seeing both “non-304 Hits” and “304 Hits” for the DEFAULT Content Group, along with a non-zero value under Memory Usage.

Integrated-Caching-10-1

Since we run in HA mode, we could consult our not-yet-upgraded, 9.3 NetScaler node.  Visiting Integrated Caching / Content Groups / DEFAULT revealed the expected values of zero for Memory Usage, Non-304 Hits, and 304 Hits.

Integrated-Caching-9-3

 

Our solution was to disable Integrated Caching in System / Settings / Configure Basic Features as it wasn’t needed.  As soon as we did this, the undesired content stopped displaying within our PeopleSoft application.

Integrated-Caching-Disable-10-1

Posted in Uncategorized | Leave a comment

Rotate images in ADFS 3.0

ADFS 3.0 is otherwise known as ADFS 2012 R2 since it is available only on Server 2012 R2. As I gain some experience with it, one of the nice configuration options is the ability to use PowerShell to customize the sign-in page.

Among the customizations we’ve made is one to help keep our sign-in page from looking stale over time. I wrote this simple PowerShell script to rotate the large “illustration” image occasionally. It runs as a Scheduled Task, and pulls approved images randomly from a file system folder. The script also logs which image was in place at any given time in case that happens to be interesting to someone at some point.

cd X:\path\images
$RandomImage = Get-ChildItem | Get-Random | %{((Get-Item $_).VersionInfo).FileName}
(Get-Date -format G) + " $RandomImage" | Out-File X:\path\Logs\IllustrationRandomizer.log -Append
Set-AdfsWebTheme -TargetName Custom_Theme -Illustration @{path=$RandomImage}

Posted in Uncategorized | Tagged | Leave a comment

Redirect URL for SSL_BRIDGE Virtual Server on NetScaler

When you create an SSL_BRIDGE Virtual Server (VIP) in NetScaler, there is no way to specify a Redirect URL (the field is grayed out).  So if your back-end servers are down, there’s no way to specify an outage page.  If you try to create a Responder policy as a workaround, you will be unable to bind it to the SSL_BRIDGE Virtual Server if it that policy contains anything other than a DROP or RESET action (http://discussions.citrix.com/topic/336769-ssl-bridge-virtual-server-response-when-down/).

You can, however, use a Listen Policy to deliver your outage page.  Create a new SSL Virtual Server alongside the existing SSL_BRIDGE Virtual Server using the same IP address and port.  (You can’t normally do this, but you can when you specify a Listen Policy on the Advanced tab.)

Example steps for setting up the new SSL Virtual Server in version 9.3 of the GUI (no changes are made to the existing SSL_BRIDGE Virtual Server):

  • Add your new SSL Virtual Server using the same IP and same port as the existing SSL_BRIDGE Virtual Server
  • Do not bind any Services to the new SSL Virtual Server (it will always be DOWN)
  • Set http://outage_page.your_domain.com (or whatever you please) as the Redirect URL on the new SSL Virtual Server
  • Bind a Listen Policy to the new SSL Virtual Server by setting a Listen Priority of 1 and a Listen Policy Rule of SYS.VSERVER(“ssl_bridge_virtual_server_name”).STATE.NE(up)
  • Add the same SSL certificate that is bound to the SSL_BRIDGE Virtual Server to the new SSL Virtual Server

Click for larger image:

Capture

References: http://support.citrix.com/article/CTX139276 and http://support.citrix.com/proddocs/topic/ns-system-10-map/cb-br-aws-acc-encryp-mapi-smb-ssl-br-win-dom-tsk.html

Posted in Uncategorized | Leave a comment

Exchange 2010 Mailbox Move fails with MapiExceptionCallFailed

Moving a mailbox within a single Exchange 2010 SP2 RU8 server failed repeatedly with error text like the following.  Dismounting/remounting the database and running various New-MailboxRepairRequest commands did not fix the issue or provide guidance.

Error: MapiExceptionCallFailed: IExchangeFastTransferEx.TransferBuffer failed (hr=0x80004005, ec=1162)
Diagnostic context:
Lid: 55847 EMSMDBPOOL.EcPoolSessionDoRpc called [length=3004]
Lid: 43559 EMSMDBPOOL.EcPoolSessionDoRpc returned [ec=0x0][length=685][latency=15]
Lid: 23226 — ROP Parse Start —
Lid: 27962 ROP: ropFXDstCopyConfig [83]
Lid: 27962 ROP: ropTellVersion [134]
Lid: 27962 ROP: ropFXDstPutBufferEx [157]
Lid: 17082 ROP Error: 0x48A
Lid: 31329
Lid: 21921 StoreEc: 0x48A
Lid: 27962 ROP: ropExtendedError [250]
Lid: 1494 —- Remote Context Beg —-
Lid: 1238 Remote Context Overflow
Lid: 1947 StoreEc: 0x48A
[...edited...]
Lid: 1750 —- Remote Context End —-
Lid: 26849
Lid: 21817 ROP Failure: 0x48A
Lid: 22630

The mailbox’s failed Move Request Log in the EMC showed that the mailbox had 2819 folders total, which seemed high.  There was also a Litigation Hold on the mailbox, which could be part of the problem.  (In the past, a Litigation Hold in combination with an Exchange bug had resulted in a 1TB mailbox on our servers: a 2GB mailbox with 1TB of redundant calendar data.)

Opening the mailbox using OWA simply to check out those 2819 folder revealed the problem, a massive number of nested Junk E-mail folders totaling about 250MB.

Junk E-mail

After the folders were deleted using Outlook and OWA, the mailbox moved successfully.  If deleting the folders using Outlook/OWA was impossible due to the sheer number of folders, then Outlook Cached Mode would have been tried next, then perhaps a server-side tool from Microsoft, and then possibly an IMAP client.

Posted in Uncategorized | Leave a comment

PowerShell error with Get-ADUser user -Properties *

After upgrading some of our servers to Server 2012 R2, we’ve discovered a bug in the PowerShell 4.0 Get-ADUser cmdlet. When running the command Get-ADUser username -Properties *, the cmdlet returns the following error:

Get-ADUser : One or more properties are invalid.
Parameter name: msDS-AssignedAuthNPolicy
At line:1 char:1
+ Get-ADUser username -Properties *
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo          : InvalidArgument: (username:ADUser) [Get-ADUser], ArgumentException
+ FullyQualifiedErrorId : ActiveDirectoryCmdlet:System.ArgumentException,Microsoft.ActiveDirectory.Management.Comm
ands.GetADUser

Mike F Robbins researched the error in the following blog post, and determined that the issue occurs with PowerShell 4.0 run against Server 2008 R2 domain controllers. The issue is that two attributes, AuthenticationPolicy and AuthenticationPolicySilo, exist in a Server 2012 R2 Active Directory but do not exist in a Server 2008 R2 Active Directory. The Server 2012 R2 RSAT tools expect the attributes to exist in both environments, so an error is returned in the Server 2008 R2 Active Directory environment.

http://mikefrobbins.com/2013/11/07/windows-8-1-rsat-powershell-cmdlets-get-aduser-get-adcomputer-one-or-more-properties-are-invalid/

As Mike points out, a good workaround is to use PowerShell implicit remoting to process the commands on the domain controllers themselves. However, in the meantime I’m able to process the command from other downlevel OS machines, which is fine in my case for now.

I’ve submitted the case to Microsoft Premier who has confirmed the bug and escalated to the Platforms Team. I’ll post updates here as I get them.

Posted in Uncategorized | Tagged , | 3 Comments

HP Service Pack for ProLiant 2013.02.0 update broke NIC teams, VLANs using tagging

Networking was lost after updating HP’s Service Pack for ProLiant on two cluster nodes.  The servers were both BL460c G6 blades with two NICs that had first been teamed, then split back into two network connections using VLAN tagging.  (Multiple Networks had been assigned in Virtual Connect and then HP’s Network Configuration Utility (NCU) was used to team the NICs and configure the VLANs.)  Both servers were running Windows 2008 R2 SP1, with one being a file server cluster node and the other a member of an Exchange 2010 DAG.  The fact that the nodes were clustered is probably not important, but it was in the Failover Cluster Manager where the failures were most obvious, with our client-facing Cluster Network showing as “Failed,” while the connection used for backups remained functional.

On the first server, disabling/enabling the NICs in Windows’ Network Connections provided temporary relief, where the client-facing (Public) NIC would work for several seconds before failing again as seen in Failover Cluster Manager.  Disabling *both* NICs, then re-enabling the Public NIC first, then the Backup NIC second allowed both NICs to remain up.  This was not a permanent solution, obviously, so all networking as seen from Windows was torn down and recreated in Windows’ Network Connections and HP’s NCU, and the configuration was tested with a reboot just to be sure.

A call to HP revealed that this behavior was not unexpected.  Their advice when doing these updates is:

  • Break the team (and reconfigure one or more now-un-teamed NICs with production IP addresses if network connectivity during upgrade is required).
  • Perform the Service Pack for ProLiant upgrade.
  • Recreate the team.

The HP technician also noted that the previously-installed version of NCU was two years old and that there had been one or more significant updates in the interim.  When asked if frequent updates of the HP software would allow skipping breaking/recreating the team when doing these updates, he said HP’s advice would still be to do the above steps.

The second server proved more nettlesome, with HP’s Service Pack for ProLiant causing the production NIC to go offline soon after the start of the installation, losing the Remote Desktop session with the server.  The RDP session could not be re-established, and the server had to be accessed via iLO.  In iLO, the console session was not responsive to mouse or keyboard input, and the server had to be ungracefully reset (resetting iLO through its administrative web page did not help).  After logging in with cached credentials, the Service Pack for ProLiant installation was repeated “successfully” and the server rebooted.

After reboot, both NICs in Windows were showing as “unplugged” and HP’s NCU did not show any NICs at all, and both of these issues were after the NCU displaying an error that “The version of the miniport driver(s) for the following adapters are not compatible with the HP Network Configuration Utility software installed.”  Running the Service Pack for ProLiant a third time did not remedy the situation, and the CPxxxxxx.EXE file on the Service Pack for ProLiant DVD had to be tracked down and installed individually.  (The Service Pack for ProLiant was identifying the NIC driver as already upgraded, so it was not offering to upgrade it by default.  It may have been possible to force the upgrade from within the Service Pack for ProLiant, without tracking down the individual driver installation file on the DVD.)

After the reboot following the NIC driver upgrade, the NICs did not function normally on this server.  A tear-down and recreate of the team did not help as it had on the previous server either.  With eight NICs (FlexNICs) available to this blade, the solution was to create two separate teams (one for client traffic and the other for backup traffic).

With other blades in the chassis not having the luxury of eight NICs, the solution for those servers may be to remove teaming altogether.  Hopefully, this won’t be required and simply following HP’s advice to dissolve the team prior to the Service Pack for ProLiant upgrade and recreate it after will avoid the problem entirely.

Posted in Uncategorized | Leave a comment

SQL Server PowerShell Module (SQLPS)

SQL Server provides a Windows PowerShell module called sqlps that is used to import the SQL Server components into Windows PowerShell. The sqlps module loads two Windows PowerShell modules:

  • A SQL Server provider, which enables a simple navigation mechanism similar to file system paths. You can build paths similar to file system paths, where the drive is associated with a SQL Server management object model, and the nodes are based on the object model classes. You can then use familiar commands such as cd and dir to navigate the paths similar to the way you navigate folders in a command prompt window. You can use other commands, such as ren or del, to perform actions on the nodes in the path.
  • A set of cmdlets, which are commands used in Windows PowerShell scripts to specify a SQL Server action. The SQL Server cmdlets support actions such as running a sqlcmd script containing Transact-SQL or XQuery statements.

More details here: http://msdn.microsoft.com/en-us/library/hh245198.aspx

Although the SQLPS module is installed along with SQL Server, you do not have to install SQL Server to obtain the module. You simply need to install three stand-alone packages from the Microsoft® SQL Server® 2012 Feature Pack, available here:
http://www.microsoft.com/en-us/download/details.aspx?id=29065

Install the following packages in this order:

  1. Microsoft® System CLR Types for Microsoft® SQL Server® 2012 (SQLSysClrTypes.msi)
  2. Microsoft® SQL Server® 2012 Shared Management Objects (SharedManagementObjects.msi)
  3. Microsoft® Windows PowerShell Extensions for Microsoft® SQL Server® 2012 (PowerShellTools.msi)

Be sure to select the appropriate package platform for each, either x86 or x64.

To load the sqlps module in Windows PowerShell:

Import-Module sqlps

(You can include the -DisableNameChecking parameter if you’re concerned about suppressing the Encode-Sqlname and Decode-Sqlname warning.)

Thanks to Max Trinidad for his article explaining this information. http://www.maxtblog.com/2012/09/create-powershell-smo-scripts-without-installing-sql-server/

Posted in Uncategorized | Tagged | 1 Comment

ActiveDirectory module and UAC

I have noticed a few odd behaviors with PowerShell’s ActiveDirectory module, one regarding the msDS-UserPasswordExpiryTimeComputed attribute of the Get-ADUser cmdlet, and another regarding the New-ADServiceAccount cmdlet.

On a brand new Server 2008 R2 domain, the following command returns values for only a small percentage of the accounts in the domain but the vast majority of accounts return with no value.

Get-ADUser -filter * -Properties “msDS-UserPasswordExpiryTimeComputed” | ft name, “msDS-UserPasswordExpiryTimeComputed” –AutoSize

Also on this domain, issuing the following command often results in an “Access is denied” error, but not always. In fact, typically if I receive the error and then leave my RDP session to the computer open for 30 to 60 minutes and try command again (hit the Up Arrow key on the keyboard), the command runs without error.

New-ADServiceAccount svc_test

This seems to indicate a general problem with the ActiveDirectory PowerShell module.

The following article pointed me in the right direction for a workaround.
http://social.technet.microsoft.com/Forums/en-US/windowsserver2008r2general/thread/5947c0eb-9549-4161-afa1-edb451fc742f

The problem appears to be a bug in the way the ActiveDirectory module behaves when User Account Control (UAC) is enabled. The workaround is to disable UAC, which requires a reboot. When UAC is left enabled, even when PowerShell is run with elevated permissions, the problem still occurs.

I’ve confirmed the experiences above both on domain controllers and on member servers, and in two different Active Directory forests.

Posted in Uncategorized | Tagged , | 1 Comment

2003 R2 Windows Update fails with Error Number 0x80072EE2

After installing Windows Server 2003 R2 in a VM for some quick testing, I installed SP2, rebooted, then went to Windows Update to download an expected multitude of Windows patches.  However, Windows Update failed repeatedly with “Error Number 0x80072EE2″ displayed in the browser window.  Updating Windows Update to “Microsoft Update” and IE6 to IE 8 did not help, nor did deleting the SoftwareDistribution folder.

Oddly, turning off the Windows Firewall fixed the problem.  Below is a link to an article that indicated turning the firewall *on* solved the same problem.  With the server fully patched and after re-enabling the firewall, I am still seeing the same behavior (117 patches were installed post-SP2).  I don’t yet know if Automatic Updates (non-browser) will experience the same problem with the firewall enabled.

Reference: http://forums.techarena.in/windows-update/707782.htm

Posted in Uncategorized | Leave a comment