Archive for July, 2013

vCloud Director, RHEL 6.3, and Windows Server 2012 NFS

July 16th, 2013

One of the new features introduced in vCloud Director 5.1.2 is cell server support on the RHEL 6 Update 3 platform (you should also know that cell server support on RHEL 5 Update 7 was silently removed in the recent past – verify the version of RHEL in your environment using cat /etc/issue).  When migrating your cell server(s) to RHEL 6.3, particularly from 5.x, you may run into a few issues.

First is the lack of the libXdmcp package (required for vCD installation) which was once included by default in RHEL 5 versions.  You can verify this at the RHEL 6 CLI with the following command line:

yum search libXdmcp

or

yum list |grep libXdmcp

Not to worry, the package is easily installable by inserting/mounting the RHEL 6 DVD or .iso, copying the appropriate libXdmcp file to /tmp/ and running either of the following commands:

yum install /tmp/libXdmcp-1.0.3-1.el6.x86_64.rpm

or

rpm -i /tmp/libXdmcp-1.0.3-1.el6.x86_64.rpm

Next up is the use of Windows Server 2012 (or Windows 8) as NFS for vCloud Transfer Server Storage in conjunction with the newly supported RHEL 6.3.  Creating the path and directory for the Transfer Server Storage is performed during the initial deployment of vCloud Director using the command mkdir -p /opt/vmware/vcloud-director/data/transfer. When mounting the NFS export for Transfer Server Storage (either manually or via /etc/fstab f.q.d.n:/vcdtransfer/opt/vmware/vcloud-director/data/transfer nfs rw 0 0 ), the mount command fails with error message mount.nfs: mount system call failed. I ran across this in one particular environment and my search turned up Red Hat Bugzilla – Bug 796352.  In the bug documentation, the problem is identified as follows:

On Red Hat Enterprise Linux 6, mounting an NFS export from a Windows 2012 server failed due to the fact that the Windows server contains support for the minor version 1 (v4.1) of the NFS version 4 protocol only, along with support for versions 2 and 3. The lack of the minor version 0 (v4.0) support caused Red Hat Enterprise Linux 6 clients to fail instead of rolling back to version 3 as expected. This update fixes this bug and mounting an NFS export works as expected.

Further down in the article, Steve Dickson outlines the workarounds:

mount -o v3 # to use v3

or

Set the ‘Nfsvers=3′ variable in the “[ Server “Server_Name” ]”
section of the /etc/nfsmount.conf file
An Example will be:
[ Server “nfsserver.lab.local” ]
Nfsvers=3

The first option works well at the command line but doesn’t lend itself to /etc/fstab syntax so I opted for the second option which is to establish a host name and NFS version in the /etc/nfsmount.conf file.  With this method, the mount is attempted as called for in /etc/fstab and by reading /etc/nfsmount.conf, the mount operation succeeds as desired instead of failing at negotiation.

There is a third option which would be to avoid the use of /etc/fstab and /etc/nfsmount altogether and instead establish a mount -o v3 command in /etc/rc.local which is executed at the end of each RHEL boot process.  Although this may work, it feels a little sloppy in my opinion.

Lastly, one could install the kernel update (Red Hat reports as being fixed in kernel-2.6.32-280.el6). The kernel package update is located here.

Software Defined Single Sign On Database Creation

July 2nd, 2013

I don’t manage large scale production vSphere datacenters any longer but I still manage several smaller environments, particularly in the lab.  One of my pain points since the release of vSphere 5.1 has been the creation of SSO (Single Sign On) databases.  It’s not that creating an SSO database is incredibly difficult, but success does require a higher level of attention to detail.  There are a few reasons for this:

  1. VMware provides multiple MS SQL scripts to set up the back end database environment (rsaIMSLiteMSSQLSetupTablespaces.sql and rsaIMSLiteMSSQLSetupUsers.sql).  You have to know which scripts to run and in what order they need to be run in.
  2. The scripts VMware provides are hard coded in many places with things like database names, data file names, log file names, index file names, SQL login names, filegroup and tablespace information.

What VMware provides in the vCenter documentation is all well and good however it’s only good for installing a single SSO database per SQL Server instance.  The problem that presents itself is that when faced with having to stand up multiple SSO environments using a single SQL Server, one needs to know what to tweak in the scripts provided to guarantee instance uniqueness, and more importantly – what not to tweak.  For instance, we want to change file names and maybe SQL logins, but mistakenly changing tablespace or filegroup information will most certainly render the database useless for the SSO application.

So as I said, I’ve got several environments I manage, each needing a unique SSO database.  Toying with the VMware provided scripts was becoming time consuming and error prone and frankly has become somewhat of a stumbling block to deploying a vCenter Server – a task that had historically been pretty easy.

There are a few options to proactively deal with this:

  1. Separate or local SQL installation for each SSO deployment – not really what I’m after.  I’ve never been much of a fan of decentralized SQL deployments, particularly those that must share resources with vSphere infrastructure on the same VM.  Aside from that, SQL Server sprawl for this use case doesn’t make a lot of sense from a financial, management, or resource perspective.
  2. vCenter Appliance – I’m growing more fond of the appliance daily but I’m not quite there yet. I’d still like to see the MS SQL support and besides that I still need to maintain Windows based vCenter environments – it’s a constraint.
  3. Tweak the VMware provided scripts – Combine the two scripts into one and remove the static attributes of the script by introducing TSQL variables via SQLCMD Mode.

I opted for option 3 – modify the scripts to better suit my own needs while also making them somewhat portable for community use.  The major benefits in my modifications are that there’s just one script to run and more importantly anything that needs to be changed to provide uniqueness is declared as a few variables at the beginning of the script instead of hunting line by line through the body trying to figure out what can be changed and what cannot.  And really, once you’ve provided the correct path to your data, log, and index files (index files are typically stored in the same location as data files), the only variable needing changing going forward for a new SSO instance is the database instance prefix.  On a side note, I was fighting for a method to dynamically provide the file paths by leveraging some type of system variable to minimize the required.  While this is easy to do in SQL2012, there is no reliable method in SQL2008R2 and I wanted to keep the script consistent for both so I left it out.

Now I’m not a DBA myslef but I did test on both SQL2008R2 and SQL2012 and I got a little help along the way from a few great SMEs in the community:

  • Mike Matthews – a DBA in Technical Marketing at Dell Compellent
  • Jorge Segarra – better known as @sqlchicken on Twitter from Pragmatic Works (he’s got a blog here as well)

If you’d like to use it, feel free.  However, no warranties, use at your own risk, etc.  The body of the script is listed below and you can right-click and save the script from this location: SDSSODB.sql

Again, keep in mind the TSQL script is run in SQLCMD Mode which is enabled via the Query pulldown menu in the Microsoft SQL Server Management Studio.  The InstancePrefix variable, through concatenation, will generate the database name, logical and physical file names, SQL logins and their associated passwords.  Feel free to change any of this behavior to suit your preferences or the needs of your environment.

————————————————————————————-

— The goal of this script is to provide an easy, consistent, and repeatable

— process for deploying multiple vSphere SSO databases on a single SQL Server

— instance without having to make several modifications to the two VMware provided

— scripts each time a new SSO database is needed.

— The following script combines the VMware vSphere 5.1 provided

— rsaIMSLiteMSSQLSetupTablespaces.sql and rsaIMSLiteMSSQLSetupUsers.sql scripts

— into one script. In addition, it removes the static database and file names

— and replaces them with dynamically generated equivalants based on an

— InstancePrefix variable declared at the beginning of the script. Database,

— index, and log file folder locations are also defined with variables.

— This script meets the original goal in that it can deploy multiple iterations

— of the vSphere SSO database on a single SQL Server instance simply by modifying

— the InstancePrefix variable at the beginning of the script. The script then uses

— that prefix with concatenation to produce the database, .mdf, .ldf, .ndf, and

— two user logins and their required SQL permissions.

— The script must be run in SQLCMD mode (Query|SQLCMD Mode).

— No warranties provided. Use at your own risk.

— Jason Boche (@jasonboche, http://boche.net/blog/)

— with special thanks to:

— Mike Matthews (Dell Compellent)

— Jorge Segarra (Pragmatic Works, @sqlchicken, http://sqlchicken.com/)

— VMware, Inc.

————————————————————————————-

:setvar InstancePrefix “DEVSSODB”

:setvar PrimaryDataFilePath “D:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\DATA\”

:setvar IndexFilePath “D:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\DATA\”

:setvar LogFilePath “D:\Program Files\Microsoft SQL Server\MSSQL10_50.MSSQLSERVER\MSSQL\DATA\”

USE [master];

GO

————————————————————————————-

— Create database

— The database name can also be customized, but cannot contain

— reserved keywords like database or any characters other than letters, numbers,

— _, @ and #.

————————————————————————————-

CREATE DATABASE [$(InstancePrefix)_RSA] ON

PRIMARY(

NAME = N’$(InstancePrefix)_RSA_DATA’,

FILENAME = N’$(PrimaryDataFilePath)$(InstancePrefix)_RSA_DATA.mdf’,

SIZE = 10MB,

MAXSIZE = UNLIMITED,

FILEGROWTH = 10% ),

FILEGROUP RSA_INDEX(

NAME = N’$(InstancePrefix)_RSA_INDEX’,

FILENAME = N’$(IndexFilePath)$(InstancePrefix)_RSA_INDEX.ndf’,

SIZE = 10MB,

MAXSIZE = UNLIMITED,

FILEGROWTH = 10%)

LOG ON(

NAME = N’$(InstancePrefix)_translog’,

FILENAME = N’$(LogFilePath)$(InstancePrefix)_translog.ldf’,

SIZE = 10MB,

MAXSIZE = UNLIMITED,

FILEGROWTH = 10% );

GO

— Set recommended performance settings on the database

ALTER DATABASE [$(InstancePrefix)_RSA] SET AUTO_SHRINK ON;

GO

ALTER DATABASE [$(InstancePrefix)_RSA] SET RECOVERY SIMPLE;

GO

————————————————————————————-

— Create users

— Change the user’s passwords (CHANGE USER PASSWORD) below.

— The DBA account is used during installation and the USER account is used during

— operation. The user names below can be customised, but cannot contain

— reserved keywords like table or any characters other than letters, numbers, and _ .

— Please execute the scripts as a administrator with sufficient permissions.

————————————————————————————-

USE [master];

GO

CREATE LOGIN [$(InstancePrefix)_RSA_DBA] WITH PASSWORD = ‘$(InstancePrefix)_RSA_DBA’, DEFAULT_DATABASE = [$(InstancePrefix)_RSA];

GO

CREATE LOGIN [$(InstancePrefix)_RSA_USER] WITH PASSWORD = ‘$(InstancePrefix)_RSA_USER’, DEFAULT_DATABASE = [$(InstancePrefix)_RSA];

GO

USE [$(InstancePrefix)_RSA];

GO

ALTER AUTHORIZATION ON DATABASE::[$(InstancePrefix)_RSA] TO [$(InstancePrefix)_RSA_DBA];

GO

CREATE USER [$(InstancePrefix)_RSA_USER] FOR LOGIN [$(InstancePrefix)_RSA_USER];

GO

The .vmfsBalloon File

July 1st, 2013

One year ago, I wrote a piece about thin provisioning and the role that the UNMAP VAAI primitive plays in thin provisioned storage environments.  Here’s an excerpt from that article:

When the manual UNMAP process is run, it balloons up a temporary hidden file at the root of the datastore which the UNMAP is being run against.  You won’t see this balloon file with the vSphere Client’s Datastore Browser as it is hidden.  You can catch it quickly while UNMAP is running by issuing the ls -l -a command against the datastore directory.  The file will be named .vmfsBalloonalong with a generated suffix.  This file will quickly grow to the size of data being unmapped (this is actually noted when the UNMAP command is run and evident in the screenshot above).  Once the UNMAP is completed, the .vmfsBalloon file is removed.

Has your curiosity ever got you wondering about the technical purpose of the .vmfsBalloon file?  It boils down to data integrity and timing.  At the time the UNMAP command is run, the balloon file is immediately instantiated and grows to occupy (read: hog) all of the blocks that are about to be unmapped.  It does this so that during the unmap process, none of the blocks are allocated during the process of new file creation elsewhere.  If you think about it, it makes sense – we just told vSphere to give these blocks back to the array.  If during the interim one or more of these blocks were suddenly allocated for a new file or file growth purposes, then we purge the block, we have a data integrity issue.  More accurately, newly created data will be missing as its block or blocks were just flushed back to the storage pool on the array.