Sunday, December 14, 2014

vMotion Fails at 14% - Operation Timed Out

While working on a 3 host VMware cluster I was trying to vMotion some servers around and begin maintenance and updates.  I hit this error for the first time "Operation Timed Out" and the dreaded red X.

As it turns out there was a vmx-****.vswp file inside the VM folder that apparently was left over from a failed DRS migration.  After doing some research I found out that during a DRS migration the VMX file is started on both nodes.  Once the migration is completed normally one of these gets removed.  In this case it did not.

If you browse the datastore of the VM that will not move and you open up the folder of the VM you should see two vmx-****.vswp files.  If you see only one then you've got another issue and this isn't the fix.  If you do see two they will be .vswp-1 and .vswp-2.  There could be a .vswp for the VM itself.  DON'T DELETE THAT ONE!  I can't stress that enough.

To know which one to get rid of just look at the time stamps.  If the VM is running it may be hard to tell which is which so make sure you turn the VM off before you begin this process.  Once it's off then whichever file remains in the folder is the one you need to delete.

You can try deleting the file using the datastore browser but if that doesn't work then here's how you can remove it.

1. Start the SSH daemon on the host which has the VM registered and then login via SSH.  I recommend Putty.  You can download it HERE along with many other great tools for network and systems maintenance.
2. # cd /vmfs/volumes/{datastore name}
3. # cd {VM folder name}
4. # ls -lash *.vswp (this will show all the files and timestamps just to verify)
5. # rm vmx-{VM name}-[1-2].vswp

That's all there is to it.  Remember to disable the SSH daemon on your host and now you should be able to vMotion the VM again with it running.

Saturday, December 13, 2014

How Long Is Left on Database (IN RECOVERY)

While working on a SQL 2008 R2 database connection error for a medical company I found the database showing (IN RECOVERY).  Naturally no connections could be made during this process.  What I truly needed to know was how to see how long the process has left.  I found this great script HERE.  For my own purposes I decided to list it here as well.

DECLARE @DBName VARCHAR(64) = 'databasename'

DECLARE @ErrorLog AS TABLE([LogDate] CHAR(24), [ProcessInfo] VARCHAR(64), [TEXT] VARCHAR(MAX))

EXEC sys.xp_readerrorlog 0, 1, 'Recovery of database', @DBName

,SUBSTRING([TEXT], CHARINDEX(') is ', [TEXT]) + 4,CHARINDEX(' complete (', [TEXT]) - CHARINDEX(') is ', [TEXT]) - 4) AS PercentComplete
,CAST(SUBSTRING([TEXT], CHARINDEX('approximately', [TEXT]) + 13,CHARINDEX(' seconds remain', [TEXT]) - CHARINDEX('approximately', [TEXT]) - 13) AS FLOAT)/60.0 AS MinutesRemaining
,CAST(SUBSTRING([TEXT], CHARINDEX('approximately', [TEXT]) + 13,CHARINDEX(' seconds remain', [TEXT]) - CHARINDEX('approximately', [TEXT]) - 13) AS FLOAT)/60.0/60.0 AS HoursRemaining

FROM @ErrorLog ORDER BY [LogDate] DESC

Update for SQL 2012.  The syntax is a bit different:

DECLARE @DBName VARCHAR(64) = 'databasename'

DECLARE @ErrorLog AS TABLE([LogDate] CHAR(24), [ProcessInfo] VARCHAR(64), [TEXT] VARCHAR(MAX))

EXEC master..sp_readerrorlog 0, 1, 'Recovery of database', @DBName

,SUBSTRING([TEXT], CHARINDEX(') is ', [TEXT]) + 4,CHARINDEX(' complete (', [TEXT]) - CHARINDEX(') is ', [TEXT]) - 4) AS PercentComplete
,CAST(SUBSTRING([TEXT], CHARINDEX('approximately', [TEXT]) + 13,CHARINDEX(' seconds remain', [TEXT]) - CHARINDEX('approximately', [TEXT]) - 13) AS FLOAT)/60.0 AS MinutesRemaining
,CAST(SUBSTRING([TEXT], CHARINDEX('approximately', [TEXT]) + 13,CHARINDEX(' seconds remain', [TEXT]) - CHARINDEX('approximately', [TEXT]) - 13) AS FLOAT)/60.0/60.0 AS HoursRemaining

FROM @ErrorLog ORDER BY [LogDate] DESC

Just copy and paste this script in, change the database name, and you're good to go.

EDIT:  One thing to note is that it may show over 85% and yet if you check the database the process has completed.  I suspect this is because once the recovery is done this information is no longer updated.

Monday, September 08, 2014

Adjust Exchange 2013 Send-Receive Message Size Limits

After setting up and Exchange 2013 SP1 server on Server 2012 R2 I ran into this small issue with a customer.  Even though I had all of the send/receive connectors in the ECP set at 100M, I was still hitting the default 10M attachment size limits.

Here's how to fix that by changing the Global Transport Settings: (short answer)

Check to see what the size limits currently are -

Get-TransportConfig | ft MaxSendSize, MaxReceiveSize

Once you know those sizes then here's the command to change them to what you need -

Set-TransportConfig –MaxSendSize 100MB –MaxReceiveSize 100MB

I know these sizes are a little large but just using them as an example.  Go ahead and bounce the Transport service just to be sure the settings take effect and you'll be good to go.

All of the other size limits can be set using the ECP under Mailflow and then Send Connectors and Receive Connectors.

Update:  I've just completed reading almost all of Mastering Microsoft Exchange Server 2013 .  This is an excellent read that will give you tons of insight into installing, maintaining, and mastering all there is to Exchange 2013.  I highly recommend you give the book a read.

Wednesday, August 06, 2014

How to Disable the 21 Day Timer for SBS 2003, 2008, and 2011

Here’s the process to prevent SBS2003 from shutting down after 21 days of migration have passed. The basic principle is to disable the SBS Core Service (sbscrexe.exe). The problem is when you just turn off the service, it will restart itself automatically. To achieve this goal follow the steps below:
  1. Download process explorer from
  2. Run Process Explorer and SUSPEND the sbscrexe.exe file.
  3. Open regedit and navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SBCore
  4. Right click the key and click permissions. Give the Administrators group full control on both the key and child nodes.
  5. Refresh the key so you can see all of the registry settings for sbcore and change the Start DWORD value from 2 to 4 to disable the service.
  6. Using explorer navigate to the sbscrexe.exe file in the C:\WINDOWS\system32 directory and change permissions on the file to everyone deny.
  7. Go back to process explorer and kill the sbscrexe.exe process. If it doesn’t start again then you have successfully disabled the file.

Wednesday, July 30, 2014

Exchange 2010: The WS-Management Service Cannot Process The Request...

PROBLEM: The WS-Management service cannot process the request. The user load quota of 1000 requests per 2 seconds has been exceeded. Send future requests at a slower rate or raise the quota for this user. The next request from this user will not be approved for at least XX milliseconds.

This can happen if a certificate on Exchange expires (SSL or UCC) and the IIS service has not been restarted since.

All you need to do is open a command prompt (Run as Administrator if using UAC) and the type in IISRESET.  Press ENTER and wait.

Now you should be able to login to the Exchange 2010 EMC with no issues.

Really simple fix.

Good luck!

Friday, July 11, 2014

Exchange 2013 Installation Step-by-Step Cheat Sheet

Ok so I've setup a new Exchange 2013 server for a customer and got it installed with no errors.  These instructions pertain to installation on Windows Server 2012/2012R2.

I have condensed it down to make sense.  We all know TechNet instructions look like they were written by a lawyer.

I hope this helps with your install.  Good luck.

1. Install all OS updates
2. Install all the following prereqs
- .NET Framework (already on Server 2012R2).  Just use the server manager to install this as a Feature.

- Remote Tools Admin Pack ( Install-WindowsFeature RSAT-ADDS )

- Windows Components ( Install-WindowsFeature AS-HTTP-Activation, Desktop-Experience, NET-Framework-45-Features, RPC-over-HTTP-proxy, RSAT-Clustering, RSAT-Clustering-CmdInterface, RSAT-Clustering-Mgmt, RSAT-Clustering-PowerShell, Web-Mgmt-Console, WAS-Process-Model, Web-Asp-Net45, Web-Basic-Auth, Web-Client-Auth, Web-Digest-Auth, Web-Dir-Browsing, Web-Dyn-Compression, Web-Http-Errors, Web-Http-Logging, Web-Http-Redirect, Web-Http-Tracing, Web-ISAPI-Ext, Web-ISAPI-Filter, Web-Lgcy-Mgmt-Console, Web-Metabase, Web-Mgmt-Console, Web-Mgmt-Service, Web-Net-Ext45, Web-Request-Monitor, Web-Server, Web-Stat-Compression, Web-Static-Content, Web-Windows-Auth, Web-WMI, Windows-Identity-Foundation )

3. Extend the AD schema.  Use Command Prompt to drive letter with Exchange ( setup.exe /PrepareSchema /IAcceptExchangeServerLicenseTerms ).  When setup completes allow some time for AD replication to complete across all DCs.

4. Prepare AD for Exchange.  Use Command Prompt to drive letter with Exchange ( Setup.exe /PrepareAD /OrganizationName:"" /IAcceptExchangeServerLicenseTerms )

5.  Prepare all domains for Exchange 2013.  Use Command Prompt to drive letter with Exchange ( Setup.exe /PrepareAllDomains /IAcceptExchangeServerLicenseTerms )

6.  Install Office 2010 Filter Packs - Version 2.0 

7.  Install Office 2010 Filter Packs - Version 2.0 SP1 

8. Run setup.exe.  Follow prompts and allow wizard to complete install.

That's pretty much all there is to it to get it installed. 

If you want more information on detailed installation, configuration after install, and maintaining then I highly recommend you read Mastering Microsoft Exchange Server 2013.  I did and it has been a tremendous reference point for me now that Exchange 2013 deployments are becoming more common for me.

Enjoy the read and good luck!

Thursday, May 15, 2014

AD Health Checks for Domain Controllers

I recently had a request from a customer to provide some reports on how well their AD replication was working.  I found these come in helpful during routine maintenance or for specific requests on health checks.

Hope these help.

Dcdiag.exe /v >> c:\temp\pre_dcdiag.txt
This is a must!  It will always tell you if there is trouble with your DCs and/or services associated with it.

Netdiag.exe /v >> c:\temp\pre_Netdiag.txtThis will let you know if there are issues with the networking components on the DC.

Netsh dhcp show server >> c:\temp\pre_dhcp.txtYou may not want to do this but I've ran into issues with a DHCP server somehow not being authorized after a patch.  This allows me verify the server count and names.

Repadmin /showreps >> c:\temp\pre_rep_partners.txtThis shows all replication and if it was successful or not.  Be aware that Global Catalogs will have more info than a normal domain controller.

repadmin /replsum /errorsonly >> c:\temp\pre_repadmin_err.txtThis command takes a while to run but will let you know which server are having issues replicating.

Update 04/08/15:  Below is the text you can put into a script file that will dump everything you need into one text file for you to use for troubleshooting.  The commands above are great if that's all you need but now I use this script to grab it all at once.

@Echo Off
ECHO Running AD Health Checks - Notepad will open after completion
ECHO This Command Prompt will close after you close Notepad
set logfile=%userprofile%\Desktop\ADHealth.txt
echo You can share this log using > %logfile%
echo. >> %logfile%
echo. >> %logfile%
REM Finds system boot time
echo System Boot Time ------------------------------------------------------------- >> %logfile%
systeminfo | find "System Boot Time:" >> %logfile%
systeminfo | find "System Up Time:" >> %logfile%
echo. >> %logfile%
echo. >> %logfile%
REM Displays all current TCP/IP network configuration values
echo IPCONFIG ------------------------------------------------------------- >> %logfile%
ipconfig /all >> %logfile%
echo. >> %logfile%
echo. >> %logfile%
REM Analyse the state of domain controllers in a forest and reports any problems to assist in troubleshooting
echo DCDIAG ------------------------------------------------------------- >> %logfile%
dcdiag /a >> %logfile%
echo. >> %logfile%
echo. >> %logfile%
REM The replsummary operation quickly summarizes the replication state and relative health
echo Replsummary ------------------------------------------------------------- >> %logfile%
repadmin /replsummary >> %logfile%
echo. >> %logfile%
echo. >> %logfile%
REM Displays the replication partners for each directory partition on the specified domain controller
echo Showrepl ------------------------------------------------------------- >> %logfile%
repadmin /showrepl >> %logfile%
echo. >> %logfile%
echo. >> %logfile%
REM Query FSMO roles
echo NETDOM Query FSMO ------------------------------------------------------------- >> %logfile%
netdom query fsmo >> %logfile%
REM Query Global Catalogs
echo List Global Catalogs ------------------------------------------------------------- >> %logfile%
for /f "tokens=2" %%a in ('systeminfo ^| findstr Domain:') do set domain=%%a
nslookup -querytype=srv _gc._tcp.%domain% >> %logfile%
notepad %logfile%

Discover FSMO Roles Using PowerShell

Working with a rather confusing AD setup recently and trying to remove a dead domain controller I needed a quick way to identify which machines had the FSMO roles.
Just run the following commands:
Get-ADForest  | Format-Table SchemaMaster,DomainNamingMaster
Get-ADDomain | Format-Table PDCEmulator,RIDMaster,InfrastructureMaster
This gives a nice quick output as to where the roles reside and allows you to capture them as needed.

If you want to manage the roles with PS the command to move the roles is Move-ADDirectoryServerOperationMasterRole and it can be used in a variety of ways.
To transfer all 5 of the FSMO roles simply run the following command in PowerShell:
Move-ADDirectoryServerOperationMasterRole -Identity “Target_DC_name” –OperationMasterRole PDCEmulator,RIDMaster,InfrastructureMaster,SchemaMaster,DomainNamingMaster
To shorten the command line syntax you can use role numbers in place of the role names.  The following list details the role number for each of the five FSMO roles.
  • PDC Emulator – 0
  • RID Master – 1
  • Infrastructure Master – 2
  • Schema Master – 3
  • Domain Naming Master – 4
So if you wanted to transfer all 5 FSMO roles using numbers instead you would run the following command in PowerShell:
Move-ADDirectoryServerOperationMasterRole -Identity “Target_DC_Name” –OperationMasterRole 0,1,2,3,4
Now in my case since the DC was gone permanently I had to seize the roles using the –Force parameter.  This is the PowerShell command I ran to seize the roles:
Move-ADDirectoryServerOperationMasterRole -Identity “Target_DC_name” –OperationMasterRole PDCEmulator,RIDMaster,InfrastructureMaster,SchemaMaster,DomainNamingMaster -Force
Of course I could have used the short version:
Move-ADDirectoryServerOperationMasterRole -Identity “Target_DC_Name” –OperationMasterRole 0,1,2,3,4 -force
If you are just transferring or seizing a single role you will run the same command with just the name(s) or number(s) of the role(s) you want to move.  These commands can be run from any Windows Server 2008 R2 or newer as well as Windows 7 or newer with RSAT tools installed.

This is a little better than running all over the AD tools to get everything moved over.

Good luck.

Thursday, February 27, 2014

How to Show Thumbnail instead of Icon - Windows 8 or 8.1

This one is a quick fix.  Many people I've worked with so far hate how they don't see the thumbnails for their pictures any longer.  Here's the fast fix.

1. Open "This PC"
2. Click VIEW and which brings up the ribbon.
3. On the far right click OPTIONS and select "Change Folder and Search Options"
4. Click the VIEW tab.
5. Uncheck the "Always show icons, never thumbnails" box.

Once you Apply this change you'll see all of your pictures begin to change back to thumbnails like they have been in previous versions of Windows.

I'm not sure why they opted to make this change but this will fix it.

Good luck.

Wednesday, January 29, 2014

Resetting Officescan 10.X Console Password

I had a customer that was locked out of their console.  The previous IT employee had made no records that could be found of the password and shy of rebuilding the server install from scratch things were looking dim.

I found some information on the web about how to "hack" the root password for the console and get in.  To my surprise this one actually works.  No editing of the .ini files or any other things you'll see at the top of the Google list actually worked for me.  This does and here's how you do it:

1. MAKE A BACKUP OF C:\Program Files\Trend Micro\Officescan\PCCSRV\Private\AuthorStore\TrendAuthDef.xml - this is just in case this process fails.  You need a way to recover.

2. Open the C:\Program Files\Trend Micro\Officescan\PCCSRV\Private\AuthorStore\TrendAuthDef.xml file in edition.

3. Replace the Root password information at the bottom of the file as shown below:

**Note: Replace only the portion for the root. If there are other accounts showing do not delete or change them in any way.

4. Restart the OfficeScan Master Service.

5. Log in as root and use “trendmicro” as the password.

6. Change the root password to something more secure by going to Administration -> User Accounts.

This should take no longer than a few minutes. 

Good luck! 

Saturday, January 04, 2014

iTunes64.msi Missing During Install/Upgrade of iTunes

I've bumped up against this one a couple of times but being as I don't update iTunes often I had forgotten the fix.  To help you out and of course so I can reference this later here's the fast way to get around this that actually works.

1. Download iTunes64setup.exe from Apple.
2. Extract the exe using a program such as WinRAR.
3. Run the installer.  Once it stops and asks where iTunes64.msi is simply point it there and the install will continue and complete.

Now one more thing...

I have seen even this not work.  If that's the case simply run the new iTunes64.msi from inside the folder you extracted and it will bypass the need to look for itself and will install over and upgrade your existing iTunes install with no loss of information.

Hope this helps out.  It sure saves a lot of time messing with Microsoft Fix-it (doesn't work) or doing registry edits (doesn't work).