I’ve been arguing with an Exchange 2016 server lately, due to what I suspect is a dodgy IBM-badged MR10i RAID controller in a x3650 M3. It has been kicking disks that seem entirely fine out of RAID1 volumes, which effectively has the same side effect as losing a disk. I intend to publish a few posts with some of the links and practices I’ve used lately.

Original article: How to install LSI MegaRAID Storage Manager (MSM) on ESXi 5.5

The original, excellent instructions from Mike Smith at Serenity-Networks, despite being for ESXi 5.5, seemed to work with some minor adaptations for ESXi 6.5.0 Update 1 (Build 5969303), with the latest versions of software from Avago (Broadcom).

Enable SSH on ESXi host: From the web UI, in the Navigator column, select Host, then choose Actions > Enable Secure Shell (SSH):

Adjust the “acceptance level” to allow installation of unsigned VIB files: In the Navigator column, select Manage, then select the Security & users tab. Then, click the Edit settings button and choose Community.

Get the LSI downloads (SMIS provider and the latest MegaRAID MSM): I found filtering by OEM did not successfully show results. On the Broadcom website, I selected the following categories for download:

• Group: Storage Controllers, Adapters and ICs
• Family: Storage Controllers, Adapters and ICs
• OEM: (left blank – showed up as ‘OEM’ in the search interface)
• Product: All
• Asset Type: Management Software and Tools

There were 679 results; I used Ctrl+F and searched for “SMIS”, which offered a link titled “Latest SMIS Providers” for VMWare 6.0 and 6.5: https://docs.broadcom.com/docs/VMware_MR_SAS_Providers-00.67.V0.04.zip

Then I also used Ctrl+F and searched for “MegaRAID Storage Manager”, which offered MSM for a variety of platforms:

• https://docs.broadcom.com/docs/17.05.00.02_Windows_MSM.zip
  (Windows x86 and x64)
• https://docs.broadcom.com/docs/17.05.00.02_Linux-64_MSM.gz (Linux x64)

Copy the LSI SMIS provider (the file with .vib extension) to the /tmp directory on ESXi host (scp/WinSCP/your client of choice). I found that my sneaky attempt at copying it to a shared volume at /vmfs/volumes/… was hit and miss; when it was a fibre channel mount, the install worked properly, but if the datastore was on a local disk, it died with an error message.

SSH to the ESXi host with appropriate credentials (I did everything as root) and run the following install command:

esxcli software vib install -v /tmp/vmware-esx-provider-lsiprovider.vib --no-sig-check

I also had to disable the firewall on the ESXi host. Bad practice, but I don’t have a list of the specific ports to open at present.

esxcli network firewall set --enabled false

Reboot the ESXi host when complete. You can and probably should do the usual behaviour of taking it into maintenance mode, but in my case everything shut down and came up cleanly as VMWare Tools was installed on each guest.

It was hit and miss as to whether I had to add the line from /etc/hosts on the ESXi server with the hostname to my Windows box. I found that eventually creating both A and PTR records in Active Directory DNS, combined with turning off the ESXi firewall, were sufficient to get the MSM client on a domain-joined Windows server to connect – not even necessarily a guest VM on the same hypervisor.

I also had to change the MSM client settings in the Configure Host dialog to “Display all of the systems in the network of local server”, and not the “ESXi-CIMOM” option:

I had a Windows Server Update Services installation that after a reboot, failed to start the WSUS service with a fairly generic error message. Clients issued an “unable to check for updates” message with an 8-character hex error code, differing depending on the client OS.

To fix it, I followed the directions in this source article on vcloudnine.de: WSUS on Windows 2012 (R2) and KB3159706 – WSUS console fails to connect

• Run elevated Command Prompt and issue the following command:"C:\Program Files\Update Services\Tools\wsusutil.exe" postinstall /servicing
• Ensure the “HTTP Activation” feature is installed, using Server Manager > Add Roles and Features > Features > .NET Framework 4.5 Features. In my case, it was already installed.
• Restart “WSUS Service” from services.msc

Side note: several years ago Kayla caught me talking in my sleep, muttering something about “you’ve got to check the boxes!” This is the actual dialog and process in question.

Full credit to Jeremy Jameson at MSDN. Posting in case the original disappears.

• Run Server Cleanup Wizard with only the “Unused updates and update revisions” (option #1) box checked. This took about six hours on the server experiencing the problem:
• Once finished, run the wizard again with only the “Unneeded update files” (option #3) box checked.
• Once that’s finished, run the wizard with all the boxes checked.

Well, better late than never, but I’m currently in the process of cleaning up paperwork in the home office, and noted that RiteBite had given me a flyer asking for a Google review several months ago. So here’s a conclusion to the review series, which will be combined with the other content and sliced into bits and pieces for the less-verbose social media pages.

Completing the Program

Since last time I wrote, I went through about half of another series of trays with 7-day rotations. I specifically requested to have my treatment wrapped up about a week before my wedding in August 2016, and Dr. Luis and staff were very accommodating since this third set was effectively “finishing touches”. As part of the removal, I had permanent wiring bonded behind both my top and bottom teeth and was given a set of top and bottom harder, clear plastic retainers to wear overnight. One important point is that for the first two weeks, you’re expected to wear the retainers as close to 24/7 as possible, so you’re not “entirely” done. I obviously made an exception to this for the wedding.

A Few Nitpicks

The retainers are not ideal, to put a point on it. Their larger size (compared to the Invisalign trays) and increased rigidity triggers my gag reflex nearly every morning when taking them out, and I still run into similar problems with drooling on my pillow.

I also specifically requested the top permanent wire, and had to ask several times before getting a “yes” – several staff suggested that it wasn’t strictly necessary or had a higher chance of breaking. I wanted to ensure that with my financial investment, there was a “backup” in place to help the teeth from moving as much. The top wire’s presence is still noticeable when I close my mouth several months later, unlike the bottom wire. Both still have a distinct “pebbled” texture where the wire is adhered to the back of each row of teeth.

Despite asking for Google reviews as part of the “exit interview”, RiteBite seems to have several accounts under their name on Google Plus (1, 2, 3, 4, 5) and no link to the official Google profile from their website, nor any content on these pages. I was also disappointed to find that the Case Graphics section has disappeared from my profile since completing treatment.

Overall Results

The change has been quite impressive. It took slightly over a year and a half, I wasn’t seriously inconvenienced, and now that it has been paid off, I begrudgingly admit that it was probably a better personal choice than replacing the laminate flooring in the house or buying the same amount of networking gear.

Following up from my previous round of router testing, I managed to get a spare Lanner FW-7540 with an Intel Atom D525 CPU to test how my current pfSense 2.3.2 setup compared to an EdgeRouter Lite. The results were well below what I was expecting: the pfSense box topped out at 490Mbit in the 1MB test and was very spiky when looking at the netdata graphs.

The results file is also available if you’d like to look directly at the ab output.

Filesize	Average Mbit/s	Total Failed Requests	Notes
10K	145.07	87	10K concurrency test only resulted in 49Mbit. No failed requests in 10, 100 and 1000 concurrency tests.
100K	421.71	4896	No failed requests in 10, 100 and 1000 concurrency tests.
1MB	489.96	3341	No failed requests in 10, 100 and 1000 concurrency tests.

This test fairly obviously shows a ceiling. For WAN connections of over 500Mbit, it looks like something beefier than an Atom D525 is necessary to run the NAT as anticipated.

I also ran some more informal WAN to LAN iPerf3 testing on direct connection (MDI-X), the EdgeRouter Lite and the pfSense/7540 combination to get some synthetic numbers:

Connection	iPerf Result
Direct	941Mbit with no retries
EdgeRouter Lite	939Mbit with retries
pfSense/7540	829Mbit with no retries

Given how well the EdgeRouter Lite seems to perform for its price, and since it beats out the more general purpose hardware, I suspect I will be swapping out for an ERL or ER-Pro very shortly.

A friend and colleague of mine (Matt) and I have an ongoing discussion about over-specced gear for our home networks. Our core routers have been FW-7540s running pfSense (Atom D525, 4GB RAM, 4 Intel NICs) since 2013. pfSense offers a huge advantage over commercial-grade routers – I run dual WAN with failover based on ping, link, and packet loss, have extremely customizable DNS and DHCP, and can set up an OpenVPN server in just a few minutes. Matt and I also recently have had 500Mbit+ downstream connections installed, so it’d be good to know what hardware and software combination is “for sure” capable of utilizing the full pipe.

There have been a series of excellent articles at Ars Technica this year by Jim Salter that constantly get mentioned in our discussions:

• Numbers don’t lie—it’s time to build your own router
• The Ars guide to building a Linux router from scratch
• The Router rumble: Ars DIY build faces better tests, tougher competition

Continue reading

This post is a collection of my recent Windows/Exchange administrative work.

Run AD Directory Sync Manually (New Version of Start-OnlineCoexistenceSync)

Source: https://blogs.technet.microsoft.com/rmilne/2014/10/01/how-to-run-manual-dirsync-azure-active-directory-sync-updates/

Instructions:

Import-Module ADSync
Start-ADSyncSyncCycle -PolicyType Delta

or

Start-ADSyncSyncCycle -PolicyType Initial

How do I check total mailbox sizes for Office 365/Exchange Online mailboxes?

Source: https://community.spiceworks.com/how_to/93142-check-mailbox-size-and-usage-with-office-365-or-exchange-online-find-users-nearing-their-quota

Instructions:

# In PowerShell:
$LiveCred = Get-Credential
$Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://ps.outlook.com/powershell/ -Credential $LiveCred -Authentication Basic -AllowRedirection 
Import-PSSession $Session

get-mailbox | get-mailboxstatistics | ft displayname, totalitemsize 

# When done:
Remove-PSSession $Session

Error during migration: MigrationPermanentException: Cannot find a recipient that has mailbox GUID ” error message when you try to move a mailbox in an Exchange hybrid deployment

Source: https://support.microsoft.com/en-ca/kb/2956029

• Ensure the local user object doesn’t have an exchange GUID. From the local Exchange Management Shell:

  Get-RemoteMailbox <MailboxName> | Format-List ExchangeGUID

• Get the GUID from the error message, or retrieve it from the O365/Exchange Online shell (connect as above):

  Get-Mailbox <MailboxName> | Format-List ExchangeGUID

• Set the exchange GUID for the user from the local Exchange Management Shell:

  Set-RemoteMailbox <MailboxName> -ExchangeGUID <ExchangeGUID>

• Force directory sync. Using the latest Azure AD Connect commands, on the server with the directory sync tool installed:

  Import-Module ADSync
  Start-ADSyncSyncCycle -PolicyType Delta

• Monitor with “Azure AD Connect Synchronization Service Manager” GUI application if needed.

 

Error during migration:  MigrationPermanentException: Mailbox size 12.56 GB ‎(13,489,367,463 bytes)‎ exceeds target quota 2.3 GB ‎(2,469,396,480 bytes)‎.

Source: http://andywolf.com/migrating-exchange-mailbox-from-another-forestmailbox-exceeds-target-quota/

• If applicable to a single user, use ADSI Edit to set the “mDBUseDefaults” property to False on the applicable user object, then try again.
• If database or organization-wide, use the Exchange Administrative Center to remove quotas for the database.

I have a migration batch that partially failed. Now I can’t get those mailboxes to migrate.

Sources:

• https://technet.microsoft.com/en-us/library/jj218717(v=exchg.160).aspx
• http://www.expta.com/2015/08/exchange-2013-migration-batch-completes.html

Scenario: A migration batch was partially successful (one or more mailboxes in the batch migrated properly). The errors for the remaining mailboxes have been corrected. I’d like to start a new migration batch containing the failed mailboxes, but the batch bombs out with an email to the Exchange Online administrator. The batch online looks like it’s still migrating, but the CSV with the results that was emailed contains the following error messages for each account:

The user "user@example.com" is already included in migration batch "My Migration Batch Name."  Please remove the user from any other batch and try again.

In this case you need to remove user from migration batch using the Remove-MigrationUser cmdlet when connected to the Exchange Online PowerShell session:

• Get the details of all users in migration batches, or get the details for the specific user being migrated:

  Get-MigrationUser
  Get-MigrationUser user@example.com

• Remove the user from the migration batch. Use the additional -Force parameter if you aren’t running interactively.

  Remove-MigrationUser user@example.com

• Clean up any migration batches that may still be in progress with the ‘already included’ error.
• Create a new migration batch containing the affected mailboxes.

I have an Ubuntu 16.04 desktop installation with Unity and wanted to try KDE, so I ran sudo apt-get install kubuntu-desktop. apt failed with the following message:

trying to overwrite '/usr/share/accounts/services/google-im.service', which is also in package account-plugin-google [...]

The original issue at Ask Ubuntu has several suggestions but none of them worked – any apt commands returned the same requirement to run apt-get -f install, which in turn gave the original “trying to overwrite” error message. synaptic also wasn’t installed so I couldn’t use it (or install it, as all other apt installation commands failed.)

I was able to get the dpkg database out of its bad state and continue to install kubuntu-desktop by running the following:

dpkg -P account-plugin-google unity-scope-gdrive
apt-get -f install

(Link to original Kubuntu bug for posterity: https://bugs.launchpad.net/kubuntu-ppa/+bug/1451728)

This post was cross-posted to The Linux Experiment, where I haven’t written anything for months.