EdgeRouter 4: routing, VLANs and banging one’s head against the wall

I spent most of my Labour Day trying to accomplish two tasks with an EdgeRouter 4 and the other miscellaneous networking gear in the house: setting up a simple VLAN and getting my backup DSL connection working.

Two WANs and a LAN

With two WAN connections (one DHCP/cable, one PPPoE/DSL), I wanted to have specific local network ranges send traffic out to (and receive forwarded traffic from) a specific WAN connection. Note that this isn’t quite the load balancing feature (which I don’t want), but moreso “IP range A uses cable, IP range B uses DSL”. I went through the gauntlet of EdgeRouter support articles and forum posts without much success:

I haven’t yet solved the problem, but I believe the issue is related to the PPPoE connection not injecting default routes into the main table (hence the need for policy-based routing), plus my second SNAT rule didn’t seem to match traffic. The PPPoE connection has a very volatile dynamic IP address, so source NATing based on address translation rather than masquerade wouldn’t work.

In any event, I’m sure this will be another weekend problem, but it was compounded by…

Why can’t I ping hosts on the VLAN?

Using some details from the “Router on a Stick” configuration, I wanted to split out hosts that would be on the DSL network from the cable network. I added a new VLAN (16) to eth1, stood up a DHCP server in the appropriate IP block, and configured /etc/network/interfaces on my Ubuntu 16.04 box using approximately these instructions from Debian and microHOWTO. The system got a lease in the correct range, but hosts on VLAN 1 (192.168.1.0/24) were unable to ping or access the server in VLAN 16 (192.168.16.0/24).

I went through a large number of troubleshooting steps, including:

  • Can I ping from VLAN 16 to VLAN 1?
    • Yes, but the server still had an interface on VLAN 1, so this wasn’t really a valid test.
  • Can I ping the router IP address?
    • Yes, clients from VLAN 1 could ping 192.168.16.1, which is the EdgeRouter IP on VLAN 16.
  • What does tcpdump say?
    • The Linux box on VLAN 16 was getting ping packets, but not replying to them.
  • Are there firewall rules on the EdgeRouter that might be preventing VLAN-to-VLAN traffic? 
    • The default seems to be “accept”, but adding explicit accept policies including logging only showed the inbound traffic.
  • Is the switch not permitting VLAN traffic?
    • The Cisco SG500-52P purchased as surplus gear has the most awful web interface. I tried changing the port mode from “Trunk” to “General” and back again, specifically setting the port for the server as untagged/PVID 16 and then updating the config on the Linux box to avoid tagging the VLAN – no change. I also took the opportunity to upgrade the firmware.
  • Is the EdgeRouter somehow not permitting the reply ICMP traffic at a lower level that I can’t easily see?
    • At this point I busted out the old pfSense box and hooked it into an EdgeSwitch Lite, configured VLANs and firewall settings correctly there and tried to ping the server on VLAN 16 from another system. No change.

At this point I had changed out all components in the equation except for the server, so after dinner I poked around with a few more settings in the switch and then tried a different scenario:

  • Using a “known good” Netgear GS742 switch that wasn’t connected to the rest of the network, I configured port 3 with VLAN 16, untagged/PVID
  • A Windows desktop computer was connected to port 1 with VLAN 1 untagged
  • A macOS laptop was connected to port 3
  • The pfSense box was connected to port 24 and offered DHCP on VLANs 1, 16 and 32

 

 

When all components were connected, the desktop on VLAN 1 at 192.168.1.101 was able to ping the laptop on VLAN 16 at 192.168.16.101 successfully.

The next test was to move the laptop downstairs, plugit into the Cisco SG500-52P, and assign the port VLAN membership as 16, untagged, PVID. The laptop picked up a DHCP lease from the EdgeRouter, and a system on VLAN 1 elsewhere on the network was able to ping the laptop on VLAN 16!

Investigating the server

At this point, the trouble seemed to lie with the server itself. After some Googling, I ran across a Ubuntu Forums post that talked about VLAN routing issues – the last post suggested checking the rp_filter setting with the following command:

sysctl -a | grep \.rp_filter

The setting is described in sysctl.conf as:

# Uncomment the next two lines to enable Spoof protection (reverse-path filter)
# Turn on Source Address Verification in all interfaces to
# prevent some spoofing attacks

On my IBM x3650 server with a large number of interfaces, it turns out rp_filter was enabled in both the “all”, “default” and “eno2” categories:

net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.all.rp_filter = 1
net.ipv4.conf.br-0fce6441466a.arp_filter = 0
net.ipv4.conf.br-0fce6441466a.rp_filter = 1
net.ipv4.conf.br-f02b395ad2f3.arp_filter = 0
net.ipv4.conf.br-f02b395ad2f3.rp_filter = 1
net.ipv4.conf.default.arp_filter = 0
net.ipv4.conf.default.rp_filter = 1
net.ipv4.conf.docker0.arp_filter = 0
net.ipv4.conf.docker0.rp_filter = 1
net.ipv4.conf.eno1.arp_filter = 0
net.ipv4.conf.eno1.rp_filter = 1
net.ipv4.conf.eno2.arp_filter = 0
net.ipv4.conf.eno2.rp_filter = 1
...

I made the following adjustments to /etc/sysctl.conf, then ran sysctl -p:

net.ipv4.conf.default.rp_filter=0
net.ipv4.conf.all.rp_filter=0

Then manually made the adjustment for the eno2 interface:

sudo echo 0 > /proc/sys/net/ipv4/conf/eno2/rp_filter

After this command was run, I was able to successfully ping the server’s IP address in VLAN 16 from a desktop in VLAN 1.

Follow up tasks

So that I don’t forget, here are some follow up tasks that I’d like to finish for this project (in addition to sorting out the PPPoE routing):

  • Do some reading and better understand the rp_filter mechanism. Try firing up a VM or system with only one interface (instead of one on VLAN 1 and one on VLAN 16) to see if this affects the behaviour.
  • Reboot the server in question and see if the rp_filter setting persists on the eno2 interface based on the “conf.default” and “conf.all” settings.
  • Review switch port settings; see if some ports can be changed to “General” from “Trunk”. Consider replacing the switch with something that will cause less irritation.
  • See if merely tagging the port with VLAN 16 (and not setting it as untagged/primary) and configuring an eno2.16 interface still allows traffic to flow.
  • Apply firewall rules on the EdgeRouter (starting from a “deny all” basis) and confirm that only authorized traffic is permitted.
  • Ensure VLAN hardware offload is enabled on the EdgeRouter
  • Add another VLAN now that the first one was figured out!

Windows file share and NTFS permissions

For future reference when I inevitably forget whether it is more appropriate to restrict folders with NTFS permissions (Security tab) or file share permissions (Sharing tab).

https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2008-R2-and-2008/cc754178(v%3dws.10)

“For example, some experienced administrators prefer always to set share permissions to Full Control for Everyone, and to rely entirely on NTFS permissions to restrict access.”

Relevant table of examples:

Folder type Share permissions NTFS permissions
Public folder. A folder that can be accessed by everyone. Grant Change permission to the Users group. Grant Modify permission to the Users group.
Drop folder. A folder where users can drop confidential reports or homework assignments that only the group manager or instructor can read. Grant the Change permission to the Users group.

Grant the Full Control permission to the group manager.

Grant the Write permission for the users’ group that is applied to This Folder only. (This is an option available on the Advanced page.)

If each user needs to have certain permissions to the files that he or she dropped, you can create a permission entry for the Creator Owner well-known security identifier (SID) and apply it to Subfolder and files only. For example, you can grant the Read and Write permission to the Creator Owner SID on the drop folder and apply it to all subfolders and files. This grants the user who dropped or created the file (the Creator Owner) the ability to read and write to the file. The Creator Owner can then access the file through the Run command using \\ServerName\DropFolder\FileName.

Grant the Full Control permission for the group manager.

Application folder. A folder containing applications that can be run over the network. Grant Read permission for the Users group. Grant Read, Read and Execute, and List Folder Content permissions to the Users group.
Home folders. Individual folders for each user. Only the user has access to the folder. Grant the Full Control permission to each user on their respective folder. Grant the Full Control permission to each user for their respective folder.

Exchange 2016 + Outlook on iOS and Android: Message size limits and their configuration

Users with the official Microsoft Outlook client on Android or iOS kept running into ~36MB size limits when attempting to send attachments (given the megapixel sizes of most cell phone photos, this can amount to 3 to 4 pictures attached and the whole message is rejected), and none of the conventional transport/mailbox maximum size settings were the cause. I’m hoping the changes in the following articles are the fix:

The settings I specifically believe are responsible are:

  • maxAllowedContentLength in %ExchangeInstallPath%FrontEnd\HttpProxy\ews\web.config
  • maxAllowedContentLength and maxReceivedMessageSize in %ExchangeInstallPath%ClientAccess\exchweb\ews\web.config
  • maxAllowedContentLength and maxRequestLength in %ExchangeInstallPath%FrontEnd\HttpProxy\owa\web.config
  • maxAllowedContentLength, maxRequestLength and maxReceivedMessageSize in %ExchangeInstallPath%ClientAccess\Owa\web.config

Cruise experiences: NCL Epic, December 2017

Neither our first trip on the Norwegian Epic nor subsequent one on the Breakaway in December 2016 scared us off of cruising, so here are some ramblings about the NCL Epic sailing an Eastern Western Caribbean itinerary in December 2017.

Eastern… no, wait, Western

We never made it here.

Due to the hurricanes (Irma and Maria) that absolutely crushed Eastern Caribbean regions in September 2017, nearly every itinerary featuring these destinations was adjusted, regardless of cruise line. As we were only a few days away from the 90-day deadline where one can cancel for a full refund, I was closely monitoring the situation. There wasn’t any coherent news regarding NCL’s plans, mainly because their Miami headquarters was also in a state of disarray around that time.

Cruises sailing to these places in September were definitely being cancelled, cut short, or adjusted, but my theory was that by mid-December, the various Virgin Islands would be up and running again. Maybe the entire region would be worse for wear, but at least the touristy areas where ships drop off several thousand passengers would be up and running. Frankly, the best thing a tourist can do in one of these situations is to continue to visit, and spend hard-earned (or easily-earned) currency with the locals. So it was with that theory in mind that I decided not to adjust our plans.

Unfortunately a swift recovery wasn’t the case. Once NCL got things somewhat settled, they made rumblings about a possible itinerary adjustment 88 days prior to the cruise date, which was just enough time to incur a 25% penalty to switch. The replacement itinerary was officially announced at exactly 75 days out (coinciding with a 50% cancel/change fee.) Not being a common idiot, I knew that we were likely to end up at Falmouth and Grand Cayman as replacement ports, but really didn’t want another $300 in airfare changes or to have to sort out transportation from Orlando/Port Canaveral to a different port. So, we stuck with the Epic and the revised route.

Oh, what’s that you say? Shouldn’t the cruise line have to do something – I mean, they’ve changed two-thirds of the ports on your vacation – harrumph harrumph? I direct you to NCL’s guest ticket contract that basically says they don’t even have to put you on a ship (6b), and they don’t have to stick to the itinerary (6c). Also in the same section, you release NCL from any loss/damage/injury due to piracy, among other egregious things, so don’t expect compensation for any Captain Phillips experience.

Continue reading

Manage your MR10i or other LSI MegaRAID controller on a ESXi 6.5 host

I’ve been arguing with an Exchange 2016 server lately, due to what I suspect is a dodgy IBM-badged MR10i RAID controller in a x3650 M3. It has been kicking disks that seem entirely fine out of RAID1 volumes, which effectively has the same side effect as losing a disk. I intend to publish a few posts with some of the links and practices I’ve used lately.

Original article: How to install LSI MegaRAID Storage Manager (MSM) on ESXi 5.5

The original, excellent instructions from Mike Smith at Serenity-Networks, despite being for ESXi 5.5, seemed to work with some minor adaptations for ESXi 6.5.0 Update 1 (Build 5969303), with the latest versions of software from Avago (Broadcom).

Enable SSH on ESXi host: From the web UI, in the Navigator column, select Host, then choose Actions > Enable Secure Shell (SSH):

Adjust the “acceptance level” to allow installation of unsigned VIB files: In the Navigator column, select Manage, then select the Security & users tab. Then, click the Edit settings button and choose Community.

Get the LSI downloads (SMIS provider and the latest MegaRAID MSM): I found filtering by OEM did not successfully show results. On the Broadcom website, I selected the following categories for download:

  • Group: Storage Controllers, Adapters and ICs
  • Family: Storage Controllers, Adapters and ICs
  • OEM: (left blank – showed up as ‘OEM’ in the search interface)
  • Product: All
  • Asset Type: Management Software and Tools

There were 679 results; I used Ctrl+F and searched for “SMIS”, which offered a link titled “Latest SMIS Providers” for VMWare 6.0 and 6.5: https://docs.broadcom.com/docs/VMware_MR_SAS_Providers-00.67.V0.04.zip

Then I also used Ctrl+F and searched for “MegaRAID Storage Manager”, which offered MSM for a variety of platforms:

Copy the LSI SMIS provider (the file with .vib extension) to the /tmp directory on ESXi host (scp/WinSCP/your client of choice). I found that my sneaky attempt at copying it to a shared volume at /vmfs/volumes/… was hit and miss; when it was a fibre channel mount, the install worked properly, but if the datastore was on a local disk, it died with an error message.

SSH to the ESXi host with appropriate credentials (I did everything as root) and run the following install command:

esxcli software vib install -v /tmp/vmware-esx-provider-lsiprovider.vib --no-sig-check

I also had to disable the firewall on the ESXi host. Bad practice, but I don’t have a list of the specific ports to open at present.

esxcli network firewall set --enabled false

Reboot the ESXi host when complete. You can and probably should do the usual behaviour of taking it into maintenance mode, but in my case everything shut down and came up cleanly as VMWare Tools was installed on each guest.

It was hit and miss as to whether I had to add the line from /etc/hosts on the ESXi server with the hostname to my Windows box. I found that eventually creating both A and PTR records in Active Directory DNS, combined with turning off the ESXi firewall, were sufficient to get the MSM client on a domain-joined Windows server to connect – not even necessarily a guest VM on the same hypervisor.

I also had to change the MSM client settings in the Configure Host dialog to “Display all of the systems in the network of local server”, and not the “ESXi-CIMOM” option:

Fixing WSUS – error 507: “Update services failed its initialization and stopped”

I had a Windows Server Update Services installation that after a reboot, failed to start the WSUS service with a fairly generic error message. Clients issued an “unable to check for updates” message with an 8-character hex error code, differing depending on the client OS.

To fix it, I followed the directions in this source article on vcloudnine.deWSUS on Windows 2012 (R2) and KB3159706 – WSUS console fails to connect

  • Run elevated Command Prompt and issue the following command:"C:\Program Files\Update Services\Tools\wsusutil.exe" postinstall /servicing
  • Ensure the “HTTP Activation” feature is installed, using Server Manager > Add Roles and Features > Features > .NET Framework 4.5 Features. In my case, it was already installed.
  • Restart “WSUS Service” from services.msc

First cruise review: Norwegian Epic March 1, 2015 – Western Caribbean

This has been sitting in my drafts folder since mid-2015, so the information in it re: UBP charges, menu contents, etc. are all outdated at this point, but I think it’s still a good representation of how our first cruise went. Suffice it to say, I need to be more timely with these – we’ve since gone on a Bahamas cruise on the NCL Breakaway in December 2016, and have a different Caribbean cruise planned on the Epic during December 2017. So even if there are some negatives here, clearly we’ve gone back so it can’t be that bad.

 

I’ll try and note in the article where things have changed.

In Which We Decide To Cruise

In 2015, Kayla and I got tired of the snow and wind and ridiculously cold temperatures in Southwestern Ontario, and decided to get away to a warmer climate. We’d been to BlueBay Villas Doradas in the Dominican Republic to do the all-inclusive resort trip with a few friends last year and while we liked it, we wanted to try something new before falling back to the same thing. We’d also discussed various Sunwing-promoted destinations flying directly out of YKF to Mexico, but the available resorts in our price range were either too new to have a decent amount of feedback, or had recently begun “focusing on a new concept.”

Then the idea of taking a cruise came up and we started looking. A few years ago in a hotel room in Prague, I’d seen a documentary that was pretty much an expose of the entire cruise ship industry. I managed to locate it after returning – it’s CNBC’s “Cruise Inc: Big Money on the High Seas” (2009).  It went over a cruise on the Norwegian Pearl, describing how passengers are basically just walking ATMs and that the cruise line is constantly running the numbers on every aspect of shipboard operations. The conclusion was that on the last sea day, the operator broke even for the cruise, in no small part to making up $21K in alcohol sales. Being shaken up and down for cash constantly didn’t really appeal to me.

Enter a combination of a Norwegian Cruise Line and an Expedia for TD 4x points promo. First, note that cruises for Expedia for TD are booked over the phone, but you still get the highest point multiplier as if you had booked online, and there’s no jerk fee to book over the phone unlike with other types of vacation. I’d recommend checking out the regular Expedia.ca site for a price range based on room category, then adding another 13-18% for taxes and fees. Both NCL (when called directly) and the Expedia for TD agent ended up quoting the exact same price.

The March 1/2015 sailing of the Epic also had an option that if you booked your stateroom in at least a balcony class (the previous tiers are “interior” and “oceanview”, and I took interior to mean “would feel like a third class room on the Titanic”), you could pick either $300US of onboard credit, a dining package involving access to theoretically better a la carte restaurants, or what was called the Ultimate Beverage Package(NCL Epic does not actually have “oceanview” rooms, it goes from inside to balcony to mini-suite.)

Valued at $54US per person per day (at the time; I saw it listed as $59 per day on the ship) the idea of the UBP is that you’d get most booze free, with a few exceptions on the really premium stuff. Given that the cheapest beer bottles are $5.75 each, and you can easily order an $11 drink without trying, it’s not too difficult to get your value out of this choice. I would highly recommend getting your travel agent or NCL to include the UBP as a complimentary option.

(As of March 2017 this package is now $79US/day to buy outright, which is just absurd. It ends up being worth avoiding any “Canadian at par” or “Guarantee cabin” offers that don’t include the UBP, if you are at all interested in drinks.)

Here’s an example liquor menu from the Epic, which I’m spoiling now in advance of the main review content because it seems to be one of the most requested things online. Basically you want to order beers, mixed drinks and wine by the glass and avoid the “super-premium” liquors.

So, off we went. We flew into Miami directly rather than messing around trying to get a transfer from Fort Lauderdale, stayed over at the Marriott Residence Inn on the previous night, then caught a shuttle to the Port of Miami on the morning of the cruise.

At this point I must impress upon readers that nothing really went wrong and that I’m impressed with our NCL and Epic experience. For never having taken a cruise before, the experience here was definitely what I wanted out of a vacation, and I can somewhat understand the CruiseCritic members who count down the hours until their next sailing in their forum signature. My criticisms are minor dull spots in what I would consider a 95% positive experience. I don’t see a reason to pick any other cruise line at this point and I do have every reason to keep choosing to cruise as a vacation.

Day 1: Embarkation Day and Ultimate Beverage Package

After arriving at the port of Miami, we got out of the shuttle and handed our suitcases over to a baggage handler, who will be sure to let you know that they would very gladly appreciate a tip. We then went into the terminal and waited in a fairly quick moving line to provide ID, checkin documentation and (most importantly) a credit card. Each person in your party gets issued room keycards with a mag stripe and barcode, and if you’ve purchased the UBP you get a sticker on the card indicating as much. The card acts like your passport while on board and in the ports, and even to the extent that we were told it would be a good idea to keep your passport in your room safe, separate from the keycard.

keycard

Of course, the first thing I did after checking out the room was try to get a drink from the overworked bar staff. The slightly nasty surprise was that while than still docked or less than 3 miles from the port of Miami, you get charged sales tax on booze. It ended up being a matter of us paying 56 to 76 cents out of pocket per drink before hitting international waters, but it was unexpected and wasn’t explicitly disclosed in the UBP terms other than “your check may reflect applicable VAT for certain ports or itineraries.” It would have been a much better experience to be given a heads up on “if you hit the bar before we leave Miami…” when we received our room cards.

What else can I say about the UBP? There was a bit more documentation in our room by the end of the first day, but from the practical experience of using it: There is an 18% autogratuity on all drink purchases, but it is included in the package.

(March 2017: You actually pay this at booking time, it’s considered 18% per person on the retail price of the package, so about $200US for a 7-day cruise for 4 people.)

You can order something with a base cost of up to $11 (so if your receipt total is $12.98 or under per drink, there’s nothing you pay out of pocket, and it won’t show up on your stateroom charges.) You should always pick up or ask for a bar menu to be sure, and also to make sure that you take full advantage of the variety of spirits and beer.

(March 2017: drinks are now comped up to $15US up from $11US, and the “super premium” liquors are all slightly over $15.)

In practice, despite the paper documentation in the stateroom indicating that two drinks per patron is allowed, none of the bars would allow you to order doubles or two drinks at a time. The best way we found to handle the situation was to present two room cards when ordering two drinks; then the bartender appeared to have discretion as to serving two drinks at a time.

Perhaps the most irritating matter is that you are expected to sign for all booze even if it will be zero rated by the UBP, so you end up striking out the “additional tip” and total lines, ignoring the “print name” line, and scrawling a signature that may or may not be yours (if a drink was put on your spouse’s account.) The receipt process really adds time and inconvenience to the whole experience, given that a large number of cruisers on our voyage had clearly taken NCL up on their free drinks offer and the bartenders were quite busy.

(As of March 2017, receipts were issued much less frequently and typically only if you had something chargeable, so this has improved. We also had fewer issues with “I’m getting a drink for my wife” without showing two cards on the Breakaway.)

This is a slightly redacted example of a bar receipt that you still get with the UBP. The drink in question was vodka and Sprite.
This is a slightly redacted example of a bar receipt that you still get with the UBP. The drink in question was vodka and Sprite.

Our bags weren’t delivered to our stateroom until later on the first day, so you’ll want your carry-ons – really, personal items – to have the essentials. It was quite nice to have a shower right away but I didn’t have new clothes to change into immediately afterwards.

We did a tour of the ship at 2pm, which involved a large group of people traipsing around decks, being told the location of every bar and pay-for restaurant, and concluding with me deciding next time I’d skip the tour and look around at my own pace. At 3:30 there was an obligatory emergency drill, which involved sitting beside a bunch of Canadians and deciding that under no circumstances would the other people half-assing around be any help if the ship sank.

On the first evening we ran into Park West, the ship art auctioneers, who are infamous in this story. They embodied the negative stereotype of “used car salesman” through and through. We decided to attend their art auction the next day as they did have some moderately interesting pieces on display.

I don’t know that I remember much of the rest of the first night, but I will say that for the first two days on the Epic I was absurdly hungry at various points, when there was really no need to be. The standard restaurants are basically closed between 3 and 5:30pm, and O’Sheehan’s (midship 24×7 restaurant and bar) has two areas to it – if you’re not seated in the restaurant, no food. Your best option with the Epic during the day is to head up to deck 15 and see what’s available at the outdoors buffet. I ate incredibly well once finding that out.

Day 2: Sea Day 1

Woke up, skipped breakfast. There is a moderate amount of noise in the hall from the cruise director and captain’s announcements around 9am, so if you want to sleep in make use of earplugs or several pillows. We ended up getting out of bed at 10:55, just barely making the start of the cruisecritic.com forum meet and mingle.

Unfortunately the Meet and Mingle event was a bust for us. Everyone had split off into groups, and the discussions focused on secretive insider-y “Posh Passes” that people had purchased the previous day. While the folks from the forums were helpful pre-cruise, you’ll want to be an extreme extrovert or a cruise regular if you want to get anything out of the scheduled event.

(March 2017: This is really cruise by cruise, and I’ve since had a better experience with a meetup on the NCL Breakaway.)

Next up, Kayla wanted to go to a shopping tutorial/seminar/meeting at noon by Linda and her mostly silent or surly partner Albert, which was supposed to be packed full of secrets and deals when you went to the various ports of call. This was a giant waste of time and I could have spent it trying to acquire lunch instead of being hungry/angry later in the afternoon.

Linda waxed on the wonderful melanin generating properties of some “sleep band” that purportedly cured some woman of Parkinson’s symptoms (lies), told us to ask for specific people at each store to get us the “best deal”, absolutely shilled out for Diamonds International, expressed what an awesome deal we were getting because we weren’t paying tax and duty, and in general made me feel like an idiot.

Ship Hallway
One of the ship’s hallways near guest rooms.

 

The one thing we did get out of her presentation was to go to Gold and Time in Ocho Rios, Jamaica and get a free gemstone for showing the shop map. The free gemstone is topaz, and they really use it as an upsell offer to earrings and a pendant (which both actually look quite nice.) You can now all do the same without burning an hour of time in the theatre with Linda and having homeopathic “natural frequency” bracelets shoved down your gullets.

Next event on the ship was the Park West art auction. This took place from 1:30 to 3pm and involved primarily pieces from Peter Max. For those of you who don’t know, Peter Max painted murals for Woodstock and produced a whack of other American pop art. If I never hear about Peter Max being the voice of a generation again it will be too soon. There was also a contest where you could win a print for its shipping cost – $55 to Canada.

Again I was hungry so I don’t remember much of the rest of the afternoon. I do recall that getting to the sit-down restaurants (Manhattan Club and Taste) close to the start of service got you a better table. Kayla though O’Sheehan’s mid day appetizers were gross. I didn’t mind the 24/7 menu from the night prior, but the fatty chicken wings she got that afternoon were straight out of a utility grade foodservice bucket.

(March 2017: the wing quality was definitely improved on Breakaway.)

Monday night we saw Blue Man Group at 10pm, and arrived about 30 minutes early to get reasonable seats in the second row. There were a bunch of wiener kids in the first row that squirmed throughout the show and could have calmed down a bit. Despite this, both of us really enjoyed the performance and we would recommend it for anyone, especially if you’re not sure what to expect.

Day 3: Sea Day 2

The second sea day continued our relationship with Park West, who at this time had decided we might be interested in buying art and kept sending invitations to our cabin. Kayla won the “free” print, which was actually decent, but really none of the other stuff appealed to us. To get into the category of things we liked, it was in the range of $1100 per piece. There was also incessant badgering about the following topics:

  • Peter Max being highly collectible, how many famous people own his work, and the sheer amount of stuff that he’s painted. Voice of a generation, etcetera. Still not impressed.
  • Why you should pay close to $20K for a Rembrandt. Also, it’s really an ink pressing of an etching that Park West owns, which means they control the supply. Every time they mentioned that this was an etching I heard the Buzz Killington “etching” cutaway from Family Guy.
  • The child prodigy Autumn DeForest, whose art is allegedly so in demand that she can’t paint it fast enough. Why? Because she has to go to school. Also, the art is good but not great.

The other memorable point of the day was attending the Legends in Concert show in the Epic Theatre. Online reviews and NCL propaganda indicated that this was a show not to miss. I’ll save you the trouble: you could probably miss it.

At the theatre bar, I ordered a Crown and ginger ale for myself, and tried to order a coffee for Kayla. The server gave me the stink-eye and said that coffee wasn’t included in the UBP, and that it fell into the same category of fresh-squeezed fruit juice and energy drinks that were specifically excluded. This is despite the fact that coffee was freely available in our room, in the 15th deck buffet, and also by the poolside. This was really the only time onboard that I felt cheated. For anyone reading from NCL, this was the single thing that stuck out at me as really poor form.

(March 2017: This still seems to be the case. Solution? Order a coffee with booze in it.)

The best part of the show was the third performer, who did an awesome Aretha Franklin tribute. Not having heard enough Jimmy Buffett, I don’t know if the performer was spot on or just sounded like an elderly man trying to cash in on prior fame – maybe that’s actually what you’re supposed to expect. The Adele impersonator was not great. She wasn’t hitting the necessary vocal range and made a whole bunch of cockney/British jokes that fell flat. It was a good thing Aretha closed the show out because otherwise I’d have considered it a total loss.

Having now had meals at both Taste and the Manhattan Room, I feel like I’m qualified to say that the reviewers on Yelp are morons – the food comes from the exact same place for both of these restaurants, and going to one dining room over another does not change the quality. We had mediocre to good service in both of these places and didn’t ever have to wait more than a minute for a table for two. The exact opposite could be said for the Garden Cafe, where you had to prowl the deck for a table and then get food in alternate shifts lest your newly-acquired food be cleared away.

Day 4: Ochos Rios, Jamaica

Before I detail the ports of call, full disclosure: we booked all three days through NCL’s online portal before leaving, taking their recommended tours and excursions rather than trying to organize our own itinerary. With these excursions you’re realistically in for at least $100US per person per day, but there were some definite advantages that I’ll get into shortly.

Beginning the ports of call was Jamaica, which seemed to be organized well. You exited the ship through deck 4 and it was pretty clear where to go. We met up with our tour group, and took a Toyota Coaster to a tourist-centric plantation. We then boarded a wagon hitched to a Massey Ferguson tractor and got driven around the grounds, stopping in various places to see animals, a tree climbing and coconut de-husking demonstration, and the many artifacts from Pierre Trudeau and family. Apparently they like the Trudeaus pretty well in Ochos Rios.

The main portion of the plantation excursion involved taking a camel ride. If our guides are to be believed, there are only eight camels in Jamaica and all of them are at the plantation. Only five camels were actively giving rides, and with a two person capacity, we waited around for 20 minutes while the first group of ten plodded around. When our turn came around it was pretty fun; the camel was like riding a slightly less stable horse and the animal had a singular focus on eating vegetation.

Cindy the camel wanted nothing more than to chomp leaves all day.
Cindy the camel wanted nothing more than to eat leaves all day. I did not get chomped.

In the afternoon, we were escorted to Dunn’s River Falls and opted to climb them. One of the tour organizers seemed to have a rough time getting the necessary number of admission wristbands for our group, and I felt like the entire excursion was rushed because of it. You’ll need to rent a locker for $10 ($3 refund on key/receipt return) because you can’t have anything in your hands, and a backpack would also not be suitable. Once we did get going, the climb was a great experience. I would also definitely recommend buying water shoes before the trip.

Also keep in mind that upon exit you’ll have to run the gauntlet of fairly aggressive peddlers trying to sell Jamaican souvenirs before making your way back to the tour bus. Props to our driver and one of the other tour guides for getting us out of a parking space, with the dialog between the guide and another driver sounding very similar to this GTA IV clip of Little Jacob:

Day 5: Grand Cayman

In Grand Cayman, we chose the Sunken Ship Snorkel & Tiki Beach excursion, and were called to the Epic Theatre for 8:20am. Of the three ports here, Grand Cayman is one where the ship doesn’t pull up to the pier directly, and uses the ship’s lifeboats to “tender” passengers to and from shore. I had heard some disastrous reviews about this process from previous sailings and was prepared for a fiasco.

After waiting about 30 minutes in the theatre, our group was called to load up a lifeboat – looking down one deck, we managed to skip a large line of people without NCL excursions that had either set up something on their own or just wanted to go ashore. Based on the onboard announcements, there was a swell interfering with loading from one side of the ship, and I’d overheard staff conversations that some people were really angry. Therefore, my recommendation is to book an NCL-managed excursion for any port where tendering is involved, even only if for the priority boarding.

A view of the ship from the shore in Grand Cayman.
A view of the ship from the shore in Grand Cayman. Note the lifeboats on the side used for transporting passengers to and from shore.

Also keep in mind that there are tours with very similar names: the “Reef & Wreck Snorkel” excursion is not the same as the “Sunken Ship” snorkel, despite the physical sunken ship being the same for both.

I’m not a big snorkeling person myself, but it was pretty neat seeing all the fish surrounding the sunken aircraft carrier. Other than that, the snorkeling was pretty average – if you’re really into it, I could see this being a better experience. We did use the flippers on boat but brought our own masks and air tubes – the communal equipment just goes into a large rubber trash can and I’m not sure how often it gets sanitized. Going from the large boat to a smaller one also was a bit of a change; we were glad to get off by the time we’d arrived back at shore as both of us had started feeling a bit queasy.

The afternoon Tiki Beach experience was also underwhelming for me. You get a complimentary rum punch on arrival, then you locate a beach lounger. Everything else is at additional cost, and pricing was in Cayman Islands Dollars in what I can only assume is a deliberate attempt to obscure the exchange rate to USD. If you do choose this excursion, I’d suggest bringing along snacks, as well as headphones, music and a book. I was fairly bored in the afternoon and sitting in the sun wasn’t entirely thrilling.

Days 6 and 7: Cozumel, Mexico and Sea Day 3

Most of this review ended up being written initially on the third sea day, which was the last whole day of the cruise, trying to remember most of the onboard experiences. On Day 6, we attended a “Salsa and Salsa” class in Cozumel, which had delicious food, tons of authentic tequila and a reasonable amount of dancing (at the end, when your inhibitions are lowered from the booze.) It is a group-style class so you share your table and ingredients with others. There was wifi in the hotel lobby where the class was offered – you just had to ask the desk representative for the credentials.

The last sea day basically involved me milling around the ship bars and Kayla reading/decompressing. I also paid about $4US for an excellent Singapore Noodles at Shanghai’s Noodle Bar (as of March 2017, this is a complimentary restaurant and gets very busy.) I walked around all the decks and ended up settling on Shaker’s Martini Lounge, occasionally ordering a drink and bar snacks, and trying to figure out when we could schedule our next cruise.

Return to Miami

Probably the least fun part of our cruse, we returned back to Miami at early o’clock and quickly got a taxi to the airport. Unfortunately due to price and the repeated suggestion to not book anything too early in case the ship came in late, we had nearly twelve hours to kill at MIA waiting for our flight. We were quickly bored and I would definitely suggest picking an afternoon flight, or buying an airport lounge pass where you can just kill time and perhaps have some beverages.

So, if you’re willing to deal with the occasional service charge and are willing to just “go with the flow”, I think our first NCL experience was quite decent.

Fix: WSUS Server Cleanup Wizard hangs/stalls when deleting unused updates

Side note: several years ago Kayla caught me talking in my sleep, muttering something about “you’ve got to check the boxes!” This is the actual dialog and process in question.

Full credit to Jeremy Jameson at MSDN. Posting in case the original disappears.

  • Run Server Cleanup Wizard with only the “Unused updates and update revisions” (option #1) box checked. This took about six hours on the server experiencing the problem:screen-shot-2016-11-22-at-11-32-29-am
  • Once finished, run the wizard again with only the “Unneeded update files” (option #3) box checked.
  • Once that’s finished, run the wizard with all the boxes checked.

RiteBite and Invisalign Review: Conclusion

Well, better late than never, but I’m currently in the process of cleaning up paperwork in the home office, and noted that RiteBite had given me a flyer asking for a Google review several months ago. So here’s a conclusion to the review series, which will be combined with the other content and sliced into bits and pieces for the less-verbose social media pages.

Completing the Program

Since last time I wrote, I went through about half of another series of trays with 7-day rotations. I specifically requested to have my treatment wrapped up about a week before my wedding in August 2016, and Dr. Luis and staff were very accommodating since this third set was effectively “finishing touches”. As part of the removal, I had permanent wiring bonded behind both my top and bottom teeth and was given a set of top and bottom harder, clear plastic retainers to wear overnight. One important point is that for the first two weeks, you’re expected to wear the retainers as close to 24/7 as possible, so you’re not “entirely” done. I obviously made an exception to this for the wedding.

A Few Nitpicks

The retainers are not ideal, to put a point on it. Their larger size (compared to the Invisalign trays) and increased rigidity triggers my gag reflex nearly every morning when taking them out, and I still run into similar problems with drooling on my pillow.

I also specifically requested the top permanent wire, and had to ask several times before getting a “yes” – several staff suggested that it wasn’t strictly necessary or had a higher chance of breaking. I wanted to ensure that with my financial investment, there was a “backup” in place to help the teeth from moving as much. The top wire’s presence is still noticeable when I close my mouth several months later, unlike the bottom wire. Both still have a distinct “pebbled” texture where the wire is adhered to the back of each row of teeth.

Despite asking for Google reviews as part of the “exit interview”, RiteBite seems to have several accounts under their name on Google Plus (1, 2, 3, 4, 5) and no link to the official Google profile from their website, nor any content on these pages. I was also disappointed to find that the Case Graphics section has disappeared from my profile since completing treatment.

Overall Results

The change has been quite impressive. It took slightly over a year and a half, I wasn’t seriously inconvenienced, and now that it has been paid off, I begrudgingly admit that it was probably a better personal choice than replacing the laminate flooring in the house or buying the same amount of networking gear.

Continuing the “Router rumble” with pfSense 2.3.2 and a FW-7540

Following up from my previous round of router testing, I managed to get a spare Lanner FW-7540 with an Intel Atom D525 CPU to test how my current pfSense 2.3.2 setup compared to an EdgeRouter Lite. The results were well below what I was expecting: the pfSense box topped out at 490Mbit in the 1MB test and was very spiky when looking at the netdata graphs.

The results file is also available if you’d like to look directly at the ab output.

d525_pfsense

Filesize Average Mbit/s Total Failed Requests Notes
10K 145.07 87 10K concurrency test only resulted in 49Mbit. No failed requests in 10, 100 and 1000 concurrency tests.
100K 421.71 4896 No failed requests in 10, 100 and 1000 concurrency tests.
1MB 489.96 3341 No failed requests in 10, 100 and 1000 concurrency tests.

This test fairly obviously shows a ceiling. For WAN connections of over 500Mbit, it looks like something beefier than an Atom D525 is necessary to run the NAT as anticipated.

I also ran some more informal WAN to LAN iPerf3 testing on direct connection (MDI-X), the EdgeRouter Lite and the pfSense/7540 combination to get some synthetic numbers:

Connection iPerf Result
Direct 941Mbit with no retries
EdgeRouter Lite 939Mbit with retries
pfSense/7540 829Mbit with no retries

Given how well the EdgeRouter Lite seems to perform for its price, and since it beats out the more general purpose hardware, I suspect I will be swapping out for an ERL or ER-Pro very shortly.