Neither our first trip on the Norwegian Epic nor subsequent one on the Breakaway in December 2016 scared us off of cruising, so here are some ramblings about the NCL Epic sailing an Eastern Western Caribbean itinerary in December 2017.
Eastern… no, wait, Western
Due to the hurricanes (Irma and Maria) that absolutely crushed Eastern Caribbean regions in September 2017, nearly every itinerary featuring these destinations was adjusted, regardless of cruise line. As we were only a few days away from the 90-day deadline where one can cancel for a full refund, I was closely monitoring the situation. There wasn’t any coherent news regarding NCL’s plans, mainly because their Miami headquarters was also in a state of disarray around that time.
Cruises sailing to these places in September were definitely being cancelled, cut short, or adjusted, but my theory was that by mid-December, the various Virgin Islands would be up and running again. Maybe the entire region would be worse for wear, but at least the touristy areas where ships drop off several thousand passengers would be up and running. Frankly, the best thing a tourist can do in one of these situations is to continue to visit, and spend hard-earned (or easily-earned) currency with the locals. So it was with that theory in mind that I decided not to adjust our plans.
Unfortunately a swift recovery wasn’t the case. Once NCL got things somewhat settled, they made rumblings about a possible itinerary adjustment 88 days prior to the cruise date, which was just enough time to incur a 25% penalty to switch. The replacement itinerary was officially announced at exactly 75 days out (coinciding with a 50% cancel/change fee.) Not being a common idiot, I knew that we were likely to end up at Falmouth and Grand Cayman as replacement ports, but really didn’t want another $300 in airfare changes or to have to sort out transportation from Orlando/Port Canaveral to a different port. So, we stuck with the Epic and the revised route.
Oh, what’s that you say? Shouldn’t the cruise line have to dosomething – I mean, they’ve changed two-thirds of the ports on your vacation – harrumph harrumph? I direct you to NCL’s guest ticket contract that basically says they don’t even have to put you on a ship (6b), and they don’t have to stick to the itinerary (6c). Also in the same section, you release NCL from any loss/damage/injury due to piracy, among other egregious things, so don’t expect compensation for any Captain Phillips experience.
I’ve been arguing with an Exchange 2016 server lately, due to what I suspect is a dodgy IBM-badged MR10i RAID controller in a x3650 M3. It has been kicking disks that seem entirely fine out of RAID1 volumes, which effectively has the same side effect as losing a disk. I intend to publish a few posts with some of the links and practices I’ve used lately.
The original, excellent instructions from Mike Smith at Serenity-Networks, despite being for ESXi 5.5, seemed to work with some minor adaptations for ESXi 6.5.0 Update 1 (Build 5969303), with the latest versions of software from Avago (Broadcom).
Enable SSH on ESXi host: From the web UI, in the Navigator column, select Host, then choose Actions > Enable Secure Shell (SSH):
Adjust the “acceptance level” to allow installation of unsigned VIB files: In the Navigator column, select Manage, then select the Security & users tab. Then, click the Edit settings button and choose Community.
Copy the LSI SMIS provider (the file with .vib extension) to the /tmp directory on ESXi host (scp/WinSCP/your client of choice). I found that my sneaky attempt at copying it to a shared volume at /vmfs/volumes/… was hit and miss; when it was a fibre channel mount, the install worked properly, but if the datastore was on a local disk, it died with an error message.
SSH to the ESXi host with appropriate credentials (I did everything as root) and run the following install command:
I also had to disable the firewall on the ESXi host. Bad practice, but I don’t have a list of the specific ports to open at present.
esxcli network firewall set --enabled false
Reboot the ESXi host when complete. You can and probably should do the usual behaviour of taking it into maintenance mode, but in my case everything shut down and came up cleanly as VMWare Tools was installed on each guest.
It was hit and miss as to whether I had to add the line from /etc/hosts on the ESXi server with the hostname to my Windows box. I found that eventually creating both A and PTR records in Active Directory DNS, combined with turning off the ESXi firewall, were sufficient to get the MSM client on a domain-joined Windows server to connect – not even necessarily a guest VM on the same hypervisor.
I also had to change the MSM client settings in the Configure Host dialog to “Display all of the systems in the network of local server”, and not the “ESXi-CIMOM” option:
I had a Windows Server Update Services installation that after a reboot, failed to start the WSUS service with a fairly generic error message. Clients issued an “unable to check for updates” message with an 8-character hex error code, differing depending on the client OS.
This has been sitting in my drafts folder since mid-2015, so the information in it re: UBP charges, menu contents, etc. are all outdated at this point, but I think it’s still a good representation of how our first cruise went. Suffice it to say, I need to be more timely with these – we’ve since gone on a Bahamas cruise on the NCL Breakaway in December 2016, and have a different Caribbean cruise planned on the Epic during December 2017. So even if there are some negatives here, clearly we’ve gone back so it can’t be that bad.
I’ll try and note in the article where things have changed.
In Which We Decide To Cruise
In 2015, Kayla and I got tired of the snow and wind and ridiculously cold temperatures in Southwestern Ontario, and decided to get away to a warmer climate. We’d been to BlueBay Villas Doradas in the Dominican Republic to do the all-inclusive resort trip with a few friends last year and while we liked it, we wanted to try something new before falling back to the same thing. We’d also discussed various Sunwing-promoted destinations flying directly out of YKF to Mexico, but the available resorts in our price range were either too new to have a decent amount of feedback, or had recently begun “focusing on a new concept.”
Then the idea of taking a cruise came up and we started looking. A few years ago in a hotel room in Prague, I’d seen a documentary that was pretty much an expose of the entire cruise ship industry. I managed to locate it after returning – it’s CNBC’s “Cruise Inc: Big Money on the High Seas” (2009). It went over a cruise on the Norwegian Pearl, describing how passengers are basically just walking ATMs and that the cruise line is constantly running the numbers on every aspect of shipboard operations. The conclusion was that on the last sea day, the operator broke even for the cruise, in no small part to making up $21K in alcohol sales. Being shaken up and down for cash constantly didn’t really appeal to me.
Enter a combination of a Norwegian Cruise Line and an Expedia for TD 4x points promo. First, note that cruises for Expedia for TD are booked over the phone, but you still get the highest point multiplier as if you had booked online, and there’s no jerk fee to book over the phone unlike with other types of vacation. I’d recommend checking out the regular Expedia.ca site for a price range based on room category, then adding another 13-18% for taxes and fees. Both NCL (when called directly) and the Expedia for TD agent ended up quoting the exact same price.
The March 1/2015 sailing of the Epic also had an option that if you booked your stateroom in at least a balcony class (the previous tiers are “interior” and “oceanview”, and I took interior to mean “would feel like a third class room on the Titanic”), you could pick either $300US of onboard credit, a dining package involving access to theoretically better a la carte restaurants, or what was called the Ultimate Beverage Package. (NCL Epic does not actually have “oceanview” rooms, it goes from inside to balcony to mini-suite.)
Valued at $54US per person per day (at the time; I saw it listed as $59 per day on the ship) the idea of the UBP is that you’d get most booze free, with a few exceptions on the really premium stuff. Given that the cheapest beer bottles are $5.75 each, and you can easily order an $11 drink without trying, it’s not too difficult to get your value out of this choice. I would highly recommend getting your travel agent or NCL to include the UBP as a complimentary option.
(As of March 2017 this package is now $79US/day to buy outright, which is just absurd. It ends up being worth avoiding any “Canadian at par” or “Guarantee cabin” offers that don’t include the UBP, if you are at all interested in drinks.)
Here’s an example liquor menu from the Epic, which I’m spoiling now in advance of the main review content because it seems to be one of the most requested things online. Basically you want to order beers, mixed drinks and wine by the glass and avoid the “super-premium” liquors.
So, off we went. We flew into Miami directly rather than messing around trying to get a transfer from Fort Lauderdale, stayed over at the Marriott Residence Inn on the previous night, then caught a shuttle to the Port of Miami on the morning of the cruise.
At this point I must impress upon readers that nothing really went wrong and that I’m impressed with our NCL and Epic experience. For never having taken a cruise before, the experience here was definitely what I wanted out of a vacation, and I can somewhat understand the CruiseCritic members who count down the hours until their next sailing in their forum signature. My criticisms are minor dull spots in what I would consider a 95% positive experience. I don’t see a reason to pick any other cruise line at this point and I do have every reason to keep choosing to cruise as a vacation.
Day 1: Embarkation Day and Ultimate Beverage Package
After arriving at the port of Miami, we got out of the shuttle and handed our suitcases over to a baggage handler, who will be sure to let you know that they would very gladly appreciate a tip. We then went into the terminal and waited in a fairly quick moving line to provide ID, checkin documentation and (most importantly) a credit card. Each person in your party gets issued room keycards with a mag stripe and barcode, and if you’ve purchased the UBP you get a sticker on the card indicating as much. The card acts like your passport while on board and in the ports, and even to the extent that we were told it would be a good idea to keep your passport in your room safe, separate from the keycard.
Of course, the first thing I did after checking out the room was try to get a drink from the overworked bar staff. The slightly nasty surprise was that while than still docked or less than 3 miles from the port of Miami, you get charged sales tax on booze. It ended up being a matter of us paying 56 to 76 cents out of pocket per drink before hitting international waters, but it was unexpected and wasn’t explicitly disclosed in the UBP terms other than “your check may reflect applicable VAT for certain ports or itineraries.” It would have been a much better experience to be given a heads up on “if you hit the bar before we leave Miami…” when we received our room cards.
What else can I say about the UBP? There was a bit more documentation in our room by the end of the first day, but from the practical experience of using it: There is an 18% autogratuity on all drink purchases, but it is included in the package.
(March 2017: You actually pay this at booking time, it’s considered 18% per person on the retail price of the package, so about $200US for a 7-day cruise for 4 people.)
You can order something with a base cost of up to $11 (so if your receipt total is $12.98 or under per drink, there’s nothing you pay out of pocket, and it won’t show up on your stateroom charges.) You should always pick up or ask for a bar menu to be sure, and also to make sure that you take full advantage of the variety of spirits and beer.
(March 2017: drinks are now comped up to $15US up from $11US, and the “super premium” liquors are all slightly over $15.)
In practice, despite the paper documentation in the stateroom indicating that two drinks per patron is allowed, none of the bars would allow you to order doubles or two drinks at a time. The best way we found to handle the situation was to present two room cards when ordering two drinks; then the bartender appeared to have discretion as to serving two drinks at a time.
Perhaps the most irritating matter is that you are expected to sign for all booze even if it will be zero rated by the UBP, so you end up striking out the “additional tip” and total lines, ignoring the “print name” line, and scrawling a signature that may or may not be yours (if a drink was put on your spouse’s account.) The receipt process really adds time and inconvenience to the whole experience, given that a large number of cruisers on our voyage had clearly taken NCL up on their free drinks offer and the bartenders were quite busy.
(As of March 2017, receipts were issued much less frequently and typically only if you had something chargeable, so this has improved. We also had fewer issues with “I’m getting a drink for my wife” without showing two cards on the Breakaway.)
Our bags weren’t delivered to our stateroom until later on the first day, so you’ll want your carry-ons – really, personal items – to have the essentials. It was quite nice to have a shower right away but I didn’t have new clothes to change into immediately afterwards.
We did a tour of the ship at 2pm, which involved a large group of people traipsing around decks, being told the location of every bar and pay-for restaurant, and concluding with me deciding next time I’d skip the tour and look around at my own pace. At 3:30 there was an obligatory emergency drill, which involved sitting beside a bunch of Canadians and deciding that under no circumstances would the other people half-assing around be any help if the ship sank.
On the first evening we ran into Park West, the ship art auctioneers, who are infamous in this story. They embodied the negative stereotype of “used car salesman” through and through. We decided to attend their art auction the next day as they did have some moderately interesting pieces on display.
I don’t know that I remember much of the rest of the first night, but I will say that for the first two days on the Epic I was absurdly hungry at various points, when there was really no need to be. The standard restaurants are basically closed between 3 and 5:30pm, and O’Sheehan’s (midship 24×7 restaurant and bar) has two areas to it – if you’re not seated in the restaurant, no food. Your best option with the Epic during the day is to head up to deck 15 and see what’s available at the outdoors buffet. I ate incredibly well once finding that out.
Day 2: Sea Day 1
Woke up, skipped breakfast. There is a moderate amount of noise in the hall from the cruise director and captain’s announcements around 9am, so if you want to sleep in make use of earplugs or several pillows. We ended up getting out of bed at 10:55, just barely making the start of the cruisecritic.com forum meet and mingle.
Unfortunately the Meet and Mingle event was a bust for us. Everyone had split off into groups, and the discussions focused on secretive insider-y “Posh Passes” that people had purchased the previous day. While the folks from the forums were helpful pre-cruise, you’ll want to be an extreme extrovert or a cruise regular if you want to get anything out of the scheduled event.
(March 2017: This is really cruise by cruise, and I’ve since had a better experience with a meetup on the NCL Breakaway.)
Next up, Kayla wanted to go to a shopping tutorial/seminar/meeting at noon by Linda and her mostly silent or surly partner Albert, which was supposed to be packed full of secrets and deals when you went to the various ports of call. This was a giant waste of time and I could have spent it trying to acquire lunch instead of being hungry/angry later in the afternoon.
Linda waxed on the wonderful melanin generating properties of some “sleep band” that purportedly cured some woman of Parkinson’s symptoms (lies), told us to ask for specific people at each store to get us the “best deal”, absolutely shilled out for Diamonds International, expressed what an awesome deal we were getting because we weren’t paying tax and duty, and in general made me feel like an idiot.
The one thing we did get out of her presentation was to go to Gold and Time in Ocho Rios, Jamaica and get a free gemstone for showing the shop map. The free gemstone is topaz, and they really use it as an upsell offer to earrings and a pendant (which both actually look quite nice.) You can now all do the same without burning an hour of time in the theatre with Linda and having homeopathic “natural frequency” bracelets shoved down your gullets.
Next event on the ship was the Park West art auction. This took place from 1:30 to 3pm and involved primarily pieces from Peter Max. For those of you who don’t know, Peter Max painted murals for Woodstock and produced a whack of other American pop art. If I never hear about Peter Max being the voice of a generation again it will be too soon. There was also a contest where you could win a print for its shipping cost – $55 to Canada.
Again I was hungry so I don’t remember much of the rest of the afternoon. I do recall that getting to the sit-down restaurants (Manhattan Club and Taste) close to the start of service got you a better table. Kayla though O’Sheehan’s mid day appetizers were gross. I didn’t mind the 24/7 menu from the night prior, but the fatty chicken wings she got that afternoon were straight out of a utility grade foodservice bucket.
(March 2017: the wing quality was definitely improved on Breakaway.)
Monday night we saw Blue Man Group at 10pm, and arrived about 30 minutes early to get reasonable seats in the second row. There were a bunch of wiener kids in the first row that squirmed throughout the show and could have calmed down a bit. Despite this, both of us really enjoyed the performance and we would recommend it for anyone, especially if you’re not sure what to expect.
Day 3: Sea Day 2
The second sea day continued our relationship with Park West, who at this time had decided we might be interested in buying art and kept sending invitations to our cabin. Kayla won the “free” print, which was actually decent, but really none of the other stuff appealed to us. To get into the category of things we liked, it was in the range of $1100 per piece. There was also incessant badgering about the following topics:
Peter Max being highly collectible, how many famous people own his work, and the sheer amount of stuff that he’s painted. Voice of a generation, etcetera. Still not impressed.
Why you should pay close to $20K for a Rembrandt. Also, it’s really an ink pressing of an etching that Park West owns, which means they control the supply. Every time they mentioned that this was an etching I heard the Buzz Killington “etching” cutaway from Family Guy.
The child prodigy Autumn DeForest, whose art is allegedly so in demand that she can’t paint it fast enough. Why? Because she has to go to school. Also, the art is good but not great.
The other memorable point of the day was attending the Legends in Concert show in the Epic Theatre. Online reviews and NCL propaganda indicated that this was a show not to miss. I’ll save you the trouble: you could probably miss it.
At the theatre bar, I ordered a Crown and ginger ale for myself, and tried to order a coffee for Kayla. The server gave me the stink-eye and said that coffee wasn’t included in the UBP, and that it fell into the same category of fresh-squeezed fruit juice and energy drinks that were specifically excluded. This is despite the fact that coffee was freely available in our room, in the 15th deck buffet, and also by the poolside. This was really the only time onboard that I felt cheated. For anyone reading from NCL, this was the single thing that stuck out at me as really poor form.
(March 2017: This still seems to be the case. Solution? Order a coffee with booze in it.)
The best part of the show was the third performer, who did an awesome Aretha Franklin tribute. Not having heard enough Jimmy Buffett, I don’t know if the performer was spot on or just sounded like an elderly man trying to cash in on prior fame – maybe that’s actually what you’re supposed to expect. The Adele impersonator was not great. She wasn’t hitting the necessary vocal range and made a whole bunch of cockney/British jokes that fell flat. It was a good thing Aretha closed the show out because otherwise I’d have considered it a total loss.
Having now had meals at both Taste and the Manhattan Room, I feel like I’m qualified to say that the reviewers on Yelp are morons – the food comes from the exact same place for both of these restaurants, and going to one dining room over another does not change the quality. We had mediocre to good service in both of these places and didn’t ever have to wait more than a minute for a table for two. The exact opposite could be said for the Garden Cafe, where you had to prowl the deck for a table and then get food in alternate shifts lest your newly-acquired food be cleared away.
Day 4: Ochos Rios, Jamaica
Before I detail the ports of call, full disclosure: we booked all three days through NCL’s online portal before leaving, taking their recommended tours and excursions rather than trying to organize our own itinerary. With these excursions you’re realistically in for at least $100US per person per day, but there were some definite advantages that I’ll get into shortly.
Beginning the ports of call was Jamaica, which seemed to be organized well. You exited the ship through deck 4 and it was pretty clear where to go. We met up with our tour group, and took a Toyota Coaster to a tourist-centric plantation. We then boarded a wagon hitched to a Massey Ferguson tractor and got driven around the grounds, stopping in various places to see animals, a tree climbing and coconut de-husking demonstration, and the many artifacts from Pierre Trudeau and family. Apparently they like the Trudeaus pretty well in Ochos Rios.
The main portion of the plantation excursion involved taking a camel ride. If our guides are to be believed, there are only eight camels in Jamaica and all of them are at the plantation. Only five camels were actively giving rides, and with a two person capacity, we waited around for 20 minutes while the first group of ten plodded around. When our turn came around it was pretty fun; the camel was like riding a slightly less stable horse and the animal had a singular focus on eating vegetation.
In the afternoon, we were escorted to Dunn’s River Falls and opted to climb them. One of the tour organizers seemed to have a rough time getting the necessary number of admission wristbands for our group, and I felt like the entire excursion was rushed because of it. You’ll need to rent a locker for $10 ($3 refund on key/receipt return) because you can’t have anything in your hands, and a backpack would also not be suitable. Once we did get going, the climb was a great experience. I would also definitely recommend buying water shoes before the trip.
Also keep in mind that upon exit you’ll have to run the gauntlet of fairly aggressive peddlers trying to sell Jamaican souvenirs before making your way back to the tour bus. Props to our driver and one of the other tour guides for getting us out of a parking space, with the dialog between the guide and another driver sounding very similar to this GTA IV clip of Little Jacob:
Day 5: Grand Cayman
In Grand Cayman, we chose the Sunken Ship Snorkel & Tiki Beach excursion, and were called to the Epic Theatre for 8:20am. Of the three ports here, Grand Cayman is one where the ship doesn’t pull up to the pier directly, and uses the ship’s lifeboats to “tender” passengers to and from shore. I had heard some disastrous reviews about this process from previous sailings and was prepared for a fiasco.
After waiting about 30 minutes in the theatre, our group was called to load up a lifeboat – looking down one deck, we managed to skip a large line of people without NCL excursions that had either set up something on their own or just wanted to go ashore. Based on the onboard announcements, there was a swell interfering with loading from one side of the ship, and I’d overheard staff conversations that some people were really angry. Therefore, my recommendation is to book an NCL-managed excursion for any port where tendering is involved, even only if for the priority boarding.
Also keep in mind that there are tours with very similar names: the “Reef & Wreck Snorkel” excursion is not the same as the “Sunken Ship” snorkel, despite the physical sunken ship being the same for both.
I’m not a big snorkeling person myself, but it was pretty neat seeing all the fish surrounding the sunken aircraft carrier. Other than that, the snorkeling was pretty average – if you’re really into it, I could see this being a better experience. We did use the flippers on boat but brought our own masks and air tubes – the communal equipment just goes into a large rubber trash can and I’m not sure how often it gets sanitized. Going from the large boat to a smaller one also was a bit of a change; we were glad to get off by the time we’d arrived back at shore as both of us had started feeling a bit queasy.
The afternoon Tiki Beach experience was also underwhelming for me. You get a complimentary rum punch on arrival, then you locate a beach lounger. Everything else is at additional cost, and pricing was in Cayman Islands Dollars in what I can only assume is a deliberate attempt to obscure the exchange rate to USD. If you do choose this excursion, I’d suggest bringing along snacks, as well as headphones, music and a book. I was fairly bored in the afternoon and sitting in the sun wasn’t entirely thrilling.
Days 6 and 7: Cozumel, Mexico and Sea Day 3
Most of this review ended up being written initially on the third sea day, which was the last whole day of the cruise, trying to remember most of the onboard experiences. On Day 6, we attended a “Salsa and Salsa” class in Cozumel, which had delicious food, tons of authentic tequila and a reasonable amount of dancing (at the end, when your inhibitions are lowered from the booze.) It is a group-style class so you share your table and ingredients with others. There was wifi in the hotel lobby where the class was offered – you just had to ask the desk representative for the credentials.
The last sea day basically involved me milling around the ship bars and Kayla reading/decompressing. I also paid about $4US for an excellent Singapore Noodles at Shanghai’s Noodle Bar (as of March 2017, this is a complimentary restaurant and gets very busy.) I walked around all the decks and ended up settling on Shaker’s Martini Lounge, occasionally ordering a drink and bar snacks, and trying to figure out when we could schedule our next cruise.
Return to Miami
Probably the least fun part of our cruse, we returned back to Miami at early o’clock and quickly got a taxi to the airport. Unfortunately due to price and the repeated suggestion to not book anything too early in case the ship came in late, we had nearly twelve hours to kill at MIA waiting for our flight. We were quickly bored and I would definitely suggest picking an afternoon flight, or buying an airport lounge pass where you can just kill time and perhaps have some beverages.
So, if you’re willing to deal with the occasional service charge and are willing to just “go with the flow”, I think our first NCL experience was quite decent.
Well, better late than never, but I’m currently in the process of cleaning up paperwork in the home office, and noted that RiteBite had given me a flyer asking for a Google review several months ago. So here’s a conclusion to the review series, which will be combined with the other content and sliced into bits and pieces for the less-verbose social media pages.
Completing the Program
Since last time I wrote, I went through about half of another series of trays with 7-day rotations. I specifically requested to have my treatment wrapped up about a week before my wedding in August 2016, and Dr. Luis and staff were very accommodating since this third set was effectively “finishing touches”. As part of the removal, I had permanent wiring bonded behind both my top and bottom teeth and was given a set of top and bottom harder, clear plastic retainers to wear overnight. One important point is that for the first two weeks, you’re expected to wear the retainers as close to 24/7 as possible, so you’re not “entirely” done. I obviously made an exception to this for the wedding.
A Few Nitpicks
The retainers are not ideal, to put a point on it. Their larger size (compared to the Invisalign trays) and increased rigidity triggers my gag reflex nearly every morning when taking them out, and I still run into similar problems with drooling on my pillow.
I also specifically requested the top permanent wire, and had to ask several times before getting a “yes” – several staff suggested that it wasn’t strictly necessary or had a higher chance of breaking. I wanted to ensure that with my financial investment, there was a “backup” in place to help the teeth from moving as much. The top wire’s presence is still noticeable when I close my mouth several months later, unlike the bottom wire. Both still have a distinct “pebbled” texture where the wire is adhered to the back of each row of teeth.
Despite asking for Google reviews as part of the “exit interview”, RiteBite seems to have several accounts under their name on Google Plus (1, 2, 3, 4, 5) and no link to the official Google profile from their website, nor any content on these pages. I was also disappointed to find that the Case Graphics section has disappeared from my profile since completing treatment.
The change has been quite impressive. It took slightly over a year and a half, I wasn’t seriously inconvenienced, and now that it has been paid off, I begrudgingly admit that it was probably a better personal choice than replacing the laminate flooring in the house or buying the same amount of networking gear.
Following up from my previous round of router testing, I managed to get a spare Lanner FW-7540 with an Intel Atom D525 CPU to test how my current pfSense 2.3.2 setup compared to an EdgeRouter Lite. The results were well below what I was expecting: the pfSense box topped out at 490Mbit in the 1MB test and was very spiky when looking at the netdata graphs.
A friend and colleague of mine (Matt) and I have an ongoing discussion about over-specced gear for our home networks. Our core routers have been FW-7540s running pfSense (Atom D525, 4GB RAM, 4 Intel NICs) since 2013. pfSense offers a huge advantage over commercial-grade routers – I run dual WAN with failover based on ping, link, and packet loss, have extremely customizable DNS and DHCP, and can set up an OpenVPN server in just a few minutes. Matt and I also recently have had 500Mbit+ downstream connections installed, so it’d be good to know what hardware and software combination is “for sure” capable of utilizing the full pipe.
There have been a series of excellent articles at Ars Technica this year by Jim Salter that constantly get mentioned in our discussions:
The first two initial articles were mildly interesting – we do plenty of Linux-based routing at the office, but I don’t really want to build a router from scratch at home if there is a distribution that works as well. The results in Jim’s latest Router rumble article with pfSense 2.3.1 and the homebrew Celeron J1900 were described as “tweaky” and didn’t seem to hold up against the homebrew variant running Linux. I found this a bit odd because FreeBSD is widely assumed to have a hardened, robust and performant network stack; the general impression amongst networking folks I’ve talked to that Linux isn’t quite as good for this use case.
Coming from 2.2, the 2.3 series of pfSense is not exactly everything I’m looking for. I had to ‘factory reset’ the unit shortly after the 2.2 to 2.3 upgrade to avoid firewall rules displaying errors in the web configuration UI. As a personal irritation, the development team also took out the RRD-style graphs and replaced them with a “Monitoring” page, which I am not a fan of.
The Router rumble article, though, tested the UniFi Security Gateway but not the 3-port EdgeRouter Lite, which is my preferred option for users that need more capability than their ISP-provided modem/router combination. Jim did mention that they were both not up to routing gigabit from WAN to LAN, so I figured I’d see if I could replicate the results and if the ERL was any better than the USG.
Configuration and Setup
Following the posts, I configured two machines to act as client and server. Both were booted to Ubuntu 16.04.1 live USB sticks and had ‘apt-get update; apt-get upgrade’ run before any tests were performed. I also had to run “rm -rf /var/lib/apt/lists” to get apt to start working.
The “client” machine at 192.168. running the test script and the netdata graphing and collection system is a Core i7 4770K, 16GB RAM and a PCI-Express Intel 82574L gigabit network card.
The “server” machine with nginx and the sample files is a Lenovo X230, Core i5 3320M, 16GB RAM and an onboard Intel 82579LM gigabit NIC.
Some additional changes from the Ars Technica article are more suitable for my configuration and testing. On Ubuntu 16.04, the command to install ab and nginx should be apt-get install apache2-utils nginx (the ‘ab’ package doesn’t exist.) I made the same configuration changes to /etc/nginx/nginx.conf, /etc/default/nginx and /etc/sysctl.conf as suggested in the article:
# The key to high performance - have a lot of connections available
# Each connection needs a filehandle (or 2 if you are proxying)
# ... existing content
# ... existing content
# Note: You may want to look at the following page before setting the ULIMIT.
# Set the ulimit variable if you need defaults to change.
# Example: ULIMIT="-n 4096"
After cloning the Git repository and running the suggested install steps, you may also need to edit /etc/netdata/netdata.conf and add the following sections (replacing enp5s0 with your network interface from ifconfig) in order to get the same graphs:
You can download the test runs in a ZIP file, which contains the ‘ab’ output from the tests. Note that some of the graphs show a larger separation between the ab runs with different filesizes; this was due to different ‘sleep’ values being tested in the script.
Direct Connection (Auto MDI-X)
Many NICs support auto MDI-X, which allows a standard Ethernet cable to act like a crossover cable if both network cards support it. I ran a test with the server directly connected to the client and the graph appeared very cleanly.
Total Failed Requests
10K concurrency test only resulted in 308Mbit. Failed requests only in 10K concurrency test.
10K concurrency test only resulted in 417Mbit. Failed requests only in 10K concurrency test.
All tests had a similar speed. Failed requests only in 10K concurrency test.
With both systems connected to a Netgear GS108T switch, the graphs were fairly consistent with one unexplained valley in the 1MB/-c 100 test – but there were no failed requests to nginx noted in the ab results. This seemed to be a fluke; I wasn’t able to reproduce the problem in the exact same spot later. However, the valley did appear during other tests, lending suspicion that the GS108T may be causing a problem.
Total Failed Requests
10K concurrency test only resulted in 131Mbit. No failed requests in 10, 100 and 1000 concurrency tests.
10K concurrency test only resulted in 319Mbit. No failed requests in 10, 100 and 1000 concurrency tests.
All tests had a similar speed. Failed requests only on 1000 and 10K concurrency tests.
The ERL was flashed with 1.9.0 firmware and configured using the “Basic Setup” wizard, which sets configuration back to default values. The eth0 port acts as the WAN interface and provides NAT to the eth1 (LAN) interface. The wizard also configures some default firewall rules. I set up the WAN interface with a static IP of 192.168.99.2, and the laptop at 192.168.99.101 was plugged into eth0. The LAN interface (eth1) had an IP range of 192.168.1.1/24 and provided an IP via DHCP to the desktop. The resulting config.boot file is also available for inspection.
Unfortunately the scale and size of this image is slightly off from the direct switched test, but the peaks and dips in the graph should be sufficient to demonstrate the differences in performance. We can see that the 10KB test is particularly brutal on the EdgeRouter Lite, with speeds topping out at about 215Mbit/s. The 100KB test is slightly better in terms of bandwidth, with the lowest test result at 626.82Mbit, but the top of the graph is not smooth on each test. Finally, the ERL with this firmware pulls out a great performance on the 1MB test, with only the last 10K concurrency run showing a few dips in the graph; the lowest result from ab sits at 904.73Mbit.
Total Failed Requests
10K concurrency test was especially terrible at 51.25Mbit/s. No failed requests in 10, 100 and 1000 concurrency tests.
10K concurrency test only resulted in 626.82Mbit/s. Failed requests in 1000 (3) and 10K (45) concurrency tests.
10K concurrency test failed more requests than completed.
Followup and Further Testing
These test runs raised some additional questions. For now, it convinced me to not immediately run out and get an EdgeRouter Pro, since according to these results, at 100KB to 1MB filesizes I’d still be able to utilize my full download bandwidth on an ERL. What I really need to do is pull my pfSense box out of line and run it through this test scenario to compare it directly to the EdgeRouter Lite and a direct connection.
Performance and Bandwidth
I am surprised at the performance difference between the Ars tests of the UniFi Security Gateway and the EdgeRouter Lite in this configuration. Since they have similar specs (512MB RAM, promised 1 million packets per second at 64 bytes, promised line rate at >=512 byte packets), I would expect to see similar results. I’m wondering whether the USG was not using Cavium hardware offload support or if there were significant changes in the 1.9.0 firmware from the tested 1.8.5 version.
The 100KB test in all configurations had its average bandwidth brought down significantly by the 10K concurrency run. It is not very clear what the ‘receive’ and ‘exceptions’ fields in the ab output indicate, but I suspect these are contributing factors. During further testing I would be curious to find out if there is a concurrency parameter between 1000 and 10,000 that would result in no errors in the output.
The 1MB/10K concurrency test through the ERL, while it returned >900Mbit in throughput, failed more HTTP requests than it completed. What is interesting is that there is nothing in the nginx error log on the laptop to indicate a failed response on the server side, and a brief packet capture didn’t return any non-200 status codes for responses.
Tweaking and Tuning the Test
sysctl parameters could likely use some additional tweaking for the two systems. The original Ars article didn’t document each option and while I trust Jim’s parameters, there may be something more we can do with the 16GB of RAM in the test clients.
Consider changing the nginx web root where the .jpg files are stored to a ramdisk, to avoid the risk of the webserver process having to repeatedly read from the SSD. Of course, nginx may already be caching these files in memory; I could look at iotop during the ab run to see what disk access patterns look like.
Consider if there is a better way to simulate NNTP and BitTorrent downloads rather than HTTP traffic, because that’s really what people are doing with gigabit-to-the-home on the downstream end. NNTP traffic, for example, generally looks like TLS inside TCP. For most copyright-infringing purposes, also requires the client to reassemble yEncoded chunks – so there is a CPU impact on the client that is not necessarily present with straight TCP + HTTP. It would be interesting to come up with a “minimum system requirement” to be able to download and reasonably process NNTP data at 1000Mbit line rate.
Consider varying contents of the data in each file downloaded – that is, a performant enough server should be able to spew out different data content
The netdata graphs presented in the latest Ars article do not seem to match mine with respect to width of each segment. Given that the filesizes are changing during each test (so obviously there will be more data and packets transferred in the 1MB test, which will take more time on the horizontal axis), I’m curious as to what causes this difference.
I have concerns about the GS108T and whether it is causing drops during the testing; I’ll have to bring in several switches and re-run the tests.
Unrelated, but I also happened to notice the netdata statistics were indicating TCP errors and handshakes when the desktop was plugged into a different switch on my main home network segment, despite ethtool and ifconfig not indicating any issues on the interface. This concerns me; I’m wondering if there is a misbehaving device on the LAN and if I can isolate it with packet captures or unplugging sections of the network until the problems disappear.
Scenario: A migration batch was partially successful (one or more mailboxes in the batch migrated properly). The errors for the remaining mailboxes have been corrected. I’d like to start a new migration batch containing the failed mailboxes, but the batch bombs out with an email to the Exchange Online administrator. The batch online looks like it’s still migrating, but the CSV with the results that was emailed contains the following error messages for each account:
The user "firstname.lastname@example.org" is already included in migration batch "My Migration Batch Name." Please remove the user from any other batch and try again.
In this case you need to remove user from migration batch using the Remove-MigrationUser cmdlet when connected to the Exchange Online PowerShell session:
Get the details of all users in migration batches, or get the details for the specific user being migrated:
I have an Ubuntu 16.04 desktop installation with Unity and wanted to try KDE, so I ran sudo apt-get install kubuntu-desktop. apt failed with the following message:
trying to overwrite '/usr/share/accounts/services/google-im.service', which is also in package account-plugin-google [...]
The original issue at Ask Ubuntu has several suggestions but none of them worked – any apt commands returned the same requirement to run apt-get -f install, which in turn gave the original “trying to overwrite” error message. synaptic also wasn’t installed so I couldn’t use it (or install it, as all other apt installation commands failed.)
I was able to get the dpkg database out of its bad state and continue to install kubuntu-desktop by running the following: