First cruise review: Norwegian Epic March 1, 2015 – Western Caribbean

This has been sitting in my drafts folder since mid-2015, so the information in it re: UBP charges, menu contents, etc. are all outdated at this point, but I think it’s still a good representation of how our first cruise went. Suffice it to say, I need to be more timely with these – we’ve since gone on a Bahamas cruise on the NCL Breakaway in December 2016, and have a different Caribbean cruise planned on the Epic during December 2017. So even if there are some negatives here, clearly we’ve gone back so it can’t be that bad.

 

I’ll try and note in the article where things have changed.

In Which We Decide To Cruise

In 2015, Kayla and I got tired of the snow and wind and ridiculously cold temperatures in Southwestern Ontario, and decided to get away to a warmer climate. We’d been to BlueBay Villas Doradas in the Dominican Republic to do the all-inclusive resort trip with a few friends last year and while we liked it, we wanted to try something new before falling back to the same thing. We’d also discussed various Sunwing-promoted destinations flying directly out of YKF to Mexico, but the available resorts in our price range were either too new to have a decent amount of feedback, or had recently begun “focusing on a new concept.”

Then the idea of taking a cruise came up and we started looking. A few years ago in a hotel room in Prague, I’d seen a documentary that was pretty much an expose of the entire cruise ship industry. I managed to locate it after returning – it’s CNBC’s “Cruise Inc: Big Money on the High Seas” (2009).  It went over a cruise on the Norwegian Pearl, describing how passengers are basically just walking ATMs and that the cruise line is constantly running the numbers on every aspect of shipboard operations. The conclusion was that on the last sea day, the operator broke even for the cruise, in no small part to making up $21K in alcohol sales. Being shaken up and down for cash constantly didn’t really appeal to me.

Enter a combination of a Norwegian Cruise Line and an Expedia for TD 4x points promo. First, note that cruises for Expedia for TD are booked over the phone, but you still get the highest point multiplier as if you had booked online, and there’s no jerk fee to book over the phone unlike with other types of vacation. I’d recommend checking out the regular Expedia.ca site for a price range based on room category, then adding another 13-18% for taxes and fees. Both NCL (when called directly) and the Expedia for TD agent ended up quoting the exact same price.

The March 1/2015 sailing of the Epic also had an option that if you booked your stateroom in at least a balcony class (the previous tiers are “interior” and “oceanview”, and I took interior to mean “would feel like a third class room on the Titanic”), you could pick either $300US of onboard credit, a dining package involving access to theoretically better a la carte restaurants, or what was called the Ultimate Beverage Package(NCL Epic does not actually have “oceanview” rooms, it goes from inside to balcony to mini-suite.)

Valued at $54US per person per day (at the time; I saw it listed as $59 per day on the ship) the idea of the UBP is that you’d get most booze free, with a few exceptions on the really premium stuff. Given that the cheapest beer bottles are $5.75 each, and you can easily order an $11 drink without trying, it’s not too difficult to get your value out of this choice. I would highly recommend getting your travel agent or NCL to include the UBP as a complimentary option.

(As of March 2017 this package is now $79US/day to buy outright, which is just absurd. It ends up being worth avoiding any “Canadian at par” or “Guarantee cabin” offers that don’t include the UBP, if you are at all interested in drinks.)

Here’s an example liquor menu from the Epic, which I’m spoiling now in advance of the main review content because it seems to be one of the most requested things online. Basically you want to order beers, mixed drinks and wine by the glass and avoid the “super-premium” liquors.

So, off we went. We flew into Miami directly rather than messing around trying to get a transfer from Fort Lauderdale, stayed over at the Marriott Residence Inn on the previous night, then caught a shuttle to the Port of Miami on the morning of the cruise.

At this point I must impress upon readers that nothing really went wrong and that I’m impressed with our NCL and Epic experience. For never having taken a cruise before, the experience here was definitely what I wanted out of a vacation, and I can somewhat understand the CruiseCritic members who count down the hours until their next sailing in their forum signature. My criticisms are minor dull spots in what I would consider a 95% positive experience. I don’t see a reason to pick any other cruise line at this point and I do have every reason to keep choosing to cruise as a vacation.

Day 1: Embarkation Day and Ultimate Beverage Package

After arriving at the port of Miami, we got out of the shuttle and handed our suitcases over to a baggage handler, who will be sure to let you know that they would very gladly appreciate a tip. We then went into the terminal and waited in a fairly quick moving line to provide ID, checkin documentation and (most importantly) a credit card. Each person in your party gets issued room keycards with a mag stripe and barcode, and if you’ve purchased the UBP you get a sticker on the card indicating as much. The card acts like your passport while on board and in the ports, and even to the extent that we were told it would be a good idea to keep your passport in your room safe, separate from the keycard.

keycard

Of course, the first thing I did after checking out the room was try to get a drink from the overworked bar staff. The slightly nasty surprise was that while than still docked or less than 3 miles from the port of Miami, you get charged sales tax on booze. It ended up being a matter of us paying 56 to 76 cents out of pocket per drink before hitting international waters, but it was unexpected and wasn’t explicitly disclosed in the UBP terms other than “your check may reflect applicable VAT for certain ports or itineraries.” It would have been a much better experience to be given a heads up on “if you hit the bar before we leave Miami…” when we received our room cards.

What else can I say about the UBP? There was a bit more documentation in our room by the end of the first day, but from the practical experience of using it: There is an 18% autogratuity on all drink purchases, but it is included in the package.

(March 2017: You actually pay this at booking time, it’s considered 18% per person on the retail price of the package, so about $200US for a 7-day cruise for 4 people.)

You can order something with a base cost of up to $11 (so if your receipt total is $12.98 or under per drink, there’s nothing you pay out of pocket, and it won’t show up on your stateroom charges.) You should always pick up or ask for a bar menu to be sure, and also to make sure that you take full advantage of the variety of spirits and beer.

(March 2017: drinks are now comped up to $15US up from $11US, and the “super premium” liquors are all slightly over $15.)

In practice, despite the paper documentation in the stateroom indicating that two drinks per patron is allowed, none of the bars would allow you to order doubles or two drinks at a time. The best way we found to handle the situation was to present two room cards when ordering two drinks; then the bartender appeared to have discretion as to serving two drinks at a time.

Perhaps the most irritating matter is that you are expected to sign for all booze even if it will be zero rated by the UBP, so you end up striking out the “additional tip” and total lines, ignoring the “print name” line, and scrawling a signature that may or may not be yours (if a drink was put on your spouse’s account.) The receipt process really adds time and inconvenience to the whole experience, given that a large number of cruisers on our voyage had clearly taken NCL up on their free drinks offer and the bartenders were quite busy.

(As of March 2017, receipts were issued much less frequently and typically only if you had something chargeable, so this has improved. We also had fewer issues with “I’m getting a drink for my wife” without showing two cards on the Breakaway.)

This is a slightly redacted example of a bar receipt that you still get with the UBP. The drink in question was vodka and Sprite.
This is a slightly redacted example of a bar receipt that you still get with the UBP. The drink in question was vodka and Sprite.

Our bags weren’t delivered to our stateroom until later on the first day, so you’ll want your carry-ons – really, personal items – to have the essentials. It was quite nice to have a shower right away but I didn’t have new clothes to change into immediately afterwards.

We did a tour of the ship at 2pm, which involved a large group of people traipsing around decks, being told the location of every bar and pay-for restaurant, and concluding with me deciding next time I’d skip the tour and look around at my own pace. At 3:30 there was an obligatory emergency drill, which involved sitting beside a bunch of Canadians and deciding that under no circumstances would the other people half-assing around be any help if the ship sank.

On the first evening we ran into Park West, the ship art auctioneers, who are infamous in this story. They embodied the negative stereotype of “used car salesman” through and through. We decided to attend their art auction the next day as they did have some moderately interesting pieces on display.

I don’t know that I remember much of the rest of the first night, but I will say that for the first two days on the Epic I was absurdly hungry at various points, when there was really no need to be. The standard restaurants are basically closed between 3 and 5:30pm, and O’Sheehan’s (midship 24×7 restaurant and bar) has two areas to it – if you’re not seated in the restaurant, no food. Your best option with the Epic during the day is to head up to deck 15 and see what’s available at the outdoors buffet. I ate incredibly well once finding that out.

Day 2: Sea Day 1

Woke up, skipped breakfast. There is a moderate amount of noise in the hall from the cruise director and captain’s announcements around 9am, so if you want to sleep in make use of earplugs or several pillows. We ended up getting out of bed at 10:55, just barely making the start of the cruisecritic.com forum meet and mingle.

Unfortunately the Meet and Mingle event was a bust for us. Everyone had split off into groups, and the discussions focused on secretive insider-y “Posh Passes” that people had purchased the previous day. While the folks from the forums were helpful pre-cruise, you’ll want to be an extreme extrovert or a cruise regular if you want to get anything out of the scheduled event.

(March 2017: This is really cruise by cruise, and I’ve since had a better experience with a meetup on the NCL Breakaway.)

Next up, Kayla wanted to go to a shopping tutorial/seminar/meeting at noon by Linda and her mostly silent or surly partner Albert, which was supposed to be packed full of secrets and deals when you went to the various ports of call. This was a giant waste of time and I could have spent it trying to acquire lunch instead of being hungry/angry later in the afternoon.

Linda waxed on the wonderful melanin generating properties of some “sleep band” that purportedly cured some woman of Parkinson’s symptoms (lies), told us to ask for specific people at each store to get us the “best deal”, absolutely shilled out for Diamonds International, expressed what an awesome deal we were getting because we weren’t paying tax and duty, and in general made me feel like an idiot.

Ship Hallway
One of the ship’s hallways near guest rooms.

 

The one thing we did get out of her presentation was to go to Gold and Time in Ocho Rios, Jamaica and get a free gemstone for showing the shop map. The free gemstone is topaz, and they really use it as an upsell offer to earrings and a pendant (which both actually look quite nice.) You can now all do the same without burning an hour of time in the theatre with Linda and having homeopathic “natural frequency” bracelets shoved down your gullets.

Next event on the ship was the Park West art auction. This took place from 1:30 to 3pm and involved primarily pieces from Peter Max. For those of you who don’t know, Peter Max painted murals for Woodstock and produced a whack of other American pop art. If I never hear about Peter Max being the voice of a generation again it will be too soon. There was also a contest where you could win a print for its shipping cost – $55 to Canada.

Again I was hungry so I don’t remember much of the rest of the afternoon. I do recall that getting to the sit-down restaurants (Manhattan Club and Taste) close to the start of service got you a better table. Kayla though O’Sheehan’s mid day appetizers were gross. I didn’t mind the 24/7 menu from the night prior, but the fatty chicken wings she got that afternoon were straight out of a utility grade foodservice bucket.

(March 2017: the wing quality was definitely improved on Breakaway.)

Monday night we saw Blue Man Group at 10pm, and arrived about 30 minutes early to get reasonable seats in the second row. There were a bunch of wiener kids in the first row that squirmed throughout the show and could have calmed down a bit. Despite this, both of us really enjoyed the performance and we would recommend it for anyone, especially if you’re not sure what to expect.

Day 3: Sea Day 2

The second sea day continued our relationship with Park West, who at this time had decided we might be interested in buying art and kept sending invitations to our cabin. Kayla won the “free” print, which was actually decent, but really none of the other stuff appealed to us. To get into the category of things we liked, it was in the range of $1100 per piece. There was also incessant badgering about the following topics:

  • Peter Max being highly collectible, how many famous people own his work, and the sheer amount of stuff that he’s painted. Voice of a generation, etcetera. Still not impressed.
  • Why you should pay close to $20K for a Rembrandt. Also, it’s really an ink pressing of an etching that Park West owns, which means they control the supply. Every time they mentioned that this was an etching I heard the Buzz Killington “etching” cutaway from Family Guy.
  • The child prodigy Autumn DeForest, whose art is allegedly so in demand that she can’t paint it fast enough. Why? Because she has to go to school. Also, the art is good but not great.

The other memorable point of the day was attending the Legends in Concert show in the Epic Theatre. Online reviews and NCL propaganda indicated that this was a show not to miss. I’ll save you the trouble: you could probably miss it.

At the theatre bar, I ordered a Crown and ginger ale for myself, and tried to order a coffee for Kayla. The server gave me the stink-eye and said that coffee wasn’t included in the UBP, and that it fell into the same category of fresh-squeezed fruit juice and energy drinks that were specifically excluded. This is despite the fact that coffee was freely available in our room, in the 15th deck buffet, and also by the poolside. This was really the only time onboard that I felt cheated. For anyone reading from NCL, this was the single thing that stuck out at me as really poor form.

(March 2017: This still seems to be the case. Solution? Order a coffee with booze in it.)

The best part of the show was the third performer, who did an awesome Aretha Franklin tribute. Not having heard enough Jimmy Buffett, I don’t know if the performer was spot on or just sounded like an elderly man trying to cash in on prior fame – maybe that’s actually what you’re supposed to expect. The Adele impersonator was not great. She wasn’t hitting the necessary vocal range and made a whole bunch of cockney/British jokes that fell flat. It was a good thing Aretha closed the show out because otherwise I’d have considered it a total loss.

Having now had meals at both Taste and the Manhattan Room, I feel like I’m qualified to say that the reviewers on Yelp are morons – the food comes from the exact same place for both of these restaurants, and going to one dining room over another does not change the quality. We had mediocre to good service in both of these places and didn’t ever have to wait more than a minute for a table for two. The exact opposite could be said for the Garden Cafe, where you had to prowl the deck for a table and then get food in alternate shifts lest your newly-acquired food be cleared away.

Day 4: Ochos Rios, Jamaica

Before I detail the ports of call, full disclosure: we booked all three days through NCL’s online portal before leaving, taking their recommended tours and excursions rather than trying to organize our own itinerary. With these excursions you’re realistically in for at least $100US per person per day, but there were some definite advantages that I’ll get into shortly.

Beginning the ports of call was Jamaica, which seemed to be organized well. You exited the ship through deck 4 and it was pretty clear where to go. We met up with our tour group, and took a Toyota Coaster to a tourist-centric plantation. We then boarded a wagon hitched to a Massey Ferguson tractor and got driven around the grounds, stopping in various places to see animals, a tree climbing and coconut de-husking demonstration, and the many artifacts from Pierre Trudeau and family. Apparently they like the Trudeaus pretty well in Ochos Rios.

The main portion of the plantation excursion involved taking a camel ride. If our guides are to be believed, there are only eight camels in Jamaica and all of them are at the plantation. Only five camels were actively giving rides, and with a two person capacity, we waited around for 20 minutes while the first group of ten plodded around. When our turn came around it was pretty fun; the camel was like riding a slightly less stable horse and the animal had a singular focus on eating vegetation.

Cindy the camel wanted nothing more than to chomp leaves all day.
Cindy the camel wanted nothing more than to eat leaves all day. I did not get chomped.

In the afternoon, we were escorted to Dunn’s River Falls and opted to climb them. One of the tour organizers seemed to have a rough time getting the necessary number of admission wristbands for our group, and I felt like the entire excursion was rushed because of it. You’ll need to rent a locker for $10 ($3 refund on key/receipt return) because you can’t have anything in your hands, and a backpack would also not be suitable. Once we did get going, the climb was a great experience. I would also definitely recommend buying water shoes before the trip.

Also keep in mind that upon exit you’ll have to run the gauntlet of fairly aggressive peddlers trying to sell Jamaican souvenirs before making your way back to the tour bus. Props to our driver and one of the other tour guides for getting us out of a parking space, with the dialog between the guide and another driver sounding very similar to this GTA IV clip of Little Jacob:

Day 5: Grand Cayman

In Grand Cayman, we chose the Sunken Ship Snorkel & Tiki Beach excursion, and were called to the Epic Theatre for 8:20am. Of the three ports here, Grand Cayman is one where the ship doesn’t pull up to the pier directly, and uses the ship’s lifeboats to “tender” passengers to and from shore. I had heard some disastrous reviews about this process from previous sailings and was prepared for a fiasco.

After waiting about 30 minutes in the theatre, our group was called to load up a lifeboat – looking down one deck, we managed to skip a large line of people without NCL excursions that had either set up something on their own or just wanted to go ashore. Based on the onboard announcements, there was a swell interfering with loading from one side of the ship, and I’d overheard staff conversations that some people were really angry. Therefore, my recommendation is to book an NCL-managed excursion for any port where tendering is involved, even only if for the priority boarding.

A view of the ship from the shore in Grand Cayman.
A view of the ship from the shore in Grand Cayman. Note the lifeboats on the side used for transporting passengers to and from shore.

Also keep in mind that there are tours with very similar names: the “Reef & Wreck Snorkel” excursion is not the same as the “Sunken Ship” snorkel, despite the physical sunken ship being the same for both.

I’m not a big snorkeling person myself, but it was pretty neat seeing all the fish surrounding the sunken aircraft carrier. Other than that, the snorkeling was pretty average – if you’re really into it, I could see this being a better experience. We did use the flippers on boat but brought our own masks and air tubes – the communal equipment just goes into a large rubber trash can and I’m not sure how often it gets sanitized. Going from the large boat to a smaller one also was a bit of a change; we were glad to get off by the time we’d arrived back at shore as both of us had started feeling a bit queasy.

The afternoon Tiki Beach experience was also underwhelming for me. You get a complimentary rum punch on arrival, then you locate a beach lounger. Everything else is at additional cost, and pricing was in Cayman Islands Dollars in what I can only assume is a deliberate attempt to obscure the exchange rate to USD. If you do choose this excursion, I’d suggest bringing along snacks, as well as headphones, music and a book. I was fairly bored in the afternoon and sitting in the sun wasn’t entirely thrilling.

Days 6 and 7: Cozumel, Mexico and Sea Day 3

Most of this review ended up being written initially on the third sea day, which was the last whole day of the cruise, trying to remember most of the onboard experiences. On Day 6, we attended a “Salsa and Salsa” class in Cozumel, which had delicious food, tons of authentic tequila and a reasonable amount of dancing (at the end, when your inhibitions are lowered from the booze.) It is a group-style class so you share your table and ingredients with others. There was wifi in the hotel lobby where the class was offered – you just had to ask the desk representative for the credentials.

The last sea day basically involved me milling around the ship bars and Kayla reading/decompressing. I also paid about $4US for an excellent Singapore Noodles at Shanghai’s Noodle Bar (as of March 2017, this is a complimentary restaurant and gets very busy.) I walked around all the decks and ended up settling on Shaker’s Martini Lounge, occasionally ordering a drink and bar snacks, and trying to figure out when we could schedule our next cruise.

Return to Miami

Probably the least fun part of our cruse, we returned back to Miami at early o’clock and quickly got a taxi to the airport. Unfortunately due to price and the repeated suggestion to not book anything too early in case the ship came in late, we had nearly twelve hours to kill at MIA waiting for our flight. We were quickly bored and I would definitely suggest picking an afternoon flight, or buying an airport lounge pass where you can just kill time and perhaps have some beverages.

So, if you’re willing to deal with the occasional service charge and are willing to just “go with the flow”, I think our first NCL experience was quite decent.

Fix: WSUS Server Cleanup Wizard hangs/stalls when deleting unused updates

Side note: several years ago Kayla caught me talking in my sleep, muttering something about “you’ve got to check the boxes!” This is the actual dialog and process in question.

Full credit to Jeremy Jameson at MSDN. Posting in case the original disappears.

  • Run Server Cleanup Wizard with only the “Unused updates and update revisions” (option #1) box checked. This took about six hours on the server experiencing the problem:screen-shot-2016-11-22-at-11-32-29-am
  • Once finished, run the wizard again with only the “Unneeded update files” (option #3) box checked.
  • Once that’s finished, run the wizard with all the boxes checked.

RiteBite and Invisalign Review: Conclusion

Well, better late than never, but I’m currently in the process of cleaning up paperwork in the home office, and noted that RiteBite had given me a flyer asking for a Google review several months ago. So here’s a conclusion to the review series, which will be combined with the other content and sliced into bits and pieces for the less-verbose social media pages.

Completing the Program

Since last time I wrote, I went through about half of another series of trays with 7-day rotations. I specifically requested to have my treatment wrapped up about a week before my wedding in August 2016, and Dr. Luis and staff were very accommodating since this third set was effectively “finishing touches”. As part of the removal, I had permanent wiring bonded behind both my top and bottom teeth and was given a set of top and bottom harder, clear plastic retainers to wear overnight. One important point is that for the first two weeks, you’re expected to wear the retainers as close to 24/7 as possible, so you’re not “entirely” done. I obviously made an exception to this for the wedding.

A Few Nitpicks

The retainers are not ideal, to put a point on it. Their larger size (compared to the Invisalign trays) and increased rigidity triggers my gag reflex nearly every morning when taking them out, and I still run into similar problems with drooling on my pillow.

I also specifically requested the top permanent wire, and had to ask several times before getting a “yes” – several staff suggested that it wasn’t strictly necessary or had a higher chance of breaking. I wanted to ensure that with my financial investment, there was a “backup” in place to help the teeth from moving as much. The top wire’s presence is still noticeable when I close my mouth several months later, unlike the bottom wire. Both still have a distinct “pebbled” texture where the wire is adhered to the back of each row of teeth.

Despite asking for Google reviews as part of the “exit interview”, RiteBite seems to have several accounts under their name on Google Plus (1, 2, 3, 4, 5) and no link to the official Google profile from their website, nor any content on these pages. I was also disappointed to find that the Case Graphics section has disappeared from my profile since completing treatment.

Overall Results

The change has been quite impressive. It took slightly over a year and a half, I wasn’t seriously inconvenienced, and now that it has been paid off, I begrudgingly admit that it was probably a better personal choice than replacing the laminate flooring in the house or buying the same amount of networking gear.

Continuing the “Router rumble” with pfSense 2.3.2 and a FW-7540

Following up from my previous round of router testing, I managed to get a spare Lanner FW-7540 with an Intel Atom D525 CPU to test how my current pfSense 2.3.2 setup compared to an EdgeRouter Lite. The results were well below what I was expecting: the pfSense box topped out at 490Mbit in the 1MB test and was very spiky when looking at the netdata graphs.

The results file is also available if you’d like to look directly at the ab output.

d525_pfsense

Filesize Average Mbit/s Total Failed Requests Notes
10K 145.07 87 10K concurrency test only resulted in 49Mbit. No failed requests in 10, 100 and 1000 concurrency tests.
100K 421.71 4896 No failed requests in 10, 100 and 1000 concurrency tests.
1MB 489.96 3341 No failed requests in 10, 100 and 1000 concurrency tests.

This test fairly obviously shows a ceiling. For WAN connections of over 500Mbit, it looks like something beefier than an Atom D525 is necessary to run the NAT as anticipated.

I also ran some more informal WAN to LAN iPerf3 testing on direct connection (MDI-X), the EdgeRouter Lite and the pfSense/7540 combination to get some synthetic numbers:

Connection iPerf Result
Direct 941Mbit with no retries
EdgeRouter Lite 939Mbit with retries
pfSense/7540 829Mbit with no retries

Given how well the EdgeRouter Lite seems to perform for its price, and since it beats out the more general purpose hardware, I suspect I will be swapping out for an ERL or ER-Pro very shortly.

Replicating the Ars Technica “Router rumble” with a Ubiquiti EdgeRouter Lite

A friend and colleague of mine (Matt) and I have an ongoing discussion about over-specced gear for our home networks. Our core routers have been FW-7540s running pfSense (Atom D525, 4GB RAM, 4 Intel NICs) since 2013. pfSense offers a huge advantage over commercial-grade routers – I run dual WAN with failover based on ping, link, and packet loss, have extremely customizable DNS and DHCP, and can set up an OpenVPN server in just a few minutes. Matt and I also recently have had 500Mbit+ downstream connections installed, so it’d be good to know what hardware and software combination is “for sure” capable of utilizing the full pipe.

There have been a series of excellent articles at Ars Technica this year by Jim Salter that constantly get mentioned in our discussions:

The first two initial articles were mildly interesting – we do plenty of Linux-based routing at the office, but I don’t really want to build a router from scratch at home if there is a distribution that works as well. The results in Jim’s latest Router rumble article with pfSense 2.3.1 and the homebrew Celeron J1900 were described as “tweaky” and didn’t seem to hold up against the homebrew variant running Linux. I found this a bit odd because FreeBSD is widely assumed to have a hardened, robust and performant network stack; the general impression amongst networking folks I’ve talked to that Linux isn’t quite as good for this use case.

Coming from 2.2, the 2.3 series of pfSense is not exactly everything I’m looking for. I had to ‘factory reset’ the unit shortly after the 2.2 to 2.3 upgrade to avoid firewall rules displaying errors in the web configuration UI. As a personal irritation, the development team also took out the RRD-style graphs and replaced them with a “Monitoring” page, which I am not a fan of.

screen-shot-2016-10-01-at-6-36-22-pm

The Router rumble article, though, tested the UniFi Security Gateway but not the 3-port EdgeRouter Lite, which is my preferred option for users that need more capability than their ISP-provided modem/router combination. Jim did mention that they were both not up to routing gigabit from WAN to LAN, so I figured I’d see if I could replicate the results and if the ERL was any better than the USG.

Configuration and Setup

Following the posts, I configured two machines to act as client and server. Both were booted to Ubuntu 16.04.1 live USB sticks and had ‘apt-get update; apt-get upgrade’ run before any tests were performed. I also had to run “rm -rf /var/lib/apt/lists” to get apt to start working.

  • The “client” machine at 192.168. running the test script and the netdata graphing and collection system is a Core i7 4770K, 16GB RAM and a PCI-Express Intel 82574L gigabit network card.
  • The “server” machine with nginx and the sample files is a Lenovo X230, Core i5 3320M, 16GB RAM and an onboard Intel 82579LM gigabit NIC.

Some additional changes from the Ars Technica article are more suitable for my configuration and testing. On Ubuntu 16.04, the command to install ab and nginx should be apt-get install apache2-utils nginx (the ‘ab’ package doesn’t exist.) I made the same configuration changes to /etc/nginx/nginx.conf, /etc/default/nginx and /etc/sysctl.conf as suggested in the article:

/etc/nginx/nginx.conf

events {
    # The key to high performance - have a lot of connections available
    worker_connections  19000;
}

# Each connection needs a filehandle (or 2 if you are proxying)
worker_rlimit_nofile    20000;

http {
  # ... existing content
  keepalive_requests 0;
  # ... existing content
}

/etc/default/nginx

# Note: You may want to look at the following page before setting the ULIMIT.
#  http://wiki.nginx.org/CoreModule#worker_rlimit_nofile
# Set the ulimit variable if you need defaults to change.
#  Example: ULIMIT="-n 4096"
ULIMIT="-n 65535"

/etc/sysctl.conf

kernel.sem = 250 256000 100 1024
net.ipv4.ip_local_port_range = 1024 65000
net.core.rmem_default = 4194304
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 262144
net.ipv4.tcp_wmem = 262144 262144 262144
net.ipv4.tcp_rmem = 4194304 4194304 4194304
net.ipv4.tcp_max_syn_backlog = 4096
net.ipv4.tcp_mem = 1440715	2027622	3041430

The testing script was modified to use a -s 20 parameter as indicated in the latest article, as well as sleeping for 10 and 20 seconds at appropriate times to distinguish each test run in the graphs:

test.sh

#!/bin/bash
mkdir -p ~/tests
mkdir -p ~/tests/$1
ulimit -n 100000

ab -rt180 -c10 -s 20 http://192.168.99.101/10K.jpg 2>&1 | tee ~/tests/$1/$1-10K-ab-t180-c10-client-on-LAN.txt; sleep 10
ab -rt180 -c100 -s 20 http://192.168.99.101/10K.jpg 2>&1 | tee ~/tests/$1/$1-10K-ab-t180-c100-client-on-LAN.txt; sleep 10
ab -rt180 -c1000 -s 20 http://192.168.99.101/10K.jpg 2>&1 | tee ~/tests/$1/$1-10K-ab-t180-c1000-client-on-LAN.txt; sleep 10
ab -rt180 -c10000 -s 20 http://192.168.99.101/10K.jpg 2>&1 | tee ~/tests/$1/$1-10K-ab-t180-c10000-client-on-LAN.txt
sleep 20
ab -rt180 -c10 -s 20 http://192.168.99.101/100K.jpg 2>&1 | tee ~/tests/$1/$1-100K-ab-t180-c10-client-on-LAN.txt; sleep 10
ab -rt180 -c100 -s 20 http://192.168.99.101/100K.jpg 2>&1 | tee ~/tests/$1/$1-100K-ab-t180-c100-client-on-LAN.txt; sleep 10
ab -rt180 -c1000 -s 20 http://192.168.99.101/100K.jpg 2>&1 | tee ~/tests/$1/$1-100K-ab-t180-c1000-client-on-LAN.txt; sleep 10
ab -rt180 -c10000 -s 20 http://192.168.99.101/100K.jpg 2>&1 | tee ~/tests/$1/$1-100K-ab-t180-c10000-client-on-LAN.txt
sleep 20
ab -rt180 -c10 -s 20 http://192.168.99.101/1M.jpg 2>&1 | tee ~/tests/$1/$1-1M-ab-t180-c10-client-on-LAN.txt; sleep 10
ab -rt180 -c100 -s 20 http://192.168.99.101/1M.jpg 2>&1 | tee ~/tests/$1/$1-1M-ab-t180-c100-client-on-LAN.txt; sleep 10
ab -rt180 -c1000 -s 20 http://192.168.99.101/1M.jpg 2>&1 | tee ~/tests/$1/$1-1M-ab-t180-c1000-client-on-LAN.txt; sleep 10
ab -rt180 -c10000 -s 20 http://192.168.99.101/1M.jpg 2>&1 | tee ~/tests/$1/$1-1M-ab-t180-c10000-client-on-LAN.txt

I also generated ‘JPEG’ files with /dev/urandom and placed them in /var/www/html (default nginx directory):

dd if=/dev/urandom of=/var/www/html/10K.jpg bs=1024 count=10
dd if=/dev/urandom of=/var/www/html/100K.jpg bs=1024 count=100
dd if=/dev/urandom of=/var/www/html/1M.jpg bs=1024 count=1024

Finally, installing netdata on the client needed a different set of dependencies (16.04 may have changed some of them):

sudo apt-get install zlib1g-dev uuid-dev libmnl-dev gcc make git autoconf libopts25-dev libopts25 autogen-doc automake pkg-config curl

After cloning the Git repository and running the suggested install steps, you may also need to edit /etc/netdata/netdata.conf and add the following sections (replacing enp5s0 with your network interface from ifconfig) in order to get the same graphs:

/etc/netdata/netdata.conf

[net.enp5s0]
  enabled = yes

[net_packets.enp5s0]
  enabled = yes

Results

You can download the test runs in a ZIP file, which contains the ‘ab’ output from the tests. Note that some of the graphs show a larger separation between the ab runs with different filesizes; this was due to different ‘sleep’ values being tested in the script.

Direct Connection (Auto MDI-X)

Many NICs support auto MDI-X, which allows a standard Ethernet cable to act like a crossover cable if both network cards support it. I ran a test with the server directly connected to the client and the graph appeared very cleanly.

direct_mdix

 

Filesize Average Mbit/s Total Failed Requests Notes
10KB 700.34 3117 10K concurrency test only resulted in 308Mbit. Failed requests only in 10K concurrency test.
100KB 785.03 3368 10K concurrency test only resulted in 417Mbit. Failed requests only in 10K concurrency test.
1MB 912.16 5533 All tests had a similar speed. Failed requests only in 10K concurrency test.

Switched Connection

With both systems connected to a Netgear GS108T switch, the graphs were fairly consistent with one unexplained valley in the 1MB/-c 100 test – but there were no failed requests to nginx noted in the ab results. This seemed to be a fluke; I wasn’t able to reproduce the problem in the exact same spot later. However, the valley did appear during other tests, lending suspicion that the GS108T may be causing a problem.

direct_x230

Filesize Average Mbit/s Total Failed Requests Notes
10KB 651.75 3939 10K concurrency test only resulted in 131Mbit. No failed requests in 10, 100 and 1000 concurrency tests.
100KB 760.61 1085 10K concurrency test only resulted in 319Mbit. No failed requests in 10, 100 and 1000 concurrency tests.
1MB 908.38 6690 All tests had a similar speed. Failed requests only on 1000 and 10K concurrency tests.

EdgeRouter Lite

The ERL was flashed with 1.9.0 firmware and configured using the “Basic Setup” wizard, which sets configuration back to default values. The eth0 port acts as the WAN interface and provides NAT to the eth1 (LAN) interface. The wizard also configures some default firewall rules. I set up the WAN interface with a static IP of 192.168.99.2, and the laptop at 192.168.99.101 was plugged into eth0. The LAN interface (eth1) had an IP range of 192.168.1.1/24 and provided an IP via DHCP to the desktop. The resulting config.boot file is also available for inspection.

erl_cropped

Unfortunately the scale and size of this image is slightly off from the direct switched test, but the peaks and dips in the graph should be sufficient to demonstrate the differences in performance. We can see that the 10KB test is particularly brutal on the EdgeRouter Lite, with speeds topping out at about 215Mbit/s. The 100KB test is slightly better in terms of bandwidth, with the lowest test result at 626.82Mbit, but the top of the graph is not smooth on each test. Finally, the ERL with this firmware pulls out a great performance on the 1MB test, with only the last 10K concurrency run showing a few dips in the graph; the lowest result from ab sits at 904.73Mbit.

Filesize Average Mbit/s Total Failed Requests Notes
10KB 153.81 55 10K concurrency test was especially terrible at 51.25Mbit/s. No failed requests in 10, 100 and 1000 concurrency tests.
100KB 800.28 48 10K concurrency test only resulted in 626.82Mbit/s. Failed requests in 1000 (3) and 10K (45) concurrency tests.
1MB 908.81 23723 10K concurrency test failed more requests than completed.

Followup and Further Testing

These test runs raised some additional questions. For now, it convinced me to not immediately run out and get an EdgeRouter Pro, since according to these results, at 100KB to 1MB filesizes I’d still be able to utilize my full download bandwidth on an ERL. What I really need to do is pull my pfSense box out of line and run it through this test scenario to compare it directly to the EdgeRouter Lite and a direct connection.

Performance and Bandwidth

  • I am surprised at the performance difference between the Ars tests of the UniFi Security Gateway and the EdgeRouter Lite in this configuration. Since they have similar specs (512MB RAM, promised 1 million packets per second at 64 bytes, promised line rate at >=512 byte packets), I would expect to see similar results. I’m wondering whether the USG was not using Cavium hardware offload support or if there were significant changes in the 1.9.0 firmware from the tested 1.8.5 version.
  • The 100KB test in all configurations had its average bandwidth brought down significantly by the 10K concurrency run.  It is not very clear what the ‘receive’ and ‘exceptions’ fields in the ab output indicate, but I suspect these are contributing factors. During further testing I would be curious to find out if there is a concurrency parameter between 1000 and 10,000 that would result in no errors in the output.
  • The 1MB/10K concurrency test through the ERL, while it returned >900Mbit in throughput, failed more HTTP requests than it completed. What is interesting is that there is nothing in the nginx error log on the laptop to indicate a failed response on the server side, and a brief packet capture didn’t return any non-200 status codes for responses.

Tweaking and Tuning the Test

  • sysctl parameters could likely use some additional tweaking for the two systems. The original Ars article didn’t document each option and while I trust Jim’s parameters, there may be something more we can do with the 16GB of RAM in the test clients.
  • Consider changing the nginx web root where the .jpg files are stored to a ramdisk, to avoid the risk of the webserver process having to repeatedly read from the SSD. Of course, nginx may already be caching these files in memory; I could look at iotop during the ab run to see what disk access patterns look like.
  • Consider if there is a better way to simulate NNTP and BitTorrent downloads rather than HTTP traffic, because that’s really what people are doing with gigabit-to-the-home on the downstream end. NNTP traffic, for example, generally looks like TLS inside TCP. For most copyright-infringing purposes, also requires the client to reassemble yEncoded chunks – so there is a CPU impact on the client that is not necessarily present with straight TCP + HTTP. It would be interesting to come up with a “minimum system requirement” to be able to download and reasonably process NNTP data at 1000Mbit line rate.
  • Consider varying contents of the data in each file downloaded – that is, a performant enough server should be able to spew out different data content

Outstanding Questions

  • The netdata graphs presented in the latest Ars article do not seem to match mine with respect to width of each segment. Given that the filesizes are changing during each test (so obviously there will be more data and packets transferred in the 1MB test, which will take more time on the horizontal axis), I’m curious as to what causes this difference.
  • I have concerns about the GS108T and whether it is causing drops during the testing; I’ll have to bring in several switches and re-run the tests.
  • Unrelated, but I also happened to notice the netdata statistics were indicating TCP errors and handshakes when the desktop was plugged into a different switch on my main home network segment, despite ethtool and ifconfig not indicating any issues on the interface. This concerns me; I’m wondering if there is a misbehaving device on the LAN and if I can isolate it with packet captures or unplugging sections of the network until the problems disappear.

Office 365 and Exchange Migration Notes

This post is a collection of my recent Windows/Exchange administrative work.

Run AD Directory Sync Manually (New Version of Start-OnlineCoexistenceSync)

Source: https://blogs.technet.microsoft.com/rmilne/2014/10/01/how-to-run-manual-dirsync-azure-active-directory-sync-updates/

Instructions:

Import-Module ADSync
Start-ADSyncSyncCycle -PolicyType Delta

or

Start-ADSyncSyncCycle -PolicyType Initial

How do I check total mailbox sizes for Office 365/Exchange Online mailboxes?

Source: https://community.spiceworks.com/how_to/93142-check-mailbox-size-and-usage-with-office-365-or-exchange-online-find-users-nearing-their-quota

Instructions:

# In PowerShell:
$LiveCred = Get-Credential
$Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://ps.outlook.com/powershell/ -Credential $LiveCred -Authentication Basic -AllowRedirection 
Import-PSSession $Session

get-mailbox | get-mailboxstatistics | ft displayname, totalitemsize 

# When done:
Remove-PSSession $Session

Error during migration: MigrationPermanentException: Cannot find a recipient that has mailbox GUID ” error message when you try to move a mailbox in an Exchange hybrid deployment

Source: https://support.microsoft.com/en-ca/kb/2956029

  • Ensure the local user object doesn’t have an exchange GUID. From the local Exchange Management Shell:
    Get-RemoteMailbox <MailboxName> | Format-List ExchangeGUID
  • Get the GUID from the error message, or retrieve it from the O365/Exchange Online shell (connect as above):
    Get-Mailbox <MailboxName> | Format-List ExchangeGUID
  • Set the exchange GUID for the user from the local Exchange Management Shell:
    Set-RemoteMailbox <MailboxName> -ExchangeGUID <ExchangeGUID>
  • Force directory sync. Using the latest Azure AD Connect commands, on the server with the directory sync tool installed:
    Import-Module ADSync
    Start-ADSyncSyncCycle -PolicyType Delta
  • Monitor with “Azure AD Connect Synchronization Service Manager” GUI application if needed.

 

Error during migration:  MigrationPermanentException: Mailbox size 12.56 GB ‎(13,489,367,463 bytes)‎ exceeds target quota 2.3 GB ‎(2,469,396,480 bytes)‎.

Source: http://andywolf.com/migrating-exchange-mailbox-from-another-forestmailbox-exceeds-target-quota/

  • If applicable to a single user, use ADSI Edit to set the “mDBUseDefaults” property to False on the applicable user object, then try again.
  • If database or organization-wide, use the Exchange Administrative Center to remove quotas for the database.

I have a migration batch that partially failed. Now I can’t get those mailboxes to migrate.

Sources:

Scenario: A migration batch was partially successful (one or more mailboxes in the batch migrated properly). The errors for the remaining mailboxes have been corrected. I’d like to start a new migration batch containing the failed mailboxes, but the batch bombs out with an email to the Exchange Online administrator. The batch online looks like it’s still migrating, but the CSV with the results that was emailed contains the following error messages for each account:

The user "user@example.com" is already included in migration batch "My Migration Batch Name."  Please remove the user from any other batch and try again.

In this case you need to remove user from migration batch using the Remove-MigrationUser cmdlet when connected to the Exchange Online PowerShell session:

  • Get the details of all users in migration batches, or get the details for the specific user being migrated:
    Get-MigrationUser
    Get-MigrationUser user@example.com
  • Remove the user from the migration batch. Use the additional -Force parameter if you aren’t running interactively.
    Remove-MigrationUser user@example.com
  • Clean up any migration batches that may still be in progress with the ‘already included’ error.
  • Create a new migration batch containing the affected mailboxes.

Fix: trying to overwrite ‘/usr/share/accounts/services/google-im.service’ installing kubuntu-desktop

I have an Ubuntu 16.04 desktop installation with Unity and wanted to try KDE, so I ran sudo apt-get install kubuntu-desktop. apt failed with the following message:

trying to overwrite '/usr/share/accounts/services/google-im.service', which is also in package account-plugin-google [...]

The original issue at Ask Ubuntu has several suggestions but none of them worked – any apt commands returned the same requirement to run apt-get -f install, which in turn gave the original “trying to overwrite” error message. synaptic also wasn’t installed so I couldn’t use it (or install it, as all other apt installation commands failed.)

I was able to get the dpkg database out of its bad state and continue to install kubuntu-desktop by running the following:

dpkg -P account-plugin-google unity-scope-gdrive
apt-get -f install

(Link to original Kubuntu bug for posterity: https://bugs.launchpad.net/kubuntu-ppa/+bug/1451728)

This post was cross-posted to The Linux Experiment, where I haven’t written anything for months.

RiteBite and Invisalign, just over a year in

I’m just over a year in since starting Invisalign treatment with RiteBite Orthodontics – and here’s how things stand.

Positive Experience

I want to reiterate that I’m quite pleased with the experience I’ve had with Dr. Luis and RiteBite. Everyone at the Waterloo office has been friendly, professional and my appointments have always started on time. I feel like Invisalign was definitely a better option over braces. Even under the perpetually ticking clock of their Terminal Services-hosted dental software, everyone that’s put their hands in my mouth has done a great job.

Don’t you just love the graphics?

One of the best improvements RiteBite has made since I signed up has been the addition of the Case Graphics / Patient Records section to their patient portal. Despite its dated “win a 4th-gen iPod” banner on the landing page, it has X-rays and full sets of mouth and jaw pictures from every appointment where the digital camera comes out.

These photos are perhaps the most convincing tool they could use to convince customers that money spent on orthodontic/Invisalign treatment is worth it. The progress made after just eight months of trays was phenomenal. Teeth are shifting into their proper positions and I have much higher confidence in a successful result.

Social Media Milling, aka Poisoning the Well

RiteBite’s Internet presence/social media strategy is intended to attract new customers. They have a decent website and the usual Twitter / Facebook / Pinterest / Instagram accounts. Current patients are enrolled in the Patient Rewards Program, where 10 points = $1 in gift card value, redeemable with a minimum 100 points.

ritebite2
Straight out of 2001, the RiteBite Rewards Hub.

At a typical appointment (6-8 weeks apart) you might get 2 points for “brushing after signing in”, another 2 for “being on time”, and 3 for “wearing appliance as instructed”. The higher point values in this program are designed to encourage social media interaction – a YouTube video testimonial will get you 20 points, and 10 points goes to the author of a Google Maps review.

Given these values, it’s a bit of a grind to make it to your $10/100 point Tim Card.

Online review and social media activity for RiteBite is inevitably going to skew on the positive side, because there’s a reward for doing so. As a cynical tech worker, I’m also highly allergic to anything like a “selfie contest”. Occasionally I’ll get an email promoting one and I scowl before remembering that a large portion of RiteBite’s patients are teenagers with nothing better to do than hashtag.

Full disclosure: I was credited with a whopping 250 points for referring a friend to the practice shortly after I signed up, but I have yet to exchange them for anything.

Align Technologies Inc.

Be aware that Align Technologies, Inc. is also very heavily involved in managing their online presence and regularly comps “mommy bloggers” with treatment either for themselves or their kids. You can usually find these disclosures at the bottom of the page or post in question in FTC-compliant language. These posts exclusively skew positively for Invisalign over other types of treatment, and hammer home the main marketing points (can remove trays, easy to use, comparable in cost to braces, no metal mouth.)

They also appear to engage in patent-troll like behaviour, but I don’t currently have any solid opinion on the merits of their legal maneuvering.

Invisalign Drawbacks

I’d still pick Invisalign if I had to choose between it and conventional braces again, but consider the following:

  • For best results, trays have to be in for 20-22 hours per day, and you’re not supposed to drink anything other than water with the trays in. So it’s really only for meals that removing the trays is practical. I can’t just try a drink or have a bite of food – it becomes a whole ordeal to remove them, and then they’re supposed to be replaced as soon as possible. In what might be seen as a net positive for my health, I’ve switched to drinking soda water (rather than cola or coffee) during the work day because of this inconvenience.
  • Plastic in my mouth during the night sucks. I tend to drool overnight with the trays in, and even through a pillow protector I’ve ruined at least one pillow.
  • You still have to have attachments bonded to your teeth, which are initially rough on the inside of the mouth. The installation process is also unpleasant affixing as it requires your jaw to remain open in an odd position for several minutes for each attachment.
  • A surprise to me – and not really fully described at my initial appointment – is that my second set of trays required installation of a “button” (a metal protrusion cemented to a tooth) and use of an elastic. This also complicates insertion and removal. More complicated cases are likely to have more elastics and buttons.
  • It’s not completely painless. Switching to a new set of trays causes pressure and occasional tooth pain. I find popping two Advil is necessary on the first day of a new set, or otherwise I can’t concentrate at work.
  • Don’t lose or break your aligners; it’s a $150 replacement fee per set. I have heard that depending on where you are in your treatment process or cycle, you may be able to skip to the next set instead. With braces you have to be cautious about breaking brackets or loose wires, but with a set of trays it’s incredibly easy to leave them in a napkin at a restaurant or misplace the Invisalign case.

“4 Strikes and They’re Off!”

At RiteBite, apparently you can get kicked out of braces (or Invisalign) if you don’t have decent oral hygiene at four appointments. According to the initial contract I received, RiteBite also can “rat you out” to your dentist with a letter and won’t perform whatever orthodontic process was scheduled for the day.

Since I haven’t heard about this system since the initial package of paperwork, I think this is more of a way for parents to threaten their kids into compliance for orthodontic treatment – “if you get a 3 or less on this arbitrary grading system, you’re in trouble!” I suspect it’s not frequently employed to its full extent.

With this in mind, Invisalign itself does encourage better oral hygiene. You won’t want to put the aligners back on without cleaning your teeth well – a fragment of steak in between molars becomes very painful when compressed with plastic trays.

Aligner Use and Abuse: Beer, Whisky and Vodka

One of the really frequent questions online is “can I drink (beer) with Invisalign in?” I’ll refer you to “Another Invisalign Blog“, where the author has written specifically about Drinking Beer With Invisalign and Pros And Cons of Invisalign: Revised After 2+ Years of Wearing Aligners. Although the author’s recent posts have gone into the realm of what I’d consider unnecessary surgery, her writing was crucial for me in my early research.

I am sure that Dr. Luis would not approve of drinking anything other than water, but here’s my experience:

  • It is definitely possible to drink light-coloured beer with aligners in. In 2014 I tried this at Oktoberfest, matching each beer with a glass of water, and didn’t notice any discolouration afterwards. In contrast, for 2015’s festival of German-style debauchery I decided to remove them entirely for the evening.
  • Do not drink dark-coloured liquors with aligners in – it will absolutely stain the trays.
  • I will drink cider (Grower’s 1927 Premium Dry), vodka/soda or vodka/7Up with a set of trays still in, but am meticulous about removing, cleaning, and reinstalling them for the night and going to bed. The trays don’t seem to be any worse for wear as long as they are cleaned with a separate toothbrush dedicated to this task.

What’s Next

Since this post was initially drafted, I am presently waiting on a third set of trays, targeting completion in August 2016. The “button” remains on in between sets but the attachments get taken off.

I’ll follow up in a few months with progress on the next set.

Another “Let’s Encrypt” post for nginx

I’ve replaced the certificate on this site with one issued by Let’s Encrypt and plan to do so for all clients (or enable SSL in the first place) as their domains come up for renewal, or other maintenance work is contracted. The big downside is a 90 day expiry time, which requires a service nginx reload at least that often.

I had no end of issues using the official client as it wouldn’t create the .well-known/acme-challenge files necessary to get the domain to validate (yes, I checked directory permissions.) Instead, Vincent Composieux has some excellent instructions on just using the certonly parameter inside a script. Rundown, including my changes in case the article disappears:

  • clone letsencrypt repository to /opt/letsencrypt
  • create /usr/local/etc/le-example.com-webroot.ini:
# We use a 4096 bit RSA key instead of 2048
rsa-key-size = 4096

email = email@example.org
domains = example.com, www.example.com

authenticator = webroot

# This is the webroot directory of your domain in which
# letsencrypt will write a hash in /.well-known/acme-challenge directory.
webroot-path = /var/www/example.com/
ssl_session_timeout 5m;
ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:50m;
ssl_dhparam /etc/ssl/dhparams.pem;
ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';

if ($myroot = false) {
        set $myroot $realpath_root;
}

location '/.well-known/acme-challenge' {
    root $myroot/;
    try_files $uri /$1;
}
  • For each site in /etc/nginx/sites-enabled, update the SSL definition to store the webroot in the $myroot variable, then have the root directive (and ssl.conf) reference it:
server {
        listen 443 ssl;
        server_name example.com www.example.com;
        # [...]
        set $myroot /var/www/example.com;
        root $myroot;
        include global/ssl.conf;
        # [...]
}
  • Create the certificate: sudo /opt/letsencrypt/letsencrypt-auto certonly --config /usr/local/etc/le-example.com-webroot.ini
  • Add the certificate paths to each site in sites-enabled:
server {
        # [...]
        include global/ssl.conf;
        ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
        # [...]
}
  • To automatically renew certificates 30 days before expiry, checking each day: ln -snf /usr/local/bin/renew-certificates.sh /etc/cron.daily/renew-certificates.sh

Some adjustments are obviously necessary for multiple sites but this got me past the point where site validation failed.

Invisalign on my own dime: orthodontics in Kitchener-Waterloo

I’ve been meaning to write about my experiences with Invisalign and the orthodontic consultation and treatment process since I started investigating various options in July 2014. On advice from my dentist Dr. Reddy and her staff at King Street Dental, I received several referrals to orthodontists in the KW area, and did my own research into reputation, pricing and treatment options.

Before getting into the orthodontic part of the piece, I would definitely recommend Dr. Reddy. In my experience, she handles both routine and emergency dental work to a very high standard.

Initially, Dr. Reddy suggested that she could extract one or more teeth to correct crowding in my lower jaw, but also indicated that I should look at orthodontic treatment as an alternative.

Evaluations

I received evaluations from three orthodontic practices in the Kitchener/Waterloo area:

TL;DR: Out of these three options, I opted for Invisalign treatment with RiteBite in August 2014 and began wearing the aligner trays in October 2014. As of October 2015 I am on a second box of trays, but from everything I’ve heard, I am on target to finish within 24 months.

My main concerns with orthodontic work were the following:

  • What is the cost? Despite the fact that I have health coverage through my employer, orthodontic coverage is generally limited to dependents under 19 years of age, so I’m on the hook for the whole bill. This is a common theme with corporate health benefits – even if you have 100% dental, orthodontic work is generally provided for your kids only.
  • Is it going to be a gigantic pain in the ass? I have heard horror stories of people breaking brackets and popping wires, unable to eat anything but soup after getting braces tightened, and having to use crappy plastic mouthguards or slimy retainers for the rest of their natural life. I also didn’t want things to drag on for months or years past the quoted timeframe.

I received significantly different options and opinions from each practice, so I’d highly recommend getting multiple evaluations performed. Payments are typically 0% financing with monthly installments over the expected course of the treatment, plus an upfront deposit. (You get a 2% to 5% discount at these practices for a lump sum payment, which I didn’t find to be worth it.)

Nicolucci

I first went to Nicolucci Orthodontics, based on the initial recommendation by Dr. Reddy. I’ll note that my experience may have been negatively biased by the fact that it was my first consultation, and I wasn’t quite ready to make a decision on braces vs. tooth yanking.

Dr. Mai performed an initial evaluation, and the results complicated the situation. Before they would perform any orthodontic work, they’d want one lower tooth extracted. She and her assistant also indicated that I would likely require gum grafts, and that they wouldn’t begin treatment until my gum health improved. Treatment time was 24 to 30 months and the only option available was traditional braces.

Pricing was the least expensive (not by much) of the three practices at $5500. After reviewing the documentation in preparation for this post, on top of that was a “diagnostic records fee” of $300. The initial deposit requested was $2200 (so really $2500), then $3300 spread out over 24 months.

Even though I knew my oral health wasn’t especially great, I didn’t think it was bad enough to warrant a hard stop. The experience was really discouraging. I wasn’t impressed that tooth extraction and additional procedures were going to be needed on top of braces.

TriCity

After the initial experience at Nicolucci, I wanted to price compare and see if there were other options available. TriCity was one of two additional referrals from Dr. Reddy’s office.

When I initially called to schedule the consultation, the receptionist indicated that there would be a $50 initial examination fee (which wasn’t listed on their website or referral card.) I balked a bit, and they were willing to waive the fee because “my dentist hadn’t mentioned it.” This was the only practice I went to that wanted to charge for the evaluation.

I was very impressed with Dr. Phan. He addressed all my questions, explained everything in a satisfactory fashion, and was upfront about timeline (24 months) and expected results. He and his assistant had no issues with my oral health and was willing to begin treatment immediately, using clear braces for the upper teeth and conventional metal brackets for the lower ones. He indicated this would give a better result in fixing the bottom crowding.

More critically, Dr. Phan did not want to extract any teeth, and suggested that if I did go ahead with any extraction operations, it could put me into a situation where I’d need up to four upper and lower teeth removed in order to get the results he wanted. His recommendation was to complete my other evaluations and then make a decision, but not to have any teeth pulled in the meantime.

Cost was the highest of the three options, at $6300. I don’t have a precise payment plan breakdown available but it also involved an upfront deposit followed by 24 months of equal payments.

Dr. Phan and TriCity ended up being a really close #2 in my evaluation – only beaten out by the later option of Invisalign with RiteBite.

RiteBite

My last stop was at RiteBite, which is Dr. Luis’ practice. They have three locations: Waterloo, Cambridge and Listowel. I’ve only ever been to the Waterloo location but apparently you can book appointments and receive treatment at any one of the offices.

Going into the office was a stark contrast to the other practices I had visited. All the chairs in the lobby were occupied by children and their parents, and this has been consistent at nearly every appointment I’ve been to since. It’s a bit of a zoo compared to the other options – TriCity was completely serene and had a very upscale waiting room, and Nicolucci had much more of a high-end surgical practice feel.

I was seen promptly, though, and one of the treatment coordinators took digital photos of various angles of my face and teeth, rather than having the orthodontist examine my mouth directly. I thought this was a novel and sensible approach. After a bit of evaluation of the pictures, Dr. Luis came in for a short discussion. The takeaway was his claim that “whatever I can do with braces, I can do with Invisalign” and that he also recommended against any tooth extraction prior to orthodontic work. Timeframe was quoted at 24 months as well.

Initially I wasn’t as comfortable with Dr. Luis as I was with Dr. Phan. Dr. Luis has an Bluetooth earpiece perpetually attached, and he seems in quite a hurry to get from patient to patient. The treatment room in the Waterloo office is an assembly line –  there are PCs at each station precisely timing the length of the visit based on the treatment plan for that appointment.

During later sessions, despite him clearly being torn in many directions, Dr. Luis has been quite friendly and given me his full attention when I posed questions. I also have to compliment the treatment staff: they are clearly on a tight schedule but are professional and perform tasks the right way, not just the fast way.

Pricing was the middle option – $5880 total (equal cost, regardless of braces or Invisalign choice), and the financial coordinator was easy to work with. My initial deposit could have been as low as $500, with the remainder of the balance spread out into payments over 24 months. They were also willing to charge my credit card on a recurring monthly basis for the instalments.

After a few email followups with my treatment coordinator and some research about Invisalign versus conventional braces, I ended up signing the treatment forms and going with Dr. Luis and RiteBite for this work.

What’s next?

A subsequent post will provide further details on Invisalign and RiteBite, having spent a year living with the trays and treatment. As a preview, though, I definitely recommend RiteBite/Dr. Luis and Invisalign as an orthodontic treatment option if it’s available.