I had 45 minutes to go to the store and get my son one more Christmas present before picking him up from school. After arrived and picking up the item I wanted to purchase, I proceeded to the customer service counter only to realize that there were roughly twelve people in line. No big deal since none of the other patrons were strapped with large purchases. I kept a mindful eye on the time, when I realized 15 minutes later and no one moved. Five more minutes, and finally, one person moved. What was going on? Another 25 minutes, and I was just seven people away, away from screaming. This was unacceptable, my patience was being tested, and on several occasions I was ready to just drop the items and walk out.
As I got just a little closer, I could see that the cashier was on the phone, holding the headset in one hand and a credit card on the other, obviously phoning in for approval. A lot of thoughts raced through my mind… isn't this 2014, 21st century and we're still calling in approvals, what failed, and don't they have a redundant system? I just had to know, so finally, 50 minutes into waiting, I asked the cashier what the problem was. In a very unconsoling look, the person said “the entire mall network is down, so we're unable to process credit cards, we'll just need to phone it in”. At that point, I didn't care, just go ahead and phone it in, and I'll be on my way. But during this whole ordeal, I noticed people walk out the store, dropping whatever they planned on purchase, and looking back in disgust shaking their heads in disbelief. Luckily, my daughter was able to pick up my son from school, but this was truly irritating.
The next day, I just had to know more about what happened. I phoned a friend who works for a very large telecom company and he had heard about the mishap at the mall. He explained that they only have one, yes one WAN link, and the mall then provides wireless to the retailers. The WAN link went down to some farming accident hundreds of miles away, cut one of the fiber lines to the carrier and quit a few customers suffered. It took over 12 hours to restore the fiber cut. It happens quite a lot, and you would think someone at the mall would have had a contingency plan to deal with a single carrier outage. Apparently not. I called the management firm that owns the mall, and asked to speak to someone in charge of the IT department. I really didn't expect to get very far, but I was able to reach the director of networking. I explained what my experience was and how a hybrid network solution would help. The response I got back was alarming. In a very monotone voice, she explained that “we have no budget and they've been asking for a redundant network for over two years”. I felt bad for her because she knew exactly what the problem was but her hands were tied. I explained to her that in the short time I was at the mall, how much revenue retailers must have lost because they couldn't afford to wait in line or simply grew frustrated. She apologized and abruptly ended the call to run to a meeting, hopefully to procure more budget.
I couldn't stop thinking about this because for the past six months, I've been helping companies with overcoming this exact situation by employing technology to use hybrid networks. Sure, cost is always a factor, but the hidden cost, i.e., lost customers can eclipse the actual cost without anyone realizing it. But its not always as simple as getting another WAN link and problem solved. How do you fail over? Is it a manual process? Do you rely on routing, or other technologies to determine which is the best path to take and how much does that cost? When talking to large retailers, they count every penny, yes penny that goes into the cost of a store front. I've heard the term when approaching a retailers IT department on hybrid networking solutions that “how many shirts do I have to sell per month to pay for your solution”?
After working with some of the largest networking/software companies in the industry over my career, yes, these retailers would have to sell a lot of shirts to pay for a proper solution. But when I look at how affordable the Ipanema solution is compared to others, I want to go back and say I've got an answer. I found the holy grail, please let me in. See, Ipanema Technologies is not a hardware company, its truly a software company.
The appliances that a retail store could use are cheaper to throw away than to RMA, and that's what retail's look for. They don't want expensive servers or proprietary hardware that cost thousands of dollars to support. Ipanema's hybrid solution is called “Dynamic WAN Selection”, or DWS. DWS will automatically detect when one link becomes congested or goes down, and fail-over to an alternate link. Its entirely possible to have up to three WAN links. For example, in the mall case, the retailer could have:
- WAN-1: Mall Wireless
- WAN-2: Sat Comm
- WAN-3: 4G LTE
If the retailer I visited had these options, it would have made for a pleasant experience, and I'm sure other retailers would have been satisfied as well.
Have a Merry Christmas and Happy New Year, or”¨Happy Holidays, which ever you prefer.
See, two options are better than one.