Ron is the Vice President of Network Operations and is responsible for technology and service management functions including; the Network Operations Center, planning and engineering, central office support and the technology and networks at the heart of the company’s residential, business and wholesale services, for both Cincinnati and Hawaii. Ron has been with the company since his co-op days in 1993 and has held leadership positions in several areas of the organization including IT, Contact Center, Managed Services and Field Operations. We sat down with him to ask how Cincinnati Bell has prepared for the COVID-19 pandemic and how the network is holding up to the increased demand.
Network Traffic
Q: Since the COVID-19 pandemic started what kind of traffic are you seeing to the network and how does that compare to what is typical?
A: We have seen the traffic steadily increase since even before the Stay at Home Orders. We are currently ranging at about 10% to 20% additional aggregate traffic coming across the network. It has been a noticeable bump, but fortunately not one we haven’t been able to handle. We have also seen a noticeable shift in the pattern of the traffic throughout the day. You now have parents working over VPN and pushing traffic and multiple kids streaming, gaming, etc. While traffic has increased during the usual peak evening hours, it has increased quite a bit more during the daytime hours.
The capacity across our core network (where data traffic leaves your house and works back through network), is in excellent shape. Thanks to the hard work of our planners and engineers, we have the capability to handle increases to critical data and voice traffic as utilization grows.
"Thanks to the hard work of our planners and engineers, we have the capability to handle increases to critical data and voice traffic as utilization grows."
Network Preparation
Q: As circumstances continue to unfold, how are you preparing for additional volume spikes from Ohio’s stay at home order as well as unforeseen circumstances such as inclement weather.
A: Our Network Operations Center (NOC), voice and data network engineers have been closely watching the network and capacity as the situation around us continues to change. Where we are seeing traffic increase or patterns shift, we are able to make quick adjustments to augment capacity and update the networks. We are watching not only the short-term shifts in the network, but also planning ahead for long-term shifts in usage, such as the long-lasting onset of telehealth capabilities, or the likelihood that some businesses may increase their work from home capabilities on a more permanent basis.
We have found ourselves in good shape to handle the current Work from Home Orders, but we are heading into a typical Midwest spring, which adds a chance for issues across the network due to severe weather at a time when people are working and learning from home. There are things that are out of our control, like power to the customer, but where we have cable cuts, or other storm related issues, our central office and field operations teams are ready to act quickly to restore service.
Critical Infrastructure
Q: How are we helping critical local organizations stay connected during this pandemic?
A: With the Work from Home Orders came an increase in escalations for extra capacity and speed for businesses. Schools, small businesses and Hospitals have been quickly increasing their capacity to provide services in new ways. As usual, our network, operations and central office teams have really stepped up in assisting wherever possible. These types of scenarios really give our employees a chance to shine, as they are extremely customer focused and quick to respond when needed.
Beyond additional requests, we have a responsibility to plan and be on top of as many scenarios as possible to help keep our customers up and running. In certain areas, this has been imperative. For example, we manage the delivery of local 911 calls, and handle the voice and data communications for local governments, hospitals and health services. This is all especially important now! Again, our operations teams continue to step up in these unusual times.
COVID-19 Preparation
Q: How have you prepared network operations for the COVID-19 pandemic?
A: Cincinnati Bell has a responsibility to keep our employees safe, but also to keep the network and critical infrastructure up and running no matter the situation. We have business continuity plans that have been tested and are now being put in place. However, as you can imagine, there is no good way to plan for all of the situations we are currently seeing, and no playbook to pull from, so it's really about building an infrastructure for fast decision making to respond and act as quickly as possible. Over-communication, clear decision-making and immediate responsiveness are important aspects of our current business continuity team.
"We have business continuity plans that have been tested and are now being put in place."
Moving our Network Operations Center (NOC) employees home is just one example of the complexity that we are facing with this unprecedented situation. Our previous Pandemic Business Continuity plan calls for us to separate the teams between a primary and secondary location in order to mitigate exposure risk. We have tested for this and had even put in place a concept called “air gapping” before many businesses were even thinking about working from home. But sending our NOC employees home had previously been something we didn’t believe was technically possible. Once again, the team stepped up to ensure we could offer any employee who preferred to work from home the opportunity to do so safely and effectively. It speaks to the dedication our team has to our employees and keeping them safe.
Network Connection Troubleshooting
Q: What can customers do to troubleshoot issues they are seeing with connectivity? How do they determine whether it is an application, or their internet connection causing the issue?
A: The work and learning from home environment can be challenging, especially as new tools such as Zoom become more popular in the market (and overwhelmed with new users and traffic). For users having issues, I recommend starting by going to speedtest.net to test your internet speed and connectivity. Try other applications and if it’s the case where everything is slow, then follow your typical troubleshooting: rebooting a gateway, call help desk, etc. Another good place to start is downdetector.com where you can see if the application you are using is getting reports of issues across the community.
With added users and volume in the home, Wi-Fi connectivity and bandwidth may also be an issue. If you don’t already have Cincinnati Bell’s Premier Wi-Fi solution, I would recommend it. As a best practice, run a hardwire connection from your device to the gateway and take away the wireless connection altogether. It’s not always technically possible, but if you are the working parent in a house full of online kids, this will help to fully take advantage of the high speed internet coming into your home.
Network Redundancy
Q: What kind of network redundancy do we have in place?
A: With internet traffic lighting up all over the country, one of our biggest concerns has been our connectivity to the broader Internet so that all of your cloud applications, games, and streaming continue to function as expected. To ensure we have redundancy, we connect to the internet through multiple points, including Chicago, New York, Atlanta and Los Angeles. We have multiple paths to connect to those bigger providers so if one link goes down the other ones pick up the slack without impact.
Tips for Working From Home
Q: Are there any special considerations for customers working from home?
A: As an employee now working out of the home, we must take steps to make the house look like an office. Buying a monitor, connecting directly to the gateway, etc. There are a lot of employees who have access to data that should not be shared with others, so security should be considered when finding your perfect spot so setup. Make sure sensitive customer data is not available to all walking through the kitchen or close down applications that you aren’t using when you are done. To be sensitive to other home workers, consider logging out of VPN during lunch or breaks. And of course, make sure others in the home know when you are on a video call so you don’t have unexpected participants joining in!
Our Commitment
Q: What would you tell people who have concerns about Cincinnati Bell and their internet service during this difficult time?
A: Right now it is critically important that we as a company:
- Keep our employees safe and healthy.
- Keep our customers up and running. We are doing everything we can to keep our services running the way they expect and need. We know how important these services are, because they’re important to us as well. We won’t always be perfect, but we hope people understand that no matter how bad things get, we will have people constantly focused on services and our customers because that is what we do. We are a hometown provider, it’s our friends and family using our services. I swear we have the most dedicated employees in the industry, I’ve seen it already, they will do everything and anything to keep their family and friends working and using our services the way we expect them to. We will keep doing that, as long as it takes.
"No matter how bad things get, we will have people constantly focused on services and our customers because that is what we do"