Manufacturing operational challenges abound in all industries. Water and waste water are no exception and share some common challenges. Unlike many manufacturing organizations which operate for profit, the water and waste water industry consists of two main types of entities: government-run and cost centers that operate in private companies. The processes themselves are outwardly simple: filter the water, then biologically and chemically treat it. The chemistry involved makes this challenging. The biologicals make it an art form. However, the business model is inherently problematic because the drivers are different between the two types of entities.
Strategic planning for water and wastewater infrastructure investments and transitioning the operation and management of existing assets to incorporate new advances in technology is critical. Challenges and issues the industry must address include:
- Aging infrastructure, increased demand for water supply, and impact of climate change
- Inadequate and/or ineffective wastewater infrastructure in many parts of the world
- Obtaining funding for large projects during periods of tight budgets and economic uncertainty
- Privatization and public/private partnerships (PPP) require efficient transition of legacy systems to meet new demands from all stake-holders
- Increasing concern for physical and cybersecurity
- Increasing demands placed upon the aging, shrinking workforce and associated loss of knowledge, experience and skills
- Energy costs, which may account for a large share of overall operating costs.
In the developed world, the mission of the water and wastewater industry is:
- to treat and transport high-quality drinking water at affordable rates in a reliable and sustainable manner and
- to collect and treat wastewater and storm runoff.
The industry faces significant issues, and municipalities are challenged to maintain operations. Infrastructure maintenance competes with other government programs for attention. As a result, infrastructure is failing dramatically, and it will cost even more to rebuild in the future. The U.S. EPA announced in 2013 that ensuring safe drinking water for Americans in 20 years will require a $384 billion investment in infrastructure alone. The industrial side has similar challenges because water treatment is not associated to a revenue stream. Instead, it is a cost of doing business; if they could avoid it, they would.
In many cases, information management systems and networks in the water industry are isolated. Control systems are not integrated with business systems. Divisions and departments operate independently. Business and operational analytics are not in place. Decisions are often made based on experience and “tribal knowledge,” rather than current data. Tactical approaches and fixes are employed, instead of strategic planning and decision making. As a result, organizations focus their limited resources on reacting and mitigating, rather than anticipating and preventing issues and incidents.
Organizations also must be able to attract a modern workforce capable of deploying the latest Smart Manufacturing technologies, and provide these technologies to its workers. Currently, many of these processes are operated by experienced baby boomers that make up about 50 percent of a total workforce, but are approaching retirement age. Operational technology and its application are complex and losing on-the-job experience to retirement has a major impact on quality operations and uptime. Organizations must engage in knowledge capture and management, and the incorporation of new workflow processes. A new workforce, mostly consisting of millennials, needs and expects new tools and modern workflows to meet the challenges of flexible, agile production. Workflows guide users in their day to day tasks ensuring uptime and repeatable quality, facilitate collaboration, and reduce paperwork, enabling the digital transformation of these plants. For example, mobility allows secured access to any information from anywhere. Central operation and maintenance centers allow participation of remote experts in problem solving. The result is improved efficiency in operations and maintenance tasks, along with improved work conditions for a new dynamic workforce.
How a modern OT/IT infrastructure helps address these challenges
Employing digital transformation concepts in key operations can enable better water and chemicals management operations. Digital technologies enable the collection of information needed to document evidence of proper treatment and to detect causes of wasted chemicals, energy, and water. The resulting increases plant productivity, reduces water waste, energy and overuse of chemical treatments; and the ability to document operations for better reporting. Predictive analytics and demand forecasting – being prepared for the halftime surge from a national sports event, say, or a major rain event in the forecast – can save operating costs and prevent outages or spills.
None of the industry challenges addressed with modern technologies can be deployed without the convergence of operations technology (OT) and information technology (IT). This has led to a rapid learning curve for both IT and OT groups. The side effect of OT/IT convergence has meant that IT personnel often have to learn what terms such as “real time,” “non-stop,” and “deterministic” mean in the operations context, and OT personnel are rapidly discovering the advantages of leveraging the latest IT-based approaches. This convergence helps address the change in expertise, increase overall equipment availability, and improves data flow because legacy automation assets often need IT technology to be connected to the Cloud or to act as edge devices. Continuously available real-time or near real-time data is essential for any water and waste water organization to provide safe water on demand. This OT/IT convergence trend increases integration and provides better information flow to and from all assets, including legacy automation systems. It also contributes to the ease of adoption of cloud computing and Big Data applications that help eliminate unscheduled downtime.
OT/IT convergence also helps enable edge control, which is a converged control architecture that gives facilities the critical capability to manage their assets on premise as well as from the Cloud based on their particular needs. This includes connected control platforms with remote access, advanced automation, and operator override capabilities. Local control and firewall protection for cybersecurity are also available to maximize the benefits, especially for mission-critical applications. Edge control applications can be deployed for energy monitoring, equipment automation, and process automation systems that provide monitoring, control, and safety management.
Facilities deploying a simpler OT/IT infrastructure can reduce both technical and business risks by deploying solutions that are easier to maintain by operations and maintenance personnel. OT/IT
infrastructure enables a very reliable integration of disparate equipment and system information for end-to-end production visibility and traceability, along with automated data collection for efficiency and analysis. A flexible and robust OT/IT infrastructure is also the easiest path to address both existing application needs, as well as leveraging digital transformation to meet business challenges. In summary, a modern OT/IT infrastructure is essential to help water and waste water facilities address their processing challenges and make a positive impact to their quality, safety, and cost reduction.
ARC Advisory Group and Stratus are teaming up for a live webinar on addressing operational challenges within water/wastewater organizations. Register now to attend live or receive the recording.
Welcome back to the second blog in the series relating to a recent Twitter poll we launched to gauge industry audience insights and sentiments in specific Industrial Automation (IA) sectors. These IA vertical markets include Oil & Gas, Food & Beverage, and Water & Wastewater. As a reminder, we had just over 32,000 individuals engage with us on their understanding of their industry’s organizational strategies and plans.
This week, we will be covering some thought provoking results involving the Water & Wastewater sector. The question we asked via Twitter was “How much unplanned system downtime per incident is acceptable when it comes to water treatment?”. Almost half of respondents (45%) simply did not know. This blog will discuss the importance of continuous uptime in water treatment and will educate readers about the implications of downtime in wastewater facilities.
Can you recall a high-profile incident where a water crisis made national headlines? Just looking within North America, how about the poor water sourcing choices that led to the poisoned water of Flint, Michigan? Or the drought in California? Or the wastewater sewage pollution that occurred on the East Coast after Hurricane Sandy? Although not all these issues may not have been a result of downtime – they certainly stress the criticality of water treatment safety. High profile cases like these really make people think about the quality and management of their drinking water.
Let’s talk about how unplanned downtime occurs in the Water sector.
According to the 2016 AWWA SOTWI Report, here are the top issues facing the Water industry:
- Renewal & replacement (R&R) of aging water and wastewater infrastructures
- Financing for capital improvements
- Public understanding of the value of water systems & services
- Long-term water supply availability
- Public understanding of the value of water resources
While these are all relevant issues, all with their own domino effect of consequences, I am going to focus on the top two. R&R of aging infrastructures, as well as the struggle of financing capital improvements and subsequent delays can both materially contribute to the increased probability of unplanned downtime of SCADA systems that govern the operation of water facilities.
Obviously, the increasing challenges related of raising the financing for capital improvements can further delay the start to refresh aging infrastructures, consequently deepening financial risk exposure. It’s a dangerous and costly cycle that is a strong contributor to unplanned system downtime – forcing the water facility to perform what would normally be an automated task, manually. This further increases costs while opening the possibility for human error when dealing with, for example, disinfection/chemical additives or filtration systems/UV operation.
Now that we know how downtime can occur, here are some effects of unplanned SCADA downtime for water facilities:
- “Blindness” from the plant level to the entire district that an outage has occurred
- Inability to control remote locations
- Loss in integration with reliability systems
- Loss of data/report generation
- Could affect compliance
- Loss of analytics
So, now that we know what could go wrong with even momentary unplanned downtime, let’s talk about how much it could cost. Now, this metric certainly depends on the company size, industry, production volume, etc., but according to the Aberdeen Group, the cost of unplanned downtime went up $260,000 per hour on average between 2014 and 2016. However, this metric is only monetary. We must remember that unplanned outages in IA industries, like water and wastewater, can result in damage to the environment, people’s health – and even death. Unfortunately, as practitioners in this field know, it can require a very critical outage to create the momentum needed to provide capital and accelerate decision making to update an infrastructure.
So – let’s ask the question again:
How much unplanned system downtime per incident is acceptable when it comes to water treatment? And what if you were not sure of the answer, how could you better inform yourself – and your stakeholders?
Here are some options for you to get started
Another great way to explore the challenges associated with automation and instrumentation in the water and wastewater sectors would be to attend WEFTEC – the water quality event, September 30 – October 4, 2017 in Chicago, IL. Stratus will be there. We are proud to join this event because this is an Industry that we are at the forefront of, and so are many of our customers. If you are attending, please visit us at booth #7847.
Follow us on Twitter @StratusAlwaysOn to watch out for the next in the series!
Imagine annually hosting 25 million theme park visitors that expect a magical and safe experience and the public health consequences if water contamination occurred at the site. That’s why Shanghai Disney Resort, the newest and largest Disney theme park in Asia, takes no chances. They run their water monitoring and control systems on Stratus continuous availability solutions.
Shanghai Disney Resort is a massive theme park with six “lands” and shopping, dining, and entertainment venues. It spans a lush, 963-acre property rich with scenic lakes and public green spaces. Behind the scenes, a comprehensive water circulation and irrigation system keeps these natural areas safe and inviting for visitors by ensuring strict adherence to water quality standards.
The complex water treatment system has the capacity to treat 24,000 cubic meters of water per day. It relies on an automated software system to track, control, and analyze the equipment managing water flow, as well as data measuring water quality. Even brief unplanned system downtime could lead to water contamination and turn a nice day at the park into a public health nightmare.
To avoid such disasters, Stratus built the core infrastructure of Shanghai Disney Resort’s water treatment system on Stratus ftServers. Intelligent, self-healing ftServers prevent unplanned downtime from occurring in the first place. As a result, the park’s critical systems run continuously with no data loss.
In a Stratus case study about the park, Mr. Pang, Shanghai Disney Resort’s chief engineer, explains, “We can count on the Stratus ftServer solution to eliminate single points of technical failure so this critical system runs continuously—without unscheduled interruptions—for smooth, reliable operations.”
In fact, since going live, the ftServer solution has delivered continuous transmission of vital water quality and control system data with zero unplanned downtime. This allows the resort to consistently meet— and exceed—quality standards for second-class surface water throughout the property.
The Stratus solution also has been easy for Mr. Pang’s team to manage and maintain. They didn’t have to write any special failover scripts as in a traditional disaster recovery solution, or modify their applications to be cluster-aware. The software simply runs just as it would on any industry-standard server, except that the ftServer is continuously available.
As further testimony of the easy serviceability of Stratus solutions, the park’s IT staff uses a single ftServer dashboard to monitor the water treatment system infrastructure. The team can automatically restart applications if an unrecoverable error occurs and avoid any downtime or loss of in-flight data.
Stratus keeps water systems like the one at Shanghai Disney Resort up and running for organizations worldwide. That’s one of the key reasons the resort chose the ftServer. It’s proven to be reliable and easy to manage in numerous other water treatment facilities. By removing the complexity of preventing downtime, Stratus translates to low total cost of ownership and high return on investment. For Shanghai Disney Resort, that return is best demonstrated by the delight and safety of tens of thousands visitors that enjoy its vast facility every day.