Tweet
January 1, 2021 - On January
1, a friend was posting about what he was doing 20 years ago today. He
was on duty and close to work, ready for the great Y2K meltdown—a
meltdown that did not occur in part because of a lot of work done in
advance…
Early that morning, 20 years ago, I was
standing on a one-lane bridge on a country road at midnight, watching
neighbors shoot off fireworks to greet the new century. My phone, not
yet a smart phone, was in my pocket, waiting for phone calls from the
campus Power Plant and the Electric Distribution group reporting any
issues. If I did not get those calls, I planned to call in, as the
consultants had warned that the phone switches might fail after
midnight as well. I was on that country road so I could be on site in
fifteen minutes if any of the phone calls were not good.
At the time, I had the integration of
distribution systems for steam and for chilled water in my wheelhouse
as well. I felt good, having spent four years remediating system issues
from low-level BIOS to inter-system communications. The University did
not yet provide complete networking services so that work included
patches to routers and switches as well. The parallel work to patch the
line-of-business client-server systems had been relatively painless.
My largest issue involved the
integration of a hundred buildings with advanced control systems,
mostly for energy management. After the oil-price shock of 1973, a few
dozen of the most critical high-energy-consuming buildings had been
migrated to a single centralized energy management system—one that was
still based on RSX running on an ageing PDP-11. Since then, each new
building was equipped with digital controls, purchased from and
installed by the lowest bidder in accord with State Construction law.
To make matters worse, with a couple years to go before Y2K, the local
utility had begun giving us each afternoon with 15-minute pricing for
power the next day. The need to coordinate building responses had never
been greater.
(For the nerdy, the
prevailing interaction between control systems in buildings in the 90s
was DCOM with even a little DDE. It was a brittle insecurable mess. The
interactions were all concrete and low level, in some cases simulating
typing on virtual keyboards. Most progress since then has been based on
making the interactions more abstract and thereby less fragile.)
For many of the control systems,
preparing for Y2K would begin with prying out PROMs from boards and
replacing them. Upgrading one system would inevitably break all
integration with the next.
The talks I gave after that crisis
contributed to the wide-spread adoption of web services to building
automation control systems, including a middleware standard in wide use
internationally. That work became the roots of the US National Smart
Grid roadmap. The common vision at the start that project of direct
utility control over building operations was not only bad for tenants,
bad for owners, and bad for privacy, but was far more complex than they
imagined. The roadmap described distributed autonomous power management
systems (microgrids) with high-level abstract communications between
them.
Microgrids need only coordinate supply
and demand over time. Attempting to manage internal mechanisms and
motivations adds complexity, reduces resilience, and creates a
cybersecurity nightmare. That work is still percolating as after a
decade of false-starts and one-offs, and only now is the are those
sites that most value power reliability and resilience, homing in on a
standard model for service integration of microgrids.
Hacking critical infrastructure has
matured from a loner’s hobby into coordinated incursions by
professionals and nation-states. Since 2000, there have been two (or
perhaps three) flat-out nation-on-nation cyberwars between Russia and
Lithuania. A SCADA worm deployed to take out foreign nuclear weapons
facilities is considered a likely contributor to the largest oil spill
ever in US waters, friendly fire that is astonishing in its scope.
Russian operatives casually took out the entire power grid of Ukraine
along with other infrastructure. The attacks on the Ukraine are widely
considered to be practice runs for attacks on US infrastructure.
Military war games delicately model infrastructure threats, including
EMP and physical attacks on substations as zombie outbreaks. Today,
military planners will not approve wide deployment of any technology
for critical infrastructure until a common model for cyberdefense of
the control systems is in place.
There is a growing recognition that
Cybersecurity systems for critical infrastructure require integration
with those for traditional networking and IT. To prevent technology
lock-in and stagnation of innovation, this cybersecurity must be
abstract, not reliant on direct controls. These new systems must work
in effect as distributed situation awareness, informing highly
distributed systems of autonomous components what dangers are present
or anticipated, and receiving from those systems added situation
awareness in return.
In
2017, the US promoted USCYBERCOM to unified force command, that is, a
top-level inter-service organization able to coordinate responses and
technologies across branches of the US military. In the new world, all
critical infrastructure systems named above, and more, must fit into
common abstract cybersecurity models. Consistent training programs for
cybersecurity must prepare personnel to work with them all, even as the
accelerating pace of innovation increases the technical diversity among
these systems.
The DotCom Boom provided an opportunity
to re-write commercial applications, which included removing duplicate
code by enabling these applications to communicate with each other.
These communications required us to solve the problems of identity and
security between applications.
Much of today’s Operational Technology
(OT) was also re-written for Y2K, but the work is not done. They share
identity with enterprise systems. When OT applications interact with
other applications, it is almost always with an application from the
same vendor. Today’s systems are highly connected. Lagging best
practice, as usual, OT is evolving to lean on cloud-based AI for
decision making even as those who want privacy and reliability are
leaning toward new techniques bringing even the most sophisticated AI
to inexpensive local systems.
The challenge of secure integration of
rapidly evolving OT systems has just begun. BACnet/SC (BACnet Secure
Connect) is both a necessary and welcome improvement but does not
address wider security integration. (Be sure to learn more if you
attend the AHR Show this month.)
To me, it feels like I never left working on Y2K integration issues, and will continue to do so for at least another decade.
[Click Banner To Learn More]
[Home Page] [The
Automator] [About] [Subscribe
] [Contact
Us]