By Jesse Drake
"Automate all the things" fittingly resides in a tool that — what else? — automates at least some of the things.
The Online Automated Services Toolkit, or TOAST, is a custom-built platform that manages tools running automated network tasks. These tasks were selected for automation because they're routine and repeatable — "anything that a machine can reliably perform faster than our engineers, and allows our engineers' time to be better served doing something else," said Chadd Barney, network automation engineer in UIT Network Services.
Developed five years ago by UIT's Network Operations Center (NOC), TOAST started as a small bash script created to trace IP addresses on the network. A year and a half ago it hosted eight automated tools. Today, TOAST's toolbox boasts more than 30.
"Moving forward, all of the engineers within the network team have been asked to start creating their own scripts that can be integrated into TOAST so we can continue to improve the efficiency of the network as a whole," said Rosalia Villegas, business data analyst in UIT Network Services.
"TOAST is all about is gathering and presenting data in a useful way," Barney said. "It is not a single source of truth, nor does it generate its own data; it's more of a data aggregation tool."
TOAST is also used by teams outside of the NOC. Only NOC engineers have administration rights, but they can give read-only or read-write access based on a user's role in the Active Directory — the directory service developed by Microsoft that automates network management of user data, security, and distributed resources.
One such use is a VLAN changer developed to switch upward of 1,000 thin clients in U hospitals and clinics to zero clients, allocated to their own specified subnets. A tool in TOAST allowed installation techs in the field to "run the commands an engineer would normally run to make those changes and let them know when it succeeded," Barney said. "The last report I heard about this project is that they're three months ahead of schedule."
Another tool has helped delegated users in the Information Security Office quickly identify and disable access ports by tracking down IP addresses or Media Access Control (MAC) addresses of devices conducting nefarious activity on the U's networks. This, Barney said, significantly decreased ISO's response time. The tool also scans for the protected status of a port so if numerous MAC addresses are attached to a port, and taking it offline will impact a large number of users, built-in protocols prevent the action without further review.
Other tools within TOAST help the NOC operate more efficiently, like an automated calendar to schedule on-call engineers and a feedback module where you find "Automate all the things" and somewhat more actionable ideas. TOAST is integrated with a Kanban-style task board that lives in ServiceNow and includes ideas, tasks in progress, and a log of completed tasks. It even has a "Lunch Train" — a fun tool for employees who can't decide where to eat. An algorithm randomizes and serves up three choices based on proximity and Yelp reviews.
Barney hopes to integrate more documentation to make TOAST even more of a self-service tool in the future.
"It's so valuable to have tools that live on past us being here. We want to operate in a partnership that reinforces redundancy and allows other people to come in and take up the mantle," he said.