Blog

Challenge

Two separate industrial baking companies, Amoroso Bread and Ginsburg Baking,  started a joint venture and built a new manufacturing facility. The new company, 151 Foods LLC, is an industrial baking company that strives to consistently produce the highest quality products in the most effective manner while providing world-class customer service.

151 Foods was building out five production lines and had a number of machines that were being brought over from the two different companies. The machines would benefit from utilizing a network so they could connect with one another. Still, the expense of the overall project seemed prohibitive until the company eventually realized that it was necessary for interim machine communication and management purposes such as remote monitoring.

“We operate a very large facility. When you bring multiple companies under one roof there’s a lot of equipment, and a lot of things can go wrong, explained Emil Andrae, facility manager at 151 Foods. “Having the ability to get in remotely to access any issues is a tremendous help because it saves time and money. With the amount of equipment we had, it was decided that going with an OT network would be the most beneficial. It opened up a lot of options for our plant, and it offers the ability to remotely fix things when necessary.”

Cornerstone Automation was involved in bringing some of the equipment over from the previous facilities and refurbishing it. Most of the older equipment was running with legacy control systems. For example, many of the machines weren’t network capable. Once they were upgraded, they had ethernet capability, so it made sense to tie them together and communicate with them. About half of the plant was new equipment that was purchased when they put the five lines in, and half of it was older equipment that came from the other two plants. 

Originally, the machines were going to be connected through hardwired interlocks. In other words, simply running a wire to enable the other machines. Instead, the network allows interlocking. The network also permits information sharing such as knowing which recipes are going between machines, and what speeds the machines are running from machine to machine so that another piece of equipment could follow the rate of a downstream machine.

Additionally, before the upgrade, the company had been experiencing countless access issues with operators inadvertently changing settings and passwords. These errors were time-consuming to fix, having to rewrite the program, open it on an external device, change the programming and then download it back to the HMI. As part of their ongoing service contract, Cornerstone created a more secure hierarchy allowing different levels of access to authorized users and preventing unauthorized changes.

Cornerstone recommended the OT network for two reasons:

• Inter-Machine Communication
  • It’s helpful to communicate between machines on the line to set up interlocks, speed control, and more

• Management Purposes

• • Connect to machines to remotely to support them
  • Connect to machines while on-site
  • Connect to multiple machines at the same time for troubleshooting

Solution

Cornerstone Automation was brought in as a longtime partner, trusted resource, and consultant by Ginsburg Baking out of Atlantic City, one of the two companies starting 151 Foods.

Cornerstone offered a solution to help 151 Foods manage its security and data collection needs, as well as accommodate the size of its large plant. They utilized multiple IDFs (intermediate distribution frames) to cover the square footage and used layer 3 managed switches connected through a fiber optic backbone to reduce latency and interference. Each switch covers a specific geographical area and the network is also segmented into its referred VLANs or (virtual local area networks) with one VLAN assigned per line to further segregate the traffic. This setup ensures that the individual production line’s communications don’t interfere with other lines. It also helps with logical organization, making the network easier to understand from a conceptual point of view.

Using a Zyxel Network allowed for the managed switch functionality and the ability to create VLANs, as VLANs can’t be created without using a managed switch. The managed switch allowed for the segmentation between the lines, reducing the communication demands on the network. 

On an industrial bakery line, there is an ingredient system that usually spans multiple lines. Each line will have one to three dough mixers. The dough then travels down the line into a machine that molds it, then there is a rack or sometimes the dough will go down the conveyor line into a proofer. Occasionally there are other machines that score or seed the bread before sending it to the oven. After the bread is baked, it goes into a cooler before going through packaging equipment and is finally stored in a freezer. For this particular manufacturer, about half of their lines have freezers, which equates to about 10 pieces of equipment per line on average.

The production lines are completely segregated with the exception of shared equipment. There is only one oxidizer for the whole plant, so each of the ovens communicates with the oxidizer. The lines also all share the same ingredient system that delivers flour and water to all the mixers. The oxidizer and ingredient system are the only two pieces of equipment that communicate across VLANs. This particular network has over 100 different pieces of distinct equipment on it, with each piece of equipment having anywhere from two to 20 different devices connected to the network. Overall, that’s an average of a little over 10 machines per line.

Security

For a firewall, Cornerstone utilized a layer 3 switch to route within their network with one uplink to the IT-controlled firewall that allows authorized remote connections. 151 Foods provided VPN access to Cornerstone Automation’s network. That way it’s company controlled and not under the external control of a contractor.

Cornerstone implemented different levels of security in terms of logins on the machines to prevent the previous constant stream of access issues. Now, there are four different permissions categories ranging from operator to plant manager. Operators have the most basic access, essentially pressing start and stop, supervisors can make minor changes, plant managers are authorized to make recipe modifications, and then maintenance engineers also have a level of access to perform maintenance functions.

The Cornerstone team created a floor plan layout illustrating where all the network connections are. The Cornerstone engineers coordinated with many new equipment vendors to get them IP addresses, as well as assigned an IP address to any of the older equipment that had been upgraded. They also maintain a database of IP addresses showing what they are used for.

“Sometimes system integrators keep their information close to the chest. The Cornerstone Automation engineers are extremely knowledgeable and tell me anything I need to know. Nathan is an open book. He’ll make up prints for me, and get me anything I need. The Cornerstone team is forward with all that kind of information, which is a necessity when there’s a problem,” Emil Andrae detailed. “They’re always there to answer the phone. I’ve called Nathan when he’s up in a bucket truck 60 feet in the air, and we’ll talk through a problem. Their team always makes time for us and that’s a big help. Cornerstone Automation provides excellent service. I would have no problem recommending them and giving them a glowing review.”

The following general list of hardware is included:

• Zyxel Networks

• IDFs (intermediate distribution frames)

• Fiber Optics

• Layer 3 Managed Switches

• VLANs

• Allen Bradley PLCs

• Allen Bradley HMI

• Allen Bradley Drives

Results

The remote access, increased data collection, added network security, and ongoing plant maintenance have had a large impact on 151 Foods’ operations.

• Remote Access – Remote access capability saves 151 Foods a tremendous amount of money on a daily basis. Remote access saves an hour of downtime and the cost of travel. 151 Foods can call for emergency maintenance at any time 24/7/365, and the engineers can simply log in online to complete the service call.
  
  
• Data collection – Monitoring resource consumption, specifically with their flour and water systems, allows for more detailed fine-tuning of recipes which ultimately increased throughput and quality. Cornerstone also created a “central time server” for the system, allowing everything to be timed in relation to that, which in turn eliminated many problems with timing, increased throughput, and reduced downtime.
  
  

“We had some information before about the flour and water consumption, but now that we have the network, we have access to much more detailed information,” said Emil Andrae. “We’ve also seen the benefit of monitoring the chilling system. We use chilled glycol, which is an antifreeze-type liquid, to chill the mixers because the mixing process generates a lot of heat. We’ve redesigned all that and now just looking at the screen, I can see the temperature in, temperature out, pressure, and all kinds of things, which is wonderful. We didn’t have that level of insight before. Now we have much better control, and less chance for error.”

• Security – Each operator has been assigned a user account that is propagated throughout the network so that there are now individual log-ins at each machine. Digital security is a common issue for many companies. In this particular instance, there had been numerous problems with people accessing machines that they shouldn’t have, and making unauthorized changes. Now, when anybody tries to log into a machine, their credentials are checked to the server, and then they have access limited to whatever they need to do.
  
  

“We’ve incorporated a facility-wide access system on all the equipment, and it has been very helpful as far as keeping people from making unauthorized changes. That has improved downtime tremendously and gives us a higher level of insight so that we can track the changes made. We know what time it happened, what was changed, and who changed it, and it’s all logged and recorded,” Emil stated.

• Plant Maintenance – Cornerstone has a service contract with 151 Foods. Controls Specialist, Nathan Ferrin, is on-site once a week where he is constantly connecting to different machines and working on various improvements to the plant. Without this network in place, this fine-tuning of the system would be nearly impossible.

“A lot of plants just have one giant flat network that’s not segmented. It’s all on unmanaged switches and any one hiccup on the network can cause the entire plant to go down. I’ve seen it happen,” explained Nathan. “If it’s all unmanaged switches, there are no diagnostics so engineers are out there on the plant floor unplugging things and plugging things in and just looking for a needle in a haystack. When you have a network that works well, a lot of the headaches go away. The ease of access to everything on the network saves so much time in the long run.”

• Energy Savings – Environmental/energy savings have been realized by connecting new equipment for ventilation and heating. The plant can move the air around to spread the heat from the ovens to the more exterior locations within the plant, reducing the need for unit heaters, and ultimately saving money. 
  
  

Conclusion

Overall, this solution has resulted in tremendous cost savings through reduced downtime, increased quality, and throughput, and has drastically reduced the frequency of emergency calls. It has also opened the door to new possibilities for energy savings and other helpful upgrades.

After the network project was complete, Cornerstone has continued to streamline operations for 151 Foods by integrating building automation including upgrades such as a chilled potable water system and ice water maker, which also communicate over the plant network. Having the network in place has been highly beneficial for interconnecting the different building automation systems.

“Once we got the OT network, we had much more flexibility. We’ve been able to get creative with a lot of different things, including environmental systems, which were never actually part of the original plan. We’ve incorporated things like building management controls, heating, ventilation, the chiller, air compressor, and all sorts of things. We’re trying to incorporate as much as possible onto this network to give us better central control because we’re getting close to half a million square feet here. There’s no direct line anywhere, and there’s a lot of walking around this place. The more centralized I can get the controls, the easier my life is,” said Emil.

For assistance with developing an OT Network for your industrial plant contact Cornerstone Automation.