A couple of years back, whilst I was between train-wreck automation project contracts, I somehow got to looking at overall factory floor integration in the context of modern internet software development technology that was upending every other industry except ours. The more I looked at industrial automation, the more I thought that the only way to design it holistically was to at least have a vision of what you want, and then try and implement it. What we had was a wickerwork marvel of bespoke interfaces, disparate database sources and brittle software that couldn’t change without breaking everything else. A vision was needed that could be encapsulated in an universally accessible model or prototype or some form of conference room pilot. It wouldn’t be a complete panacea; if no systems are out there that can do what is asked, then we are not really a lot further on. But at least we have reduced the ‘unknown unknowns’. ..
I felt that there were a number of ‘blockers’ to effective industrial automation engineering:
Flashpoint 1: no way to demonstrate the vision
- Inability to rapidly iterate through a control system design in order to get process engineer buy-in.
- Inability to easily demonstrate control philosophy to key stakeholders.
- Inability to get access to the target control system to perform a mockup or even to train people on how aspects of a DCS system work.
- Inability to iterate iterate iterate – design by doing. After all its how we write code.
Flashpoint 2: control philosophy misalignment
When it came to batch control system design reviews what I found was that reviewing these MS word documents ‘cold’, was prone to missing huge chunks of requirements functionality entirely. You have to imagine all the what-if scenarios and I always felt an interactive system would be a better way.
In addition, I was uncomfortable with the insufficient level of capture of S88 batch design before handing off this key design task to a system integrator. I frequently came across questionable batch design philosophy while test witnessing a batch control system. How did we let this project get so far without challenging the design? Maybe it was because there were insufficient guidelines for how it should work. Maybe a paper document was not the best way to review and capture these requirements in the first place. I felt that a high resolution mockup model would be better, and looked to the use of 3D models during major construction design for inspiration.
Flashpoint 3: testing 3rd party vendor package equipment
If platform selection, batch control philosophy , MES interface and requirements elicitation were the major battles throughout the development of an automation project, then interfaces to 3rd party systems such as modular equipment were the nasty little skirmishes requiring special ops and strong tactical coordination. These design issues would appear out of nowhere and invariably be handled in a different way by different teams; different vendors with different capabilities dealing with different client groups with widely differing views on implementation strategies.
Where was the overwatch? The guiding implementation?
Vendor package equipment was often badly programmed ; and they were not entirely to blame for this : At the eleventh hour, they were told to change their control platform from what they knew to be reliable to what they had little experience of. Why couldn’t we just provide a software stack of what is acceptable from a coding perspective?
Interfaces between systems programmed by different vendors were particularly problematic, especially when they were tightly coupled. Neither vendor really knew how the other side worked. A model was really needed here, as was an object oriented communication technology such as OPCUA.
Flashpoint 4: aspects of software factory acceptance testing are flawed
In industrial automation, engineers test against a document specification. So the quality of the language, description, structure and tester is key. The quality varies across organisations and language boundaries and between engineers. Automated regression testing ensures a certain level of software quality. Why on earth were we still manually testing complex and expensive control systems?! In addition, the factory acceptance test always ended up doubling as first contact between control system and end user operations and maintenance teams. This never went well: we are one week from delivering the software and we find out that that is not how we do that particular process at all… Operations continuously turned the testing into a new design meeting.
3rd party vendor packages systems had their own unique set of test problems; these systems were tested in isolation, and this required some form of faking / mocking of the particular interface. Invariably this was done by manually forcing values in the PLC/DCS/OPC code. The programming and testing of PLC-PLC communications has always been laborious and time consuming and difficult to setup. I would much rather have a pre-built model to fake the interface; its better and more indicative of the target system! OK, you can’t emulate a profibus point to point network in OPCUA….but that is how we should be thinking.
Flashpoint 5: the MES & ERP interface
For those of us well practised in the dark arts of SAP/R3, SAP iDocs and other pre-XML arcane file formats, congratulations – you are still indispensable. For everyone else, trying to grapple with these interfaces is like being captured behind enemy lines; no part of this is going to be enjoyable for you!
On one project, when I was engaged by a client to clarify and standardise and document their MES ERP interfaces, I had extreme difficulty even finding the people that actually knew how stuff worked. When I did eventually track them down, this other MES vendor wouldn’t tell me anything until they were on the job too. ‘Homeland’ style CIA spymasters had nothing on them. It is in a companys interest to make automation more accessible. Again, a living breathing accessible model is a great way – perhaps even the only way.
On another project, I was engaged to do a fit-gap analysis of an MES-ERP interface. The first job was to figure out how the horrendously complex system currently worked. Wow – it was almost undocumentable. 6 months of document after document from the annals of the factory codex for 10 different systems and we didn’t even know whether they were up to date. An interactive model would have helped. I think its easier to engage people when they are actively creating something (for example an interactive wireframe). The next job was to figure out how long to program the replacement system. Again, the model would have provided accurate metrics for this.
Operation ‘Spike’
Enter the aspiration to try and improve the automation engineers lot. So I listened to the Honda TV ad and asked ‘What if?’ and assembled a minimum viable product (MVP) 1 wishlist:
Use case 1 – animate that P&ID / PFD
Its quite difficult to mark out a CIP valve route on a P&ID. It can be done, with dashed lines to represent toggling valves, but it is so much easier to just record then play back and modify these routes – and you can still print out the routes if necessary. Large projects will probably still want the documents, but I still think that the interactive model helps to better capture requirements.
Use case 2 – expedite functional specification capture
Its difficult to write detailed software design specifications in isolation of the proprietary target system. This is borne out by system integrators – most detailed software design specifications are indeed reverse engineered from their respective platforms. That’s one way of doing it, but its a little late in the project. How to capture earlier, especially if you don’t know what platform you are using yet? Develop on an abstract control system with 80% of the functionality of the target systems.
Use case 3 – Estimate automation project costing?
Batch control system work estimation – physical model /procedural model /formulae. Wireframe it all out to estimate cost. It is much easier just to model rather than write word docs in the authors view.
Use case 4 – enable iterative design
The iterative nature of batch control system design when it comes to recipe parameters (and whether they should be deferred or not) means you really need a system to interact with. Normally this info is captured only during software development, but it can be captured earlier with a model.
Use case 5 – enable faster reverse engineering of existing control philosophies
In my work in the semiconductor industry, I regularly had to modify sequence of operations to bring them in line with a newly modified P&ID or vice versa. Oftentimes, while we were updating the specification, the client would have to explain how the system currently worked by demonstrating on the live system because the sequences of operations documents were too difficult to understand in a short period. The real system was easier to understand. We were getting to a point whereby we didn’t have time to read a 30 page document and needed to learn and understand faster. An interactive model is the faster way to learn. It would have been a lot easier to generate this specification from a model.
Use case 6 – make communication and collaboration more effective
I have been in many meetings that have been ineffective because there is lack of clarity on how systems should work. I always wanted an easy to use tool to explain S88 to process engineers in order to try and persuade them to take a ‘class based design’ approach to the P&IDs. Similarly I wanted to be able to better communicate Batch to MES interface points to the MES engineer.
Use case 7 – harness the factory or organisation global lessons learned and body of knowledge more effectively.
Its great that corporations standardise on control philosophies and document these philosophies. But they are useless to me as an automation engineer if they are embedded in a 60 page specification that I don’t have time to read, or are contained on a DCS system that I don’t have access to. A universally available model encompassing these quick wins and company standard design patterns is what I want, so I can access it at anytime from anywhere.
And the list goes on.. a list that would ideally be developed in co-operation with clients. Modern software needs to be a living thing that develops with its clients in a manner that consistently addresses their pain points. Smaller, more responsive software companies are best suited to providing this continuously adapting software capability to their larger corporate clients.
Making the case for change
I might be the only person in manufacturing industry that thinks a control ecosystem model of what you are trying to achieve is valuable, but its unlikely. Models are used in many other industries to develop software. For me, integration and testing quality were the biggest issues. A model can step in as a mock or fake reciprocal system (if everyone uses the same tech such as OPCUA). The ability to play – to rapidly iterate – to rapidly demonstrate to a process engineer. These are seriously valuable traits of a model. And fundamental to good engineering.
So if there’s one reason why I asked ‘what if?’, its simply the fact that I was tired of not being able to do my job.
So in my special ops room, there’s a wireframe model of how I want my entire factory control and MES system to work. Period. It gets updated as and when required in keeping with ‘Agile’. Its there for everyone, including vendors to play and interact with. Its there to keep everyone aligned and on the same page. Its a vision of what we want, encapsulating corporate standards of best known methods and how we do things. Some people call this a plan. I call it a prototype.
See Spike in action or try it yourself, no sign up or login required.
Comments are closed.