This is a story of developing Army Command, Control and Information System, ACCIS for Finnish Land Forces. Development program was called MATI 2 (in Finnish MAavoimien TIetojarjestelma 2). This story is composed from personal observations of the author when he was the head of G6 and later inspector of signals between 2008 – 2012 in Army Command.
System engineering as development model
System engineering or V-model has been the main method in military systems development since 1950’s and famous especially from Apollo program. The model, which is depicted in fig 1, is originated from need to identify and manipulate the properties of a system as a whole rather than only properties of each part. This approach has been followed successfully to develop military systems, which has been integrated to force via manual processes. Problem has occurred when force integration requires machine to machine interoperability. Especially command, control and communications systems that has been creating a common bus for data exchange are facing development challenges. Defence vendors have legacy for closed architectures in the name of security, military coalitions are not flexible enough to define interoperability standards, fast evolving civilian information and communication technology is creating new pressures towards traditional military development and contemporary multinational operations have required quick solutions implemented during operation. Linear planning and investment driven development has been too slow to answer ongoing operations requirements.
Figure 1: Classical system engineering approach to development
This is a story of all good intentions to follow system engineering development model while mitigating risks by Commercial-Of-The-Shelf procurement, phased delivery via Initial and Full Operational Capability, special implementation of AQAP-quality measures and full involvement of functional, hierarchical organizations.
A short history of Army Command, Control and Information Systems in Finland
In 1994 there was the first field exercise in Finland, where main command post of an infantry brigade was connected by email service (VAXMAIL) to upper echelons. Common Operational Picture, COP was created still manually on paper maps and sheets of transparent plastic. Situational Information though travelled quickly by using digital messaging system, email or telefax. Orders were delivered verbally, written on paper and drawn on plastic sheet. Computers were used to help in creating documents, calculations and presentations within command post. Digital files were shared via Ethernet connections between offices, shared storages, closed email systems or by dispatchers with portable mediums.
In 1995 there was a field exercise where first Army Command, Control and Information System (ACCIS) was established and connected all the way from brigade command post to army corps and further to theater headquarters. It was called OPJO-95 (OPeratiivinen JOhtaminen), a Unix-based Graphical Information System (GIS) with virtual database to give processing speed, integrated message system to update COP -information and replication system that enabled transfer of larger amounts of data at time either on line or with courier. For first time Army orders were given on a digital map and delivered to subunits by electrical sheets over IP- or message networks. Typical Graphical User Interface (GUI) of Unix and even more complex GIS GUI made operator training time consuming and only very seasoned operators were able to beat “manual – pedal” planning and ordering system. This ended with situation where only those officers, that were forced to utilize system, possessed some skills to operate with it. Evolution of ACCIS in Finland is depicted in figure 2.
In 1997 artillery did use first time information system (JOHtamispaikkaLAitteisto, JOHLA-97) for their fire control centers. It was integrated with message system so forward observer (FOB) was able to send formatted message with messaging system -93 directly to JOHLA that presented fire command on digital map, calculated the position of FOB and delivered fire steering automatically to batteries. This really fastened indirect fires by removing human operators from calculation and signaling functions.
Figure 2: A short description of ACCIS evolution in Finland
The 2000 field exercise witnessed next generation ACCIS at every brigade level command post called JOTI JOhtamisen TIetojarjestelma). JOTI was a cousin of OPJO but with Microsoft GUI, easier graphical interface and it was running faster on a PC in each office of a command post. Staff work was accelerated by both office tools and GIS- system that enabled distributed working, cut briefing times half and squeezed planning time to one quarter from earlier. Since JOTI was integrated with message system all situational information was send in formatted messages to JOTI systems and semi-automatically presented in GIS-screens. Top – down development automated all COP –and Intelligence processes with brigade command posts but left lower echelons with manual maps and message terminals. Program failed to automate whole process from bottom up.
In 2005 JOTI update came up unsuccessful, sustenance of JOHLA-family ceased and Army Command had to make decision to downgrade their ACCIS to JOHLA-97, which had been updated without problems. Since functions were almost identical in both systems, there was no major issues in training. With sudden stop in their major ACCIS line (JOTI), Army Command was forced to initiate a program to procure and field next generation ACCIS called MATI 2 (MAavoimien TIetojarjestelma), since everyone understood that JOHLA-97 extension was only a temporary solution.
Initial phases of MATI 2 -development program
Army Command planned to have reasonably quick program to field next ACCIS called MATI. Since they did not have similar strong technical and tactical development force as was with OPJO/JOTI/JOHLA –family in early 90’s, Army Command decided to go after MOTS–product, which should have mitigated major risks of development. All national tactical requirements were changed to international since NATO –compatible was thought to be main driver at that time. National tactical symbols were changed to APP-6A and planning process was changed accordingly to NATO STANAG 2014. With Multinational Interoperability Program (MIP) there was also new datamodel for information exchange called C2IEDM (Command and Control Information Exhange DataModel). That was selected as a base for all datamodels in Defence Forces and Army Command required to follow model within the whole MATI-system.
Although MATI was planned to be bought as a turnkey solution mainly off the shelf, there was extensive planning and design work with field tests to capture primary requirements that all possible products would be tested against. Staff processes were changed to NATO compatible ones and they were role played and finalized as process documents to be utilize in further testing and training. Technical requirements were definite with Microsoft middleware platform, ability to operate standalone and within networked group over tactical non-IP communications. MIP –compatibility requirement was given from Joint –level and Service Oriented Architecture (SOA) was recommended although no specific requirements for open interfaces or structure were stated.
Possible vendors were screened with Request For Information –process that created lots of documentation produced from marketing point of view. Actual procurement process included Request for Proposals – Proposals – Prove of Concept – Shortlisting – Negotiation – Decision and contraction –phases. Prominent vendors were invited to intensive laboratory tests where both users and system administrators screened MOTS-applications. One vendor of Battle Management application (BMS) was chosen amongst many with good scores in usability tests and the only one that promised full MIP compliance with SOA structure. Major defence industry was either not able to provide functional system or they proposed old systems like existing JOHLA and JOTI.
Since chosen vendor was able to provide only BMS –application, a loose consortium was negotiated between application, communications and quality assurance providers. Joint level had created a theoretical model where 3rd party was utilized between customer and providers to help managing relations and deliverables. It remained unclear in contract structure, who was responsible of what deliverables and how they were approved. This model was new for all parties of consortium and customer. It was planned that application vendor provides off-the-shelf BMS-software to communications integrator, which in turn delivers BMS application with right communications configuration to customer. Because Defence Material Command had adopted AQAP -quality system, a 3rd party vendor was contracted only to do quality assurance.
Since this seemed straightforward linear procurement, customer distributed responsibilities in functional way as depicted in Fig 3. Material Command was executing procurement of Material parts supervised by G10 of Army Command. Army School was given testing and C3 processes development responsibilities. J6 did coordinate development with architecture, whereas C4I Centre did provide common components like Operation System, interfaces like MIP and XML gateways and joint level C3 system (LEIJONA). Defence University and R&D Centre were developing special functions supporting operational analyses. Everything was integrated by Inspector of Signals supported by G6 of Army Command, that also took care of other capability components of DOT(M)LPFII structure and prepared to field and operate the future system.
Figure 3: Distribution of responsibilities in MATI 2 procurement
Contract was signed for 2.5 years and deliveries were phased in iterative way: by the beginning of 2008 a integration version of BMS, by mid 2008 an Initial Operational version of both BMS and Communications and by the end of 2009 a Full Operational Capability versions to be fielded. Clear milestones and documented requirements to be fulfilled at time of delivery. This seems to be typical system engineering procurement and delivery project with responsibilities distributed to appropriate functions and risks mitigated by simple linear approach.
Challenges occur
Everything started well. All functions were in place and there was no big risk since BMS application was coming off-the-shelf. Contract was phased to three clear and measurable deliverables where all vendors had their own responsibilities. Funding was fastened for 3 years and risks were managed. Two important questions - WHY and HOW - were answered. But were they?
Why this program was established and how shall it enable the capabilities of multiarms Battalion task force? Program was established to replace temporary BMS solution in Brigade command post and it was not planned to improve C3 processes or multiarms effect majorly. Sure all investment justifications were communicated within orientation for force digitalization and enabling awareness, but effectively there were no plans to extend BMS further than Battalion command posts, no plan how to integrate artillery, sappers, reconnaissance and surveillance or logistics processes into one system, no plan to integrate sensors or other awareness sources into system. From initial program intent project adapted only the simple task of replacing existing ACCIS.
How to integrate all layers together to create functioning system as defined in Fig 4? Only command post in Brigade had Local Area Network support. Battalions were still using analog radios and messaging system -93. Land forces was using their own messaging format that was not interpreted to MIP-datamodel. No one had an idea of how full synchronized BMS is able to replicate all necessary information throughout Brigade. There was no cross domain gateways (IEG) in between Land components and Joint C3 system nor were there any MIP-interoperable Joint C3 systems (LEIJONA) to be integrated to. No single model of the whole system either in small world installation or logical model was created to simulate and see how all these parts come together.
Figure 4: Technical challenges in ACCIS structure
Engineering logic was simple. If all these components were seen functioning separately, why should not they be easily integrated and function together. Unfortunately that was not the case!
Tactical communications challenges
As noted earlier, BMS was functioning over 100 M Ethernet between few nodes, but there was problem to fit replication traffic over existing Combat Radio Network. Communications integrator had assumed similar messaging as in older version of JOTI and JOHLA. BMS application session level required integrity of connection that messaging was not able to provide. Throughput of CRN digital channel was not near what was required by replication between BMS instances. Tactical IP connection with 128 kbps gross throughput was feasible but since Bit Error Rate was around 10 EXP-4 BER, the TCP-session was not stable enough for BMS client-server utilization. Application assumed fixed IP-addresses when mobility of command posts required changing addresses. Existing encryption was optimized for short messaging (32 bit block encryption) purposes and was not supporting continuous flow of longer IP-packets.
At best tactical communications were useful if stations were fixed, running with maximum transmission and plain text. This was not what was required and communications integrator announced that project would need at least one year more and millions of euros additional resources.
Application developer was testing his system on LAN with very limited number of nodes. Communications integrator was honing existing messaging system with unrealistic assumptions. AQAP vendor did not understand this kind of system of systems development but concentrated merely in documentation transparency and governance.
Datamodel challenges
J6 had made architecture decision of adapting MIP information exchange model, C2IEDM and further JC3IEDM as a base for all Defence operational and tactical information management. They appreciated an idea of using one information model from soldier level up to Joint Command level since that would support best the transparency of common operational picture and provide interoperability for both national and international level. Decision was understandable from the view that MIP was the only information model development program running at that time, NATO indicated to adapt it fully and it was experimented successfully in exchanging COP information elements between nations CCIS entities.
Unfortunately no one had utilized model with larger force and with more exhausted replication. It appeared that application was not able to manage more than 2 000 to 10 000 units and thus was incapable to manage the whole Land Forces order of battle not to speak of Defence Forces. It also appeared that C2IEDM model implementation in relational database was using processing power extensively. Complex queries took long time to be executed within normal PC instance. C2IEDM replication protocol and policy was not planned to tactical communications thus introduced several errors and frequent disconnection from replication network. Replication was also meant to work between two instances which automatically restricted topology to a star. Even within Headquarters there was not possible to sustain longer than two level hierarchies in replication. Replication with three levels of hierarchy was not feasible.
C2IEDM as relational datamodel was not flexible for changes and did not understand recent features of operational space like NGO’s, supported population, communal administration, integrated effect of multiarms, jamming as fire element, etc. Datamodel took also long time to be established in first time. There was no practical way of cleaning old data but dumping whole system. If unit was not part of original Order Of Battle, ORBAT, it was impossible to add it on the run. All information was owned by troop that was part of hierarchical structure. Datamodel had problems to understand terrain, time or phenomenon based information. Datamodel was binding land forces tactics to international compromised understanding without national special features and did not support tactical adaptation or learning.
Testing challenges
There was no reference environment to test whole system of systems until it was installed for field exercise. This testing procedure did not detect main problems but until staff was employed in command posts and all vendors started parallel remedy onsite, which produced even bigger problems without shared knowledge of root causes. Everyone ended up frustrated and untrusting in each other’s capabilities and products.
Several field tests were executed but not with same topology, ORBAT or situation, thus it was difficult to compare previous test results. There was also problem with staff officers using legacy procedures rather than NATO planning guide, GOP or Common Operational Picture fusing and sharing procedures. New SOP’s were not captured by testers, which produced one hand false expectations and other hand wrong perception of new system.
Basic performance and integration failures prevented test programs to proceed, which denied further feedback to developer. Challenges were managed more sequential than parallel, which did not give full picture of systems possibilities or challenges. Without laboratory environment and reference system there were no tools for gradual/parallel testing nor bringing in testing crew as technical issues matured. Lots of resources were lost in full scale field tests and MATI 2 got very negative brand value, which was further communicated by testers coming from different units.
Management challenges
Managing whole DOTMLPFII capability development was challenging since main focus was on material component. There was no aligned or coordinated plan to proceed with other components. G6 tried to come up with holistic plan, but major parts were missing. Army School did good job with staff work and BMS coordination, but Signal School was outside with their own agendas. A value chain of MATI 2 development and support organization is depicted in Fig 5.
Doctrine connection to tactical or C3 process development was nonexistent. Other arms were missed and most importantly intelligence, reconnaissance and surveillance was not included. The big Why was lost since initial commander’s intention was lost during haste execution and along many partially optimizing tentacles of functional organization.
Command post organization and C3 procedures were not really questioned although G6 tried to define major processes and published Land Forces C3 doctrine. There was a major mistake in attempt to adapt NATO based procedures and terminology as part of national art of operation and tactics. Outcome was long unsureness of used terminology and basic procedures.
Training focused only to have field exercise testers understanding of new processes and skills. Further fielding was not planned and education of new basic skills remained to be initiated. One should start to change basic tactical understanding amongst leaders for them to first understand WHY and then be able to instruct the HOW to troops when skills based training starts.
Even in material issues there were challenges since Material Command was not able to provide terminal devices for field use. C4I Agency failed to provide hardened operating system to field terminals. Since Joint operational C3 program failed totally, land forces were left alone without provincial above level C4-system to connect with and without necessary information services. Information Assurance solutions that C4I Agency proposed were not applicable in tactical environment. Even with best technical solutions off-the-shelf, there is no capability if understanding, integration and training skills of the whole system of systems are missing.
Leadership and management doctrine was still based on authoritarian and deep line organization without idea of emphasizing mission command or self-synchronized troops that would be able to swarm around adversary using terrain like infantry used to do during II WW. It was remarkable that conscripts and reservists adapted new tools and methods faster than enlisted, NCO’s and officers. It seems that learning new things and adapting new habits requires first unlearning of old things which takes both time and effort.
There was haste to build up required service support maturity within Signals personnel, since both technology, interconnections and system of systems interdependency required more Information Technology System Management, ITSM process approach than hierarchical line organization orders and control. If there is no legacy for process driven value chain like in artillery or logistics, then simple provider-client cooperation might appear hard to adapt by line commanded troops and their authoritarian leaders.
Maintenance and repair personnel were mainly sustaining individual devices that did not need software updates or complex configurations. Complex networked software defined systems required online warehousing, were all devices were powered and connected to maintenance network. Whole fleet of systems required weekly updates to keep interoperability and minimize vulnerability.
Information management was started to create and maintain training ORBAT for blue, neutral and red parties. Since there was a major delay at joint level C3 systems development, land forces were required to create alternative information feeds that provided weather forecast information, interagency Blue Force Track, BFT information, enemy target information from air and sea, logistics information from Defence Forces Enterprise Resource Management system, etc.
Interoperability required a standing Battalion task force reference system of fighting systems to be implemented. This reference was a small world where different experts and vendors met in integration testing, experimenting, system hardening, change management, root cause analyses and configuration management. The Reference environment sustained the integrity of complex interconnected system of systems and enabled quick changes when required.
Program management did have long hierarchical and administrative heavy lines of command that were slow, loosed information, prevented to create holistic awareness and were prone to small function optimization. Neither hierarchy nor customer driven management was followed but each function acted individually as they seemed best without understanding of causality and when failed they were protected by thick layer of irresponsible administration.
Figure 5: Planned value chain of MATI 2 development and support organization
The value chain pictured in Fig 5 never functioned since there was no maturity for matrix cooperation, lines of communication were long, functional priorities of each sub provider caused friction and holistic understanding of chains operation was missing.
Getting out
When IOC tests failed a third time in a row, it was clear that this program has failed as a whole. Both communications integrator and quality vendor had pulled out in silence and only customer and BMS provider were left in confrontation situation. When analyzed the procurement process it became clear that there was no clear violation of agreement since both sides had misunderstood intentions and failed in communicating requirements.
It was responsible single people that were managing this situation since organizations were busy doing their administration. Inspector of Signals, head of G6, project manager from G10 and project engineer from Material Command together with BMS vendor came up with settlement which they proposed to Ministry of Defence where the final decision was made.
Settlement included dismissal of original agreement by both sides. It included a new contract to develop BMS solution to a smaller international force and sustain it to the future.
Success in the end
G6 of Army Command took responsibility to manage this new international capability. They flattened development organization in to Integrated Project Team where representatives from G6, G10, Material Command and vendor worked together. It also helped that vendor had recruited two experts from Defence Forces to help in delivering understanding trough entire program. Since Special Operations Regiment had best experience in using BMS from patrol level upwards, they were tasked to test and develop holistic capability. Signals School was included with task to build IP-based mobile communications infrastructure for peace keeping operations for a company sized force. Development plan with parallel paths of advance was created, clear intermediate goals set and effort to communicate these goals to all stake holders was intensified.
As result BMS was fielded to international operations within companies that were operating in Afghanistan and in Lebanon. Both systems were integrated with other nations either at BFT or MIP –levels to create situational awareness over all forces committed within the same area of operation.
Lessons Identified
There rarely is a military capability that one can purchase as-is from vendors. Capabilities are often composed from DOTMLPFII parts and material component is nothing without other components adaptation.
As fighting system of systems becomes more complex and interrelated there is rarely a one vendor that is able to provide even all technical parts. The basic system engineering model is not best practice in developing complex system of systems capabilities.
Organization is often slave of its origins, culture and legacy systems. If one is not careful, future development is only renewing existing capabilities and not seeking asymmetric and strategic advantages over possible opponents. Since partial optimization and egocentricity is a risk for all projects a program or portfolio level should be very careful to overcome these pits.
There are cases where lack of maintenance or negligence in updating key modules of one system of systems has led to dispersion of the whole capability. Software defined and interconnected systems should be kept up to date or their will easily flip from strength to weakness.
If single arm is concentrated only to improve their inner capability, there is a danger that it would not support multiarms operations. With integrated force there should not be any separate development program nor project but at least coordination with clear milestones at force capability level. NATO and US Forces have utilized ongoing operation in Afghanistan to improve their integrated effects.
System of systems capability is not based on single subsystem functions but subsystems cooperation. Planning, design and testing should include always whole system of systems structure – either live or simulated.
All datatransfer issues in tactical environment should be modelled and simulated to have better understanding of volatile environment and evolving requirements. If tested scenario is following only existing procedures or topology, there is a danger of major blindness.
If no other modelling is utilized then a small world of larger system of systems implementation should be created as reference environment. Around this sandbox one can both test all changes in configuration but also experiment with possible new components.
One should be careful in introducing long term architecture decisions since installed legacy does not follow new good principles, but needs to be integrated. When introducing datamodel practices one should have clear understanding of the magnitude of one’s force and sample data from existing and possible situations. With this understanding one should start vigorous modelling and simulation before advancing to any implementation. Laboratory environment does not full truth and field tests are very expensive way to find out design failures.
Data and information flow in tactical domain is very much dependent on tactics and battle technics used. Adopting models from foreign forces does not give similar capability to your force until you change whole stack of DOTMLPFII components.
If customer is not able to create testing procedures or test environment with use cases, there is no clear understanding of required capability and the way how new subsystems are integrated to existing structure and processes. Testing, even in systems engineering way, is a sequence of tests that should utilize information collected from whole test track record.
If so called ‘show stopper’ emerges during testing, one should have parallel courses of action to proceed. Otherwise one failure in pure sequential procedure will become a major risk for whole project. One should not neglect failures but encapsulate them in order to proceed with other features and come back next time with remedy.
Modern software defined processes may be developed either tailored to customer defined processes or adapted as-is implemented in turn-key system. If force is not aiming to follow fully for example NATO based art of operations, tactics, order of battle and has similar weapon systems, one should not adapt NATO based processes, symbols or terminology as-is. Adaptation will most probably fail since there is no required maturity of tactical understanding or battle technical skills. Old habits are hard to get rid of at staff processes and combat level leading. It requires multitude of simulation training to get rid of old habits and adapt new ones.
With software defined technical systems also maintenance and sustenance are to be changed from legacy model of ‘box’ or component based repair. Immaterial asset management and software updates become major challenges for logistics.
Document driven development is suitable for small system development and quick delivery projects. ‘Sprints’ should have clear stories and prompt delivery. When complexity and interrelationships increases a more iterative development approach should be implemented with several parallel courses of action. Spiral development with many practical testing phases through laboratory, reference and pilot environments is more risk driven approach that is recommended for CIS system of systems development programs.
Normal military planning with different scenarios, decisive points, commander’s intention and ‘WHAT IF’ sensitivity testing should be implemented also in capability development programs. Clear customer driven approach normally helps to align number of different units along value chain because people has tendency to do meaningful work.
No comments:
Post a Comment