Why introduce new specifications instead of building on existing cloud-native standards?

General
1

Hi everyone,

I’ve been reading through the Margo specification and the roadmap over the last few days. Coming from industrial automation and cloud-native engineering, I’m trying to better understand the long-term architectural vision.

My impression so far is that many of the problems Margo aims to solve are already addressed by well-established cloud-native technologies and CNCF projects:

  • OCI Images
  • Kubernetes
  • Helm
  • GitOps (Argo CD / Flux)
  • SPIFFE/SPIRE
  • …

Industrial Edge platforms such as Siemens Industrial Edge, ABB, Schneider, Rockwell and others already provide application lifecycle management, fleet management and deployment capabilities. The implementations differ, but conceptually they solve very similar problems.

This leads me to a fundamental question:

Why introduce new Margo specifications instead of standardizing on existing cloud-native best practices wherever possible?

For example, if an application is already packaged as an OCI image, deployed through Helm and managed via GitOps, what additional value does a Margo-specific application model provide?

One concern I have is that Margo could unintentionally become another abstraction layer on top of already established industry standards. If vendors implement Margo differently internally while still maintaining their proprietary edge platforms, the interoperability benefits may remain limited.

Personally, I would find it more compelling if Margo clearly stated:

“Cloud-native standards first. Only define new specifications where industrial requirements cannot reasonably be addressed by existing CNCF standards.”

So my questions are:

  1. Is this the long-term philosophy of the project?
  2. Where does the working group currently see the limitations of existing cloud-native standards?
  3. Which parts are expected to remain vendor-specific, and which parts are intended to become fully interoperable through Margo?

I’m genuinely interested in understanding the architectural direction, as I believe avoiding unnecessary duplication of existing standards is essential for long-term adoption.

Looking forward to your thoughts.

Felix Kammerer

2

Hello Felix,
Thank you for asking questions.
And if you want to have a deeper conversation, feel free to reach out to me.
Andrii.Melashchenko@belden.com

To answer your concerns quickly, imagine a situation:
You are a battery producer with a well-established cell design. You scale by cloning cells, and each cell runs 5 apps.
You have 20 cells in your factory.

Now, to run the same 5 apps 20 times, you need https://landscape.cncf.io/
What will be the ROI of this investment?
What HW should I buy? It is the shop floor, a standard RPi would not work,
Firewall rules prohibit inbound connections…

K8S can solve almost any problem - the question is whether industrial customers’ use cases are a good fit for such a high price tag?

More here: https://youtu.be/tN_4B0CbiJw?si=xRmacGHPaYoTDE2F

Best regards,
Andrii

Andrii.Melashchenko@belden.com

3

I think we’re talking about two different questions.

I fully agree that a full Kubernetes stack is not the right solution for every industrial edge deployment.

However, today there are lightweight edge platforms such as Talos Linux, k3s or MicroK8s that significantly reduce this complexity while still using open cloud-native standards.

My concern is therefore not Kubernetes itself, but why Margo introduces new specifications instead of building on existing standards like OCI, Helm, Compose and GitOps wherever possible.

As someone working in industrial automation, I’d rather see less fragmentation and more reuse of existing standards.

Where does the working group draw that line?

4

Hello Felix,

Please check Edge Compute Devices | Margo Documentation page.

Margo is not reinventing CNCF, it reusing all standards that you mentioned, but adding a well defined workflow around them.

You can read more here:docs: SUP-00 The Helm Way — canonical OCI component publishing pattern by javatask · Pull Request #67 · margo/specification-enhancements · GitHub

Andrii

5

I share your concerns WRT “reinventing the (cloud) wheel” and I’ve raised it at least once already, but…

We have in the Margo community different participants asking for Docker Compose support and therefore K8s-only compatibility is not an option. Additionally I agree with @andrii that even those tiny K8s solutions (like K3s) are too big (always running daemon) for some of Margo’s target devices.

We are also using OCI as much as possible. I proposed OCI on the communication device ↔ management to have an OCI-based GitOps approach (like Flux does), but the voting went for a lightweight approach that reuses some of the concepts. I don’t see any other place where we could use it and aren’t.

Summarizing, we are aware of the cloud technologies that you are mentioning and try to reuse as much as possible, consider the reuse potential and then vote. I would say that at least Andrii and me are both pushing for the reuse of cloud technologies a lot, but they don’t always fit.

In any case, please feel free to make concrete proposals and we can discuss them.

6

Thanks for the insight.

Could you give a concrete example where existing cloud-native standards don’t fit?

For example, is there a specific industrial use case where OCI, Helm or GitOps cannot express the required semantics, making a Margo-specific specification necessary?

In our projects, we already build industrial edge platforms using technologies such as Talos Linux, GitOps (Argo CD), OCI and Helm. We deploy and manage edge devices this way, integrate machines through applications like Node-RED and even run industrial workloads such as soft PLCs, industrial apps or HMI integrations.

From that perspective, I’d really like to understand which industrial use cases cannot be addressed by extending existing cloud-native standards, and therefore require Margo-specific specifications instead.

7

Hello Felix,
We are happy you came and expressed your concerns. I would be happy to jump on a call and discuss any of the topics mentioned in this chat.

Key point in my feedback to your concerns:
I think about Margo as a collection of proven cloud native standards that are prescribed for OT environments. We only create what is necessary to enable the vendor interoperability the community is aiming to achieve. Our goal is always adopt proven technologies where available.

Could you give a concrete example where existing cloud-native standards don’t fit?

  • If this is a question of what Margo has created, outside of exisitng cloud-native standards, it really comes down to two items.
    • First, the Workload Fleet Managmeent(WFM) cloud to edge API was created to enable vendor interoperability between the Fleet manager and Device client
      • GitOps was introduced initially, but was not chosen as the standard interface for retrieving desired state files from the Workload Fleet Manager. Reasons can be found in the approved SUP linked here
      • Although your solutions utilize GitOps, many other cloud to edge managers maintain their own proprietary management interface, that requires device vendors to comply limiting largescale compute production.
      • Additional information on GitOps can be found here as well from a previous post to the community.
    • Second, Margo’s Application Description was created to enable application developers to describe their apps once, for inclusion in application catalogs and marketplaces. We did not find anything that fit our needs, mostly proprietary solutions requiring the app dev to manage one to many descriptors per WFM.
      • See details of the application description here

For example, is there a specific industrial use case where OCI, Helm or GitOps cannot express the required semantics, making a Margo-specific specification necessary?

  • Margo has adopted the following standards you are referencing holistically, with minimal limitations.
    • Helm
      • Minimal limitations regarding server side kubernetes API access.
    • OCI runtimes
    • OCI packaging and distribution
  • GitOps was the only solution from your list here that we did not adopt for reasons outlined above.

In our projects, we already build industrial edge platforms using technologies such as Talos Linux, GitOps (Argo CD), OCI and Helm. We deploy and manage edge devices this way, integrate machines through applications like Node-RED and even run industrial workloads such as soft PLCs, industrial apps or HMI integrations.

Some comments on the items you mentioned:

  • Talos Linux
    • This is considered a “black box” selection via the device supplier. Margo does not dictate a specific Linux or other OS.
  • Gitops
    • See notes above
  • OCI / Helm
    • Margo holistically adopts both of these technologies. OCI for runtime and packaging, and helm for manifests targeting kubernetes.

I appreciate @silvano and @andrii pitching in and replying.

@Felix I hope I addressed your concerns. If not, please follow up with additional questions/concerns.

Armand
TWG Chair

8

Hi Felix,

Allow me to provide my perspectives to.

First of all, thanks for your candid feedback. This is exactly what we hoped for by moderating this feedback forum. I’ll let the community members continue the discussion, but can confirm that MArgo’s DNA is to avoid reinventing the wheel as much as possible. (see our Margo promise)

As Margo chair, I was wondering how the community, and by extension the industry, could take advantage of your valuable experience.

Would your organization be interested joining Margo’s Advisory Board? You could be the representative for your organization.

This participation would allow you and your organization to actively participate in providign guidance to our 45+ members collaborating in the community.

If interested, we can setup a meeting to better understand such an engagement.

Of course, if you and your organization want to actively contribute to the Margo community, by making specification update proposals, defining strategy etc. you can always become a Margo member! We would be grateful for your contributions. Membership is free, with the option to sponser the project, if you wish to. https://margo.org/about/members/join/

Looking forward to hearing your thoughts!

Bart

Chair of Margo

9

Thanks for the detailed explanation. It definitely helped me understand the current architecture much better.

However, I still have a fundamental concern.

From my perspective, an open-source project under the Linux Foundation should primarily drive the industry towards open, widely adopted standards instead of adapting to today’s proprietary platform landscape.

Reading the discussion, I get the impression that some architectural decisions are influenced by the need to accommodate existing vendor-specific fleet management solutions. I understand the practical reasons for that, but I question whether this should be the direction of an open standard.

If vendors expose different proprietary management APIs today, shouldn’t the vendors evolve towards open standards instead of creating another abstraction layer on top?

Cloud-native has spent years converging around OCI, Kubernetes, Helm, GitOps and related standards. Platform Engineering is built on the idea of composing proven technologies instead of inventing new ones.

I completely understand that Kubernetes is not suitable for every edge device. We also use lightweight solutions where appropriate. But that still doesn’t explain why new specifications are preferable over extending or profiling existing open standards.

My concern is that Margo could unintentionally preserve today’s fragmented vendor ecosystem instead of helping to reduce it.

10

Hi Bart,

Thank you for the invitation. I really appreciate the openness of the Margo community and the constructive discussion.

To be honest, my available time is quite limited, and as a non-native English speaker I’m probably not the right person to actively drive specifications or participate in frequent meetings.

However, I do enjoy reviewing architectures and specifications from both an industrial automation and cloud-native perspective. I tend to challenge whether new specifications are really necessary or if existing open standards could be adopted instead.

If that kind of review and feedback is valuable for the project, I’d be happy to contribute whenever time allows.

11

Reading the discussion, I get the impression that some architectural decisions are influenced by the need to accommodate existing vendor-specific fleet management solutions.

Hrmm, that might be a misrepresentation of what a few from the community have described. Can you point to a specific concern you had in the conversation that lead you to this opinion?

If vendors expose different proprietary management APIs today, shouldn’t the vendors evolve towards open standards instead of creating another abstraction layer on top?

I believe you just described Margo :slight_smile:

Today, many fleet managers in the community, including those with public offerings, rely on proprietary management interfaces. BUT, many of those same vendors are actively participating in the development of the Margo specification to build exactly the open standard you’re describing.
You can read the full details of that open standard in the specification section of our documentation website.

12

You’re right, that wasn’t the best wording. :slightly_smiling_face:

I don’t think Margo is protecting proprietary platforms, and I actually think it’s great that all these vendors are working together on an open specification.

What I’m really trying to understand is the technical decision behind creating a new Workload Fleet Management API.

Was there a specific limitation in existing open approaches that made a new API necessary? Or was it mainly because today’s fleet managers all expose different proprietary interfaces?

I’m asking because I generally like the “adopt before invent” mindset. Whenever there’s already an established open standard, I’d rather see the industry converge around it than introduce another one.

Why the community decided that a new API was the better option?

13

What I’m really trying to understand is the technical decision behind creating a new Workload Fleet Management API.

Was there a specific limitation in existing open approaches that made a new API necessary?

What existing specification for a standard Workload Fleet Management API are you referring to?


The replies above attempted to respond to the questions about OCI, Kubernetes, Helm, GitOps, etc., so maybe there’s a misunderstanding over what your concerns are?

Kubernetes:

The responses above have indicated why we want more options than just Kubernetes.

Are you looking for the specific reasons why Margo doesn’t feel that only supporting Kubernetes-based devices is enough?

Helm

This also addresses why only using Helm isn’t enough.

Do you have reasons to believe that just using Helm should be enough, given the intent to support more than Kubernetes-based devices?

OCI

From the responses above, you can see Margo is trying to use OCI when/wherever possible. The one exception that was noted was with the desired state. You can see in the proposal’s voting results why members voted against using OCI for this.

There are also a couple of proposals being voted on soon that will make OCI the standard delivery package for all deployment types, not just Helm charts.

Are there places you see Margo still not using OCI in a way you would expect?

GitOps

The response above addresses some of this. The reasons members were against having a dependency on Git clients/libraries on a device were mentioned above and are also documented here.

The reasons for not wanting to use OCI to deliver the desired state were mentioned above.

Margo does not dictate implementation, so how the WFM chooses to implement the management of the desired state behind the scenes is not Margo’s concern. If a fleet manager wanted to apply some GitOps patterns internally, there is nothing preventing this.

Also, at least at this point, Margo is not dealing with “infrastructure as code.” Margo is dealing only with deploying applications (at this point), not using the fleet manager to build the host environments. While there will be cases where customers want to build their host environments, there are just as many cases where customers will be buying devices from suppliers that are pre-configured and are not allowed to be changed apart from installing applications.

Also, GitOps flows in one direction, so even if pure end-to-end GitOps patterns could be applied, it only solves the problem of getting the desired state to the device and is not able to be used for the device to communicate with the WFM.

Given this information, are there additional specific questions you have related to GitOps patterns?

SPIFFE/SPIRE

SPIFEE is (will be added soon anyway) part of the specification now. A SPIFFE-based identity and authentication framework proposal was recently approved.

Spire is an implementation. Margo’s specification does not dictate how the interfaces Margo defines are implemented. If implementers choose to use SPIRE to implement Margo’s identity and authentication framework, they can choose to do this, but we won’t dictate it.

I think we’re trying to address your questions, but maybe we’re being too general with the questions and answers. If you still have questions, maybe trying to make them as specific as you can might help us to answer the specific concerns you have.

14

I think the WFM API is the only part I’m still trying to understand.

From a Platform Engineering perspective, I’d normally expect the platform to build on existing open standards rather than define a new API.

Was that approach discussed during the design of the WFM API? Or was there simply no existing standard that fit the problem?

I’m just trying to understand where the community decided that creating a new specification was the better long-term option.

15

From a Platform Engineering perspective, I’d normally expect the platform to build on existing open standards rather than define a new API.

Was that approach discussed during the design of the WFM API? Or was there simply no existing standard that fit the problem?

The content on the Margo mission Statement provides a good foundation for what problem Margo is trying to solve.

The founding members were seeing and hearing from customers that the increasing introduction of siloed, vendor-specific, end-to-end edge solutions for delivering applications to the OT environments was becoming challenging to manage.

While vendors used some common technologies, patterns, and practices (ones you mentioned), every implementation was unique. There wasn’t an existing specification for solving the specific problems the Margo founders needed to solve in the way they needed to solve them.

There are some ongoing specification projects with parallels, but the founding members felt the scopes of these projects were much larger than the edge interoperability problems they wanted to solve. From the beginning, the goal was to keep Margo focused on solving specific edge interoperability concerns to control scope and speed up delivery of an implementable specification.

With Margo, we’re trying to balance two difficult things: interoperability and implementation choice. For OT use cases, there are too many special situations to be very explicit about how everything must be implemented, so we cannot dictate implementation. But on the flip side of that, if we don’t dictate enough, there is too much fragmentation to achieve interoperability. We need to give vendors choices, but still be opinionated about what choices they have. This is one of the reasons why Margo is focusing mostly on how app developers, device manufacturers, and fleet manager vendors communicate, not on implementation decisions.

16

Ty for the details! which existing open-source projects or specifications did the founding members evaluate?

For example, projects like Open Horizon (LF Edge), Eclipse ioFog,ä / hawkBit, Open Cluster Management, Cluster API, or even GitOps-based approaches.

Was there a specific technical limitation that made these approaches unsuitable, or was it mainly that none of them covered the required scope?

17

I’m assuming you’re specifically asking about the following APIs:

Correct?

There was no community-wide decision not to use any of the projects you’ve referenced specifically for these APIs. Each of the technical working group members is free to evaluate existing specs/technologies and propose specification changes following our specification update proposal(SUP) processes. You can see we have several in-progress SUPs proposing various changes to the specification. The previous comments also highlight some areas where SUPs have been submitted and approved already.

When the project first kicked off, there was an in-person meeting where the original technical leads white-boarded out some ideas for the direction we wanted to head. The initial takeaway was to acknowledge we needed some way to get the desired state to the devices, which we originally indicated would be a declarative approach using Git repositories, as indicated in previous comments. We also knew we needed to get some information from the device to the fleet manager, so we needed the device capabilities API. So, the APIs started very basic, solving specific problems we knew needed to be solved. As mentioned in the previous reply, scope is important, so we want to be careful about adding things to the APIs that aren’t necessary.

From this point, the specification has continued evolving as we’ve identified problems and solutions. For example, as mentioned above, the technical working group recently approved using SPIFFE for identity and authentication. The early specification drafts had basic certificate exchange, which we knew wasn’t suitable enough. So, following the SUP process, a member submitted a proposal to incorporate the SPIFFE specification into Margo. As we’ve indicated, we’re not looking to “reinvent the wheel,” so when there are pre-existing things that might fit, we want members to propose those ideas for consideration. But at the same time, we don’t want to use something pre-existing if it also brings along things we don’t need.

As Margo continues to evolve and finds solutions to the problems Margo needs to solve, if a technical contributor feels there is an existing specification that addresses the problems and isn’t introducing additional things Margo doesn’t need, then they are free to create an update proposal for the technical working group to vote on. I would expect that as we continue to work towards solving these problems before our official general availability 1.0 release, there will be other places we are able to incorporate other existing specifications as well.

I hope this addresses your questions. If it doesn’t, I apologize because I’m having a difficult time trying to understand exactly what information you’re looking for.