Developers, Developers, Developers
Nur ein Idiot glaubt, aus den eigenen Erfahrungen zu lernen. Ich ziehe es vor, aus den Erfahrungen anderer zu lernen, um von Vornherein eigene Fehler zu vermeiden.
As a computing buff, I would like to see more CPU architectures flourish. But of late I have seen quite a few manufacturers with interesting and innovative systems stumbling when attempting to engage with the ‘non-commercial’ world of open-source software.
These new and innovative systems need good software support to be viable — because, as the old aphorism goes, “silicon without software is just sand rearranged”. Since open-source software dominates the software landscape these days,[1] these new manufacturers need open-source software to work well on their platforms, lest their products remain at the ‘sand rearranged’ stage.
I hope the suggestions below could help these firms to engage better with the open-source world.
First, a few engagement strategies that do not work well in practice.
A personal incident (with a few details changed): at one point I was asked for “help” with an innovative system that used a non-mainstream CPU. The system looked interesting — it had a clean instruction set and its specifications were also impressive. But it was also priced out of the reach of most individual users 🫤. To the best of my knowledge, nobody in my circle used those machines.
So I asked the company, “That is a cool system — what do you intend to do to seed your hardware among developers?”
The answer was: “Nothing.”
The most the company was willing to do was to offer remote access to one of their systems. In return for this “favour” to the open-source community (i.e., of granting one developer remote access to one machine) they expected the open-source community to rally around and improve software support for their platform, for free.
Consider what was being asked: the company was asking a skilled developer to spend multiple weeks studying an unfamiliar CPU and system architecture, coming up to speed on all of the nit-picky detail needed to improve software support for their platform, and then implementing the changes needed to solve whatever problem they were facing, all for free. The developer was then to support their platform indefinitely into the future, also for free.
The person seemed puzzled when I declined to engage.
This is not the first time that corporates have reached out to me with a request that I work on some item that was of interest to them.[2] It appears that some decision-makers in the corporate world believe that open-source developers are somehow motivated to solve other people’s problems, for free.[3]
But, sadly, this belief in the munificence of open-source developers is misguided. For the most part, open-source developers work for themselves, no different from any other kind of software developer:
-
They may work for pay at a corporate-sponsored open-source project.
-
Or, they may work on a university-run open-source project, perhaps while earning their degree.
-
Or, they may, as volunteers, take up problems that are of personal interest to them.
I have also observed manufacturers assembling in-house development teams to port and maintain key open-source software for their new platform. While this approach could be more effective than asking people to work for them for free, it too does not scale well.
Consider a typical real-world problem: a customer reports that the Chromium web browser’s video playback ‘stutters’ noticeably on the new platform when a Javascript-heavy page is scrolled. Where would the in-house developers need to look to find the root cause of this behavior?
-
Is the bug inside Chromium’s video playback subsystem?
-
Or, is the jumpiness caused by an undesirable interaction between Chromium’s built-in task scheduler and Pthreads?
-
Or, is there a bug in the Javascript (V8) runtime?
-
Or, is the root cause inside the OS kernel’s thread scheduling code? Could it be that deadline scheduling in the kernel is occasionally missing a deadline?
-
Or, is there unexpected thermal throttling happening somewhere? If so, why?
-
Or, could the stutter be caused by bursty network behaviour?
-
Etc.
The range of expertise needed to tackle real-world problems is simply enormous.
Even assuming that the in-house team can track down the root cause of that jitter and implement a fix, it could be difficult to persuage upstream open-source projects to accept platform-specific changes for a platform that its developers do not use.
Let us also look at the economics involved: let us assume for the sake of this discussion that the bill-of-materials cost for the new platform is $5,000 (let us assume that the manufacturer does not yet have the economies of scale on their side). Nevertheless, this high BOM cost is still a fraction of the cost to the company of hiring an engineer with the necessary depth, assuming a CTC of $250,000 to $300,000 per annum per engineer.
Then again, it would be hard to find a single engineer with all of the expertise needed to solve the possible issues that crop up in real-world deployments (e.g. “this particular SQL query is slow on your platform, but not on X86”). At the same time, there will not be a sufficient number of such issues in any one area to justify paying for a full-time developer for that particular area.
Companies that succeeded with new platforms (the Raspberry Pi & the Beagle Board) spent considerable effort nurturing their developer ecosystems.
They took the following steps:
-
Taking steps to make hardware available at a reasonable price.
The rPI and Beagle devices were pitched at an affordable price point from the start but many systems have a higher BOM cost. For more expensive systems one way to seed the developer ecosystem could be to offer the devices ‘at cost’ or at a reduced price to interested developers.
-
Providing good quality documentation.
It is hard to over-emphasize how important this aspect is. The availability of high-quality documentation for the rPI platforms had a huge positive influence on developer adoption. Budget for quality documentation from the start.
-
Focusing on ‘play’ first, and on ‘work’ later.
This is an important point that is often missed out.
A large number of open-source developers work on their projects out of personal interest, i.e., more as ‘play’ rather than as ‘work’. Successful platforms acknowledge this aspect of open-source, and try to make their platforms “cool” for people to use.
-
Supporting software diversity.
There are usually multiple implementations of basic software building blocks in open-source: multiple kernels and operating systems, many programming languages, many compilers, web browsers, databases, web servers, and so on. This is in part due to the ‘play’ aspect of open-source.
Instead of picking favorites, successful platform makers provide the mechanism for people to support themselves (e.g., good documentation) and then they step out of the way of the resulting innovation.
-
Showing value alignment.
The value being generated by the open-source movement is enormous.
So say “thank you”, demonstrate gratitude, and ‘pay it forward’ in some way — that kind of attitude will be noticed and will make it more likely that developers will spend their valuable time on your platform.
Without widespread developer adoption the commercial market cannot take the risk of committing to a new platform — the ‘long tail’ of potential software problems for a new platform cannot be managed without adequate developer support. Microsoft executive Steve Ballmer’s exhortation remains as true today as when it was in the year 2000:
The key to industry transformation; the key to success is: developers, developers, developers, developers, developers, developers, developers, developers, developers, developers, developers, developers.
Microsoft Developer Conference, 2000