Subject: Add an ARM CI for Hadoop


> On Aug 11, 2019, at 11:55 PM, Zhenyu Zheng <[EMAIL PROTECTED]> wrote:
> So, is here the correct place to ask for help and propose to add this CI?

This is the place to come ask questions related to Apache Yetus, not necessarily any particular project’s usage of it.

> We are willing to provide resources for the CI and man power to setup the
> CI and keep it green in the future. We have already found some issues and
> started to working on fixing them in the community, such as
> https://issues.apache.org/jira/browse/HADOOP-13363 , but it will be better
> to show the process of what we have fixed and overall view of what is still
> wrong if we can have an ARM CI in upstream. Here is an example for what we
> have recently done for Apache Flink:
> https://issues.apache.org/jira/browse/INFRA-18822

There are two routes to take:

#1: Getting an ARM machine hooked up to the ASF Jenkins system

INFRA is your best contact for doing that.  After that, you’ll need to work with the Hadoop community on getting jobs/processes in place to make ARM a first class citizen. (One of the big ones will be making an ARM-specific Dockerfile!)

Unfortunately, the folks in charge of Hadoop have a track record of not being particularly friendly to contributions outside of the clique or if it doesn’t further goals by their employers. The only way to really grab their attention is to start an email campaign on [EMAIL PROTECTED].  Just be aware that you’ll need to be extremely persistent to get anywhere.  It’s probably also worth mentioning that attempts to get other architectures/OSes (Windows, PowerPC, OS X, etc) as first class CI tests have failed multiple times because, again, the majority of the community’s employers don’t work with that architecture and do not have a vested interested in making it succeed.

#2: Going a completely different route with OpenLab CI

If you plan on following the same route as you did with Apache Flink (bypass Jenkins and do something completely custom), then you’ll likely be interested in integrating Apache Yetus with OpenLab CI. This is where we can help.  But to get started, you’ll want to create some robot support:

Code:

https://github.com/apache/yetus/tree/master/precommit/src/main/shell/robots.d

Docs:

http://yetus.apache.org/documentation/in-progress/precommit-robots/

I’d recommend starting with a source tree you know well or is relatively simple while building the support before tackling Hadoop straight on. Yetus tends to shed a lot of light on short-signed CI system design so there may be some workarounds required and/or new Yetus APIs we’ll need to add.  To make matters worse, Hadoop is notoriously broken in lots of ways, so has a tendency to light up quality test results that will make it seem like your robot code is broken.

After you get the robot support, you’re likely back to talking with the Hadoop community about how to integrate your CI system with their contribution procedures. So ultimately, you’ll be back to convincing them that this is something important to work on.  You really need to find a champion on the PMC to help make it happen.

Good luck!