Open Source Summit + ELC North America 2019

High performance computing (HPC) community is increasingly demanding big data processing beyond traditional simulation-based computation. Hadoop ecosystem has a roadmap that includes HPC support including GPU, FPGA. With HPC and Big-data converging into one huge ecosystem, we launched the Chameleon project to develop a HPC based big-data platform operation management system.
Chameleon was developed based on Apache Ambari, which is well-known Hadoop management system and extended to support Lustre filesystem management, which is widely used in HPC community for massive storage and HPC resource monitoring including GPU and Infiniband. Chameleon also added advanced YARN app monitoring functionalities based on Linux performance tools so that rich linux performance tools from Linux ecosystem can be dynamically used for YARN app monitoring. Finally, Chameleon has dynamic dashboard for Hadoop and HPC which streamlines HPC based Big-data platform operation and management.