Specializing Data Access in a Distributed File System (Generative Pearl)
We propose DDLoader, a system that embeds information about data partitioning and data distribution in distributed file systems using a metaprogramming framework. We demonstrate that this technique has practical benefits for building applications that interact with distributed file system applications. By using metaprogramming, we bring the traditional benefits of staging, such as partial evaluation to the domain of distributed file system access. Furthermore, this approach also fuses the data access code with the computation code, allowing end-to-end optimization. We test our framework on a real-world use case and show that this approach allows users to generate code that performs faster than traditional data loading by up to 12.56x on a single thread and even more when parallelized.
Tue 22 OctDisplayed time zone: Pacific Time (US & Canada) change
11:00 - 12:30 | |||
11:00 30mTalk | Yin-Yang: Concealing the Deep Embedding of DSLsMost Influential Paper Award GPCE Vojin Jovanovic Oracle Labs, Amir Shaikhha University of Edinburgh, Sandro Stucki Amazon, Vladimir Nikolaev , Christoph E. Koch EPFL, Martin Odersky EPFL | ||
11:30 30mTalk | Applying Self-Extension Mechanism to DSLs for Establishing Model Libraries GPCE Arkadii Gerasimov RWTH Aachen University, Nico Jansen Software Engineering, RWTH Aachen University, Judith Michael RWTH Aachen University, Bernhard Rumpe RWTH Aachen University | ||
12:00 30mTalk | Specializing Data Access in a Distributed File System (Generative Pearl) GPCE |