Software Framework

The Data Analytics Software Framework: DASF

The success of scientific projects increasingly depends on using data analysis tools and data in distributed IT infrastructures. Scientists need to use appropriate data analysis tools and data, extract patterns from data using appropriate computational resources, and interpret the extracted patterns. Data analysis tools and data reside on different machines because the volume of the data often demands specific resources for their storage and processing, and data analysis tools usually require specific computational resources and run-time environments. The data analytics software framework: DASF, which we developed in Digital Earth, provides a framework for scientists to conduct data analysis in distributed environments.

A Framework for Research

The Data Analytics Software Framework DASF supports scientists to conduct data analysis in distributed IT infrastructures by sharing data analysis tools and data. For this purpose, DASF defines a remote procedure call (RPC) messaging protocol that uses a central message broker instance. Scientists can augment their tools and data with this protocol to share them with others. DASF supports many programming languages and platforms since the implementation of the protocol uses WebSockets. It provides two ready-to-use language bindings for the messaging protocol, one for Python and one for the Typescript programming language. In order to share a python method or class, users add an annotation in front of it. In addition, users need to specify the connection parameters of the message broker. The central message broker approach allows the method and the client calling the method to actively establish a connection, which enables using methods deployed behind firewalls. DASF uses Apache Pulsar as its underlying message broker.

The Typescript bindings are primarily used in conjunction with web frontend components, which are also included in the DASF-Web library. They are designed to attach directly to the data returned by the exposed RPC methods. This supports the development of highly exploratory data analysis tools. DASF also provides a progress reporting API that enables users to monitor long-running remote procedure calls.

The Structure

DASF facilitates using data analysis tools in distributed IT infrastructures. It consists of three major modules:

  • DASF-Web collects all web components for the data analytics software framework DASF. It provides ready-to-use interactive data visualization components like time series charts, radar plots, stacked-parameter-relation (spr), and map components to support the visual analysis of spatio-temporal data. Moreover, DASF-Web includes the web bindings for the DASF RPC messaging protocol. It is implemented in Typescript and uses Vuejs/Vuetify, Openlayers and D3 as a technical basis.
  • DASF-Messaging-Python is a RPC (remote procedure call) wrapper library for the python programming language. As part of the data analytics software framework DASF, it implements the DASF RPC messaging protocol.
  • DASF-Progress-API provides a lightweight tree-based structure to be sent via the DASF RPC messaging protocol. Its generic design supports deterministic as well as non-deterministic progress reports. While DASF-Messaging-Python provides the necessary implementation to distribute the progress reports from the reporting backend modules, DASF-Web includes ready-to-use components to visualize the reported progress.

The Framework in Real Life Scenarios

We used this framework to create the Flood Event Explorer prototype, that combines multiple scientific workflows in order to address a common overarching research question in the domain of flood events. Moreover, we realized multiple generic stand-alone modules, implementing specific scientific methods, that are applicable to various problem sets. Further details on the Digital Earth Flood Event Explorer and the implemented workflows can be found here.

License

DASF is released under the Apache-2.0 license.

Software Citation

Eggert, Daniel; Dransch, Doris (2021): DASF: A data analytics software framework for distributed environments. GFZ Data Services. https://doi.org/10.5880/GFZ.1.4.2021.004

Contact

  • Daniel Eggert, Section Remote Sensing and Geoinformatics, GFZ: daniel.eggert(at)gfz-potsdam.de