NAOC Open IR  > 光学天文研究部
DALiuGE: A graph execution framework for harnessing the astronomical data deluge
Wu, C.1; Tobar, R.1; Vinsen, K.1; Wicenec, A.1; Pallot, D.1; Lao, B.2; Wang, R.3; An, T.2; Boulton, M.1; Cooper, I.1; Dodson, R.1; Dolensky, M.1; Mei, Y.4,5; Wang, F.4,5
2017-07-01
Source PublicationASTRONOMY AND COMPUTING
Volume20Pages:1-15
AbstractThe Data Activated Liu(1) Graph Engine - DALiuGE(2) - is an execution framework for processing large astronomical datasets at a scale required by the Square Kilometre Array Phase 1 (SKA1). It includes an interface for expressing complex data reduction pipelines consisting of both datasets and algorithmic components and an implementation run-time to execute such pipelines on distributed resources. By mapping the logical view of a pipeline to its physical realisation, DALiuGE separates the concerns of multiple stakeholders, allowing them to collectively optimise large-scale data processing solutions in a coherent manner. The execution in DALiuGE is data-activated, where each individual data item autonomously triggers the processing on itself. Such decentralisation also makes the execution framework very scalable and flexible, supporting pipeline sizes ranging from less than ten tasks running on a laptop to tens of millions of concurrent tasks on the second fastest supercomputer in the world. DALiuGE has been used in production for reducing interferometry datasets from the Karl E. Jansky Very Large Array and the Mingantu Ultrawide Spectral Radioheliograph; and is being developed as the execution framework prototype for the Science Data Processor (SDP) consortium of the Square Kilometre Array (SKA) telescope. This paper presents a technical overview of DALiuGE and discusses case studies from the CHILES and MUSER projects that use DALiuGE to execute production pipelines. In a companion paper, we provide in-depth analysis of DALiuGE's scalability to very large numbers of tasks on two supercomputing facilities. (C) 2017 Elsevier B.V. All rights reserved.
SubtypeArticle
KeywordDataflow Graph Execution Engine Data Driven Square Kilometre Array Many-task Computing
WOS HeadingsScience & Technology ; Physical Sciences ; Technology
Funding OrganizationAustralian Government ; Australian Government ; Government of Western Australia ; Government of Western Australia ; Amazon Web Services ; Amazon Web Services ; AstroCompute project ; AstroCompute project ; National Natural Science Foundation of China (NSFC)(61221491 ; National Natural Science Foundation of China (NSFC)(61221491 ; Open Fund from HPCL(201401-01) ; Open Fund from HPCL(201401-01) ; State Government of Western Australia ; State Government of Western Australia ; U1231205 ; U1231205 ; U1631129) ; U1631129) ; Australian Government ; Australian Government ; Government of Western Australia ; Government of Western Australia ; Amazon Web Services ; Amazon Web Services ; AstroCompute project ; AstroCompute project ; National Natural Science Foundation of China (NSFC)(61221491 ; National Natural Science Foundation of China (NSFC)(61221491 ; Open Fund from HPCL(201401-01) ; Open Fund from HPCL(201401-01) ; State Government of Western Australia ; State Government of Western Australia ; U1231205 ; U1231205 ; U1631129) ; U1631129)
DOI10.1016/j.ascom.2017.03.007
Indexed BySCI
Language英语
Funding OrganizationAustralian Government ; Australian Government ; Government of Western Australia ; Government of Western Australia ; Amazon Web Services ; Amazon Web Services ; AstroCompute project ; AstroCompute project ; National Natural Science Foundation of China (NSFC)(61221491 ; National Natural Science Foundation of China (NSFC)(61221491 ; Open Fund from HPCL(201401-01) ; Open Fund from HPCL(201401-01) ; State Government of Western Australia ; State Government of Western Australia ; U1231205 ; U1231205 ; U1631129) ; U1631129) ; Australian Government ; Australian Government ; Government of Western Australia ; Government of Western Australia ; Amazon Web Services ; Amazon Web Services ; AstroCompute project ; AstroCompute project ; National Natural Science Foundation of China (NSFC)(61221491 ; National Natural Science Foundation of China (NSFC)(61221491 ; Open Fund from HPCL(201401-01) ; Open Fund from HPCL(201401-01) ; State Government of Western Australia ; State Government of Western Australia ; U1231205 ; U1231205 ; U1631129) ; U1631129)
WOS Research AreaAstronomy & Astrophysics ; Computer Science
WOS SubjectAstronomy & Astrophysics ; Computer Science, Interdisciplinary Applications
WOS IDWOS:000412033600001
Citation statistics
Cited Times:1[WOS]   [WOS Record]     [Related Records in WOS]
Document Type期刊论文
Identifierhttp://ir.bao.ac.cn/handle/114a11/20102
Collection光学天文研究部
Affiliation1.Univ Western Australia, ICRAR, M468,35 Stirling Highway, Perth, WA 6009, Australia
2.Shanghai Astron Observ, Shanghai, Peoples R China
3.Oak Ridge Natl Lab, Oak Ridge, TN 37831 USA
4.Kunming Univ Sci & Technol, Kunming, Yunnan, Peoples R China
5.Chinese Acad Sci, Yunnan Observ, Kunming, Yunnan, Peoples R China
Recommended Citation
GB/T 7714
Wu, C.,Tobar, R.,Vinsen, K.,et al. DALiuGE: A graph execution framework for harnessing the astronomical data deluge[J]. ASTRONOMY AND COMPUTING,2017,20:1-15.
APA Wu, C..,Tobar, R..,Vinsen, K..,Wicenec, A..,Pallot, D..,...&Wang, F..(2017).DALiuGE: A graph execution framework for harnessing the astronomical data deluge.ASTRONOMY AND COMPUTING,20,1-15.
MLA Wu, C.,et al."DALiuGE: A graph execution framework for harnessing the astronomical data deluge".ASTRONOMY AND COMPUTING 20(2017):1-15.
Files in This Item:
There are no files associated with this item.
Related Services
Recommend this item
Bookmark
Usage statistics
Export to Endnote
Google Scholar
Similar articles in Google Scholar
[Wu, C.]'s Articles
[Tobar, R.]'s Articles
[Vinsen, K.]'s Articles
Baidu academic
Similar articles in Baidu academic
[Wu, C.]'s Articles
[Tobar, R.]'s Articles
[Vinsen, K.]'s Articles
Bing Scholar
Similar articles in Bing Scholar
[Wu, C.]'s Articles
[Tobar, R.]'s Articles
[Vinsen, K.]'s Articles
Terms of Use
No data!
Social Bookmark/Share
All comments (0)
No comment.
 

Items in the repository are protected by copyright, with all rights reserved, unless otherwise indicated.