ActiveFolders

 

ActiveFolders is a data transfer tool designed for ease of use within the research and HPC community. ActiveFolders uses the 'science DMZ' pattern to separate and optimise local-area network traffic and wide-area traffic.

ActiveFolders

The Challenge:

To design, implement, and deploy a shared folder service for data, specifically for the research and HPC community. The challenges of a shared folder for this community include the large size of some of the files and the security requirements of data centres. Existing tools such as Dropbox are not well-­optimized to transport and synchronize tens of thousands of files, amounting to dozens of gigabytes or terabytes of data, across high-speed networks. In addition, tools like drop Dropbox itself have limited native encryption abilities. Finally, research and HPC centres would prefer familiar authentication mechanisms, such as grid certificates or standard public/private keys instead of proprietary solutions.

The Solution:

ActiveFolders is a high-speed data transfer tool designed to meet the security and access needs of scientists working on large datasets or computation files that need to be transferred to another computer. Traditionally, users had to use a system of tools and techniques to move their large files, which required in-depth technical learning and know-how. Otherwise, they simply had to put up with long download times. ActiveFolders adopts the familiar concept of a “shared folder” for files that are synchronized between multiple computers and utilizes existing tools and architectures to create a data-movement service for researchers.

How does it work and how is it different from Globus:

ActiveFolders is an experimental platform for data sharing, where file transfer is but one use of our technology. Beyond what the Compute Canada Globus Portal supports (offered by Compute Canada), ActiveFolders also supports file synchronization (not just transfer), automatic detect-and-transfer functionality, and a Python-based application programmer's interface (API) that Globus does not. This platform is open source and can be downloaded from the CANARIE Research Software Registry.

Who should use this:

Canadian researchers with specialized needs such as file synchronization, long lived security credentials, or would like to integrate or customize a file synchronization functionality into their system, are invited to to download and utilize ActiveFolders.

Timeline:

ActiveFolders was funded by CANARIE and ran from:

  • Development phase: January 1 – November 2014
  • Maintenance phase: December 1, 2014 – March 31, 2015

Participants: