Background

These days, developers In digital pathology frequently use OpenSlide to read whole slide images (WSIs). Since WSIs are typically large, transferring and sharing them can be challenging. Cloud storage provides a scalable, durable, and secure solution for storing and sharing WSIs remotely. By mounting cloud storage as a local file system, existing tools can access WSIs without additional modifications.

AWS S3 (Simple Storage Service) and MinIO (S3 compliant, onsite and open-source) are a widely used, robust, and highly scalable cloud storage service. While Mountpoint for AWS S3 is a promising option for treating cloud storage as a local file system, several open-source projects also allow users to mount an S3 bucket as a local file system. Among the most well-known are s3fs and Goofys. In this post, we’ll compare the performance of these two popular open-source solutions.

What are s3fs and goofys

s3fs is an open-source FUSE (Filesystem in Userspace) module that allows users to mount an Amazon S3 (or Minio) bucket as a local file system on Linux or macOS. This enables seamless interaction with S3-like storage using standard file operations, making it appear as part of the local file system.

Goofys, on the other hand, is an open-source FUSE-based file system designed for performance. It optimizes metadata operations and makes trade-offs in POSIX compliance to achieve higher efficiency. Goofys allows users to mount an S3 bucket as a local file system on Linux, macOS, and other Unix-like systems, offering a faster and more efficient way to access cloud storage.

what is the difference between s3fs and goofys?

s3fs provides POSIX-compliant behavior, meaning it supports standard file operations like chmod, mv, and chown. However, this compliance comes at a cost—s3fs frequently communicates with S3 to maintain consistency, making it slower, especially for workloads involving frequent small file operations. While it supports caching to improve performance, its reliance on S3’s object storage model can still be inefficient for high-throughput tasks. It is best suited for scenarios requiring strict POSIX compatibility, such as applications expecting a traditional filesystem interface. To use s3fs, you need an AWS profile (set up an AWS profile) or a .passwd-s3fs file (configure .passwd-s3fs).

Goofys, on the other hand, prioritizes performance over full POSIX compliance. It handles S3 metadata more efficiently, making it significantly faster than s3fs for many operations, particularly for large files and streaming reads/writes. However, since it does not fully support operations like chmod, chown, or truncate, it may not work well with software that depends on those features. Goofys is a great choice for workloads that need high-speed access to S3, such as data processing pipelines or applications that don’t require strict file system semantics.

If you have any questions about installation and configuration of s3fs or goofys, please check their GitHub repository – s3fs and goofys

Test environment setup

For the s3fs test, we mounted S3 directly and ran s3fs on bare metal.

For the Goofys test, we evaluated two scenarios. First, we mounted the S3 bucket directly and ran Goofys on bare metal. Second, we mounted the S3 bucket using Goofys on the host OS and then mounted that folder into a running Docker container. The test results showed no significant difference between the two approaches.

The following steps outline the setup process in a Docker container for Goofys:

• Edit /etc/fuse.conf and enable user_allow_other
• Mount the S3 bucket as a local folder using Goofys:
  goofys -o allow_other <s3_bucket> <local/path/folder>
• Run the Docker container and mount the folder: docker run -it -v <local/path/folder>:<container/path/folder> <docker_image> bash

Tip: To allow other users (including processes inside Docker) to access the mounted S3 bucket, you must enable user_allow_other in /etc/fuse.conf and use the -o allow_other flag when running Goofys.

Software and hardware specifications used for testing:
s3fs – 1.90
Goofys – 0.24.0
OS: Ubuntu 22.04.4 LTS 64-bit
CPU: 13th Gen Intel® Core™ i5-1350P × 16
Memory: 32GB
Disk: 1TB SSD

Test results

The test compared s3fs on bare metal vs Goofys on bare metal. We use 7 various sizes of WSI files as test samples. For each WSI, we use openslide to test and time to initialize an OpenSlide object, as well as the get_thumbnail, and read_region functions on local, s3fs mounted and goofy mounted.

Local: directly read the WSIs from local storage.
s3fs: mounted S3 directly via s3fs and read the WSIs from mounted folders.
goofys: mounted S3 directly via goofys and read the WSIs from mounted folders.

The following table shows the test results:

name	size(mb)	mode	open_time(s)	thumbnail_time(s)	read_time(s)
0590XY112001_01_06.svs	1585.873	local	0.0022	0.2053	0.0214
		goofys	0.0517	0.2752	0.0358
		s3fs	3049.7225	395.4129	1033.0831
0590XY112001_01_07.svs	2224.478	local	0.0024	0.2467	0.0232
		goofys	0.0575	0.2927	0.0387
		s3fs	2724.0224	613.4214	977.6505
0590XY112001_01_08.svs	1715.34	local	0.0037	0.1849	0.0214
		goofys	0.0598	0.237	0.0325
		s3fs	2935.5364	401.4624	959.4518
HTT-TILS-001-03B.ndpi	229.806	local	0.0009	0.0059	0.0104
		goofys	0.0059	0.0068	0.046
		s3fs	72.155	0.193	6800.1757
HTT-TILS-001-26B.svs	301.336	local	0.0044	0.1381	0.0486
		goofys	0.0686	0.145	0.0575
		s3fs	941.999	70.6746	640.7553
HTT-TILS-001-27B.ndpi	238.319	local	0.0009	0.011	0.015
		goofys	0.0052	0.0118	0.0635
		s3fs	71.6927	0.2181	5610.9871
HTT-TILS-001-37B.svs	1126.718	local	0.0078	0.4373	0.0489
		goofys	0.1063	0.5584	0.0613
		s3fs	1733.025	173.3216	748.63

We calculated the mean, min, and max of the time of executing OpenSlide, read_region, and get_thumbnail in different ways.

For executing OpenSlide time:

mode	mean	min	max
goofys	0.050714	0.0052	0.1063
local	0.003186	0.0009	0.0078
s3fs	1646.879000	71.6927	3049.7225

For executing read_region time:

mode	mean	min	max
goofys	0.047900	0.0325	0.0635
local	0.026986	0.0104	0.0489
s3fs	2395.819071	640.7553	6800.1757

For executing get_thumbnail time:

mode	mean	min	max
goofys	0.218129	0.0068	0.5584
local	0.175600	0.0059	0.4373
s3fs	236.386286	0.1930	613.4214

FYI: The s3fs and Goofys test results are only for reference since they are affected by the speed of the local network status.

Conclusion

Mountpoint for S3, Goofys, and s3fs are tools that enable applications expecting traditional file systems to interact with Amazon S3. s3fs is ideal for scenarios requiring POSIX compatibility, such as backup, archiving, and caching, though it comes with performance trade-offs. Goofys prioritizes speed and efficiency, making it a great choice for lightweight, resource-constrained environments where full POSIX support isn’t necessary. For enterprise-ready, high-performance S3 access, Mountpoint for S3 is the best option, offering fast, reliable throughput, particularly for read-heavy applications like data lakes and machine learning. When choosing an S3 access method, consider whether compatibility, simplicity, or performance best aligns with your project’s needs.