-
Notifications
You must be signed in to change notification settings - Fork 0
Open
pydata/xarray
#11216Description
Explore the possibility of changing the default cache option in xarray to make the default experience better.
Spinning this into a new issue stemming from conversation which @maxrjones started in #3 (comment)
I did some initial exploration based off of @hrodmn's notebook https://gist.github.com/hrodmn/5a377531c19da6f6d2c2bb149c450b3d
!pip install git+https://github.com/jsignell/xarray.git@blockcache --no-deps
import time
import earthaccess
import xarray as xr
auth = earthaccess.login()
results = earthaccess.search_data(
short_name="NISAR_L2_GCOV_BETA_V1",
count=1,
cloud_hosted=True,
)
g = results[0]
s3_link = g.data_links(access="direct")[0]
creds = auth.get_s3_credentials(endpoint="https://nisar.asf.earthdatacloud.nasa.gov/s3credentials")
start_time = time.perf_counter()
datatree = xr.open_datatree(
s3_link,
engine="h5netcdf",
decode_timedelta=False,
phony_dims="sort",
storage_options=dict(
anon=False,
key=creds["accessKeyId"],
secret=creds["secretAccessKey"],
token=creds["sessionToken"],
),
)
read_time = time.perf_counter() - start_time
print(f"Read time: {read_time:.2f}s")I am getting ~3s on VEDA hub, but I'm not 100% positive I am cache busting properly
Reactions are currently unavailable
Metadata
Metadata
Labels
No labels
Type
Projects
Status
In progress