[Sc] PI 26.2 Objective 3: Open-Source Contributions

[jsignell]

Motivation/Description

In order to support scientists and promote usage of open source libraries, we need to make updates to upstream libraries.This includes bug fixes, maintenance, enhancements, and docs on any of the tools listed as best practices in the VEDA docs as well as their underlying libraries (xarray, dask, geopandas...) when necessary. As part of this work we plan to stay engaged in ongoing community conversations.

Success Criteria

• Upstream libraries are improved
• DSE is an active participant in discussions and aware of new efforts
• Upstream documentation is clear and helpful

Tasks

🟡on demand 🟣milestones 🔵ongoing services

• 🟣Coordinate reporting with ODD - Julia
• 🟣Upstream Open Source Contributions - Julia/Tyler
  • 🟣Pystac V2 rewrite effort - same API easier to maintain codebase
  • 🟣STAC Best Practices improvements
  • 🟣Stretch: Xarray backend for rio-stac-io (xpystac or other)
  • 🟣Stretch: Zarr bug with sharding (zarr-python #2834)
  • 🟣Stretch: Improve xarray performance of large datatrees (xarray #9511)
  • 🟣Stretch: Push down slicing to the store for uncompressed chunks in zarr-python
  • 🟣 Stretch: More informative representation for groups and arrays in zarr-python (zarr-python #2026)
  • 🟣 Stretch: Additional Openscapes R documentation contributions
  • 🟡 Upstream fixes to Jupyter, and other libraries Adhoc
• 🔵Community meetings
  • 🔵 STAC, OpenScapes, Pangeo, EarthAccess, xarray-developer
  • Conference presentations?
  • 🟡 Other
• 🟣Release Black Marble package - Tyler
  • User Guide
  • Make the repo public - ensure there is a license file
  • Release check with Science team
  • Release to pypi and conda-forge

InterDependencies

• Communicate with ODD to ensure that we are not stepping on each other’s toes.

Level of Effort

50% FTE

Risks

• External reviews

[maxrjones]

@jsignell, do you mind sharing your thoughts on an xarray issue?

@hrodmn and I have talked a lot about how xarray could benefit from smarter fsspec defaults. It occurred to me today that this is basically the 100x speedup applied in earthaccess-dev/earthaccess#1061, but at a more widely used level of the stack. What are your general thoughts on this idea? If it's a worthwhile idea, do you think this would fit under Sc objectives?

For transparency, I made the connection between our VEDA discussions about fsspec + xarray pain and the earthaccess PR in a meeting about a different collaboration with @espg.

[jsignell]

👀 I am reading through the issue now, but I was chatting about something potentially similar with @hrodmn yesterday about whether it would be feasible to get xarray to read the header first and then set the fsspec options based on those.

[jsignell]

It looks like the earthaccess PR uses the filesize to pick a sane default rather than reading the header, so maybe that is good enough?

Another option would be to just pick a static size - for instance 4MB or 8MB and just see what the performance improvement is there. It could be just switching the cache_type to "blockcache" with a constant block size would already represent a big improvement.

I took a peak at the xarray side and it looks like pydata/xarray#9797 gives a nice isolated mechanism for hooking this up specifically for hdf5 files. We'd basically just be setting defaults for that storage_option.