add pancreas dataset

[MUCDK]

No description provided.

[selmanozleyen]

Before going more into the fixes @MUCDK, I think there is a problem with this function _load_from_url

moscot/src/moscot/datasets.py

Line 535 in c699dc9

def _load_dataset_from_url(

.

In its code force_download is not being used at all. I also noticed it locally, that even if I set it to true it uses the cache

def _load_dataset_from_url(
    fpath: PathLike,
    type: Literal["h5ad", "h5mu"],
    *,
    backup_url: str,
    expected_shape: Tuple[int, int],
    force_download: bool = False,
    **kwargs: Any,
) -> Union[ad.AnnData, mu.MuData]:
    # TODO: make nicer once https://github.com/scverse/mudata/issues/76 resolved
    fpath = os.path.expanduser(fpath)
    if type == "h5ad" and not fpath.endswith(".h5ad"):
        fpath += ".h5ad"

    if type == "h5mu" and not fpath.endswith(".h5mu"):
        fpath += ".h5mu"

    if not os.path.exists(fpath):
        with tempfile.TemporaryDirectory() as tmpdir:
            tmp = pathlib.Path(tmpdir) / f"data.{type}"
            urllib.request.urlretrieve(backup_url, tmp)
            if type == "h5ad":
                data = ad.read_h5ad(filename=tmp, **kwargs)
            if type == "h5mu":
                data = mudata.read(tmp, **kwargs)
            with contextlib.suppress(FileNotFoundError):
                os.remove(fpath)
            shutil.move(tmp, fpath)
    else:
        if type == "h5ad":
            data = ad.read_h5ad(filename=fpath, **kwargs)
        else:
            raise NotImplementedError("MuData download only available with `force_download=True`.")

    if data.shape != expected_shape:
        data_str = "MuData" if type == "h5mu" else "AnnData"
        raise ValueError(f"Expected {data_str} object to have shape `{expected_shape}`, found `{data.shape}`.")

    return data

but I think the problem with the tests are still a different issue

[MUCDK]

Yeah true, can we adapt it such that it's used again? It used to be passed to scanpy.read(), but force_download is not supported by mudata

[selmanozleyen]

@MUCDK I used scanpy's download function (it does more checks and better error handling) and it seems to work. However I think due to MuData addition pytest 3.9 tests are failing (MuData is trying to import a class that is removed from AnnData AlignedViewMixin). Also the mypy errors are also in main branch.

[MUCDK]

Yeah this is why I changed the implementation.

I actually opened an issue, shouuld we continue on that side? wdyt? scverse/mudata#76

[selmanozleyen]

What I meant is I actually download with scanpy.readwrite._check_datafile_present_and_download because it seems like it handles url requests better. It works for the notebooks and the tests above 3.10.

I think the CI got an update and now the MuData doesn't support AnnData in python 3.9. I will check the versions if they fit. But I agree that this should be added to MuData so that it has a similar interface to scanpy

Update: Yes, it turns out now importing MuData is enough for 3.9 tests to fail. I ran the tests again to a commit that passed the tests before my changes https://github.com/theislab/moscot/actions/runs/10558788448/job/29875691253

[selmanozleyen]

@MUCDK I also realized that sparseargument was being ignored. Because scanpy doesn't have that option and it just ignores it.

[MUCDK]

we can deprecate python=3.9, then it should work ,right?

[selmanozleyen]

Yes, I checked MuData commits and thats the case.

[MUCDK]

ok then let's deprecate python 3.9, was planning to do that anyways, see #740