Save intermediate results

[wangjiawen2013]

Hi,
Dynamo performed well on my datasets. However, I have trouble on saving the results. Now there are intermediate results stored in adata.uns that can may lead to errors when writing the h5ad object. Sometimes I cannot run all dynamo pipeline at one time, so I must store the intermediate results and load it again another day to continue. In this case, I must store it in pickle file and load it another day. However, for some datasets the objects is too large for pickle to handle it (verflowError: cannot serialize a bytes object larger than 4 GiB). Hope we can save dynamo in h5ad without loss information.

[Xiaojieqiu]

Dear @wangjiawen2013 , the current simplest way to get over this that you can run dyn.cleanup(adata) before saving the h5ad. When you load in the adata, you may need to use dyn.convert2float function.

Of course these are general advice, if you provide more details, we can address in more details.

[wangjiawen2013]

dyn.pp.recipe_monocle(adata) and dyn.cleanup(adata) cause the problem. I saves the adata after running dyn.pp.recipe_monocle(adata) and dyn.cleanup(adata), then load adata using dyn.read(adata), the following dyn.tl.dynamics cannot be performed.

In [14]: dyn.tl.dynamics(adata,model="stochastic",cores=10)
|-----> dynamics_del_2nd_moments_key is None. Using default value from DynamoAdataConfig: dynamics_del_2nd_moments_key=False

KeyError Traceback (most recent call last)
Cell In[14], line 1
----> 1 dyn.tl.dynamics(adata,model="stochastic",cores=10)

File ~/programs/miniconda3/envs/cellpy/lib/python3.10/site-packages/dynamo/tools/dynamics.py:300, in dynamics(adata, filter_gene_mode, use_smoothed, assumption_mRNA, assumption_protein, model, est_method, NTR_vel, group, protein_names, concat_data, log_unnormalized, one_shot_method, fraction_for_deg, re_smooth, sanity_check, del_2nd_moments, cores, tkey, **est_kwargs)
298 raise ValueError(f"\nPlease run dyn.pp.receipe_monocle(adata) before running this function!")
299 if tkey is None:
--> 300 tkey = adata.uns["pp"]["tkey"]
301 (experiment_type, has_splicing, has_labeling, splicing_labeling, has_protein,) = (
302 adata.uns["pp"]["experiment_type"],
303 adata.uns["pp"]["has_splicing"],
(...)
306 adata.uns["pp"]["has_protein"],
307 )
309 X_data, X_fit_data = None, None

KeyError: 'tkey'

[wangjiawen2013]

I found that after loading the adata, the 'tkey' and 'norm_method' (both the value is None) were lost !
In [21]: adata.uns['pp']
Out[21]:
{'has_splicing': True,
'has_labeling': False,
'splicing_labeling': False,
'has_protein': False,
'tkey': None,
'experiment_type': 'conventional',
'norm_method': None}

In [22]: adata.write("test.h5ad")

In [23]: adata = dyn.read("test.h5ad")

In [24]: adata.uns['pp']
Out[24]:
{'experiment_type': 'conventional',
'has_labeling': False,
'has_protein': False,
'has_splicing': True,
'splicing_labeling': False}

[wangjiawen2013]

Perhaps it's a bug of anndata
scverse/anndata#673

[Xiaojieqiu]

thanks! we will wait for the anndata team for a solution for now

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 14 days