Add support to dill serializer

[GregoirePelegrin]

Fixes #405.
Must be merged along with celery/celery#9100

[GregoirePelegrin]

I can't seem to find the requirements file to add dill>=0.3.8 in it...

[auvipy]

I'm thinking of compatibility between dill and pickle

[auvipy]

E ModuleNotFoundError: No module named 'dill'

[GregoirePelegrin]

Yes, as I've stated in my previous comment, I've failed to locate the requirements file. I haven't been able to include dill, which is the cause of this error. When it comes to the compatibility, the admittedly limited tests I've done have thus far been successful. This said, I am not aware of every uses made by the community, and I only tested simple objects.

[Nusnus]

Converting to draft until the PR is ready and working.

Thank you 🙏

[GregoirePelegrin]

Converting to draft until the PR is ready and working.

Thank you 🙏

I am currently searching for the place to add the dill requirement. It seems there are no packages imported for now, so no requirements.txt exists for the package (not for the tests). I assume that adding the dependency to the test environment would solve the testing issue but would lead to errors when trying to use the package itself. I am not sure how the build works here, so if someone was able to help me with this, I'd be grateful!

[GregoirePelegrin]

According to the test suite, it seems it is ready for review. Please tell me if it isn't!

[Nusnus]

It appears this PR only changes the default from pickle to dill, but the matching celery PR celery/celery#9100 is where the dill is actually “requested”.

Shouldn’t the celery use-case be enough so the billiard changes won’t be needed?
This is a follow-up to my other comment: https://github.com/celery/billiard/pull/406/files#r1663972900

[GregoirePelegrin]

It appears this PR only changes the default from pickle to dill, but the matching celery PR celery/celery#9100 is where the dill is actually “requested”.

Shouldn’t the celery use-case be enough so the billiard changes won’t be needed? This is a follow-up to my other comment: https://github.com/celery/billiard/pull/406/files#r1663972900

It is necessary to have the dill deserializer here, as it would otherwise be unable to process the message sent by the dill serializer in the matching celery PR celery/celery#9100. I initially only modified (locally) the celery library but the error I was trying to remove was then raised here:

billiard/billiard/pool.py

Line 444 in 81cc942

def _recv(timeout, loads=pickle_loads):

This is what led me to also modify the billiard deserializer.

[auvipy]

wah I can see that the CI is passing and there are unit tests added! but are the unit tests running in the CI? can you also add/update some unit tests?

[GregoirePelegrin]

wah I can see that the CI is passing and there are unit tests added! but are the unit tests running in the CI?

As the test was implemented the same way as the others, and in an already existing file, it indeed should be running the same way as the others.

can you also add/update some unit tests?

What additional tests would you like me to add?

I must also say that I do not understand the pre-commit error? Is it related to this PR or is it a more global issue?

[auvipy]

You can ignore the precommit error for now. Btw, why dill and not cloudpickle? Im curious

[GregoirePelegrin]

I had no strong feelings towards one or the other, I first found out about dill and then (quite a while later) about cloudpickle. dill had more GitHub stars than cloupickle even though they were both quite old (2013 and 2015 respectively), so I chose to go with dill.

On a side note, I currently try to fix the related PR in celery for PyPy.

[auvipy]

yeah please, focus on the celery pypy fix please