conda

Add `pyproject.toml` support for `conda env create`

#12666

Closed

Description

Fixes #10633

This is a fairly rough hack to see if this is an acceptable solution. It just reads the project.dependencies list in the supplied pyproject.toml and adds them as pip dependencies -- this is because I'm assuming that they will be using the PEP 508 syntax, rather than the conda one (for example, in conda's own pyproject.toml, "menuinst >=1.4.11,<2 ; platform_system=='Windows'" seems to be invalid for conda).

One thing in particular that I'm not sure whether it's confusing is that this only installs the project.dependencies list, and not the actual project itself. I think this is probably the minimally useful thing.

Checklist - did you ...

• Add a file to the news directory (using the template) for the next release's release notes?
• Add / update necessary tests?
• Add / update outdated documentation?

I'll add a news file and some text in the conda env create docs if this feature is acceptable.

We require contributors to sign our Contributor License Agreement and we don't have one on file for @ZedThree.

In order for us to review and merge your code, please e-sign the Contributor License Agreement PDF. We then need to manually verify your signature, merge the PR (conda/infrastructure#762), and ping the bot to refresh the PR.

Hi @ZedThree,

Thanks for submitting this pull request and finding an issue that has support of others.

My first impressions of this pull request is that it probably needs to handle some more complicated test cases. For example, the one that you mentioned in the linked issue ("menuinst >=1.4.11,<2 ; platform_system=='Windows'"). It might also be nice to find other pyproject.toml files from various projects and run them through this. If we are unable to parse these lines, it might be nice to let our users know somehow (maybe a warning message?).

Also, when adding a new dependency to conda you need to add it to the following files:

• conda/tests/requirements.txt
• conda/recipe/meta.yml

I'm sure others will also have some input on this pull request. I'm looking forward to the discussion.

Also, please be sure to sign the CLA agreement. I went through some recent submissions and noticed yours was still missing.

Also, when adding a new dependency to conda you need to add it to the following files: [...]

Ah, thanks @travishathaway ! I was a bit unsure about the build process of conda itself. Might be worth adding those to [project.optional-dependencies.tests] or something in conda's own pyproject.toml? Obviously a separate issue from this though.

My first impressions of this pull request is that it probably needs to handle some more complicated test cases [...]

Very happy to add these, but just in case it wasn't clear from my description and to make sure we're on the same page, the packages including any PEP508 markers are passed through as pip dependencies and not parsed. This also only affects the conda env create command -- I wasn't sure if the separate conda create command already supports pip/pypi packages.

I'll source some interesting cases and add them to the test suite.

Also, please be sure to sign the CLA agreement. I went through some recent submissions and noticed yours was still missing.

Will do! I was just holding off in case this PR was not of interest -- sounds like it is, so I will get the CLA signed.

Hi @ZedThree,

Thanks for clarifying the piece about it being a pass-thru for PyPI. Yes, you correct in thinking that conda create does not support the environment files.

I had a discussion with some colleagues about this yesterday, and we are still unsure whether this should be merged in as new feature, so feel free to add some more tests, but don't spend too much more time on it.

Let me know once you are done, I will bring this up with the other maintainers once more.

Just want to show my support for this. I often create an environment.yml to create a conda environment and the dependencies listed in the environment file are usually the same as the ones listed in the pyproject.toml file. It feels redundant to define the same dependencies in each file. I would like conda to at least support the option of creating a conda environment from a pyproject.toml even if it may not support some complicated version requirements.

I'm worried that merging in a hack without special syntax will create a backwards-compatibility nightmare, once conda implements a way to use pyproject.toml files without relegating to pip. Because then support for both ways of doing things would be needed. And both ways would need different syntax.

Syntax along the lines of conda install -f pyproject.toml --pip would alleviate my concern. Then conda install -f pyproject.toml should fail with an error like

"Warning: Installing from pyproject.toml only works via pip. Note the caveats at https://www.anaconda.com/blog/using-pip-in-a-conda-environment . You have to use the command 'conda install -f pyproject.toml --pip'`!

This gives conda the ability to later allow conda install -f pyproject.toml in a way where packages are pulled from the configured conda channels without breaking backwards compatibility for scripts that rely on the old way.

Hi there, thank you for your contribution!

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed automatically if no further activity occurs.

If you would like this pull request to remain open please:

1. Rebase and verify the changes still work
2. Leave a comment with the current status

NOTE: If this pull request was closed prematurely, please leave a comment.

Thanks!

Hi, I want to know if you would like to continue working on this feature. If you are currently unavailable I am willing to work on this.

I also would like to know from @travishathaway whether this feature is still desirable for conda.

I personally believe this would be a critical feature. Keeping both pyproject.toml and environment.yml is not worth the effort: I don't want to write a unit-test just to check consistency between these two, but sometimes I would really forget to update another. There are currently quite a few environment manager (e.g. pixi and poetry) that have this feature, and using them remove the unnecessary burden.

Still a useful idea. whl2conda, https://github.com/conda-incubator/conda-pypi, https://github.com/conda/grayskull have code to translate pypi to conda dependencies. I'm also interested in fixing "conda-build develop ." with something like the linked code.

Still a useful idea.

I am currently trying to find a less hacky way to implement this feature. Before opening another PR I want to discuss some general ideas&principles about the implementation with maintainers. Is conda forum the right place to go?

whl2conda, https://github.com/conda-incubator/conda-pypi, https://github.com/conda/grayskull have code to translate pypi to conda dependencies.

Could you be more specific about where I can find the implementation of translation? Would be nice if I could utilize your code for this feature.

@harnvo,

The appropriate place for this discussion is this issue:

• #10633

I also highly recommend joining our chat room:

• http://bit.ly/conda-chat-room

I'm closing this as a) I don't have time to work on it further, b) it's really only a hacky proof-of-concept, and I don't understand all the relevant issues myself, and c) conda-pypi (#10633 (comment)) looks like a much better and well thought out solution