find_packages() doesn't find PEP 420 packages

Originally reported by: gwideman (Bitbucket: gwideman, GitHub: gwideman)

---

#!python

setup(...
    packages=find_packages(..)
    )

on the developer machine will fail to find packages that lack a __init__.py file, as is allowed in Python 3.3. However, such packages listed explicitly: packages=['mypkg'] do appear to get included and later installed.

Note: When testing this, before each test be sure to delete all generated metadata, including that which setup may previously have placed in the original source directory, as it seems that setup may use metadata created on a previous run in order to include files.

This is part of a general problem reported in issue #83, but I've logged it separately as it's specifically about setuptools.

---

• Bitbucket: https://bitbucket.org/pypa/setuptools/issue/97

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

I'd like to say that Setuptools should support PEP 420 packages. The harder question is how should find_packages discover such packages? PEP 420 allows importing of any importable filename ending in .py in any directory in sys.path. However, sys.path isn't relevant when building distributions and discovering packages.

I imagine setuptools could include all suitable directories, possibly omitting explicitly excluded paths. This change would be backwards-incompatible, but could provide packagers with options to produce compatible dists.

I considered using Python's import machinery to find packages, but I don't believe the import machinery provides any discovery tools.

The other thing to consider - if setuptools supports this mode, it will not work on other versions of Python, so will require Python 3.3+.

This gives me an idea - perhaps setuptools can use the package metadata to determine when PEP 420 package discovery is enabled. It could inspect the Trove classifiers and if Python versions are specified and only include versions >= 3.3, the feature would be enabled.

That technique would be somewhat implicit, but properly-documented, it could provide an elegant way to allow a packager to both include PEP 420 packages and declare the Python requirement at the same time.

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

I believe this issue is more than a minor one and should be addressed sooner than later.

Original comment by gwideman (Bitbucket: gwideman, GitHub: gwideman):

---

Yes, the more I look at PEP 420-style packages, the less I am able to see what distinction there is between a package, and simply a directory tree containing python files at any level, a tree that import+sys.path can navigate arbitrarily.

Packages become whatever directories can be reached by import combined with sys.path. And neither the imports or sys.path are readily available to packaging software.

It may be that "packages" as a special level of modularity were an unnecessary idea in the first place, and it's as well to discard them.

But if packages are thought to be an important level of modularity, then it should be up to the package developer to positively ordain them (as with __init__.py), not left up to the caller and whatever the caller's sys.path happens to be. At least, so I'm thinking at the moment.

So I'm a bit mystified what the big plan is regarding packages.

Original comment by wyatt (Bitbucket: wyatt, GitHub: wyatt):

---

I ran into this issue yesterday and took a quick stab at a patch.

It works pretty much the same as before, but it's necessary to specify what to include when there are top level, non-package directories that contain .py files.

#!python
setup(
    # ...
    find_packages(include=['my_pep420_ns_package'])
    # ...
)

It's slightly more complicated than I'd prefer because of the need to specify include in some cases. I was thinking about putting the where directory on sys.path and then trying to import subdirectories, but I wanted to try a simple(r) approach first.

On the other hand, maybe allowing includes to be specified is generally useful. It also makes find_packages() work in a way that's more similar to how the basic packages list works (i.e., you only specify what to include). In fact, if distutils.core.setup() would allow packages to be specified using wildcards, find_packages() probably wouldn't be necessary in most cases.

Anyway, here's the patch:

#!diff

diff -r f19978009ac8 setuptools/__init__.py
--- a/setuptools/__init__.py    Thu Feb 06 09:31:48 2014 -0500
+++ b/setuptools/__init__.py    Thu Feb 06 13:09:25 2014 -0800
@@ -1,5 +1,6 @@
 """Extensions to the 'distutils' for large or complex distributions"""

+import glob
 import os
 import sys
 import distutils.core
@@ -27,31 +28,58 @@
 # Standard package names for fixer packages
 lib2to3_fixer_packages = ['lib2to3.fixes']

-def find_packages(where='.', exclude=()):
+def find_packages(where='.', exclude=(), include=()):
     """Return a list all Python packages found within directory 'where'

     'where' should be supplied as a "cross-platform" (i.e. URL-style) path; it
-    will be converted to the appropriate local path syntax.  'exclude' is a
-    sequence of package names to exclude; '*' can be used as a wildcard in the
-    names, such that 'foo.*' will exclude all subpackages of 'foo' (but not
-    'foo' itself).
+    will be converted to the appropriate local path syntax.
+
+    'exclude' is a sequence of package names to exclude; '*' can be used
+    as a wildcard in the names, such that 'foo.*' will exclude all
+    subpackages of 'foo' (but not 'foo' itself).
+
+    'include' is a sequence of package names to include. If it's
+    specified, only the named packages will be included. If it's not
+    specified, all found packages will be included (this is the same as
+    specifying include='*'). 'include' can include wild card patterns
+    just like 'exclude'.
+
+    The list of includes is built up first and then any excludes are
+    removed from it.
+
     """
+    from fnmatch import fnmatchcase
+    include = list(include or '*')
     out = []
     stack=[(convert_path(where), '')]
+    pep420 = sys.version_info[:2] >= (3, 3)
+    in_regular_pkg = False
     while stack:
         where,prefix = stack.pop(0)
         for name in os.listdir(where):
             fn = os.path.join(where,name)
+            candidate = prefix + name
+            for pat in include:
+                if fnmatchcase(candidate, pat):
+                    break
+            else:
+                continue
+            in_regular_pkg = os.path.isfile(os.path.join(fn, '__init__.py'))
+            # Namespace packages can't be nested inside regular packages
+            in_pep420_ns_pkg = (
+                not in_regular_pkg
+                and pep420
+                and glob.glob(os.path.join(fn, '*.py'))
+            )
             looks_like_package = (
                 '.' not in name
                 and os.path.isdir(fn)
-                and os.path.isfile(os.path.join(fn, '__init__.py'))
+                and (in_regular_pkg or in_pep420_ns_pkg)
             )
             if looks_like_package:
-                out.append(prefix+name)
-                stack.append((fn, prefix+name+'.'))
+                out.append(candidate)
+                stack.append((fn, candidate + '.'))
     for pat in list(exclude)+['ez_setup']:
-        from fnmatch import fnmatchcase
         out = [item for item in out if not fnmatchcase(item,pat)]
     return out

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Nice. The patch looks quite good.

Would you consider putting together a test as well?

Original comment by wyatt (Bitbucket: wyatt, GitHub: wyatt):

---

I'll see if I can find some time tomorrow to write some tests and submit a PR.

Original comment by wyatt (Bitbucket: wyatt, GitHub: wyatt):

---

I just pushed two more alternative approaches. The first uses __import__ to determine which directories are packages, and the second works similarly to the existing version but also considers all directories under where to be namespace packages when running under Python 3.3+.

• __import__ version
• Similar to existing version

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Add support for PEP 420 namespace packages to find_packages()

On Python 3.3+, find_packages() now considers any subdirectory of the
start directory that's not a regular package (i.e., that doesn't have an
__init__.py) to be a namespace package.

The other way this supports PEP 420 is by making sure __pycache__
directories are never added to the list of packages.

Fixes issue #97

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

I've merged the pull request, but I find that the output includes quite a few directories not previously covered. Consider the comparison on setuptools itself:

> python
Python 3.4.0rc3 (v3.4.0rc3:8a81cdab3e9d, Mar  9 2014, 20:24:50) [MSC v.1600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import setuptools
>>> setuptools.find_packages()
['build', 'command', 'dist', 'docs', 'foo', 'setuptools', 'temp', 'tests', '_markerlib', 'build.lib', 'docs.build', 'docs._templates', 'docs._theme', 'setuptools.command', 'setuptools.tests', 'tests.shlib_test', 'build.lib.setuptools', 'build.lib._markerlib', 'docs.build.doctrees', 'docs.build.html', 'docs._theme.nature', 'setuptools.tests.indexes', 'setuptools.tests.svn_data', 'build.lib.setuptools.command', 'build.lib.setuptools.tests', 'docs.build.html._sources', 'docs.build.html._static', 'docs._theme.nature.static', 'setuptools.tests.indexes.test_links_priority', 'setuptools.tests.indexes.test_links_priority.simple', 'setuptools.tests.indexes.test_links_priority.simple.foobar']

> py -2.7
Python 2.7.6 (default, Nov 10 2013, 19:24:24) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import setuptools
>>> setuptools.find_packages()
['setuptools', '_markerlib', 'setuptools.command', 'setuptools.tests']

I think perhaps the example above using the heuristic of testing for the presence of *.py files may be better. On the other hand, Python will import any directory.

> py
Python 3.4.0rc3 (v3.4.0rc3:8a81cdab3e9d, Mar  9 2014, 20:24:50) [MSC v.1600 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> os.listdir('foo')
[]
>>> import foo
>>>

If we stick with the generous (and probably more correct) implementation to accept any directory as a package, package maintainers are going to have to be a lot more explicit about which directories are packages. It almost renders find_packages useless.

Some more consideration is going to have to go into this idea.

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

Add support for PEP 420 namespace packages to find_packages()

On Python 3.3+, find_packages() now considers any subdirectory of the
start directory that's not a regular package (i.e., that doesn't have an
__init__.py) to be a namespace package. Because this will often
include non-package directories, a new include argument has been added
to find_packages().

include can make it easier to narrow which directories are considered
packages instead of having to specify numerous excludes. In
particular, it's an easy way to keep top-level, non-source directories
from being considered packages.

The other way this supports PEP 420 is by making sure __pycache__
directories are never added to the list of packages.

Refs issue #97

Original comment by gwideman (Bitbucket: gwideman, GitHub: gwideman):

---

Jason:

If we stick with the generous (and probably more correct) implementation
to accept any directory as a package, package maintainers are going
to have to be a lot more explicit about which directories are packages.
It almost renders find_packages useless.

I think you are reaching a level of head scratching similar to my 2013-11-05 comment. It seems to me that PEP 420's implications weren't fully thought through, as it undermines the ideas about what the package concept facilitates and what it protects against (not just causing problems for find_packages() ).

Perhaps someone at a higher pay grade could distill a little better what the big-picture goal is for packages, and figure out whether PEP 420 gets there. To that end, it might be worthwhile recording in more detail the implications that you've encountered.

For example, your discussion of package maintainers needing "to be a lot more explicit" -- what form do you envision that taking?

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

By more explicit - I mean that other package maintainers will have to do something like I've done in 78c8cfbe3e10.

Original comment by jaraco (Bitbucket: jaraco, GitHub: jaraco):

---

By the way, I don't think there's a "big plan" regarding packages. PEP-420 was the plan and where it's deficient, more discussion should probably happen in python-dev.

I was there for the implementation of PEP-420. I feel like it's a good thing to simply and directly allow folders to represent packages. In the past, I was always bothered by the fact that it took two entities (a directory and an __init__ file) to represent a package.

Regardless, the implementation for Python 3.3 and 3.4 is set, so setuptools should support those models to the best of its ability. I think for find_packages, it might be worthwhile to do more heuristics, namely -- only inferring a directory is a package if it has modules somewhere within it.

Original comment by arfrever (Bitbucket: arfrever, GitHub: arfrever):

---

+1 for not including directories without any modules.

Original comment by arfrever (Bitbucket: arfrever, GitHub: arfrever):

---

I suggest to skip directories matching some hardcoded patterns by default (unless such directories are matched by element of include parameter):

• build* (When e.g. python setup.py build is run multiple times, contents of build directory would be copied to a subdirectory of itself (build/lib/build/lib/build/lib/...) :( .)
• doc* (Often with Sphinx-specific conf.py file)