PEP 713: Callable Modules

TemporalClepsydra · 2023-06-21T23:28:06.621Z

part if the problem with the datetime example is that datetime.datetime is a class, with its own special behavior, and not just a function

TemporalClepsydra · 2023-06-21T23:29:38.530Z

you can define the __isinstance__ method of datetime

Erotemic · 2023-08-13T00:17:29.466Z

We aren’t seeing a strong argument in favor here

Considering the questions:

How much does this benefit people who write Python?
How much harder does this make the language for beginners?
How much does this impact maintainers?

In terms of teaching, I think this might be easier to teach the concept of “calling” something if modules - like virtually every other object - could be callable.

In terms of maintenance, I don’t think this is a big lift considering this is just syntactic sugar for something that already is possible. It might be an easier heuristic for mypy to use in type checking than the current way it must be implemented if modules want this feature. In fact this also mitigate the isinstance concern, because if mypy implemented this heuristic it could detect incorrectly using modules as classes.

So, how much does this benefit people who write Python? Well, I would have liked to use it in: Allow the Timerit class to be used without being explicitly imported by kalekundert · Pull Request #29 · Erotemic/timerit · GitHub instead of the current boilerplate. To me it makes so much sense how __call__ would work, and I wish I could use that instead of the boilerplate that is currently necessary.

As a 3rd party module developer, I would argue in favor of this PEP. I think it’s a good idea to avoid adding it to the stdlib (for now). I also think it would be useful especially if you are teaching the more advanced concept of module dunder methods. More correspondence between what you can do with modules and what you can do with a class feels right and more “complete”, because a module is a class.

jamestwebber · 2023-08-13T00:44:19.717Z

Jon Crall:

In terms of teaching, I think this might be easier to teach the concept of “calling” something if modules - like virtually every other object - could be callable.

While every object could be callable (including modules, already), the everyday objects that beginners are using (number, strings, dictionaries, lists, sets) are not. I think this makes teaching harder, not easier:

you can call functions, which are pieces of reusable code
functions can be stored in modules. you can import modules and call their functions
(and now!) sometimes you can call a module, it will perform one of the functions it contains (or a secret new thing)

That seems more confusing to me, not less.

It seems like the primary motivation is “I want to import a function from a module that only has one”. I think it’d be more useful (but more of a lift) if that had its own syntax (like def import time, but I’m not seriously suggesting that), so that the interpreter could enforce a rule like “there’s only one public function” and it’s clear what the author intended.

Erotemic · 2023-08-13T02:19:56.779Z

the everyday objects that beginners are using (number, strings, dictionaries, lists, sets) are not. I think this makes teaching harder, not easier:

Like __getattr__ and other dunder methods, __call__ is not a basic topic. Most modules will not be callable. The way you teach the basics doesn’t change. But this does make teaching the dunder methods easier.

It seems like the primary motivation is “I want to import a function from a module that only has one”. I think it’d be more useful (but more of a lift) if that had its own syntax (like def import time, but I’m not seriously suggesting that), so that the interpreter could enforce a rule like “there’s only one public function” and it’s clear what the author intended.

The motivation I have is interactive use. There is some discussion of this in the linked timerit PR. A callable module gives maintainers like me the option of establishing more concise patterns of use. Of course it is our responsibility to document this. The stdlib is not going to make its modules callable, so the only concern is that new module developers could use this to make their code more concise.

oscarbenjamin · 2023-08-13T17:40:59.504Z

Jon Crall:

So, how much does this benefit people who write Python? Well, I would have liked to use it in: Allow the Timerit class to be used without being explicitly imported by kalekundert · Pull Request #29 · Erotemic/timerit · GitHub instead of the current boilerplate. To me it makes so much sense how __call__ would work, and I wish I could use that instead of the boilerplate that is currently necessary.

I would not use the word “boilerplate” here. Boilerplate code refers to a situation where you repeat the same (or very similar) code many times in order to do things that are done frequently. Making a module callable is not exactly something that would ever be done very frequently though: it can happen at most once per module but actually most modules will not do this at all. Needing to write a few lines of code to do something unusual is not “boilerplate”: it’s just code.

If this PEP is accepted then I will still consider that making a module callable is an anti-pattern that should be avoided/rejected. It will just confuse people and mean another variation on how a module exposes its API that would need to be remembered. Just the existence of some packages that have this sort of design will add to confusion for users of all packages including those that do not have callable modules (why can’t I call this module?). If this PEP is ever approved then I can guarantee that I will end up arguing in a PR about why it is still a bad idea to make modules callable regardless of the idea being seemingly blessed by an approved PEP. The PEP should be rejected simply because accepting it gives the impression that having callable modules is considered reasonable in the first place.

I can think of only one situation where it might make sense to make a module callable which is if there is a redesign that introduces a submodule with the same name as a prior function so e.g. you had module mod with function func that downstream code uses like

from mod import func

Now if you decide that this design was a mistake and actually you want a submodule mod.func then you might consider making the module object callable like:

class DeprecatedCall(sys.modules[__name__].__class__):
    def __call__(self, *args, **kwargs):
        warn("deprecated: Use from mod.func import new_func instead")
        return new_func(*args, **kwargs)

sys.modules[__name__].__class__ = DeprecatedCall

This is not a common thing that should be done and so it does not need to be made any easier (unless perhaps by a library that helps with making deprecation warnings).

Erotemic · 2023-08-13T19:36:15.098Z

I would not use the word “boilerplate” here.

I stand by my use of the word “boilerplate”. This stack overflow post demonstrates that this non-varying pattern has been used multiple times. It doesn’t have to be a daily activity to be boilerplate.

The idea that you shouldn’t do this in most circumstances (which I agree with) is not a strong argument against the PEP. I assert that you shouldn’t use __getattr__ in most cases as well. But it’s nice that it’s there when you want it.

accepting it gives the impression that having callable modules is considered reasonable in the first place.

This assertion is not supported. You might not like the pattern - just like I don’t like decorators - but that doesn’t mean it’s an unreasonable pattern.

There are several reasonable use cases. I have an RSI, and using callable modules can allow for more concise usage patterns. Reducing my typing is health consideration. Now, you might counter: should we allow for all syntactic sugar under the sun? No, because then they all have to be documented. However, __call__ is already an established pattern in Python, and it’s a special case that it doesn’t work for modules like it does for classes, so it shouldn’t be considered something new, wild, or unreasonable.

mean another variation on how a module exposes its API that would need to be remembered.

It’s no different than adding any other complexity to an API. It’s the API deveper’s responsibility to ensure all paths are documented.

Just the existence of some packages that have this sort of design will add to confusion for users of all packages including those that do not have callable modules (why can’t I call this module?).

The same question currently exists for objects. Why can I call this object but not that one? The answer is: “well it’s time to learn about __call__”.

oscarbenjamin · 2023-08-13T20:19:43.348Z

Jon Crall:

I have an RSI, and using callable modules can allow for more concise usage patterns. Reducing my typing is health consideration.

If I understand correctly then what you are referring to here is the difference between needing to type

from mod import func
func()

as compared to

import mod
mod()

and specifically you mean when typing in an interactive context. Personally I use ipython for interactive work and it has many ways to reduce typing so e.g. if I regularly type from mod import func then I can do f, <up>, <enter> to repeat the previously typed command that begins with f. Alternatively I don’t use this but presumably there are ways to configure ipython automatically to import things that you want to use regularly. I actually usually use isympy rather than ipython which is just a wrapper that imports and defines a few extra things that are useful when using SymPy (to reduce typing for those of us who use it regularly).

Basically my point is that there are many ways to reduce typing in an interactive context that do not require bending the basic semantics of imports, modules, functions etc.

Jon Crall:

Just the existence of some packages that have this sort of design will add to confusion for users of all packages including those that do not have callable modules (why can’t I call this module?).

The same question currently exists for objects. Why can I call this object but not that one? The answer is: “well it’s time to learn about __call__”.

I disagree with this strongly. The usual expected semantics are that there are modules, classes, functions and objects. Users of Python do need to understand the distinction between these concepts clearly in order to use them correctly. You need to be clear about which is which not just in any import statement but also in any lines of code that follow on from that import statement. It is already problematic that some Python packages confuse what is a function and what is a class (e.g. numpy.array vs numpy.ndarray) but blurring the lines between modules and functions is much worse because their usage is starkly different in terms of import.

Yes, on some level everything is an object that may or may not have a __call__ method but no actually the vast majority of users of Python never need to know that in order to use Python correctly. What they do need to know is that you either do this:

from mod import func
func()

or otherwise you do this:

import mod
mod.func()

The fact that this is ambiguous is unfortunate:

from a.b.c import d

Is d a module or a function? It is an unfortunate ambiguity but at least attempting to call a module should be a TypeError so that we are clear about the distinction: modules are not callable.

Erotemic · 2023-08-13T21:13:54.900Z

A lot of your arguments are great for why you probably don’t want to make your module callable from a design persepctive, but only one of them is a valid argument against the PEP itself.

modules are not callable.

They could be, even in the current version of Python.

The usual expected semantics are that there are modules, classes, functions and objects.

And that doesn’t change, and every one of these is an object. From the datamodel: “All data in a Python program is represented by objects or by relations between objects.”

The fact that this has been true doesn’t change how you introduce the major flavors of objects to people. You introduce them as separate things, and later as they learn more Python they learn that these things are all unified.

You need to be clear about which is which not just in any import statement but also in any lines of code that follow on from that import statement

You should, but you don’t need to. Python is a dynamically typed language. For better or worse from mod import * is a thing.

If I understand correctly…

Correct. I use IPython too. It’s great. I also have lots of custom macros and utilties to help with my RSI.

I also want the ability to make my modules callable, which is good because I can do that right now - with or without this PEP. We don’t need to debate if I should or not. A lot of the discussion here is losing sight of that, and I think it makes it seem like there are more arguments against this PEP than there really are.

What should be debated is: should modules get syntactic sugar to allow them to access the dunder methods that are already available to them in a roundabout way? Should modules have an conceptually ellegant way to access their internal objectness? I think the answer to this is a resounding yes. Modules should be able to define __call__ and __iter__ with the same ease that you can do with any other object, but only __call__ is in the scope of this PEP, so let’s focus on that.

The strongest argument I’ve heard against this PEP is that this will “bless” the practice of defining __call__ and cause module developers to over-use the capability. I think that will happen to some extent, but I also think it won’t happen often enough to cause real problems. There are only a few cases where it is a good idea to have a callable module. As you said: “most modules will not do this at all”.

I think this argument is entirely countered by the fact that it removes the disconnect of why a(b) doesn’t find __call__ when a is a module, but it does when a is a class-instance, even though they share the same data model.

pf_moore · 2023-08-13T21:44:51.368Z

Jon Crall:

What should be debated is: should modules get syntactic sugar to allow them to access the dunder methods that are already available to them in a roundabout way? Should modules have an conceptually ellegant way to access their internal objectness? I think the answer to this is a resounding yes.

Why do modules need a second way to do what they can already do? Is it a good idea to blur the distinction between different concepts (module and function)? What does it gain us to do so?

Your assertion that this is an “elegant way to access a module’s internal objectness” is entirely subjective, and somewhat arbitrary. We don’t have an “elegant way” to access an integer’s “internal objectness”. What makes that different from this case?

Jon Crall:

The strongest argument I’ve heard against this PEP is that this will “bless” the practice of defining __call__ and cause module developers to over-use the capability.

There’s the much simpler argument that there’s no need for this feature, and far more important things for the people working on Python to focus their energies on.

And you say yourself “I also think it won’t happen often enough to cause real problems”. So you’re arguing for a feature that you don’t think should be used often, where there’s already a way of doing it for people who care enough to want to. It seems to me that your argument for the feature is also extremely weak.

Ultimately, as with any proposal, the “status quo wins” rule applies. It’s not particularly important that you counter the arguments against the feature. Or even that people offer arguments against it. What you need to do is present compelling arguments for the feature. So far it seems you’re arguing

You, personally, like the idea of making modules callable.
You believe having an alternative to the current approach is “more elegant”.
Use of the feature might mean some modules could be used with a little less typing.
You accept that there are only a few cases where this would be a good idea at all.

Unless I’ve missed something, that’s not very compelling…

oscarbenjamin · 2023-08-13T21:45:36.783Z

Jon Crall:

should modules get syntactic sugar to allow them to access the dunder methods that are already available to them in a roundabout way? Should modules have an conceptually ellegant way to access their internal objectness? I think the answer to this is a resounding yes. Modules should be able to define __call__ and __iter__ with the same ease that you can do with any other object, but only __call__ is in the scope of this PEP, so let’s focus on that.

I think that this is the major weakness of the PEP. Early on it was conflated with a separate proposal (that I would support) to allow __setattr__ for modules. The reason I would like __setattr__ is to help users avoid bugs like e.g.:

github.com/mpmath/mpmath

ENH: guard against incorrect use of `mp.dps`?

opened 11:30PM - 13 Feb 23 UTC

mdhaber

enhancement

I am a SciPy maintainer, and we frequently use mpmath to compute reference value…s. (Thank you for your work! It's very helpful.) A common mistake is for contributors to do, e.g.: ```python3 import numpy as np import mpmath as mp # should be from mpmath import mp mp.dps = 50 a, b = mp.mpf(1e-11), mp.mpf(1.001e-11) print(np.float64(mp.ncdf(b) - mp.ncdf(a))) # 3.885780586188048e-15 ``` This suffers from catastrophic cancellation just as: ```python3 from scipy.special import ndtr a, b = 1e-11, 1.001e-11 print(ndtr(b) - ndtr(a)) # 3.885780586188048e-15 ``` does, but the fact that `mpmath` is _not_ working with 50 digits is obscured by the conversion to float. (I've run into other, subtler reasons for not noticing the problem, too.) Another example in the wild: https://github.com/scipy/scipy/issues/18088#issuecomment-1712538038 I know that this is user error and not a bug in `mpmath`, but I wonder if it is possible to guard against this. For instance, `mpmath` doesn't currently have a `dps` attribute. Perhaps it could be added as a property, and a warning (or error) could be raised if modification is attempted, directing the user to `mpmath.mp.dps`?

There are probably other examples where defining methods on modules could be useful and perhaps a general mechanism for defining relevant methods on modules could be valuable to satisfy a variety of use cases. I might support a PEP that proposed something like that.

PEP 713 though focuses only on __call__ which to me is a method that I don’t really want anyone to define on a module.

merwok · 2023-08-13T22:20:46.576Z

FWIW the proposal does not entice me, and makes me worried about two kind of things:

we would be able to do import datetime then dt = datetime(y, M, d, h, m, s) but not datetime.now() or isinstance(dt, datetime): this feels incomplete and confusing
«callable modules» are already a concept: python3 -m module (usually named «calling modules as scripts»); this new capability would be something different with a smilar name

Erotemic · 2023-08-13T22:51:11.791Z

We don’t have an “elegant way” to access an integer’s “internal objectness”. What makes that different from this case?

There is an ellegant way to access an integer’s internal objectness.

In [1]: class foo(int):
   ...:     def __call__(self):
   ...:         return "you called me"
   ...: 
In [3]: foo(1)()
Out[3]: 'you called me'

There’s the much simpler argument that there’s no need for this feature , and far more important things for the people working on Python to focus their energies on.

I can mostly buy this. But I’ll add that supporting this is probably an easier feature to add.

Unless I’ve missed something

Your bullets are all correct points (although I would say that I’m not the only person who can think of compelling use cases where they would want it, and I want it enough to where I’m willing to spend my time and engery arguing for it), but it doesn’t contain what I think the strongest argument is one of corresponence with object-y things:

The data model when defining a class should be as easilly available when defining a module through dunder method of the same name.

I think @oscarbenjamin alludes to an important point when discussing __setattr__. A flaw I see in this proposal its scope is too narrow. I’m only in favor of __call__ because it moves closer to what I see as the desirable end state: All of the dunder methods that are available for objects and make syntatic sense (i.e. not __init__ because how do you even invoke it?) in terms of operators should be respected by modules. Adding all of them at once, rather than incrementally would allieviate the concern of people thinking that certain operators are now “blessed”. The only thing that is “blessed” is that dunder method are a good idea and giving users the option to hook into Python’s duck typing system concicesly is a power and responsibility that they are in control of.

@merwok Making stdlib modules callable is not part of this proposal. But “callable modules” already being an existing concept is a real issue. But if this PEP was superceded by something that unified the dunder methods when defining classes and the dunder methods that define modules, then that would be less of an issue.

flyinghyrax · 2023-08-14T02:05:10.128Z

I’m personally not convinced by the “unification of classes and modules” angle, despite the language data model ultimately using objects for everything. In a Smalltalk-like language, as I understand it, everything being an object is a core part of the language from top to bottom and integral to a good mental model of it. In Python, everything being an object feels more like an implementation detail. Modules being represented by objects at runtime is not an important part of their conceptual model. That is of course subjective, I admit it’s not a concrete reason - but I think Python already has a well-established separation of concepts, with different syntax and use cases for each, so it’s not compelling to me to increase overlap between them.

Besides, what’s that quote, “make simple things easy and complex things possible”? Based on that stack overflow link earlier subclassing ModuleType is already straightforward - it’s already possible to have a class masquerade as a module just as a class can masquerade as a function (or both).

pf_moore · 2023-08-14T05:35:22.889Z

Jon Crall:

There is an ellegant way to access an integer’s internal objectness.
In [1]: class foo(int):
   ...:     def __call__(self):
   ...:         return "you called me"
   ...: 
In [3]: foo(1)()
Out[3]: 'you called me'

That makes an int subclass callable. We already have the same mechanism for making module subclasses callable. What we don’t have is a “literal” form for those subclasses - just like we don’t have a literal form for subclasses of int, or for that matter any other user defined type.

So the point becomes - we don’t have user defined literals for any other subtype of a built in type, why should the module type be any different in this regard?

BrenBarn · 2023-08-14T06:46:07.275Z

Oscar Benjamin:

I think that this is the major weakness of the PEP. Early on it was conflated with a separate proposal (that I would support) to allow __setattr__ for modules.

I agree with this. Of all the dunders that might be useful for modules, __call__ seems like an odd one to start with. I’d actually be more receptive to a generic argument for supporting all normal customizable dunders than for just supporting __call__ on its own.

davidism · 2023-08-14T14:38:58.350Z

When discussing this or any other PEP, be sure to read the PEP first, and discus its technical merits. Don’t start discussing other ideas or expanding the scope of the PEP, as then the discussion goes off topic discussing the merits of those ideas instead of the PEP.

PEP 713 is very small. It acknowledges that a module can already be made callable. It exists because it wants a standard way for type checkers to recognize that. It does not suggest adding it to any particular modules. It does not suggest adding other special methods to modules.

The PEP author has indicated to me that this discussion is no longer about the PEP itself, and is unhelpful towards the goal of improving the PEP or moving it towards the approval process. At this point, they’ll submit it to the SC if and when they feel ready to do so.

davidism · 2023-08-14T14:39:02.869Z

davidism · 2023-09-05T13:50:13.900Z

gpshead · 2023-09-05T21:56:20.968Z

“”" Hi, thanks for writing up the nice PEP spawning the lively discussion around callable modules.

The steering council has discussed this PEP and decided to Reject it. (as I mentioned we were leaning towards in PEP 713: Callable Modules - #66 by gpshead)

We didn’t feel that there was a compelling reason to have it even though it clearly could be done from a consistency point of view. I wouldn’t be surprised if this idea comes up again in the future. If so, this is a useful prior discussion to refer to for when it does.

– The Python Steering Council
“”" - PEP 713 -- Callable Modules · Issue #191 · python/steering-council · GitHub

Tangentially related in the “modules gaining more dunder function abilities” theme… a new proposal regarding __setattr__ and __delattr__ is now also being discussed.