ParamSpec incompatible with union of callables

ParamSpec seems to be incompatible with a Union[Callable]

from typing import Callable, ParamSpec, TypeAlias, TypeVar, Union, cast

FnWithOptionalArg: TypeAlias = Union[
    Callable[[], int],
    Callable[[int], int],
]

fn = cast(FnWithOptionalArg, lambda: 1)

P = ParamSpec("P")
R = TypeVar("R")

def my_decorator(fn: Callable[P, R]) -> Callable[P, R]:
    return fn

my_decorator(fn)  # ERROR
#  :16:14 - error: Argument of type "FnWithOptionalArg" cannot be assigned to parameter "fn" of type "(**P@my_decorator) -> R@my_decorator" in function "my_decorator"
#    Type "FnWithOptionalArg" cannot be assigned to type "(int) -> int"
#      Type "() -> int" cannot be assigned to type "(int) -> int"
#        Function accepts too many positional parameters; expected 0 but received 1 (reportGeneralTypeIssues)

T_Callable = TypeVar("T_Callable", bound=Callable)

def my_decorator_2(fn: T_Callable) -> T_Callable:
    return fn

my_decorator_2(fn)  # OK

Expected behavior

I would expect my_decorator(fn) above to not raise a type error and to return a value of FnWithOptionalArg, which is what my_decorator_2(fn) does.

If your code relies on symbols that are imported from a third-party library, include the associated import statements and specify which versions of those libraries you have installed.

VS Code extension or command-line

command-line, 1.1.298-- previously I was on 1.1.293 and this error did not occur.

I think pyright's behavior is correct here. A ParamSpec captures the signature of a function. It is considered invariant, so it must capture the same signature wherever it's used. You're attempting to assign two different signatures to it (one that accepts an int parameter and one that doesn't).

Mypy generates the same error in this case.

[ParamSpec] is considered invariant, so it must capture the same signature wherever it's used. You're attempting to assign two different signatures to it (one that accepts an int parameter and one that doesn't).

Thanks for the response. Confused by the above statement. A regular TypeVar is also invariant by default-- so by your stated logic I would expect to see a similar error if we "attempt to assign two different [types]" to R. But this doesn't raise an error:

from typing import Callable, TypeVar, Union, cast
from typing_extensions import TypeAlias, ParamSpec

FnWithOptionalArg: TypeAlias = Union[
    Callable[[], int],
    Callable[[], str],
]

fn = cast(FnWithOptionalArg, lambda: 1)

P = ParamSpec("P")
R = TypeVar("R")

def my_decorator(fn: Callable[P, R]) -> Callable[P, R]:
    return fn

x = my_decorator(fn)  # OK; as expected, type is `() -> int | str`

Why does the behavior of an invariant ParamSpec not match the behavior of an invariant TypeVar here?

A regular TypeVar can take on the union of two types (or the "join" of two types in the case of mypy). A ParamSpec cannot take on multiple signatures. There's no way to "widen" a signature to include multiple signatures.

Think about what it would mean if unions of signatures were possible. How would the final type Callable[P, R] be evaluated if P were allowed to contain a union of different signatures? There is no concept in the Python type system of a callable that accepts a union of different input signatures. There is a concept of an overloaded signature. I recently added the ability for a ParamSpec to capture an overloaded signature. (Note: mypy doesn't have such support, and it's not clear from the PEP that it should be supported.) Perhaps an overload would satisfy your use case?

The other thing that you could do is define FnWithOptionalArg as a callback protocol that accepts an optional parameter. This way, you don't need to use a union to describe the type.

class FnWithOptionalArg(Protocol):
    def __call__(self, /, int=...) -> int:
        ...

Thanks for your answers and suggestions. Your explanation of why there can't be unions of signatures makes sense. I'm a bit unclear on this though:

I recently added the ability for a ParamSpec to capture an overloaded signature.

Is the correct way to express an overloaded signature as a type to use a callback protocol?

To provide a little more context-- the functions we are operating on come from our users. The function can optionally take a first argument:

def user_fn_1(arg: int) -> int:
    ...

def user_fn_2() -> int:
    ...

We've been expressing the type of user_fn_* as:

CallbackType: Union[Callable[[int], int], Callable[[], int]]`.

The type errors surfaced because we are passing a CallbackType as the wrapped arg to functools.update_wrapper, which apparently uses ParamSpec and started raising a type errors when we bumped Pyright to 1.1.298.

I experimented a little and from what I can tell, a callback protocol can't correctly express CallbackType:

class CallbackProtocol(Protocol):
    def __call__(self, /, context=...) -> int:
        ...

def my_dec(fn: CallbackProtocol) -> CallbackProtocol:
    return fn

@my_dec  # type error!
def user_fn():
    return 1

I also tried combining a callback protocol with overloads (something I'm not even sure works ever):

class CallbackProtocolOverloaded(Protocol):

    @overload
    def __call__(self) -> int:
        ...

    @overload
    def __call__(self, context= ...) -> int:
        ...

    def __call__(self, context=...) -> int:
        ...

def my_dec(fn: CallbackProtocolOverloaded) -> CallbackProtocolOverloaded:
    return fn

@my_dec  # type error!
def user_fn():
    return 1

So it doesn't look like anything can work here-- would you agree that at this time, it just isn't possible to properly express this:

CallbackType: Union[Callable[[int], int], Callable[[], int]]`.

In a way that is compatible with ParamSpec as used in functools.update_wrapper?

Can you provide a more complete example that shows how you're using CallbackType with functools.wraps currently?

Is it something like this?

from functools import wraps
from typing import Any, Callable, cast, overload

CallbackType = Callable[[int], int] | Callable[[], int]

@overload
def f(fn: Callable[[int], int]) -> Callable[[int], int]:
    ...

@overload
def f(fn: Callable[[], int]) -> Callable[[], int]:
    ...

def f(fn: CallbackType) -> CallbackType:
    return cast(Any, wraps(cast(Any, fn)))

If so, then I agree with your conclusion that functools.wraps won't work with it without the addition of some ugly cast calls (as I've added above).

Yeah that's close enough-- guess we'll go with casts for now. Thanks!

Hmmm, I've got another related question:

from typing import Any, Callable, TypeVar
from typing_extensions import ParamSpec

P = ParamSpec("P")
R = TypeVar("R")
T_Callable = TypeVar("T_Callable", bound=Callable[..., Any])

def dec_outer(f: T_Callable) -> T_Callable:
    return dec_inner(f)  # error
# /Users/smackesey/stm/code/elementl/oss/test.py:9:12 - error: Expression of type "(...) -> Any" cannot be assigned to return type "T_Callable@dec_outer"
#   Type "(...) -> Any" cannot be assigned to type "T_Callable@dec_outer" (reportGeneralTypeIssues)

def dec_inner(f: Callable[P, R]) -> Callable[P, R]:
    return f

Why the error here? Shouldn't pyright understand that dec_inner returns a function with the same type as it takes in? Or am I missing something about how ParamSpec works?

ParamSpec captures the signature of a function that is assigned to it. In your case, you're passing a TypeVar (T_Callable), so there is no signature to capture. The best a type checker can do is use the signature from the TypeVar's bound, which is .... That means the result of calling dec_inner is (...) -> Any. This isn't compatible with the return type T_Callable because the relationship with the original TypeVar is lost when you pass it through a function.

Update pyright 1.1.293 -> 1.1.298 (#12888)

Navigation Menu

Repository navigation

`ParamSpec` incompatible with union of callables #4760

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

ParamSpec incompatible with union of callables #4760

Description

Activity

erictraut commented on Mar 10, 2023

smackesey commented on Mar 10, 2023

erictraut commented on Mar 10, 2023

erictraut commented on Mar 10, 2023

smackesey commented on Mar 10, 2023

erictraut commented on Mar 10, 2023

smackesey commented on Mar 10, 2023

smackesey commented on Mar 10, 2023

erictraut commented on Mar 10, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`ParamSpec` incompatible with union of callables #4760