Advanced Python

This file covers advanced Python features that professional codebases rely on — tools that make code safer, cleaner, and more expressive. These build directly on the OOP and Intermediate Python files, so make sure you're comfortable with those first.

1. Type Hints

What is it?

Type hints let you annotate what type a variable, parameter, or return value is expected to be — Python doesn't enforce them at runtime, but tools (and other developers) use them to catch mistakes early.

Definition: Type hints are optional annotations indicating the expected data types of variables, function parameters, and return values.

Simple Example

python
def add(a: int, b: int) -> int:
    return a + b

name: str = "Aditi"
age: int = 21
scores: list[int] = [85, 90, 78]
student_info: dict[str, int] = {"age": 21, "marks": 85}

Explanation of the Code

• a: int, b: int documents that both parameters are expected to be integers; -> int documents the expected return type.
• Python itself does not enforce these — calling add("5", "3") still runs, and would concatenate the strings instead of erroring. Type hints are for tooling and readability, not runtime validation.

Checking Types with mypy

bash
pip install mypy
mypy my_script.py

Explanation: mypy is a separate static type-checking tool that reads your type hints and flags mismatches before you even run the code — catching a whole category of bugs early, especially valuable in larger codebases.

Common Mistakes

• Assuming type hints are enforced by Python itself at runtime — they aren't, unless you use an external validation tool (like mypy, or Pydantic, covered in the Web Development file).

Important Points

• Type hints are optional but strongly encouraged in professional, larger codebases.
• Optional[str] (from the typing module) indicates a value can be either that type or None.

2. Dataclasses

What is it?

The @dataclass decorator automatically generates common boilerplate methods (__init__, __repr__, __eq__) for classes that are mainly used to store data — saving you from writing repetitive code by hand.

Simple Example — Without @dataclass

python
class Student:
    def __init__(self, name, age, course):
        self.name = name
        self.age = age
        self.course = course

    def __repr__(self):
        return f"Student(name={self.name}, age={self.age}, course={self.course})"

    def __eq__(self, other):
        return self.name == other.name and self.age == other.age and self.course == other.course

The Same Class — With @dataclass

python
from dataclasses import dataclass

@dataclass
class Student:
    name: str
    age: int
    course: str

student1 = Student("Aditi", 21, "CS")
print(student1)                          # Student(name='Aditi', age=21, course='CS')
print(student1 == Student("Aditi", 21, "CS"))   # True

Explanation of the Code

• @dataclass automatically generates __init__, __repr__, and __eq__ based purely on the type-hinted attributes listed — dramatically reducing repetitive boilerplate.
• The generated __repr__ gives a clean, readable printout without you writing it manually.

Adding Default Values

python
@dataclass
class Student:
    name: str
    age: int
    course: str = "Undeclared"   # default value

student2 = Student("Rohan", 22)
print(student2.course)   # Undeclared

Important Points

• Dataclasses are ideal for simple, data-holding classes — for classes with significant custom logic/behavior, a regular class may still be more appropriate.
• Default values work exactly like default function arguments — non-default fields must come before default ones.

Practice

1. Create a Product dataclass with name, price, and a default in_stock: bool = True.

3. Protocols (Structural Typing)

What is it?

A Protocol defines an expected shape (which methods/attributes an object must have) without requiring formal inheritance — Python calls this "structural typing," informally known as "duck typing" ("if it walks like a duck and quacks like a duck...").

Simple Example

python
from typing import Protocol

class Flyable(Protocol):
    def fly(self) -> str:
        ...

class Bird:
    def fly(self) -> str:
        return "Bird is flying"

class Airplane:
    def fly(self) -> str:
        return "Airplane is flying"

def make_it_fly(flyer: Flyable) -> None:
    print(flyer.fly())

make_it_fly(Bird())        # works - Bird has a matching fly() method
make_it_fly(Airplane())    # also works - Airplane also has a matching fly() method

Explanation of the Code

• Neither Bird nor Airplane explicitly inherits from Flyable — they just happen to have a matching fly() method, which is all a Protocol cares about.
• This is different from traditional inheritance-based abstraction (like the ABC class from the OOP file), which requires an explicit class Bird(Flyable): relationship.

Important Points

• Protocols are checked primarily by static type checkers (like mypy), not enforced at runtime by Python itself.
• Useful when you want to describe "any object with this behavior," without forcing unrelated classes into an artificial inheritance hierarchy.

4. Abstract Base Classes — Deeper Look

Recap and Extension

The OOP file covered ABC and @abstractmethod for forcing subclasses to implement specific methods. Beyond that basic use, Python's abc module also supports registering existing classes as "virtual subclasses," without requiring them to actually inherit.

python
from abc import ABC, abstractmethod

class Serializable(ABC):
    @abstractmethod
    def to_json(self):
        pass

class ThirdPartyClass:
    def to_json(self):
        return "{}"

Serializable.register(ThirdPartyClass)

print(isinstance(ThirdPartyClass(), Serializable))   # True, even without direct inheritance

Important Points

• .register() lets you treat an already-existing, unrelated class as if it fulfills an abstract interface, for isinstance() checks — useful when you can't (or don't want to) modify that class's original code.

5. Descriptors

What is it?

A descriptor is a class that controls what happens when an attribute is accessed, set, or deleted on another class — this is actually the underlying mechanism that powers @property (from the Intermediate Python file).

Simple Example

python
class PositiveNumber:
    def __set_name__(self, owner, name):
        self.name = "_" + name

    def __get__(self, instance, owner):
        return getattr(instance, self.name)

    def __set__(self, instance, value):
        if value < 0:
            raise ValueError(f"{self.name} cannot be negative")
        setattr(instance, self.name, value)

class Product:
    price = PositiveNumber()   # a descriptor controlling this attribute

    def __init__(self, name, price):
        self.name = name
        self.price = price   # this actually goes through PositiveNumber.__set__

item = Product("Shirt", 500)
print(item.price)      # 500

item.price = -100        # raises ValueError!

Explanation of the Code

• __get__ and __set__ intercept every read and write to price, letting the PositiveNumber descriptor enforce validation centrally, reusable across any class that uses it.
• This is genuinely advanced — most day-to-day code uses @property instead (which is simpler for single-class use cases); descriptors shine when you need the same validation logic reused across many different classes/attributes.

Important Points

• Descriptors are the mechanism underlying @property, but are more powerful/reusable across multiple classes.
• Rarely written from scratch in everyday code, but understanding them deepens your understanding of how Python's attribute access actually works.

6. Metaclasses

What is it?

A metaclass is "the class of a class" — just as objects are instances of classes, classes themselves are instances of a metaclass (by default, type). Metaclasses let you customize how classes themselves are created.

Definition: A metaclass defines the behavior of a class itself, controlling how classes are constructed, similar to how a class controls how objects are constructed.

Simple Example

python
class LoggingMeta(type):
    def __new__(mcs, name, bases, namespace):
        print(f"Creating class: {name}")
        return super().__new__(mcs, name, bases, namespace)

class MyClass(metaclass=LoggingMeta):
    pass

Output (printed the moment MyClass is defined, not when an object is created):

Creating class: MyClass

Explanation of the Code

• LoggingMeta inherits from type (the default metaclass every class ultimately uses).
• __new__ intercepts the actual creation of the class itself, letting you inject custom logic (here, just a print statement) whenever any class using this metaclass is defined.

Real-World Example

Metaclasses are used internally by frameworks like Django's ORM to automatically register model classes and their database fields the moment they're defined.

Important Points

• Metaclasses are a genuinely advanced, rarely-needed-in-everyday-code feature — as the well-known Python saying goes, "if you're wondering whether you need metaclasses, you don't."
• Understanding they exist helps explain "magic" behavior in frameworks like Django, even if you rarely write your own.

7. Advanced Decorators

Decorators That Accept Arguments

python
def repeat(times):
    def decorator(func):
        def wrapper(*args, **kwargs):
            for _ in range(times):
                func(*args, **kwargs)
        return wrapper
    return decorator

@repeat(times=3)
def greet(name):
    print(f"Hello, {name}!")

greet("Aditi")

Output:

Hello, Aditi!
Hello, Aditi!
Hello, Aditi!

Explanation: This adds an extra layer of nesting compared to the basic decorators covered in the Intermediate Python file — repeat(times=3) first returns the actual decorator, which is then applied to greet.

Preserving Function Metadata with functools.wraps

python
from functools import wraps

def my_decorator(func):
    @wraps(func)   # preserves func's original name and docstring
    def wrapper(*args, **kwargs):
        return func(*args, **kwargs)
    return wrapper

@my_decorator
def add(a, b):
    """Adds two numbers."""
    return a + b

print(add.__name__)   # add (without @wraps, this would incorrectly show "wrapper")
print(add.__doc__)     # Adds two numbers.

Explanation: Without @wraps(func), the decorated function would lose its original name and docstring, showing wrapper instead of add — this makes debugging and introspection (and automatically generated documentation) confusing. @wraps is considered a best practice for every decorator you write.

Important Points

• Always use @functools.wraps(func) inside any decorator you write in real code.
• A decorator that accepts arguments requires an extra level of function nesting.

8. Advanced Generators

Generator Pipelines

Generators can be chained together, each one processing and passing along data lazily, without ever building a full intermediate list.

python
def read_numbers(numbers):
    for n in numbers:
        yield n

def square_all(numbers):
    for n in numbers:
        yield n ** 2

def filter_even(numbers):
    for n in numbers:
        if n % 2 == 0:
            yield n

pipeline = filter_even(square_all(read_numbers(range(1, 11))))
print(list(pipeline))   # [4, 16, 36, 64, 100]

Explanation: Each generator function processes one value at a time and passes it to the next stage — nothing is fully computed until the very end (list(pipeline)), making this extremely memory-efficient even for huge input sequences.

send() — Sending Values Into a Generator

python
def running_total():
    total = 0
    while True:
        value = yield total
        total += value

gen = running_total()
next(gen)              # "prime" the generator (runs up to the first yield)
print(gen.send(10))    # 10
print(gen.send(5))     # 15
print(gen.send(20))    # 35

Explanation: yield can also receive a value sent in via .send(), allowing two-way communication with a running generator — an advanced pattern occasionally used for building coroutine-style logic.

Important Points

• Generator pipelines are a memory-efficient, elegant way to process large data streams in stages.
• .send() is an advanced, less commonly used feature — most everyday generator use only needs yield to produce values.

9. Advanced Context Managers with contextlib

What is it?

The Intermediate Python file showed writing a context manager as a full class with __enter__/__exit__. The contextlib.contextmanager decorator offers a much shorter way to write one using a generator function.

Simple Example

python
from contextlib import contextmanager
import time

@contextmanager
def timer():
    start = time.time()
    yield
    end = time.time()
    print(f"Took {end - start:.4f} seconds")

with timer():
    total = sum(range(1_000_000))

Explanation of the Code

• Everything before yield acts like __enter__ (setup); everything after yield acts like __exit__ (cleanup) — and it all runs automatically as part of the with block.
• This is far more concise than writing a full class with two separate methods for simple cases.

Important Points

• @contextmanager is generally preferred for simple context managers; a full class is still useful for more complex cases needing extra state or methods.

10. Functional Programming Tools

functools.partial

python
from functools import partial

def power(base, exponent):
    return base ** exponent

square = partial(power, exponent=2)
cube = partial(power, exponent=3)

print(square(5))   # 25
print(cube(5))       # 125

Explanation: partial creates a new function with some arguments already "locked in" — square is just power with exponent permanently set to 2.

Immutability as a Functional Programming Principle

python
# Prefer creating new data over mutating existing data
original = (1, 2, 3)
updated = original + (4,)   # creates a NEW tuple, rather than modifying in place

print(original)   # (1, 2, 3) - unchanged
print(updated)     # (1, 2, 3, 4)

Explanation: Functional programming favors avoiding changes to existing data (mutation), instead producing new values — this makes code easier to reason about, since data can't unexpectedly change somewhere else in a program.

Important Points

• functools provides several tools (partial, reduce, lru_cache, covered earlier) supporting a more functional programming style in Python.
• Favoring immutable data structures (tuples over lists, when data shouldn't change) reduces a whole class of subtle bugs.

11. Memory Management and Garbage Collection

What is it?

Python automatically manages memory for you — allocating memory for new objects and freeing it once objects are no longer needed, primarily using a technique called reference counting.

Reference Counting (Conceptual)

python
import sys

a = [1, 2, 3]
print(sys.getrefcount(a))   # shows how many references currently point to this list

b = a   # now TWO references point to the same list
print(sys.getrefcount(a))   # count increased

Explanation: Python tracks how many variables/references point to each object in memory. When that count drops to zero (no more references exist), Python automatically frees the memory.

The Garbage Collector (For Circular References)

Reference counting alone can't detect circular references (two objects referencing each other, with nothing else pointing to either) — Python's separate garbage collector (the gc module) periodically scans for and cleans up exactly this situation.

python
import gc

class Node:
    def __init__(self):
        self.reference = None

node1 = Node()
node2 = Node()
node1.reference = node2
node2.reference = node1   # circular reference: they point to each other

del node1
del node2
# even after deleting both variables, the objects still reference each other in memory
# Python's garbage collector will eventually detect and clean this up

gc.collect()   # manually trigger garbage collection (rarely needed in normal code)

Weak References

python
import weakref

class Student:
    def __init__(self, name):
        self.name = name

student = Student("Aditi")
weak_ref = weakref.ref(student)

print(weak_ref())   # <Student object> - still accessible
del student
print(weak_ref())   # None - the object has been garbage collected

Explanation: A weak reference doesn't count toward an object's reference count — allowing the object to still be garbage collected normally, while still letting you check whether it's still alive. Useful for caches that shouldn't prevent memory from being freed.

Important Points

• Python handles memory management automatically — manual intervention (gc.collect()) is rarely needed in everyday code.
• Circular references are the main scenario where the garbage collector's extra cleanup (beyond simple reference counting) matters.

12. A Brief Note on Python Internals

CPython

CPython is the standard, most widely used implementation of Python (the one you almost certainly have installed) — written in the C programming language. Other implementations exist (like PyPy, focused on speed via just-in-time compilation), but CPython is what "Python" means for the vast majority of users.

The Global Interpreter Lock (GIL) — Brief Preview

CPython has a mechanism called the GIL, which allows only one thread to execute Python bytecode at a time, even on a multi-core processor. This has significant implications for concurrent programming, covered in full detail in the next file, Concurrency & Performance.

Important Points

• Knowing that CPython is the standard implementation, and that alternatives exist, is useful general knowledge, especially for interviews.
• The GIL is a foundational concept for understanding Python's concurrency behavior — introduced briefly here, explained fully next.

Common Beginner Mistakes — Summary for This Section

• Assuming type hints are enforced by Python at runtime.
• Forgetting @functools.wraps inside custom decorators.
• Reaching for metaclasses or descriptors when a simpler tool (a regular class, or @property) would do.
• Assuming gc.collect() needs to be called manually in typical everyday code.

Cheat Sheet — Advanced Python

python
# Type hints
def func(x: int, y: str = "default") -> bool: ...

# Dataclass
from dataclasses import dataclass
@dataclass
class Point:
    x: int
    y: int

# Protocol
from typing import Protocol
class HasArea(Protocol):
    def area(self) -> float: ...

# Advanced decorator with functools.wraps
from functools import wraps
def decorator(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        return func(*args, **kwargs)
    return wrapper

# contextlib context manager
from contextlib import contextmanager
@contextmanager
def my_context():
    # setup
    yield
    # cleanup

# functools.partial
from functools import partial
new_func = partial(original_func, fixed_arg=value)

Interview Questions

Q1. Are Python type hints enforced at runtime? Answer: No — they're purely informational/documentational by default. Enforcement requires an external tool like mypy, or a library that validates them explicitly, like Pydantic.

Q2. What does the `@dataclass` decorator do? Answer: It automatically generates common boilerplate methods (__init__, __repr__, __eq__) for a class based on its type-hinted attributes, reducing repetitive code for data-holding classes.

Q3. What is a metaclass? Answer: "The class of a class" — it controls how classes themselves are constructed, similar to how a class controls how its objects are constructed. Rarely needed in everyday code, but used internally by some frameworks.

Q4. What is the difference between a descriptor and a property? Answer: A @property is a simpler, single-class way to control attribute access. A descriptor is the more general, reusable mechanism underlying @property, allowing the same validation/access logic to be shared across multiple different classes and attributes.

Q5. Why should `functools.wraps` be used inside custom decorators? Answer: Without it, the decorated function loses its original name and docstring (showing the wrapper's instead), making debugging and introspection confusing.

Q6. How does Python primarily manage memory? Answer: Primarily through reference counting — automatically freeing an object's memory once nothing references it anymore. A separate garbage collector additionally handles circular references, which reference counting alone cannot detect.

Q7. What is CPython? Answer: The standard, most widely used implementation of Python, written in C — what most people mean when they simply say "Python."

Practice Questions

Beginner

1. Add type hints to a function that takes two floats and returns their average.
2. Convert a regular class with 3 attributes into a @dataclass.
3. Write a decorator using @functools.wraps and confirm the decorated function's __name__ is preserved.
4. Write a simple contextlib.contextmanager-based context manager that prints "Starting" and "Finished" around a block of code.
5. Use functools.partial to create a specialized version of a general function.

Intermediate

1. Write a Protocol describing objects with a save() method, and demonstrate two unrelated classes both satisfying it.
2. Write a decorator that accepts an argument (e.g., a number of retries) and applies it to a function.
3. Build a generator pipeline with at least 3 stages (e.g., read numbers → filter → transform).
4. Create a simple descriptor that validates a string attribute is never empty.
5. Demonstrate a weak reference to an object, showing it becomes None after the original object is deleted.

Challenge

1. Write a custom metaclass that automatically adds a created_at timestamp attribute to every class that uses it.
2. Build a small "functional" data pipeline using map, filter, and functools.reduce together to process a list of numbers.
3. Research and explain, in your own words (with a code example), how @property is essentially syntactic sugar over the descriptor protocol.