Import this, that, and the other thing: custom importers
-
Upload
zoom-quiet -
Category
Technology
-
view
1.379 -
download
0
description
Transcript of Import this, that, and the other thing: custom importers
If you do not know what __path__ is,
this talk is NOT for you.
Sorry.
import this, that, and the other thingCustom importers in Python
Brett Cannonwww.DrBrett.ca
Slides are sparse, so do listen to what I say.
Thanks ...
• Python Software Foundation• PyCon Financial Aid committee
• Nasuni• Jesse Noller
What the heck is an importer?
Relevant since Python 2.3
importer=
finder + loader
A finder finds modules.
A loader loads modules.
“Why do I want one?”
Customization/control, easier to work w/ than __import__
How are custom importers used by
import?
Simplified view; ignoring implicit importers
Meta pathsys.meta_path
Start
for finder in sys.meta_path:
loader = finder.find_module(name, path)
return loader.load_module(name)
...
True
False
What ‘path’ arg is
Pathsys.path or __path__, sys.path_hooks,
& sys.path_importer_cache
... Parent module
has __path__
search = parent's __path__
search = sys.path
search
True
False
for entry in search:
finder = sys.path_importer_cache[entry]
loader = finder.find_module(name)
return loader.load_module(name)
Search
path hookFalse
True
False finder
raise ImportError
True
for hook in sys.path_hooks:
finder = hook(entry)
sys.path_importer_cache[entry] = finder
sys.path_importer_cache[entry] = dummy
path hook
finder
False
True
True/False = ImportError (not) raised
how do I write my own
importer?Only masochists need apply.
Option 1:Painfully from
scratchRead PEP 302 for the gory details.
Option 2:Use importlib
Available since Python 3.1.I have suffered so you don’t have to.
Option 3: importers
http://packages.python.org/importers/
File path abstraction on top of importlib.Treating as purgatory for importlib
inclusion.
If a lesson here, then it is to use option 2 or 3 depending on your needs.Rest of talk is about lessons that led to ‘importers’.
Using azipfile importeras an example
Assuming use of importlib.Talking from perspective of using an archive.
we need a hookFor sys.path_hooks.
Refresher:Hooks look for a finder for a path
Path either from sys.path or __path__
Hooks can get funky paths
E.g. /path/to/file/code.zip/some/pkg
Search backwards looking for a file; find a directory then you have gone too far.
Consider caching archive file objects
No need to keep 3 connection objects open for the same sqlite3 file
Pass your finder the “location”:
1)the path/object &2) the package path
Import assumes you are looking in a part of a package.
Raise ImportError if you got nuthin’
Have finder, will look for code
Don’t treat modules as code but as files
Just trust me. Too many people/code make this assumption already for stuff like __file__, __path__, etc.
You did remember where in the
package you are looking, RIGHT?!?
Needed because of __path__ manipulation by user code.
fullname.rpartition(‘.’)[-1]
Need to care about packages &
modulessome/pkg/name/__init__.py
andsome/pkg/name.py
Care about bytecode if you want.Notice how many stat calls this takes?
Avoid caching within a finder
Blame sys.path_importer_cache
Tell the loader if package &path to code
Don’t Repeat Yourself ... within reason.
Nuthin’?Give back None
Now it gets tricky
Writing a loader.
Are you still thinking in terms of
file paths?
importlib.abc.PyLoader
• source_path()• Might be changing...
• is_package()• get_data()
Everything in terms of exactly what it takes to import source
importlib.abc.PyPycLoader
• source_path()• is_package()• get_data()• source_mtime()• bytecode_path()
• Might be changing...
This is what is needed to get source w/ bytecode right
Reasons to ignore .pyc
• Jython, IronPython couldn’t care less.• Safe to support, though.
• Another thing to code up.• Bytecode is just an optimization.• If you only ship .pyc for code
protection, stop it.
What to do when using
importlib ABCs
Require anchor point for paths
somewhere/mod.py is too ambiguous
Too hazy as to where a relative path is anchored; archive? Package location?
Consider cachingstat calls
Only for stand-alone loaders!Also consider caching if package or not.
Consider whether storage is read-only, append-only, or read-write.
Don’t overdo error checking
EAFP is your friend.
Perk of importers is the abstraction
Lazy loader mix-inwritten in19 lines
class Module(types.ModuleType): pass
class Mixin: def load_module(self, name): if name in sys.modules: return super().load_module(name) # Create a lazy module that will type check. module = LazyModule(name) # Set the loader on the module as ModuleType will not. module.__loader__ = self # Insert the module into sys.modules. sys.modules[name] = module return module
class LazyModule(types.ModuleType): def __getattribute__(self, attr): # Remove this __getattribute__ method by re-assigning. self.__class__ = Module # Fetch the real loader. self.__loader__ = super(Mixin, self.__loader__) # Actually load the module. self.__loader__.load_module(self.__name__) # Return the requested attribute. return getattr(self, attr)
... or you could use the importers package
http://packages.python.org/importers/
Fin