How to write your own Python Package and publish it on PyPi

Why PyPi?

The Python Package Index (PyPI) is a repository of software for the Python programming language. PyPI helps you find and install software developed and shared by the Python community. If you use pip command, you are already using PyPi.

When your package is published to PyPi, everyone can install and use it with familiar simple command:

pip install {your_package_name}

Cool, huh?

So, what do you need to do to publish your own package to PyPi? Here is a short list of steps:

• Make your code publish-ready: create a Python package, add the files needed for PyPi
• Create a PyPi account if you haven’t had one yet
• Generating distribution archives and upload to PyPi
• Install your own package using pip

Step #1. Make your code publish-ready

This tutorial will take my package called elastictools as a real example. You can found it here on PyPi, source code is here on Github.

Some rules of thumb:

• Remove all print statement from your code. If you want to inform or log something, use logging.
• Remove all code that stays outside of a class or function. Such code (if really necessary), put it under __main__ function:

if __name__ == "__main__":
    code outside of a class or function goes here

Create a python package

Package in Python is simply a folder with name of your package. This folder contains files (modules) and other sub-folders (sub-packages).

And you need to put a file __init__.py (two underscores before and after init) to mark this folder as a Python package. Inside this __init__.py file you can specify which classes you want the user to access through the package interface.

Sample __init.py__ file:

from . import indextools
from . import doctools

__all__ = [
    'indextools',
    'doctools'
]

Add files needed for PyPi

PyPi needs following file in order to work:

• setup.py (detail below)
• LICENSE.txt (the license file, if you choose MIT, get content from here)
• README.md (optional but highly recommended)
• HISTORY.md (optional but highly recommended)

Sample project structure:

.
|-- HISTORY.md
|-- LICENSE.txt
|-- README.md
|-- elastictools
|   |-- __init__.py
|   |-- __pycache__
|   |   |-- __init__.cpython-36.pyc
|   |   |-- doctools.cpython-36.pyc
|   |   `-- indextools.cpython-36.pyc
|   |-- doctools.py
|   `-- indextools.py
|-- setup.py
`-- tests
    |-- test_doc_tools.py
    `-- test_index_tools.py

The setup.py file

The setup.py file contains information about your package that PyPi needs, like its name, a description, the current version etc .We will look directly into a real simple setup.py here:

from setuptools import setup, find_packages

with open('README.md') as readme_file:
    README = readme_file.read()

with open('HISTORY.md') as history_file:
    HISTORY = history_file.read()

setup_args = dict(
    name='elastictools',
    version='0.1.2',
    description='Useful tools to work with Elastic stack in Python',
    long_description_content_type="text/markdown",
    long_description=README + '\n\n' + HISTORY,
    license='MIT',
    packages=find_packages(),
    author='Thuc Nguyen',
    author_email='gthuc.nguyen@gmail.com',
    keywords=['Elastic', 'ElasticSearch', 'ElasticStack'],
    url='https://github.com/ncthuc/elastictools',
    download_url='https://pypi.org/project/elastictools/'
)

install_requires = [
    'elasticsearch>=6.0.0,<7.0.0',
    'jinja2'
]

if __name__ == '__main__':
    setup(**setup_args, install_requires=install_requires)

Note: Do NOT confuse setuptools with distutils. We usesetuptools here.

Most of the options are self-explainable, you can just copy the content of setup.py above and modify it as your need. Please remember to list all dependencies of your package in install_requires list, so that this requirements can be installed automatically while your package is being installed.

Include data files

To include data files (config, env, templates….) to the package, at first, we need to add this line to setup.py:

setup(
    ...
    include_package_data=True
)

And then create a MANIFEST.in file next to setup.py file, and list all the file needed to include as following:

include src/templates/*

Step #2. Create a PyPi account

If you already have a PyPi account (and still remember your username/password, of course), you can skip this step. Otherwise, please go to PyPi homepage and register new account instantly (for free, of course).

Step #3. Generate distribution archives and upload to PyPi

Generating distribution archives

These are archives that are uploaded to the Package Index and can be installed by pip.

Make sure you have the latest versions of setuptools and wheel installed:

pip install --user --upgrade setuptools wheel

Now run this command from the same directory where setup.py is located:

python3 setup.py sdist bdist_wheel

This command should output a lot of text and once completed should generate two files in the dist directory, among others in build and *.egg-info folder:

.
|-- build
|   |-- bdist.linux-x86_64
|   `-- lib
|       `-- elastictools
|           |-- __init__.py
|           |-- doctools.py
|           `-- indextools.py
|-- dist
|   |-- elastictools-0.1.2-py3-none-any.whl
|   `-- elastictools-0.1.2.tar.gz
`-- elastictools.egg-info
    |-- PKG-INFO
    |-- SOURCES.txt
    |-- dependency_links.txt
    |-- requires.txt
    `-- top_level.txt

You should add all of these three folders to your .gitignore file.

Uploading the distribution archives

To do this, you can use twine. First, install it using pip:

pip install --user --upgrade twine

Then upload all the archives to PyPi:

twine upload dist/*
... enter your PyPi username and password

After successful uploading, go to PyPi website, under your project, you can found your published package.

Published package appears under `Your projects` on PyPi.

Public listing is here: https://pypi.org/project/elastictools/

Your public package page on PyPi.

Step #4. Install your own package using pip

Now everyone can install your package with familiar pip install command:

pip install {your_package_name}

And update it later:

pip install {your_package_name} --upgrade