Search

shirishweb

talk less, code more

Category

Python

When to automate?

time to build automation < (time to perform without automation x number of times task performed without automation)

Today I happen to learn the above formula and wanted to share it to the readers.

So basically, according to the formula, it is best to automate only if we can build the automated system in the time less than what it would have taken without automation. Fairly simple mathematics.

Let’s get into an example.

Let’s say an e-commerce company is manually entering the customer details after the order is received. On average, the company receives 50 orders/day and for each order, staff would take 5 minutes.

ie,

50 * 5 = 250 minutes = 3 hours and 10 mins /day

If we assume to automate this task it would take 5 days, then

5 days / 250 mins = 5 * 24 * 60 / 250 = 28.8

That means the company would be benefited after the 29th days because of this automation process.

 

Python Django: Using multiple database, inspectdb and admin for existing database

One of the most powerful parts of Django is the automatic admin interface. It allows all users including developers to manipulated data inside database table. Often people use django admin tool to manage their database rather than using the console management tool provided by the database software itself or any other tool which are very limited and specific to one type of database only.

I have an existing setup mysql database with all the required schema and stored data. I want to use Django’s powerful administration tool to manage some of the stuffs on the database.

Lets get started on.

I have setup a django project called boad with necessary configurations updated in the settings.py file. I have added my mysql server as default database for the project.

Since running migration using django management command would create default django tables like django_sessions, django_migrations etc on my database which I dont like as I would like to keep my database very neat and clean as it was prior to django setup. So I decided to use django multiple database feature using database router that would route database operations of all other django related table to other database. I have setup a database router which will route read,write and syncdb operation of all tables not related to my existing database.

First in my settings.py I set up two databases as:

DATABASES = {
    'sqlite': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.path.join(BASE_DIR, 'db.sqlite3'),
    },
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        'HOST': 'localhost'
        'USER': 'root',
        'PASSWORD': '',
        'NAME': 'boad_database'
    }
}

DATABASE_ROUTERS = ['boad.router.NonBoadAttributeRouter',] #Router's module path

I have used two database sqlite and mysql (default). I would be using sqlite to store data that are specific to django tables.

I created a router.py file in the project module directory as:

class NonBoadAttributeRouter:

    non_boad_attribute_tables = ['auth', 'admin', 'contenttypes', 'sessions', 'messages', 'staticfiles', 'migrations']
    def db_for_read(self, model, **hints):
        if model._meta.app_label in self.non_boad_attribute_tables:
            return 'sqlite'
        return None

    def db_for_write(self, model, **hints):
        if model._meta.app_label in self.non_boad_attribute_tables:
            return 'sqlite'
        return None

    def allow_relation(self, obj1, obj2, **hints):
        if obj1._meta.app_label in self.non_boad_attribute_tables or obj1._meta.app_label in self.non_boad_attribute_tables:
            return True
        return None

    def allow_migrate(self, db, app_label, model_name=None, **hints):
        if app_label in self.non_boad_attribute_tables:
            return db=='sqlite'
        return None

Here I have stated that if application label is either of labels defined in non_boad_attribute_tables python variable, they should operate by “sqlite” database. Each router’s function would check for app label and route to “sqlite” database as per required or would return “None” which would tell django to fall back to “default” database for database operation.

Django have an “inspectdb” management command which is very useful to generate Django model from the existing database. As an example here I have use inspectdb to generate model for my course_category table in mysql.

venv) D:\Learnings\Django\boad\src>python manage.py inspectdb course_category
# This is an auto-generated Django model module.
# You'll have to do the following manually to clean this up:
# * Rearrange models' order
# * Make sure each model has one field with primary_key=True
# * Make sure each ForeignKey has `on_delete` set to the desired behavior.
# * Remove `managed = False` lines if you wish to allow Django to create, modify, and delete the table
# Feel free to rename the models, but don't rename db_table values or field names.
from __future__ import unicode_literals

from django.db import models


class CourseCategory(models.Model):
 course_cat_id = models.AutoField(primary_key=True)
 coursename = models.CharField(max_length=100)
 code = models.CharField(max_length=10, blank=True, null=True)
 project = models.CharField(max_length=30, blank=True, null=True)
 created_by = models.CharField(max_length=60)
 created_date = models.DateTimeField()
 deleted_by = models.CharField(max_length=60)
 deleted_date = models.DateTimeField(blank=True, null=True)
 deleted = models.IntegerField()

class Meta:
 managed = False
 db_table = 'course_category'

I have generated an model from existing database. I can do same for other mysql tables and create put it inside their own respective django app created using

python manage.py startapp courses

app_str

Now finally in admin.py file of each django app folder, I have register each model in admin as:

# -*- coding: utf-8 -*-
from __future__ import unicode_literals

from django.contrib import admin

from course.models import CourseCategory

class CourseCategoryAdmin(admin.ModelAdmin):
    list_display = ['course_cat_id', 'coursename', 'code', 'project']
    list_filter = ['project',]
    list_editable = ['code',]

admin.site.register(CourseCategory, CourseCategoryAdmin)

 

Running django’s migration command wont create any other tables in my mysql database but instead will be created in other sqlite database.

After successful migration I can test my django app using:

python manage.py runserver

course_admin_interfaceI can now successfully setup admin interface for managing my mysql database.

This approach of creating admin interface is applicable to most of popular database systems like Oracle, postgres etc. Django is going more popular day by day and equally getting more recognition for its quick and powerful admin interface generation.

Deploying Django App using gunicorn and nginx on Linux (Ubuntu) server

According to official site,

Django is a free and open-source web framework, written in Python, which follows the model-view-template architectural pattern.

Nginx is a web server which can also be used as a reverse proxy, load balancer and HTTP cache.

We are going to use above technologies and deploy a sample Django app “dproject” with structure as

Djproject is the main Django project folder containing the import wsgi file, settings, urls. Djapp is my Django app and venv is python virtual environment folder where needed python packages and gunicorn is installed.

We can install gunicorn using pip as

pip install gunicorn

Now we will test if gunicorn can server our Django app or not

gunicorn –bind 0.0.0.0:81 dproject.wsgi

If that works well we can proceed on further creating a gunicorn service to serve my app on nginx server

Now we’ll create a system service that can manage above gunicorn process.

Create a file for the gunicorn service as

vim /etc/systemd/system/dproject_gunicorn.service

[Unit]
Descriptioin = Gunicorn Daemon for dproject
After = network.target

[Service]
User=gopal
Group=www-data
WorkingDirectory=/home/gopal/dproject
ExecStart=/home/gopal/dproject/venv/bin/gunicorn –access-logfile - --workers 3 –bind unix:/home/gopal/dproject/dproject.sock dproject.wsgi:application

[Install]
WantedBy=multi-user.target

Our service is dependent on network.target. We are using ‘gopal’ and ‘www-data’ as linux os user and group to serve our app. WorkingDirectory is pointed to our project directory.

ExecStart is a command line that will execute on service run. Here we have started our gunicorn process with 3 workers that will serve dproject through unix socket file.

We are also telling that we want our service to be managed by multiuser.target runlevel

Now we will allow our port 80 through firewall

ufw allow 80

Or

firewall-cmd –permanent –add-port=80/tcp
firewall-cmd –reload

Now we will start and enable our gunicorn service so that our service is started automatically after every boot

systemctl start dproject_gunicorn
systemctl enable dproject_gunicorn

Now we’ll create nginx configuration file for our Django app which is served through unix socket file dproject.sock via gunicorn

vim /etc/nginx/sites-available/dproject

server {
  listen: 80;
  servername dproject.com
  location /static/ {
    root /home/gopal/dproject;
  }
  location / {
    include proxy_params;
    proxy_pass http://unix:/home/gopal/dproject/dproject.sock;

We will enable this configuration file by creating a symlink to site-enabled directory of nginx configuration directory

ln –s /etc/nginx/sites-available/dproject /etc/nginx/sites-enabled/dproject

We will now test if our configuration is ok

nginx –t

It says “OK”

Now we will restart out nginx server and allow nginx ports in our firewall

systemctl restart nginx
ufw allow “Nginx Full”

 

Python defaultdict() versus dict.get()

Python is a great language. Dict is one of data structure available in python which allows data to store in the form of key/value pair. Many times we might encounter a situation where we have to retrieve a value using the dictionary key. When our key is not found in the dictionary it will throw an exception of KeyError.

KeyError

The solution to this is using defaultdict or dict.get() to return a default value if the key is not found in the dictionary.

Lets dig into both of these

defaultdict can be found in collections module of python. So, in order to use it, we have to import it first as:

from collections import defaultdict
mydict = defaultdict(int)

defaultdict constructor takes default_factory as argument which is a callable. This can be

int : default will be integer value of 0

str : default will be empty string ”

list : default will be empty list []

and so on.

If we want our own default value then we can pass function pointer.

Lets say that we want default value to string ‘default’. We can achieve this in defaultdict as

def mydefault():
        return 'default'

mydict = defaultdict(mydefault)
print mydict['test']

will output ‘default

The same result can also be achieved using the dict.get method as

mydict = {}
mydict.get('test','default')

will output ‘default

Thus same result can be achieved using both approach. dict.get have to provide a default value every time it is called whereas using defaultdict we cant setup a default value only one time.

Now lets check the efficiency of both of them in term of execution time.

For this we will be using ipython notebook since it has got %timeit command to measure the execution of any python statements over any desired looping of same statements.

In Ipython notebook environment, let us create two function that implements defaultdict in one and dict.get method in another.

defining_two_defs

Here we have implemented a() function to execute defaultdict value retrieving and b() to use dict.get method. Now lets calculate the execution time of both function in 100 loops as

defaultdict_timeit

dict.get_timeit_2

First executing a() function over 100 loops we can see that it took over 8.88 micro sec per loop using the best of three.

dict.get_timeit

Here when we run b() function over 100 loops we can see that it tooks 19.3 micro sec per loop using the best of 3 approach.

Hence defaultdict seems more efficient over dict.get method.

Lets rerun this test again but this time will be using default %timeit command of notebook which will be looping the statement for default 1000 loops

defaultdict_timeit_2

Clearly here also defaultdict seems more efficient that dict.get method and the experiment shows that defaultdict more that two times faster than dict.get method.

Create a free website or blog at WordPress.com.

Up ↑