Index by title

Django profiling

Decorator from https://code.djangoproject.com/wiki/ProfilingDjango available via DjHDGutils.profiling

Usage is simple: add @profile(‘filename’) as a decorator for view or function you wish to profile, and find results in
/tmp or into settings.PROFILE_LOG_BASE

To view, you will need to install following packages:

Generate PNG:

gprof2dot.py f pstats /yp/default/djyp/stones1.prof | dotTpng -o output.png

for a convinience I’ve made a shell script for myself.


Django project management commands

  1. Get the code.

    hg clone ssh://repos-location to-dir

Copy and edit settings from example-settings.py in your project on this step.

Install environment

fab env_install
  1. Install dependencies

    fab depup

  2. Get database

    fab get_db

  3. Upload database

    fab reinst_db

Optional steps

  1. Some projects may require downloading project data:

    fab rsync_data

  2. Sometimes some additional modules needs to be installed. This is usually done via fab, please refer to appropriate command there. For example if you get error like “Error: No module named haystack” and there are fab procedure: “fab install_haystack” so you should run it.


DjHDGutils

db_dump - Dump current django database, restore from latest sql dump

Automation of making dumps for any of your Django projects. Currently the following
database backends are supported:

There are 2 main modes for the script to run:

dbup

(outdated, please use PatchManager instead)
For a large production systems we support we prefer manual migration scripts, run in a defined
sequence. That application does exactly that thing, and nothing else.

mongo - dump & restore mongo db.

Dump and restore Mongo database.

sync_redmine_wiki

That extension helps you to maintain your documentation in both Redmine Wiki and project, and it does
make synchronization easy by uploading or downloading data from the site.


DjHDGutils.PatchManager

patch command

Introduction

PatchManager makes it easy to support large scale project migrations in the well controlled manner.
Patches can be different types, namely:

and they are run in sequentual manner one after one. The migrations are made that way that several developers
in a different branches can run their patches, merge them without conflicts (a random number at the end of the
patch name makes it low probability to patch names to interfere) and test well before going to production.

Patch names

Patches can be named automatically that is for next command is made:

(env)djyp $ ./manage.py patch next
File /someproject/migrations/00636-avk-654.sql has been created
size: 0

Filename consists from:

Commands

Compare to other systems

Most of other systems differ by their primary usage. South for example mainly used for distributed project support. Our system
is simple and straightforward to use, we always know what patch is going to be installed on a production database.
And so our advantages:

Typical migrate session

In a typical usage session you do 2 things:

Or even just directly apply them, as we usually do in our upgrade scripts.

./manage.py patch list
./manage.py patch up

fabfile.py automation for repeating maintenance tasks

In our company we have automated a lot of project life cycle repeating actions. Most of them are straightforward & easy to use.

fab up

In most of projects we implement get_db and reinst_db scripts. First one downloads the most recent copy of the production database,
2nd command installs it. Typically the end user-developer needs to run this to get his project fully updated:

hg pull -uv
fab get_db reinst_db depup patch

And usually in most of projects all this gets combined in a single command:

fab up

which implements the following:

And so Fabric abstracts it on a higher level, and makes developer life easier.

fab produp

Additionally on every production server the ‘produp’ procedure gets implemented. It is very similar to
‘up’ procedure explained above, but it also includes step to reload / restart production service software
and update required files (for example static files).

And so,

fab produp

consists from:


Hdg

Softadmin

Django-admin contrib application is great thing to see & manage application data. But it have limited functionality by design: “The primary purpose of Django’s admin interface is to let people edit data.”. All advanced actions, content managers are asking us from time to time are never going to be implemented in django admin not because its difficult or impossible, but because core concept of django admin is different. So, admin is good to do very basic actions on your data, but is not well suited for writing full content management screens.

Here is some typical tasks which content managers like to solve:

This is very typical setup and it is well suited for every application we develop. Since we don’t want to reinvent wheel every time we write new application, we decided to write some general table management functions and use them between our applications.

Another interesting trick is DataSource definition. Quite often we need to display results of some manually run query (which is impossible to run via ORM). Those queries are mostly requested by content managers, so we either need to make new application to display result of some query every time, or develop some general approach. As a general approach this admintable object can be used.

Download

svn co http://svn.halogen-dg.com/svn/repos/hdg/trunk hdg

Documentation

Every developer gets lazy when you ask about documentation. Here is epydoc results for introspection of source code for models.

Examples


Setting up desktop (AVK)

echo 'export "PATH=$PATH:$HOME/bin"' >> /.profile
cd $HOME
hg clone https://avkoval@bitbucket.org/avkoval/bin
hg clone https://avkoval@bitbucket.org/avkoval/.xorg.d
ln -s .xorg.d/.tray-progs
ln -s .xorg.d/.xbindkeysrc
ln -s .xorg.d/.xinitrc
ln -s .xorg.d/.Xresources
hg clone https://avkoval@bitbucket.org/avkoval/.i3

Add this to /etc/portage/package.use/custom:

# AVK Custom
app-editors/emacs xft toolkit-scroll-bars
media-libs/freetype bzip2 -auto-hinter -debug fontforge utils doc
media-fonts/terminus-font  X psf pcf
net-im/pidgin gstreamer gnutls
sys-kernel/calculate-sources -minimal
>=dev-lang/python-2.7.2-r3:2.7 sqlite

Now emerge programs:

emerge i3 i3status dmenu xterm

Optional: mate

As root (sudo ~~i):

layman~~a mate
emerge mate

Additional resources


Testing EMail sending in Django

class TestOnboardingEmails(TestCase):
   MSG1 = 'Congratulations! You have now set your first '
       'assignment'
   MSG2 = 'You have been using product for a week now'

   def setUp(self):
       self.teacher = factories.TeacherFactory.create()
       self.teacher.email = 'test@test.domani.com'
       self.teacher.save()
       self.student = factories.StudentFactory.create()
       self.studentset = factories.StudentSetFactory.create(owner=self.teacher)
       self.studentset.join_student(self.student)
       self.studentset.join_teacher(self.teacher)
       self.quick = factories.QuickFactory.create()

       self.original_backend = settings.EMAIL_BACKEND
       settings.EMAIL_BACKEND = 'django.core.mail.backends.locmem.EmailBackend'

   def tearDown(self):
       settings.EMAIL_BACKEND = self.original_backend


   def test_new_assignment(self):
       aset = AssignmentSet.create(teacher=self.teacher,
                                   assign_date=datetime.now(),
                                   studentset=self.studentset,
                                   students_list=[self.student],
                                   content_list=[{'obj': self.quick,
                                                  'attempt_max':1}])
       cmd = process_events.Command()
       cmd.handle(no_celery=True, debug=False, name=None, date=None,
                  quiet=False)
       self.assertTrue(len(mail.outbox)==1)
       self.assertTrue(self.MSG1 in
                       mail.outbox[0].message().as_string())
       self.assertTrue(self.MSG2 not in
                       mail.outbox[0].message().as_string())

       after_1_week_plus = datetime.now() + relativedelta(days=8)

       cmd.handle(no_celery=True, debug=False, name=None,
                  date=after_1_week_plus.strftime(DATE_FORMAT),
                  quiet=False)
       self.assertTrue(len(mail.outbox)==2)
       self.assertTrue(self.MSG1 not in
                       mail.outbox[1].message().as_string())
       self.assertTrue(self.MSG2 in
                       mail.outbox[1].message().as_string())


if __name__ == '__main__':
   unittest.main()

The important lines in setUp and tearDown:

 def setUp(self):
     self.original_backend = settings.EMAIL_BACKEND
     settings.EMAIL_BACKEND = 'django.core.mail.backends.locmem.EmailBackend'

 def tearDown(self):
     settings.EMAIL_BACKEND = self.original_backend

Using ansible

Ansible is a software to perform automated updates and configuration management on remote servers over SSH.

The site is:
http://ansible.cc/

How we are using ansible? For every project, there is file doc/ssh_config.txt which is a part of standard
/.ssh/config file related to project, contained server aliases.

To transfer those aliases into Ansible ‘hosts’ file format, use the supplied Python program.

There is admin project within UA2WEB which contains all different project groups into one file, the
repository is accessible from:

This also includes Playbook scenarios to perform various support tasks.

Some examples of using it:

Check GLSA:

ansible ua2web -u root -a "glsa-check -l -q"

Check if any of UA2WEB servers needs system update:

ansible ua2web -u root -a "emerge -up system"

Using TAGS table in emacs

When Rope fails to find definition (usually due to meta classes used in Django), you can use standard Emacs feature: TAGS to find
this tag.

Also, to make tags for project you need to find tags:

exuberant-ctags -e -R --languages=python

P.S. Useful to have project fab command “fab make_tags”


UA2WEB public API modules

DevOps

Other tips & tricks

Tips and tricks

Development procedures

Work related