You are here: Home / Blog

Pre-RuPy Python Sprint

Posted by Balázs Reé at Sep 25, 2013 02:30 PM |
Come for the Python, stay for the RuPy!

On October 11, 2013, Ruby, Python, Clojure and JavaScript enthusiasts will meet in Budapest, HU to attend RuPy - a three day conference dedicated to all things Ruby and Python based.

What's RuPy?

RuPy is a conference that brings together the two programming communities and their related technologies. Challenge yourself by learning something new about Python. Broaden your skill set by exploring Ruby. Network with other developers and share ideas. RuPy is a great place to talk about what each community can learn from one another. See how others are solving the same problems within different developer communities. Attending RuPy gives us the opportunity to see what other developers are doing. Keynote speakers include: Stu Halloway, Jessica McKellar, and Fabio Pliger. Speaker topics are still being announced. Check out the agenda for details!

RuPy also happens to be a regional conference. The combination of Python and Ruby in one event maximizes your time and your money by giving you both sets of talks all in one event. Can't make it to Brazil for the annual Plone conference? Join us at RuPy in Hungary! With over 70 presentations submitted, you're bound to find something that interests you.

Python Sprint

Before RuPy starts, plan on coming early for the Python sprints. Starting on October 8th, we'll be sprinting on Python projects. Chris McDonough will be joining us to host a Pyramid/Pylons sprint. Visit the github sprint page to see the outline of topics Chris wants to tackle. So far there are 15 packages that need bug triage. There's a lot to achieve and it's a great opportunity to learn and get involved in Pyramid.

The sprint is open for any Python topic you would like to work on. Do you have a Python topic you'd like to sprint on? Interested in Pyramid, but would like to work on Django instead? Need an excuse to come visit beautiful Budapest in the autumn? Propose a sprint topic! Add your topic to the github sprint wiki and tell us what you want to do. The page is editable if you have a github account.

Stay tuned to both the RuPy and Python sprint pages for more updates. New information is being posted everyday! Attendance is free, but please register here because we need to know the number of participants.

A regional conference

Regional conferences compliment the global conferences in a useful way. There are many developers in Eastern Europe who are on a tighter budget and cannot afford to attend on PyCon or EuroPython. A regional conference is a great place to meet people from your neighbourhood. RuPy tries to attract attendants from Hungary, Eastern Europe, and the Balcan, but it's also attractive for Western European participants.

When I attended the RuPy conference in 2012 in Brno, I must admit that it was a great experience, meeting a lot of developers that I knew and making new connections and friendships. I am very glad that this year the conference comes to my home city Budapest and I am also happy that I could become an organizer as well. Since I organized the world Plone conference in 2009, I know for sure that the conference will be great (how else could it be?), and besides the official tracks we try to organize and suggest many extra-curricular and exciting events in the city as well, for your cultural and culinar enjoyment.

The conference is organized by the community: the entire organization is non-profit, transparent and open, with the promise of the full budget disclosed at the end of the event. The Hungarian Python and Ruby communities give great help to the organization, we have several opportunities to participate and help, in case you would like to do more than just attending.

Need I mention that if you are a company that wishes to sponsor the event, it is still not late! Jump on the wagon and become a sponsor, if you are a software developer house from Eastern Europe that relies on Python or Ruby for your operations, your place is here with us!

Document Actions

A simple real-time presence portlet for Plone

Posted by Balázs Reé at Jul 18, 2013 06:17 AM |

In my previous blogs, I showed how Firebase makes it easy to quickly develop real-time responsive applications. This time, I will focus on one of these applications: a presence display that shows who is logged in. Like previously, I will show a solution integrated into the Plone CMS in the form of a portlet. Also like previously, I use AngularJS for the front-end. The result: we can display who is watching the site in real-time.

 

 

Storing connection info

A small piece of JavaScript code makes sure that Firebase "remembers" the users who are currently viewing the page from a browser. Tracking is in real-time: if a user closes the page, all the clients will immediately update their presence display.

This solution also takes into consideration that a user can open the site from an arbitrary number of browser windows. We make use of the built-in ".info/connected" database location, which we can use to bind our event handler. This way we can write our presence to the database as soon as we connect. This happens shortly after the page load.

var onlineRef = new Firebase(url);
var connectedRef = new Firebase(rootUrl + '.info/connected');

connectedRef.on('value', function(snap) {
    if (snap.val() === true) {
        // We're connected (or reconnected)! Set up our presence state and
        // tell the server to set a timestamp when we leave.
        var userRef = onlineRef.child($scope.plone_username);
        var connRef = userRef.child('online').push(1);
        connRef.onDisconnect().remove();
        userRef.child('lastSeen').onDisconnect().set(Firebase.ServerValue.TIMESTAMP);
    }
});

An important part of this solution is Firebase's ability to remove our connection status from the database via the onDisconnect directive, if the connection is lost. So, even if our browser crashes or our network connection evaporates, the database will still notice that we have been disconnected. We also instruct the server to store the timestamp of the disconnection, which could be used to display when a currently logged out user was last seen (although we do not do anything with this information in this demo).

If we look at the Forge admin interface (by visiting the database URL from the browser), we can see how the data is stored. The "online" location contains a collection of this user's active connections.

In our example the Anonymous user is currently logged in from two browser windows. You can also see that the last seen timestamps are always stored, but they only become meaningful for users that are not currently connected.

Displaying the user presence

Here is where the AngularJS + Firebase combo really shines. Displaying the presence will even be easier than it was to manage the connection information.

Since the above code takes care of managing our presence data in a guaranteed consistent way, next we only need to display it. AngularJS comes to the rescue here. We simply bind the local data to our database:

// bind the data so we can display it
var promise = angularFire(onlineRef, $scope, 'users', {});

And refer to this data from our template:

<div class="user" ng-cloak ng-repeat="(name, user) in users | isOnline">
      {{name}}
</div>

Finally, let's drop in some CSS, and here is the result:

Yes, it's really this simple with Angular. Needless to say that the same result can easily be achieved with other MVCs, or even just with hand-rolled JavaScript code.

The only missing piece of the puzzle is the isOnline filter, which you can look up from the source. This is filtering the users that are currently connected.

And here is how it works in practice. It's interesting to notice, that because I am running a Plone site on my laptop in development mode, without any upfront caching of resources, the user that logs in or out is actually getting the page reloads from Plone slower, than the other clients, who stay connected and just receive a real-time update from Firebase. So in my development environment, the remote Firebase happens to work faster than the local Plone back-end. This is not really relevant for a production setup, where Plone will be lightning fast as well, just a fun fact.

You can try how this works by yourself on the live demo. Log in with usernames "user1", "user2", "user3", "user4", each with the password "secret". You can also install the package from github and try to install the portlet on your own Plone site.

Next steps

This is a functional, but simple demo. Here are some ideas to make it a full blown app:

  • Increase Plone integration by displaying the photo of the users and link the info to the user's profile page.
  • Integrate it with the chat portlet (to be found in the same demo package).
  • Allow anonymous (not logged-in) users to set up an alias for themselves, and display them separately.
  • For inactive users, show when they were last seen from the Plone user profile page.

 

I leave to implement this as an exercise for the reader. Along these lines, a custom specified chat or help-desk application can be developed with ease.

I like simple and elegant code. I hope I could demonstrate with this article, how emerging technologies offer simple solutions to problems that used to be very hard to tackle successfully within the boundaries of our older paradigms.

Document Actions

FireBase, AngularJS, Plone: Part 2

Posted by Balázs Reé at Jul 04, 2013 07:50 AM |

In my last article, I introduced Firebase and presented too simple demos, a poll and a chat application, integrated to a Python based Plone CMS website. To continue, I would like to show the details on how this works. This time, prepare for a longer reading, or ...

TL;DR: Firebase is a versatile real-time database service that makes it easy to develop real-time responsive applications. It works especially nice with an MVC framework like AngularJS.

Let's dive straight into the gory details.

How does it work?

Let's see the simplest way of binding your data from browser JavaScript to the Firebase database. To do so, you first would need to create a Firebase account and a database at the firebase.com website. There is an url that identifies the database and it has the following form:

https://<MY-DB-NAME>.firebaseio.com/  

However, Firebase also allows any demo database to be used without being created first (provided that you do not need authentication in your demo, and you can live with the chance that your database may be erased occasionally.) So you can skip visiting the FireBase site at all, and use a database URL like this:

https://<MY-DB-NAME>.firebaseio-demo.com/  

You can try this manually from your browser console. Open your favorite browser (I use Firefox with Firebug, but you could also use Chrome), open the JavaScript console and do the followings.

To start, you will need to load the JavaScript code for Firebase. Simply enter this from your browser's console:

 var script = document.createElement('script');
script.src = "https://cdn.firebase.com/v0/firebase.js";
document.body.appendChild(script);

To connect to the database, substitute <MY-DB-NAME> with the unique name you want to use.

var messagesRef = new Firebase('https://<MY-DB-NAME>.firebaseio-demo.com/messages');

This will associate messageRef to the /messages location within your database. Following this you can read and write the data via the API that FireBase provides. Let's write some data into it:

messagesRef.push({'user_id': 'anonymous51', 'text': 'I can haz my Plone beautified!'});

The push method considers /messages as a collection, and appends the data we provided above as a new element to it. Firebase has the general characteristics that it creates empty data structures on demand, which means that the empty Array or Object will be created when we first write into it.

Following this, you can read back the data you just wrote (you can try this from a different browser to demonstrate that the data is, indeed, in the cloud). In JavaScript, we have to use an event listener for this.

messagesRef.on('value', function (messagesSnapshot) {
  var messages = messagesSnapshot.val();
var nr = 0;
for (m in messages) nr++; // this is needed because it's an object, not an array
console.log('Number of messages:', nr);
});

You can now try to add various data to the /messages array, and do whatever you want to happen as a reaction to the data change on all connected clients.

Notice how I am counting the number of messages instead of using just messages.length? This is something that you will see often: instead of an array, a dictionary object with automatic, non-numeric keys is created in Firebase to represent an ordered collection of items. Don't worry: it's also possible to use numeric values as indices and handle the collections as genuine Arrays. But at the end of the day, this will not make much difference to our application.

Connecting the database with Angular

The usage will become even simpler if we use Firebase together with an MVC. In the demos, I am not using the interface I just described. Instead, I am using the AngularJS bindings for FireBase. This offfers a higher level API when one can simply bind $scope.myVariable to a given url in the database. Following this, one can have the local data automagically replicated in the cloud in a bi-directional way.

There are two flavors of this binding which offer different data synchronization, as explained in the documentation: explicit and implicit. Both flavors are demonstrated in the chat and poll demos.

In both modes, the changes made on the server are always populated to the clients automatically. The difference berween the two flavors is how the client's changes are propagated back to the server.

In explicit mode, we need to explicitly signal the change of the client data. This means that without calling the right API method, the database will not notice your local changes. In implicit mode however the local changes are picked up automatically, and the changes are propagated to the server (thus to all the other clients as well) by default.

angularFire($scope.firebase_url + '/choices', $scope, 'choices', []).then(function () {

$scope.addChoice = function () {

$scope.choices.push({

label: '',

count: 0

});

};

$scope.removeChoice = function (choice) {

$scope.choices.splice($scope.choices.indexOf(choice), 1);

};

);

While explicit more offers more control about the network traffic, implicit mode works nice and easy with Angular.

Admin interface

FireBase provides a TTW administration interface to the database, called Forge. I've shown its screenshot above.

Besides various administrative settings (security, authentication) the interface provides introspection to the database content. The changes our app makes in the data are reflected here in real-time. It is also possible to manipulate (add, edit, delete) data records directly from here.

Access rules

The authentication scheme of Firebase is powerful and versatile. In the admin interface (Forge) we can set up the security access rules for the entire database.

{
"rules": {
"chat": 
".read": "auth !== null && auth.ploneUserid != ''",
".write": "auth !== null && auth.ploneUserid != ''"
},
"poll":
".read": "auth !== null && auth.ploneUserid != ''",
".write": "auth !== null && auth.ploneUserid != ''"
},
...
}
}
The rules are specified with a JSON file that can use expressions that evaluate to true or false, and govern if the current client can have read or write access to the requested data. Different parts of the database can have different rule definitions, allowing to define access based on the data context. The above example allows read and write access to "chat" and "poll" for logged-in Plone users.

The expressions can also contain variables that originate from the agent that authenticated us. A simple example for this is a $user variable, that must match the Plone userid. Using this it is possible to set up database locations where users can only access their own data but not the data of other users. Similarly, we could further develop these rules to also handle role based permissions, where for example an admin user could access the data of all users.

{
"rules": {
"plone_interact": {
"users": {
"$user": {
".read": "auth !== null && auth.ploneUserid == $user",
".write": "auth !== null && auth.ploneUserid == $user" 
}
}
}
}
}

Authentication

To allow security and access control, someone must authenticate the client, otherwise the database would have no way of implement access restrictions.

In the Plone demo, I used custom authentication. This means that there is a server view that knows the logged in user's Plone username, and can create an authentication token which then the client will use in the communication to Firebase. In the demos this is integrated into the portlet code, but you can easily apply it to your use case, or do the same with other Python front-ends such as Django or Pyramid.

from firebase_token_generator import create_token
    
firebase_secret = "***********************"

def gen_auth_token(context, request):
    portal_state = getMultiAdapter((context, request), name="plone_portal_state")
    plone_userid = portal_state.member().getId()
    custom_data = {
        'ploneUserid': plone_userid,
    }    
    admin = False
    options = {'admin': admin}
    token = create_token(firebase_secret, custom_data, options)
    return token

Firebase also offers authentication via other agents, most notably Facebook, Twitter and GitHub. It also offers email based authentication. So, you can make a single page web app that authenticates with one or all of these agents to Firebase, without the need to maintain a web server yourself.

Presence tracking

Firebase offers a way to track the presence of the users that are currently watching the page. This can be used for two main purposes.

First, one can notice when the client's network connection goes online or offline. This is essential for applications that must always be connected to the database, and being offline is considered to be an error condition that needs to be handled specially.

Second, it is possible to actually store in the database the logged in clients, which makes it possible that other clients can display which users are currently logged in. The following code server both purposes.

var onlineRef = new Firebase('https://<MY-DB-NAME>.firebaseio.com/users/');
var connectedRef = new Firebase('https://<MY-DB-NAME>.firebaseio.com/.info/connected');

connectedRef.on('value', function(snap) {

    console.log('CONN', snap.val());

    if (snap.val() === true) {

        // We're connected (or reconnected)! Set up our presence state and
        // tell the server to set a timestamp when we leave.
        
var userRef = onlineRef.child($scope.plone_username);
        var connRef = userRef.child('online').push(1);
        
// Tell the server what to do in case of disconnect.
connRef.onDisconnect().remove();

// Also strore the 'last seen' timestamp for this user.
        userRef.child('logout').onDisconnect().set(Firebase.ServerValue.TIMESTAMP);
    }

});

It is worth noticing that the server guarantees that the onDisconnect event executes even if the client falls out from the network without a notice, which means that the server itself will execute the desired action on going offline, which guarantees data consistency in any case. In this example, the server also stores when a user went offline, so we know when a given user was last seen online.

I am working on adding a presence portlet to the Plone demo.

Practical observations

This is very nice in theory. The question is, of course, how it will work in the practice, especially when we get to more complex use cases, and scale up the size of the application and the number of its users as well. So far, my experiences are positive, but while Firebase tries to combine the best of each concept to provide an easy to use, efficient real time database service, its decisions inevitably introduce tradeoffs. So it's important to know what these are. Here I attempt to show a glimpse of the few issues that I have experienced so far.

No database procedures

You cannot execute procedures on the database server itself. In most of the practical cases I find that this is not a big problem, as one can most of the time structure the data in a way that allows the clients to handle it directly. If this is not enough, one can deploy a daemon that connects to the database from remotely and does whatever task it needs to achieve.

Explicit vs. implicit data coupling

This is specific to the Angular support, not Firebase itself. As I mentioned above, explicit data synchronization requires more work from the programmer, but it can also be more performant in some cases. Use explicit synchronization if you want to be in better control above what gets transferred through the network.

Larger data and images

What happens if the data that has to go through the network, is large? Would that not cause performance problems?

In one application, I tried to store images in the database. In the use case, the user can upload images to the database that later get displayed and manipulated in various ways. This works easier than one would think, since it is possible to inline the images to the html itself, so we can get away without needing an additional server in addition to Firebase to serve the images. Still, larger amount of data needs to travel through the net, and I observed that this slowed down the reponsiveness of the app noticeably.

This is the moment when it is less beneficial to use the implicit synchronization of angular. Instead, one would want more control and fall back to use explicit data coupling, as I described earlier, in order to get better performance.

Limitation of the REST API and the Python bindings

Firebase supports a full grown JavaScript and Objective C API. There is also a REST API which makes it easy to write libraries to use Firebase from a web application written in other languages, such as Python.

However the REST API and thus the Python library that is based on top of it, have a serious limitation. They can read or write data, but they lack the capability to listen to real time changes from the server. Simply put, this means that while you have full access to your data, you partly loose the ability to queue management and cannot receive push notifications via the REST interface.

There are emerging libraries that would give the full Firebase functionality in Python as well which assumes the usage of network sockets instead of REST HTTP. At the time when I write this article, I have not checked the available solutions yet, so I don't know their usability status. Instead, I decided to write my simple server side applications in NodeJS instead of Python. Naturally, having a full fledged Python API would be a nice thing for anyone who is familiar with the mature libraries of this great programming language.

Limits of access rule based security and MVC

I had to implement a simple and basic use case: a queue that each user can push an item into, but that only a privileged user can read in its entirety. The following security declaration does the trick (in this use case any authenticated user counts as an admin, which means auth != null is only true for admins, but never for normal users):

{
"rules": {
"submitQ": 
".read": "auth != null",
".write": true
}
}
}

However when trying this I had to notice that this usecase does not work well with MVC style data synchronization. The basic problem is that the user cannot read the entire queue but still would need to push a new item into it, somehow. As an example, think about a landing web page where interested users can submit their email address, so that the owners of the landing page can later notify them. It is obvious however that users should not be allowed the entire mailing list collected so far, so it's out of the question to replicate the full array of emails to the client.

Luckily, this is no problem at all to carry out with Firebase's principal API that I presented in the beginning of my article. And one can freely combine this way of access with the AngularJS interface. So there is no practical limitation here, one just need to follow a golden rule of framework usage: use the proper tool in each use case, and don't think that just because you have a framework, you are bound to use it always.

Summary

Firebase is an interesting database service that ought to be kept on your radar. It is very easy to use it in combination with an MVC such as AngularJS. It has some tradeoffs when you want to implement more complex use cases with it, but its ease of use for developers makes it worth for you to consider it as a solution, when you need both data persistence and queue notification for your application.

In addition to introduce Firebase, I also tried to prove in this article, that MVC applications can play nice together with traditional (in our case Python based) server frameworks such as the Plone CMS.

Document Actions

FireBase, AngularJS, Plone: Part 1

Posted by Balázs Reé at Jun 11, 2013 01:45 PM |
An interesting way for building responsive web apps quickly

FireBase is a cloud database service, that allows developers to "power real time collaborative applications". What does this mean for you? It means that with FireBase you can easily build applications that allow users to interact via your website. FireBase provides your front-end with persistent and shared real-time data out of the box, letting you focus on your application itself. Developing applications with FireBase can be refreshingly easy, as I will try do demonstrate in this article. I will show this in combination with the AngularJS MVC framework, on an existing Plone CMS back-end.

The job of FireBase is to bind a data structure that resides in the client (in our case, JavaScript browser data), to a database server in the cloud. Data across the applications is kept in sync within the FireBase cloud so that users have access to real time data. In this sense FireBase fulfills two goals, both equally needed for interactive applications:

  • Persistence. Data is stored in the database, and the same data is shared between all clients that use it.
  • Notifications. Clients can use the cloud data to communicate with each other, making queue management and "push notifications" easy to implement without additional tools. If a single client makes a change in the data, the changes arrive to all clients that share the data simultaneously.

FireBase also enables managing your authentication in a flexible way. Both external agents (Facebook, Twitter, Github, etc.) and custom authentication is supported. I hope to reveal more about my experiences about authentication in a following blog post.

In fact, FireBase in many respects appears to be similar to the ZODB: a native object database for Python, well known for Zope and Plone developers. Just like the ZODB, FireBase also stores a tree structure of data. Its role is to synchronize this data to a native data structure of the used programming language. It also supports authentication, and just like the ZMI for the ZODB, the database content can be managed through the web. Keen readers from the Zope and Plone communities may also remember the announcement of porting the ZODB to JavaScript, an April fool joke made by David Glick and Matthew Wilkes. When we look around today we realize that the joke could have been serious.

FireBase is a commercial service, while offered completely free in its current beta release period. It is beyond the scope of this article to benchmark FireBase or compare it with alternate solutions for the same tasks: I merely focus on its ease of use and features, from a developer point of view.

Glue it to the rest of the app

FireBase offers bindings for JavaScript, Objective-C and it has a REST API as well. Bindings are available for several programming languages incuding Python. In JavaScript, FireBase can work together with other frameworks. To use FireBase in combination of an MVC framework is especially tempting: FireBase compliments the MVC's client side data synchronization capabilities and together they offer data coupling through the entire stack from database to front-end.

For the choice of MVC, we used AngularJS in these examples. Angular plays nicely with Plone, as it can work easily with Plone's built in templating system. Besides Angular, Greenfinity also supports development in other popular MVC frameworks (such as EmberJS and Obviel), and FireBase can work with these frameworks equally well.

Why Plone? The application is "just HTML". Plone does not need to know about our data, and the Firebase traffic does not enter our server. This means that this would work equally easy with any Python based web server (such as for example Pyramid), or we could even just rely on static files. The demo examples with Plone mean to demonstrate a successful integration strategy for creating a hybrid responsive application from the combination of a complex and opinionated, traditional web back-end such as Plone, and a JavaScript MVC like AngularJS.

Check out the applications

FireBase allows you to develop simple applications in a short amount of time. It can be integrated with existing code, giving you a way to build out your client site infrastructure to support the application you choose to build, instead of focusing on implementing storage and networking.

There are two applications we built in FireBase in the time span of one afternoon. They are fully functional with a minimal amount of effort. Please keep in mind that we have not tested these examples for scalaibility or performance with high volume of users. The goal here is to show you how easy the applications can be built.

The demos are hardly more than the original demo examples that accompany the FireBase API for AngularJS. We are talking about one-one page of JavaScript code each. You can visit the source code for more details.

You can install the python package in any Plone site. After installation via buildout and the Plone Quickinstaller, the package provides you a Poll portlet and a Chat portlet. You can add these portlets to your site. To set up the portlets, you need to enter Manage Portlets as an administrator, and set up your FireBase url and secret, which you can copy from the FireBase admin interface (Forge) after creating your database.

To start quickly, you can use the live demo and save yourself from the burden of installation.

Poll Application

Polling features are pretty popular on websites, especially when they show real time results. Firebase allows us to build a poll in a Plone portlet and track those results.

 

Clicking on a bubble in the poll increases the number of votes. Votes are tracked directly after the text in the bubble. Bubbles automatically sort and display based on the number of votes. As votes are changed, the user can see the bubbles resort themselves instantly.

This poll is unique in that it allows the users to edit the poll at any time. Users may add a new item, remove an existing item, or click on an existing item to edit it. AngularJS makes the implementation of the editing functionality remarkably simple to achieve. In a real application, the permission to edit the choices should be limited to some authentication role of Plone. For the sake of the demo we allowed all people to play nicely together and test the application without any necessary permission.

This application...

  • is responsive: when a user votes on an item, all the clients will immediately update their counters and reorder their bubbles.
  • uses shared data: the data is not stored in the Plone back-end, but just in the FireBase cloud. Because of this, the portlet controlling the same data can be installed on several websites simultaneously.

 

    Live Chat Application

    Live chat applications are becoming an increasingly popular way to offer customer service on a website.

    The chat application in the demo site allows you to edit your nickname (or use an Anonymous nickname). When you return to the site, the chat application will remember any nickname you previously used. If you are logged in as a Plone site user, it will default your nickname to your Plone user name. Users can chat across locations with other users via the chat window embedded in the website. No chat clients to download!

    This application...

    • is responsive: when a user sends a message, all the clients will immediately display this message.
    • uses shared data: the data is not stored in the Plone back-end, but in the FireBase cloud. Because of this, the portlet that gives access to the same chat room, can be installed on several websites simultaneously.

     

    How it works

    In our next post, I will tell you how this works. I will also tell more about my experiences about custom authentication, managing your data and administering your database through-the-web. I will also show how to access the same data from Python.

    Are you interested to hire Greenfinity to add simple, but impressive interactive features to your existing website? If so, let us know: we offer one hour of free consultation to discuss your ideas, if you are one of the first 5 clients contacting us after we publish this article. Remember: great features need not be neither complex, nor expensive! Contact us for more.

    Document Actions

    from pyramid import cookie

    Posted by Balázs Reé at Mar 21, 2013 09:21 AM |

    Pyramid is an elegant and popular web development framework that is based on the Python programming language.

    The three (no, six) tenets of Pyramid are Simplicity, Minimalism, Documentation, Speed, Reliability, Openness.

    This year Greenfinity attempted the impossible, by creating the 7th element: Cookies. We made almond shortbread cookies, featuring the Pyramid logo. We sent them to the PyCon 2013 conference, as a sweet surprise to the participants of the greatest Python event of the year.

    Did you get a look at these limited edition Pyramid cookies at PyCon? We hope you enjoyed them.

    And now for something completely different...


     




     



     

     

    Document Actions

    Installing JavaScript from zc.buildout, Part 1

    Posted by Balázs Reé at Mar 20, 2013 01:05 PM |
    A simple recipe to install and process server and client JavaScript, automated from your Python web project.

    In a typical Python web development project, you need JavaScript as well on top of Python. Here, I will show a method to satisfy your JavaScript installation and processing needs - straight from zc.buildout, the well known tool for the installation of advanced Python projects.

    TL;DR

    In this article, I will focus on how to do installation of server side JavaScript from bin/buildout. In upcoming posts, I will show what you can actually use server side JavaScript for, and also, how you can process client-side (browser) JavaScript for your web application.

    Additional dependencies

    First, we need some dependencies that buildout cannot handle itself. You need to install NodeJS and npm, the Node Package Manager. On MacOSX, this can be done in the following way, if you use MacPorts:

    $ sudo port install nodejs
    $ sudo port install npm

    On Ubuntu, you would do the following instead:

    $ sudo add-apt-repository ppa:chris-lea/node.js
    $ sudo aptitude install nodejs npm

    Check out the instructions for installing Node on your favorite OS here.

    The good news is that while this is an extra dependency, it's the only dependency you will need. With this you can use the tools and follow the practices of the JavaScript developer community, without the need to use applications ported to Python, Java, or PHP, reinventing the wheel.

    See it in action

    We can use the example buildout of Substance D for demonstration. Substance D is an application development environment built using the Pyramid web framework. Substance D is used only for demonstration here, the same method works equally well for any Python, buildout based web application. That said, you can skip trying this buildout if you want, and add a similar configuration to your own project right away.

    Let's check out the sources. Sdidev is the buildout for developing on Substance D. It is meant to be used in development only and the buildout is not needed in production. Note that for this demo, the ree-grunt-sprinting branch of the sdidev buildout has to be used.

    $ git clone git@github.com:Pylons/sdidev.git
    $ cd sdidev
    $ git checkout ree-grunt-sprinting

    Do a virtualenv:

    $ virtualenv -p python2.7 .

    And make a buildout:

    $ bin/python2.7 bootstrap.py
    $ bin/buildout

    At the end of the buildout's log you will see some interesting things happening on the console. After building out our Python, zc.buildout starts showing colorful signs of progress with building out our - surprise! - JavaScript.

    Installing node_modules.
    npm http GET https://registry.npmjs.org/grunt-cli
    npm http GET https://registry.npmjs.org/grunt
    npm http GET https://registry.npmjs.org/yo
    npm http GET https://registry.npmjs.org/bower

    --- SNIP ---

    NodeJS server side modules updated.

    ... Fun isn't it? But wait, we have not finished yet; it goes on:

    Installing bower_modules.
    bower copying /private/tmp/gruntdemo/sdidev/src/slickgrid
    bower fetching slickgrid
    bower cloning git://github.com/twitter/bootstrap.git
    bower cached git://github.com/twitter/bootstrap.git
    bower fetching bootstrap
    bower cloning git://github.com/components/jquery.git
    bower cached git://github.com/components/jquery.git
    bower fetching jquery
    bower checking out jquery#1.9.1

    --- SNIP ---

    Bower client-side modules updated.

    What just happened?

    Let me explain.

    In our project's buildout.cfg, we are extending yeoman.cfg. Yeoman is Twitter's bundle to support processing and installation tasks for web development. It contains three tools: Grunt, Bower, and Yo. I will talk about what these do in an upcoming article. Grunt can also be installed independently from Twitter's Yeoman.

    The yeoman.cfg is a buildout extension config that I created to cover all buildout tasks we deal with. You can take yeoman.cfg with you, and use it from your own buildout without the need to change it in any way. The only thing you add to your buildout.cfg is:

    [buildout]
    extends = yeoman.cfg

    The yeoman.cfg file contains the rules for building JavaScript. It contains a few parts each doing a different installation step.

    [buildout]
    parts =
    node_modules
    bower_modules
    grunt_script
    bower_script
    yo_script

    The first part does the installation of selected Node modules via npm and will be run each time you update the buildout.

    [node_modules]
    recipe = collective.recipe.cmd
    on_install = true
    on_update = true
    cmds = NODE_PATH="" npm install .; echo "\nNodeJS server side modules updated.\n"

    npm is the standard tool to install server side (Node) JavaScript. "npm install ." will install the package at the current local directory, with all its dependencies. To make it work, we need to turn our buildout into a virtual Node package, thus enabling to specify its dependencies in a package.json file. Note that this file is a data configuration in JSON format, which means that generic JavaScript is not allowed in it.


    {
    "name": "sdidev-resources",
    "version": "0.1.0",
    "devDependencies": {
    "grunt": ">=0.4"
    }
    }

    You can try to issue

    NODE_PATH="" npm install .

    from the console, if you are in the buildout root, to see how this installation works. The same will be executed from bin/buildout, via the node_modules buildout rule.

    A more complete package.json version will also install some other tools you will find useful. By adding your own package dependencies to it you can satisfy virtually all your needs. I will talk more about the usage of these tools in the upcoming parts of this post.


    {
    "name": "sdidev-resources",
    "version": "0.1.0",
    "devDependencies": {
    "yo": "*",
    "grunt-cli": "*",
    "bower": "*",
    "grunt": ">=0.4",
    "grunt-contrib-jshint": "*",
    "grunt-contrib-uglify": "*",
    "grunt-contrib-concat": "*",
    "grunt-contrib-less": "*",
    "grunt-contrib-watch": "*"
    }
    }

    How it works

    The Node Package Manager provides two ways for installation: global or local. Global installation is done with the -g parameter. Global installation is often discouraged, partly because it usually requires admin privileges. In the solution I present here, we use no global installation, instead, a local installation is utilized. This has a few advantages for us.

    • The installation will be local to the buildout, and isolated from the other buildouts, allowing each buildout to have its own private working set of installed packages and versions.
    • The NODE_PATH="" setting in the node_modules buildout rule will cause your globally installed Node packages to be ignored as well, and the usage of your local installation will be enforced. NODE_PATH has a similar role in Node as PYTHONPATH in Python.

    This will cause all packages installed locally into the node_modules directory in the buildout root. You can delete node_modules and re-run bin/buildout to regenerate its contents - in the same way as it would happen if you deleted the Python eggs directory.

    The only thing left to do is running executable applications. The following buildout rule takes care of that in case of grunt:

    [grunt_script]
    recipe = collective.recipe.template
    input = inline:
    #! /bin/sh
    export NODE_PATH=${buildout:directory}/node_modules
    ${buildout:directory}/node_modules/.bin/grunt $@
    output = ${buildout:bin-directory}/grunt
    mode = 755

    This will create a bin/grunt executable script, which will execute "./node_modules/.bin/grunt", the executable that npm has installed. This means that the JavaScript tools will be runnable from bin/* in the same way as all other buildout scripts.

    $ bin/grunt --help

    Want to use even shorter commands? Make use of virtualenv, if you prefer so. In this case you will not even need to put the bin/ in front of the script names. For example:

    $ source bin/activate
    $ python2.7 bootstrap.py
    $ buildout
    $ grunt --help

    Summary

    In this post I showed how JavaScript NodeJS installation can be integrated to zc.buildout for a Python web application, and how this integration works. Next I will show how to use your newly installed tools for your web development project.

    Document Actions

    How I learned to stop worrying, and love Sublime Text

    Posted by Balázs Reé at Jan 29, 2013 04:55 AM |
    Experience report of a long time Vim user with the Sublime Text 2 programming editor

    Sublime Text is a sophisticated text editor for code, markup and prose.

    I have been a long time Vim user, and always resisted the temptation to jump on the IDE bandwagon. Admittedly, I am a dinosaur, but it is not because of this, that I had stayed faithful to Vim just until recently. Vim has always done things the way I needed, and every time I tested a potential new editor, I realized that something was missing that I used to depend on in Vim.

    So... last Christmas I started to think about what I would like to get from myself as a present, and I thought that a new editor would be nice. I started to test Sublime Text 2, and, surprise: I found it a very satisfactory experience.

    I have a few editor features that I consider as basic. Naturally, I have my own point of view: since I am doing web application development with Python and JavaScript, some things such as tab settings or linting can be more important for me than for programmers in other languages, while for the same reason I don't care much about features related to, say, compiling.

    So let's see my first experiences with Sublime! I installed it, and started to add and configure all the plugins I need for my workflow.

    Syntax coloring, correct tabbing with 4 soft spaces

    I consider correct tabbing to be basic, which means that if an editor does not have this, I would exclude it without looking for anything more. Fortunately, most good editors do this correctly, including the ability to understand the syntax of the given programming language, as it affects auto-indentation.

    Live, "as you type" syntax checking. linting.

    I have this feature in Vim, because it is really important to see live errors and warnings as you type, especially in JavaScript, where an excess comma can simply break all your code. I make this a basic requirement in my search. And, this works well in Sublime.

    Actually, it works better for me in Sublime then on Vim. I succeeded to set this up in Vim, but since I did not follow best practices for installation and do updates rigorously, I ended up with something that was working, but never exactly matched any of my team members version. Sublime solves this problem with its package installation system.

    Editable configuration files

    This is a great one. The configuration files are editable in JSON format and they are updated in sync with the changes you make through interacting with the editor UI. For example, if you zoom in and out, and if the preferences are open, you can see the text of the configuration file updated live.

    Supports Vim keystrokes

    It knows Vim keystrokes! All you have to do is enable the Vintage module in the configuration. No more excuses for Vim users! You cannot say "...but ...but I could do this much easier in Vim."

    Git support

     

    Git support works through plugins, and it works well. Although, I would love to see a more powerful way of presenting diffs, similarly to VimDiff where you can see the two versions side by side.

     

    Multiple cursors and selection

    This is simply awesome. It is easier to show it than to explain it:

    Code Intel

    Sublime knows where your modules are at, and you can click on a symbol any time to look up its definition. This is done by the SublimeCodeIntel plugin. For python code in a buildout, this means that Sublime will know all your package locations, be it a core Python library module, or a locally installed package under "src" or "eggs". For buildout, I use the following section to generate the information that Sublime needs:

    [codeintel]
    recipe = corneti.recipes.codeintel
    eggs = ${instance:eggs}

    Debugging Python

    Python applications are debugged with pdb from a console. When I am at a breakpoint, and do a listing of the code, or step up and down the trace stack, Sublime will open the source and place the cursor. Very useful. The only thing I did to install the PdbSublimeTextSupport python package, and save the following as a .pdbrc file:

    from PdbSublimeTextSupport import preloop, precmd
    pdb.Pdb.preloop = preloop
    pdb.Pdb.precmd = precmd

    Autosave! Autosave!

    I leave this feature until the end. I know that it is also possible in Vim, but for some reason I have never used it. Sublime makes autosave very tempting. When you click away from the file you are editing, it will be saved. I was worried a bit that autosave can lead to accidental loss of code. But then I convinced myself, that the ability to undo, coupled with the fact that I version manage my code anyway, compensates the risk. The result? I type in my code, then I click the Reload button on my browser... and, I can see the updated version immediately! I don't really have to say how much this increases my work productivity.

    Problems?

    Not many, really. I find a few places where I would like to see improved support. For example, easier management of color schemes would be great. But altogether, Sublime Text 2 provides me with a great development experience.

    Getting help

    There is plenty of documentation on the web that explains how to configure Sublime and its plugins. I found the blog post of Mikko Ohtamaa especially useful.

    Price

    Finally, a word about pricing. Sublime is not free software. A single user license for Sublime Text 2 costs $70. But Sublime can be downloaded and trialed for free. Unlike other software, that lock themselves down at the end of the trial period, and perhaps at the worst and most unexpected moment, Sublime will instead periodically pop up a discreet message to you, but you will be able to continue to use the software. I immediately rewarded this user friendly behavior by buying Sublime's license.

    Verdict

    Sublime Text 2 is a programmer editor that is stable, and provides most needed editor features on a high level. At the same time, it remains easily hackable, thus connecting the useful with the pleasant. It is worth to give it a try. And, if you are a Vim user, you will find the switch to Sublime extra comfortable.

    Document Actions

    My daily routine of unit testing browser JavaScript

    Posted by Balázs Reé at Oct 28, 2012 08:55 AM |
    with BusterJS, QUnit, and more, in 2 minutes

    In this video presentation (targeted towards JavaScript Web Developers), I show how I use JavaScript unit testing in my everyday work.

    I want to demonstrate that unit testing of JavaScript code is not only important for Continuous Integration - it also becomes part of the daily working routine of a software developer. If we practice unit testing it will help us to produce better quality JavaScript code.

    And it's more fun, too. It makes it possible to do test driven development of web client JavaScript code.

    During the screencast, I show how I use the BusterJS test runner to run the QUnit tests of the KARL application. Note that in its current unfinished state, buster-qunit requires a manual preparation of your existing QUnit tests, and once this is done, the tests will be able to run both as native QUnit, and as emulated BusterJS tests.

     

     

     

    Interesting links:

     

    BusterJS: http://busterjs.org

    BusterJS is a JavaScript testing toolkit that allows unit testing of both in-browser and Node.js JavaScript.

    buster-qunit: https://github.com/reebalazs/buster-qunit

    The buster-qunit is an unreleased extension to buster.js. Even in its current unfinished state, I find it very useful, because it allows me to run the existing QUnit tests from the KARL project, with the more modern test runner of BusterJS.

    KARL: http://karlproject.org

    QUnit: http://qunitjs.org

    buster-webui: https://github.com/reebalazs/buster-webui

    Jenkins: http://jenkins-ci.org

    Document Actions