Theodo

Conseil et réalisation d'applications webs sur mesure

  

Blog

19 September 2011

“Adopt devops philosophy” at the Open World Forum

Version française plus bas

Save the date! Theodo will be present at the Open World Forum, as I have been selected to talk about “Adpoting devops philosophy” on Friday Sept. 23 at 16:30.

More info about the conference here: http://www.openworldforum.org/Conferences/Adopter-la-philosophie-DevOps

I am very happy to be able to spread the good word in such an important conference! The main purpose will be to explain how devops extends agility and its concepts to the whole lifecycle of an IT project, including deployment and system administration and how this can improve the productivity and responsiveness of your IT organization.

See you there!

–

Réservez votre vendredi 23 septembre ! Theodo sera présent à l’Open World Forum, j’ai en effet été sélectionné pour intervenir comme conférencier sur “Comment adopter la philosophie devops” ce vendredi 23/9 à 16:30.

Plus d’informations sur la conférence ici: http://www.openworldforum.org/Conferences/Adopter-la-philosophie-DevOps

Je suis très content de pouvoir répandre la bonne parole devant un nouveau public. Mon objectif sera d’expliquer comment la philosophie devops étend les concepts d’agilité à tout le cycle de vie d’un projet informatique, déploiement et maintenance incluse et comment ces concepts peuvent augmenter la productivité et la réactivité de votre organisation informatique.

Filed under : Agile and DevOps, Theodo, symfony — Fabrice Bernhard @ 15 h 07 min

1 September 2011

Massive data import! – Part 1

We often have to face the problem of importing data off an Excel file with thousands lines.
PHP is not suited for this task, it’s slow and there is a high risk for the import to crash due to “memory limit” or some other annoying stuff like that!
So instead we chose a better way by using pure SQL which is much faster at this kind of operation.

At first, you must convert your Excel file to CSV (Excel does it very well). Be careful to choose the right field separator: I generally use “~” because there is little chance of finding this character in your written data.

Steps:

  • Create a temporary table that matches exactly the structure of the Excel file
  • Fill the temporary table with the CSV file
  • Run SQL queries to fill your database

Practical example:

Suppose we have an Excel file containing thousands of users that must be dispatched to several tables depending on their type.

CSV file sample:

        User 1~user1@theodo.fr~0987564321~user~~~
        User 2~user2@theodo.fr~0134256789~user~~~
        User 3~user3@theodo.fr~0128971271~user~~~
        Agent 1~agent1@company.com~0486282688~agent~Company 1~Role 1~0987654321
        Agent 2~agent2@company.com~0176254621~agent~Company 2~Role 2~0445664332
        User 4~user4@company.com~0456789856~user~~~

1. Create the temporary table

We will create a table contain the following fields:

  • name
  • email
  • phone
  • type
  • company_name
  • agent_role
  • company_phone
DROP TABLE IF EXISTS user_tmp;
CREATE TABLE user_tmp (
        name	 VARCHAR(127),
        email VARCHAR(127),
        phone VARCHAR(20),
        TYPE VARCHAR(20),
        company_name	VARCHAR(127),
        agent_role VARCHAR(127),
        company_phone VARCHAR(20),
        id INT(11) NOT NULL AUTO_INCREMENT,
        PRIMARY KEY (`id`),
        UNIQUE KEY `IDX_ATTRIBUTE_VALUE` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

2. Fill the temporary table

Import your CSV file into the temporary table:

    LOAD DATA LOCAL INFILE 'PATH_TO_YOUR_CSV_FILE/users.csv'
        INTO TABLE user_tmp CHARACTER SET 'utf8' FIELDS TERMINATED BY '~' LINES TERMINATED BY '\n';

3. Fill your own tables

Suppose you have the following two tables:

User

  • name
  • phone
  • email

Agent

  • name
  • phone
  • email
  • company_name
  • role
  • company_phone

Insert data with SQL queries:

INSERT INTO USER (name, phone, email)
    SELECT name, phone, email FROM user_tmp WHERE TYPE = 'user';
INSERT INTO agent (name, phone, email, company_name, ROLE, company_phone)
    SELECT name, phone, email, company_name, agent_role, company_phone FROM user_tmp WHERE TYPE = 'agent';

All done! Your tables are complete.

This is a simple example, you can use this method to make more complex data imports (with joins). All you need to do is to adapt your SQL queries.

Here we have seen how we can leverage something fast but apparently limited (LOAD DATA) and make it powerful, by using a temporary table and SQL requests inserting data into the actual tables.

Filed under : Theodo, Tips — Vincent Guillon @ 10 h 53 min

24 June 2011

Graphic design for the web – Fonts

Font usage on the web is a critical matter. Well I guess it’s critical while working with other media too, but if you’re a graphic designer and you’re not that familiar with the best practices of website creation, you might come up with something rude to web designers, be it beautiful or not.

The #1 thing to keep in mind is that text is the core of most websites. In fact, websites are often comparable to newspapers since both media feature categorized articles, with headings, paragraphs, etc. In short: information.

Information also consists of images but here comes rule #2: information must be accessible to anyone. And while text can be accurately reproduced as sound by software known as screen readers (Jaws, Orca, etc.), images can not. Thus, for the blind to access the information transmitted through images we add an “alternative” text description to them, but it’s often limited. We could also put on a more complete description and then hide it to users who can see the image, yet that may require additional work.

Choosing the right fonts

Why, if today’s topic is “Fonts”, am I talking about images? Simply because in order to reproduce some fancy fonts on the web, we have no choice but to use images. Yes, a title reading “My Awesome Website” rendered using a font by Ray Larabie in Adobe Photoshop will not be displayed the same on the end user’s browser if it can’t find the exact same font on the computer.

Solutions? Sure, there are several, as can be seen in this article by Shaun Cronin.
To sum up:

  • some solutions require additional javascript (that impaired users might disable)
  • others require Adobe Flash (heavy, and disabled by even more people)
  • most make use of the @font-face property, which is somewhat new and won’t work with obsolete browsers still in use (think Internet Explorer < 9)
  • the old and dirty solution: using images, meaning that you have to make a new image for every element in each language, and change them all everytime you want to modify the wording. Plus, it won’t resize nicely and need intelligent text alternatives

In addition, Google and other search bots love real text that you can copy/paste so if the text is rendered via an image or a flash object, most of the time these bots won’t catch and index the information… not very SEO friendly.

Therefore I’m sorry to say that (at least until IE 6, 7 and 8 are all below 5% in usage statistics) the best thing to do is to find elegant ways to integrate standard fonts, also referred to as web safe fonts. They simply are the most common fonts to be expected on everybody’s computer. In that bunch you’ll find Arial, Times New Roman, Courier New, then Verdana, and to a lesser extent Georgia, Impact, and some other native Microsoft fonts (sadly…).

Also, you know that fonts can be sorted in 3 categories: serif, sans-serif and monospace fonts.
This is important because web developers/designers will call fonts like this:

font-family: helvetica, arial, sans-serif;

If the font “helvetica” is not found, then it will search for “arial”, and again, if it’s nowhere to be found then it will use the default sans-serif font defined by the browser. Therefore we must ALWAYS list at least 2 fonts: the desired one and the default font type. 3 is better.

So, in addition to working with a limited set of standard fonts, I advise graphic designers to make sure that their layout doesn’t fall apart when switching the font to Arial (a good default sans-serif font) or Times New Roman (a good default serif font). Monospace fonts are mainly used to display code so you should not have to use them a lot, but if you do, then Courier New is a good default monospace font.

If you really need a fancy font, then I would recommend choosing one from Google Web Fonts and/or to provide the said font files along with the rest of your work.

That’s all for the choice of fonts – the big part actually, but there’s more to talk about.

Styling your fonts

Because a font is not only a “font-family” affair, it’s also about the size, the color, the style you give your font.
A rather complete font declaration by a web designer would look like this:

font-family: helvetica, arial, sans-serif;
font-style: bold;
font-size: 12px;
line-height: 16px;
text-transform: uppercase;
color: #000000;

Or in a more compact way:

font: bold 12px/16px helvetica, arial, sans-serif;
text-transform: uppercase;
color: #000000;

One thing you don’t see here because it doesn’t really exist in the web world but does in Photoshop is the anti-aliasing setting (None, Sharp, Crisp, Strong, Smooth). Therefore, I beg you not to use different anti-aliasing levels and to keep it to the Crisp level.

Now, 2 things are important, mainly in order to grant the website a sufficient accessibility level:

  • consistency
  • readibility

Consistency means that elements that are to be found in different pages should be identical. For fonts, it implies for instance that the title of the page is always at the same place, using the same font at the same size, with the same color, etc.

At times you may need to lower the size to be able to insert a longer title in the same area. Should this happen, either change all titles or find another solution because this one is not acceptable. Also, you should always take into account this situation where a title or any other text variable could expand beyond the limit of its container, and suggest a solution (such as truncation).

To ensure more visual consistency, please limit yourself to 3 or 4 different font declarations.

All these measures help in not confusing readers and thus improve the readibility of the site. We can do more by making sure the text is actually visible to everyone (rule #2), and this time I’m talking about people who are not blind but still suffer from troubled vision.

For these people, we should ensure that the font size is not ridiculously small (in pixels, I would set the limit to 9) and the contrast between the text and the background is strong enough. There are precise recommendations established by the W3C addressing this concern. And hopefully a lot of tools allow you to check the validity of the chosen colors, such as Colour Contrast Check by Jonathan Snook.

Links are also interesting. One should be able to distinguish a link just by looking at the text, with no need to hover the mouse to find them. To achieve this, it is recommended to apply a different color to links AND to underline them.
Alterations in size or style (bold and italic) are to be avoided since these distinctions can apply to normal text as well, with a proper meaning.

Another thing is the letter case. You need not write titles directly in uppercase and should use the “All Caps” transformation in the Character tool panel in Photoshop instead. That’s also what web designers do thanks to the “text-transform” property. This way, we can copy/paste content text in readable form – the real content being the unformatted text, the switch to capital letters is only part of the style, the decoration. Plus, this kind of transformation preserve accents, if needed.

Visualize the site like end users would

Last and not so trivial, I know some of you are crazy about Apple products. It doesn’t matter what my opinion on these are, but it can lead to confusion and frustration since you might present your work to the client, and they will be enthustatic about it – that’s cool. But then we present them with the web rendition of your mockups and they don’t understand why the scrollbars are so ugly compared to what they had seen on the static pages you designed – showing Safari Mac like scrollbars.
Same for fonts, that are notoriously “bolder” on a Mac than they are on Linux or Windows – the OS the client and the end users are most expected to use (and thus why you shouldn’t never use the confusing “Strong” anti-aliasing).

So when dealing with fonts, you should preferably:

  • choose standard fonts
  • illustrate what happens with shorter/longer text
  • be consistent
  • make it readable

I am well aware that the points I make in this article can be regarded as restrictive to your creative spirit but please do not think of them as constraints, they’re meant as guidelines to a more efficient and “web ready” layout, for the World Wide Web to a better place for everyone :)

Filed under : Theodo — cyrillej @ 17 h 58 min

11 April 2011

DevOps conference online!

My conference about adopting DevOps philosophy on Symfony projects is now online! You can see (and listen to) me speaking here: http://symfony.com/video/Paris2011/573

In this presentation you will see what I think is the philosophy behind the DevOps movement and how to start with the 4 important aspects of adopting DevOps:

  • Configuration Management with Puppet
  • Development on the production environment with Vagrant
  • Deployment automation with Fabric
  • Continuous deployment with Jenkins

If you are interested by the DevOps movement and you happen to be in Paris, come to our Paris-DevOps meetups. After the two first meetups hosted by Theodo, the meetup is now traveling to other locations. The next one will be held on the 4th of May at Xebia’s office. More info here: http://parisdevops.fr/

Filed under : Agile and DevOps, Theodo, symfony — Fabrice Bernhard @ 13 h 21 min

17 March 2011

Start using Vagrant

I had the chance to spread the good word by talking about adopting DevOps in Symfony projects at the Symfony Live conference. The feedback was very good (for those who attended and have not done so yet, you can give some feedback here: http://joind.in/talk/view/2756) You can also find the slides here: http://www.slideshare.net/mobile/fabrice.bernhard/adopt-devops-philosophy-on-your-symfony-projects and the source code of the slides here: https://github.com/fabriceb/sflive2011-devops)

One of the very good news is that I convinced many people (as observed on twitter) to start using vagrant after the conference. The slides gave a quick introduction, so let me give here a more detailed tutorial on how to start using vagrant.

Introduction to Vagrant

Vagrant is a ruby tool that makes the process of testing your code in a virtual machine VERY easy. You are concerned:

  • If you are a developer on a complex project with a specific system configuration on the production server that you want to reproduce in your development environment painlessly. This specific system configuration can be specific versions of some packages, a specific architecture or simply a specific OS.
  • If you are a Mac user! Because the chances that the project you develop will be hosted on a Mac are quite small…

It is quite amazing to see how many people develop on Macs to deploy on Linux systems and don’t use virtual environments. This has two obvious downsides:

  • you need to install a working development environment on your Mac which can quickly become a pain.
  • even with PHP applications, which are usually quite platform-independent, it is never truly the case and it is much better to avoid last-minute system-related surprises.

Install Vagrant

sudo gem install vagrant
  • Debian/Ubuntu
# Install Ruby, Ruby Gems and Vagrant
sudo apt-get install rubygems
sudo apt-get install ruby-dev
sudo apt-get install build-essential
sudo gem install vagrant
# Add the ruby gems path to your path
PATH=$PATH:/var/lib/gems/1.8/bin
# Download and install VirtualBox
echo 'echo "deb http://download.virtualbox.org/virtualbox/debian maverick contrib" >> /etc/apt/sources.list' | sudo sh
wget -q http://download.virtualbox.org/virtualbox/debian/oracle_vbox.asc -O- | sudo apt-key add -
sudo apt-get update
sudo apt-get install virtualbox-4.0
sudo apt-get install dkms

You should now be able to type vagrant in your terminal and see the list of tasks that you can do.

$ vagrant
Tasks:
  vagrant box                        # Commands to manage system boxes
  vagrant destroy                    # Destroy the environment, deleting the created virtual machines
  vagrant halt                       # Halt the running VMs in the environment
  vagrant help [TASK]                # Describe available tasks or one specific task
  vagrant init [box_name] [box_url]  # Initializes the current folder for Vagrant usage
  vagrant package                    # Package a Vagrant environment for distribution
  vagrant provision                  # Rerun the provisioning scripts on a running VM
  vagrant reload                     # Reload the environment, halting it then restarting it.
  vagrant resume                     # Resume a suspended Vagrant environment.
  vagrant ssh                        # SSH into the currently running Vagrant environment.
  vagrant ssh_config                 # outputs .ssh/config valid syntax for connecting to this environment via ssh
  vagrant status                     # Shows the status of the current Vagrant environment.
  vagrant suspend                    # Suspend a running Vagrant environment.
  vagrant up                         # Creates the Vagrant environment
  vagrant version                    # Prints the Vagrant version information

Now fetch the base Ubuntu Ludic 32 box provided by Vagrant. This make take a few minutes depending on your connection, since it consists in downloading around 700MB

$ vagrant box add base http://files.vagrantup.com/lucid32.box

Finally, to avoid permission problems with folder sharing between your host machine and your virtual environment, I highly recommend using NFS instead of VBox, which is the default protocol used by VirtualBox.

  • On Mac OS X, you do not need to do anything, NFS is already installed.
  • On Debian/Ubuntu, you just need to install the NFS package:
$ sudo apt-get install nfs-common nfs-kernel-server

Test on a first project

I set up a test project so that you can see how it works. Create a new folder called sflive2011vm. We will clone the configuration for our Vagrant Virtual Machine and then clone our actual project inside

$ cd sflive2011vm
$ git clone git://github.com/fabriceb/sfLive2011vm.git .
$ git clone git://github.com/fabriceb/sfLive2011.git

Now all you have to do to test the project is

$ vagrant up

and after a few minutes, Vagrant will have started a virtual Ubuntu, installed all the packages needed and set up tha machine as described in the Puppet manifest. To verify that everything worked as planned just visit http://127.0.0.1:2011/hello/master

That’s it!

Understand Vagrant

Let us now understand more deeply how Vagrant works:

The base box

The base box is simply a saved hard-disk of a Virtual Machine created with VirtualBox. It can contain anything but it needs at least :

  • Ruby
  • VirtualBox guest additions
  • Puppet
  • Chef

to be boot-strapped by Vagrant and then further configured by a Chef recipe or a Puppet manifest

Vagrantfile

This is the configuration file of Vagrant. The most useful options are port forwarding, provisioning solution (Puppet or Chef) and eventually NFS

Vagrant::Config.run do |config|
  config.vm.box = "base"
  config.vm.forward_port("web", 80, 2011)
  config.vm.provision :puppet
  # config.vm.share_folder("v-root", "/vagrant", ".", :nfs =&gt; true)
end

Provisioning

The configuration of your VM is coded using Chef or Puppet. This ensures that you will reproduce exactly the same configuration in all your development VMs AND your production environment. Here is the Puppet manifest used in the example:

exec { "apt-get-update":
  command =&gt; "apt-get update",
  path =&gt; ["/bin", "/usr/bin"],
}
 
Package {
 ensure =&gt; installed,
 require =&gt; Exec["apt-get-update"]
}
 
class lighttpd
{
  package { "apache2.2-bin":
    ensure =&gt; absent,
  }
 
  package { "lighttpd":
    ensure =&gt; present,
  }
 
  service { "lighttpd":
    ensure =&gt; running,
    require =&gt; Package["lighttpd", "apache2.2-bin"],
  }
 
  notice("Installing Lighttpd")
}
 
class lighttpd-phpmysql-fastcgi inherits lighttpd
{
 
  package { "php5-cgi":
    ensure =&gt; present,
  }
 
  package { "mysql-server":
    ensure =&gt; present,
  }
 
  exec { "lighttpd-enable-mod fastcgi":
    path    =&gt; "/usr/bin:/usr/sbin:/bin",
    creates =&gt; "/etc/lighttpd/conf-enabled/10-fastcgi.conf",
    require =&gt;  Package["php5-cgi"],
  }
 
  notice("Installing PHP5 CGI and MySQL")
}
 
class symfony-server inherits lighttpd-phpmysql-fastcgi
{
 
  package { ["php5-cli", "php5-sqlite"]:
    ensure =&gt; present,
    notify  =&gt; Service["lighttpd"],
  }
 
  notice("Installing PHP5 CLI and SQLite")
}
 
class symfony-live-server inherits symfony-server
{
 
  file { "/etc/lighttpd/conf-available/99-hosts.conf":
    source =&gt; "/vagrant/files/conf/hosts.conf",
    notify  =&gt; Service["lighttpd"],
    require =&gt; Package["lighttpd"],
  }
 
  exec { "lighttpd-enable-mod hosts":
    path =&gt; "/usr/bin:/usr/sbin:/bin",
    creates =&gt; "/etc/lighttpd/conf-enabled/99-hosts.conf",
    require =&gt; File["/etc/lighttpd/conf-available/99-hosts.conf"],
    notify  =&gt; Service["lighttpd"],
  }
 
  notice("Installing and enabling Hosts file")
}
 
include symfony-live-server
notice("Symfony2 server is going live!")

Take home message

  • Vagrant is an essential part of the DevOps process: it is the solution to developing, testing and deploying in the same environment. It thus ensures a smoother transition of your project from the dev team to the ops team
  • Vagrant is EASY. And it is compatible with both Chef and Puppet
  • Vagrant is a must for Mac developers.
Filed under : Agile and DevOps, Theodo — Fabrice Bernhard @ 12 h 15 min

11 December 2010

Symfttpd: symfony’s long-awaited lightweight server

A project is never finished, and at Theodo we often have to quickly alter a project, in a small way. The issue is that the time required to having the project ready on a developer’s machine can be greater that the time required to do the the rest! But worse, setting up a project is boring.

This is why we decided to automate as much as possible the low added-value aspects of setting up a project.

There are usually four aspects to setting up a project:

  • Dependencies
  • Configuration files
  • A database
  • A web server, configured for the website, including the URL rewriting

Dependencies

In our case, it usually means only “Symfony, version 1.x”. For Python projects, a pip dependencies file is provided, and running a script will create a virtualenv and install the dependencies: no root access or hunting on the web is needed.

For Symfony projects, it’s easy to have all major versions at hand. However, these projects usually require multiple symbolic links to be created, in non-standardized places. Thankfully, we chose to adopt convention over configuration and only one line of configuration is actually needed for mksymlinks to do its magic.

If you chose to install Symfony other than in lib/vendor (likely with versions under 1.2), you can of course configure mksymlinks to handle it.

Configuration

The configuration of a project on the production servers and on the developer’s machines usually differ; database passwords, mail and database servers location, etc. The configuration files are therefore not versioned, only sample versions of them are, designed for a standard development environment.

A simple script can find the “sample” configuration files and create symbolic links (ideal for developers) or copies of these sample files. The developer shouldn’t need to alter these files to setup the project.

Running the script will for instance create a symlink from config/databases.yml to config/databases.sample.yml.

Databases

While recent versions of Symfony and Doctrine can create a database, older versions and Propel can’t. Also, database users and their credentials still have to be created, and it is really annoying to set them up. We use a simple script which will write a SQL script, and will try to run it using the credentials of ~/.my.cnf.

An alternative is to use SQLite, but it can quickly become a limitation for bigger projects.

The database is then filled with meaningful and useful fixtures.

The web server

Last but not least, the web server. Frameworks like Ruby on Rails or Django come with their own servers. These are not for production use, but are perfect for developer’s needs: no root access required, instant startup, no configuration… In the PHP world, nothing similar to be found. However, it is very well possible to configure and start a small server automatically.

That’s the job of symfttpd and its command spawn: run it and it will configure and start a lighttpd-based server, then tell you where you can view the project, and what applications are present.

The scripts handling the sample configuration and database are rather crude and around 50 lines of code; you should be able to write your own in no time if you are too impatient for a proper release of our own.

Let’s not forget documentation

While all of this works great, nothing replaces an INSTALL file with all the necessary steps, especially the unusual ones. It’s also nice to provide scripts to run all the required commands in one go; run it and your project is ready! (There’s still time to get a nice cup of coffee especially with our new awesome coffee machine.)

There is more to come

On the same principles, we use some of the tools mentioned above in our continuous integration platform, to be able to add any project in no time.

Symfttpd with its tools spawn and mksymlinks has been publicly released: https://github.com/laurentb/symfttpd. You can use them to accelerate the deployment of your development environment. As for the small scripts which automate the rest of our needs, we will surely release them in an upcoming post about continuous integration. More about it will follow, so be sure to subcribe!

Filed under : Theodo, symfony — Laurent Bachelier @ 0 h 19 min

2 December 2010

Paris Devops meetup in Theodo’s new office

Yesterday, December 1st, Theodo had the pleasure to host the improvised first Paris Devops meetup in our new office! Samuel Maftoul and Philippe Müller managed to gather a very interesting and mixed crowd of devs and ops to discuss on the promising ideas that the Devops movement is bringing in the agile develpoment world. Were present:

It was also the occasion to ask Laurent Bossavit to sign his book Gestion de projet : Extreme Programming which is our bible here at Theodo, a great honour :-)

There were too many subjects and too little time during this first session to really tackle the technologies and ideas behind devops but it was a great occasion to meet diverse people and create a first contact. A very promising start for this new meetup! A big thanks to Samuel Maftoul and Philippe Muller for the initiative and if you want to know more and be informed about the next sessions, join the Paris-devops Google Group.

Filed under : Theodo — Fabrice Bernhard @ 16 h 48 min

5 March 2009

Le blog de Theodo

Bienvenue sur le blog de Theodo !

Theodo est une jeune équipe de développement, spécialisée en symfony en particulier et plus généralement toutes les technologies de développement web opensource. Quoi de plus normal quand on a la philosophie opensource du partage des connaissances de partager à son tour ses découvertes. C’est le but de ce blog, où nous partagerons régulièrement les découvertes techniques ou les astuces des développeurs Theodo.

Filed under : Theodo — Fabrice Bernhard @ 16 h 57 min