30 September 2011

Symfony2 unit database tests

Today I will explain how to test your entities in a Symfony2 and Doctrine2 project.

To achieve our work, we will work on a location model which will look somewhat like this:

Location:
- address: string, required
- zip code: string, required
- city: string, required
- country: string, required

Test Driven Development

In the test driven development (TDD) world, a best practice is to start writing your test case before writing any code. So we will write our test case in the Tests/Entity folder of our bundle:

/**
 * Location class test
 *
 * @author Benjamin Grandfond <benjaming@theodo.fr>
 * @since 2011-07-16
 */
namespace ParisStreetPingPong\Bundle\PsppBundle\Entity;
 
class LocationTest extends \PHPUnit_Framework_TestCase
{
    protected $location;
 
    public function setUp()
    {
        parent::setUp();
 
        $this->location = new Location();
    }
 
    public function testGetAddress()
    {
        $address = '80 Rue Curial';
 
        $this->location->setAddress($address);
 
        $this->assertEquals($address, $this->location->getAddress());
    }
}

Note that the aim of this blog post is not to write a test case that covers 100% of the code, but show how to to write a database test case easily.

Once your test is written, if you run it it should not pass; don’t worry, we will write the code to make it work ;) Instead of manually creating a file as you would usually do, you can use PHPUnit! It handles the creation of classes from the test case:

$ phpunit --skeleton-class src/Theodo/Bundle/MyBundle/Tests/Entity/LocationTest.php

This will generate your Location.php class in the same folder as the LocationTest.php file, you only need to move it to the Entity folder of your bundle. The tree of your application should look like:

src/Theodo/Bundle/MyBundle
|-- Entity
|  |-- Location.php
|-- Tests
|  |-- Entity
|  |  |-- LocationTest.php

And your Location.php should already contains some code :

<?php
/**
 * Generated by PHPUnit on 2011-07-29 at 17:18:33.
 */
class Location
{
    /**
     * @todo Implement setAddress().
     */
    public function setAddress()
    {
        // Remove the following line when you implement this method.
        throw new RuntimeException('Not yet implemented.');
    }
 
    /**
     * @todo Implement getAddress().
     */
    public function getAddress()
    {
        // Remove the following line when you implement this method.
        throw new RuntimeException('Not yet implemented.');
    }
}
?>

So now, you only need to add properties with Doctrine annotations! I recommended against using the YAML or XML formats to describe your model because, when you will generate your getters and setters, Doctrine will append properties and methods to the existing source, so you will have to copy/paste a lot to clean up the code…

Finally, your class should look like this:

namespace Theodo\Bundle\MyBundle\Entity;
 
use Doctrine\ORM\Mapping as ORM;
 
/**
 * @ORM\Entity
 * @ORM\Table(name="location")
 */
class Location
{
    /**
     * @ORM\Id
     * @ORM\Column(type="integer")
     * @ORM\GeneratedValue(strategy="AUTO")
     */
    protected $id;
 
    /**
     * @ORM\Column(type="string")
     */
    protected $address;
 
    /**
     * @ORM\Column(type="string", length="7", name="zip_code")
     */
    protected $zipCode;
 
    /**
     * @ORM\Column(type="string")
     */
    protected $city;
 
    /**
     * @ORM\Column(type="string")
     */
    protected $country;
 
    /**
     * Set $address
     *
     * @param string $address
     */
    public function setAddress($address)
    {
        $this->address = $address;
    }
 
    /**
     * Get $address
     *
     * @return String
     */
    public function getAddress()
    {
 
        return $this->address;
    }
}

Actually, this sample does not prove the real utility of the skeleton class generation with PHPUnit because the class could have been generated with Doctrine generate entities command, but you can use it with a class which does not deal with Doctrine.

If you launch your test now it should pass, but we didn’t do anything that needs the database. So I will add a $localization property to the Location class which will contain the full address.

// MyBundle\Entity\Location
 
/**
 * @ORM\Entity
 */
class Location
{
 ...
/**
 * @ORM\Column(type="text", nullable="true")
 */
protected $localization;
 
...
}

Now we will complete our Location test, and after we will implement the generateLocalization() which should be called on the prePersist event.

Configuration

The first thing you must do when you run a test that use database insertion with Symfony2 and Doctrine2, is to set up the database connection. To do so, you have configure the doctrine DBAL handling the connection in the config_test.yml file:

imports:
    - { resource: config_dev.yml }
 
framework:
    test: ~
    session:
        storage_id: session.storage.filesystem
 
web_profiler:
    toolbar: false
    intercept_redirects: false
 
swiftmailer:
    disable_delivery: true
 
doctrine:
    dbal:
        driver:       sqlite
        host:         localhost
        dbname:    db_test
        user:         db_user
        password: db_pwd
        charset:     UTF8
        memory:    true

So, to run our tests, we will use SQLite in memory. While you are free to use something else, it will not be as efficient and easy to setup. Also you won’t need to use transactions to revert the data as they were before the test, you can delete anything and recreate it very quickly.

PHPUnit test case

Now that the configuration is done, you will use the kernel of your Symfony2 application which will load this configuration, Doctrine and the full application. We will do this in another class that must be abstract to not being considered as a test case by PHPUnit. It will also allow us to use it anytime we need to test something with databases interactions.

/**
 * TestCase is the base test case for the bundle test suite.
 *
 * @author Benjamin Grandfond
 * @since  2011-07-29
 */
 
namespace ParisStreetPingPong\PsppBundle\Tests;
 
require_once dirname(__DIR__).'/../../../../app/AppKernel.php';
 
use Doctrine\ORM\Tools\SchemaTool;
 
abstract class TestCase extends \PHPUnit_Framework_TestCase
{
    /**
     * @var Symfony\Component\HttpKernel\AppKernel
     */
    protected $kernel;
 
    /**
     * @var Doctrine\ORM\EntityManager
     */
    protected $entityManager;
 
    /**
     * @var Symfony\Component\DependencyInjection\Container
     */
    protected $container;
 
    public function setUp()
    {
        // Boot the AppKernel in the test environment and with the debug.
        $this->kernel = new \AppKernel('test', true);
        $this->kernel->boot();
 
        // Store the container and the entity manager in test case properties
        $this->container = $this->kernel->getContainer();
        $this->entityManager = $this->container->get('doctrine')->getEntityManager();
 
        // Build the schema for sqlite
        $this->generateSchema();
 
        parent::setUp();
    }
 
    public function tearDown()
    {
        // Shutdown the kernel.
        $this->kernel->shutdown();
 
        parent::tearDown();
    }
 
    protected function generateSchema()
    {
        // Get the metadatas of the application to create the schema.
        $metadatas = $this->getMetadatas();
 
        if ( ! empty($metadatas)) {
            // Create SchemaTool
            $tool = new SchemaTool($this->entityManager);
            $tool->createSchema($metadatas);
        } else {
            throw new Doctrine\DBAL\Schema\SchemaException('No Metadata Classes to process.');
        }
    }
 
    /**
     * Overwrite this method to get specific metadatas.
     *
     * @return Array
     */
    protected function getMetadatas()
    {
        return $this->entityManager->getMetadataFactory()->getAllMetadata();
    }
}

Complete the test

/**
 * Location class test
 *
 * @author Benjamin Grandfond <benjaming@theodo.fr>
 * @since 2011-07-16
 */
namespace ParisStreetPingPong\Bundle\PsppBundle\Entity;
 
use ParisStreetPingPong\PsppBundle\Tests\TestCase;
 
require_once dirname(__DIR__).'/TestCase.php';
 
class LocationTest extends TestCase
{
    ...
    public function testGenerateLocalization()
    {
        $this->location->setAddress('14 Rue Notre-Dame-des-Victoires');
        $this->location->setZipCode('75002');
        $this->location->setCity('Paris');
        $this->location->setCountry('FR');
 
        // Save the location 
        $this->entityManager->persist($this->location);
        $this->entityManager->flush();
 
        $this->assertEquals('14 Rue Notre-Dame-des-Victoires 75002 Paris FR', $this->location->getLocalization());
    }
}

generateLocalization implementation

And now we only need to complete our Location entity and launch again our test that must pass :)

// MyBundle\Entity\Location
 
/**
 * @ORM\Entity @ORM\HasLifecycleCallbacks
 */
class Location
{
 ...
 
    /** @ORM\PrePersist */
    public function generateLocalization()
    {
        $localization = $this->getAddress().' ';
        $localization .= $this->getZipCode().' ';
        $localization .= $this->getCity().' ';
        $localization .= $this->getCountry();
 
        $this->setLocalization($localization);
    }
}
Filed under : Symfony2 — Benjamin Grandfond @ 9 h 52 min

26 September 2011

Symfony2: Working with multiple databases

Symfony2 has been around for quite a while. Personally, I love how much PHP-oriented it is. It feels much closer to the language base than the first version of the framework. It means less of the magic and more of the important decisions in the hands of the development team. But there is no real framework without some magic, which is sometimes really hard to master.

Enough of the introduction, it is time for the technical stuff. First, let us take a look at some user cases:

  • As the bundle base grows, we see or might see soon a lot of bundles which would make heavy use of the database – blogs, forums, eCommerce… Let’s say we want to integrate such a bundle with our EnormousWebsiteBundle. Easy, isn’t it? But wait! We did not think of prefixing our table names (who does?), and neither did the author of the bundle. And all of a sudden we have conflicts everywhere.
  • We have some databases that already exist. And the client wants us to make an application that uses all of them.
  • We want to backup different sets of data at different frequencies.
  • We need to optimize our application, using different data storage solutions: a SQL database, a NoSQL database, etc.

In all those cases one of the solutions (or a necessity) is to use multiple databases. So let us do some Symfony2 magic!

It begins in config.yml

Lets say we have a simple blog bundle, which we want to adapt to use its own database. The easy part is configuring the connection:

doctrine:
    dbal:
        connections:
            ...
            blog:
                driver:   %blog.database_driver%
                host:     %blog.database_host%
                dbname:   %blog.database_name%
                user:     %blog.database_user%
                password: %blog.database_password%
                charset:  UTF8

Next, create a second entity manager:

doctrine:
    orm:
        entity_managers:
            ...
            blog:
                connection:   blog
                mappings:
                    MyAwesomeBlogBundle: ~

Well, we can say that your bundle is configured.

Try to use your second database

It is easy to find in the official documentation that you can simply do

$this->get('doctrine')->getEntityManager($name)

to use your custom entity manager. But I guess you never actually had to do it, being happy with the default one. So this will require some refactoring. The simplest solution is to specify a parameter in your config, let’s say:

parameters:
    my_awesome_blog.entity_manager.name: blog

If you’re going to publish your bundle, set it to ‘default’, in case somebody wouldn’t want to use a separate EM, and you should be safe. Now, you need to pass the parameter to each (well, most of) getEntityManager calls in your bundle. It will be a bit of work, depending on the was you used that function. Let’s hope you defined some services, like this one:

my_awesome_blog.content_repository:
    class: %my_awesome_blog.content_repository.class%
    arguments: ['@doctrine.orm.entity_manager']

or some functions like

$this->getEntityManager()

in your controllers. Don’t worry about the extra work, at least it will help you to decouple your code even more (and we like loosely coupled code, don’t we?).

Is it all?

It depends. These are the basics. Things are getting tricky when:

You need to login with an entity which is not in the default entity manager

You will need to overwrite the user provider, and pass your custom entity manager to it. In the simplest form it will be something like that:

# security.yml
security:
    providers:
        blog_user:
            id: my_awesome_blog.user_provider #this is the name of your service

Now you need to register a simple service which will use your custom entityManager:

# services.yml
parameters:
    my_awesome_blog.user_provider.class: Symfony\Bridge\Doctrine\Security\User\EntityUserProvider
    my_awesome_blog.user_provider.user.class: MyCompany\MyAwesomeBlogBundle\Entity\User
    my_awesome_blog.user_provider.user.parameter: username
 
services:
    my_awesome_blog.user_provider:
        class: %my_awesome_blog.user_provider.class%
        arguments:
            - '@doctrine.orm.blog_entity_manager'
            - %my_awesome_blog.user_provider.user.class%
            - %my_awesome_blog.user_provider.user.parameter%

This one will allow you to use a standard “form_login” configuration, as long as you pass the provider to your firewall (see security reference if you’re not familiar with the config options: http://symfony.com/doc/2.0/reference/configuration/security.html).

You have some forms that use your entities

This one is a little tricky. By default most of internal functions use

$container->get('doctrine')->getEntityManager()

which just doesn’t work with multiple EM’s. You’ll get errors saying you Entity is not an Entity (feels like JavaScript!). Don’t worry it is an Entity, just not registered in that manager. I’ve recently made a pull request about this issue (here), and it got into symfony:master, but if you still use 2.0 you have to take the matters in your own hands. So far I’ve found one class that needs to be changed (see the pull request). Simply change the few mentioned lines of code, save it in your bundle and add this to your services:

   doctrine.orm.validator.unique:
        class: MyCompany\MyAwesomeBlogBundle\Validator\Constraints\BlogUniqueEntityValidator
        tags:
            - { name: validator.constraint_validator, alias: doctrine.orm.validator.unique }
        arguments: ['@doctrine']

Well, if you ever find anything else, and you don’t feel like defining your very own service, just try to use the awesome getEntityManagerForClass() function and overload some default classes.

Good luck!

Defining your own entity manager seems easy. This part of Symfony2 configuration is awesome. It is easy, as long as you’re not trying to force it to do some more complicated stuff. After a certain point, you find a bunch of default services, which you need to redefine/overload/give up on using at all. Well, whether you really need to do this, or just want to see how it would be like… I wish you best of luck, and don’t forget to share your experience!

Filed under : Symfony2 — Marek Kalnik @ 16 h 05 min

19 September 2011

“Adopt devops philosophy” at the Open World Forum

Version française plus bas

Save the date! Theodo will be present at the Open World Forum, as I have been selected to talk about “Adpoting devops philosophy” on Friday Sept. 23 at 16:30.

More info about the conference here: http://www.openworldforum.org/Conferences/Adopter-la-philosophie-DevOps

I am very happy to be able to spread the good word in such an important conference! The main purpose will be to explain how devops extends agility and its concepts to the whole lifecycle of an IT project, including deployment and system administration and how this can improve the productivity and responsiveness of your IT organization.

See you there!

Réservez votre vendredi 23 septembre ! Theodo sera présent à l’Open World Forum, j’ai en effet été sélectionné pour intervenir comme conférencier sur “Comment adopter la philosophie devops” ce vendredi 23/9 à 16:30.

Plus d’informations sur la conférence ici: http://www.openworldforum.org/Conferences/Adopter-la-philosophie-DevOps

Je suis très content de pouvoir répandre la bonne parole devant un nouveau public. Mon objectif sera d’expliquer comment la philosophie devops étend les concepts d’agilité à tout le cycle de vie d’un projet informatique, déploiement et maintenance incluse et comment ces concepts peuvent augmenter la productivité et la réactivité de votre organisation informatique.

Filed under : Agile and DevOps, Theodo, symfony — Fabrice Bernhard @ 15 h 07 min

1 September 2011

Massive data import! – Part 1

We often have to face the problem of importing data off an Excel file with thousands lines.
PHP is not suited for this task, it’s slow and there is a high risk for the import to crash due to “memory limit” or some other annoying stuff like that!
So instead we chose a better way by using pure SQL which is much faster at this kind of operation.

At first, you must convert your Excel file to CSV (Excel does it very well). Be careful to choose the right field separator: I generally use “~” because there is little chance of finding this character in your written data.

Steps:

  • Create a temporary table that matches exactly the structure of the Excel file
  • Fill the temporary table with the CSV file
  • Run SQL queries to fill your database

Practical example:

Suppose we have an Excel file containing thousands of users that must be dispatched to several tables depending on their type.

CSV file sample:

        User 1~user1@theodo.fr~0987564321~user~~~
        User 2~user2@theodo.fr~0134256789~user~~~
        User 3~user3@theodo.fr~0128971271~user~~~
        Agent 1~agent1@company.com~0486282688~agent~Company 1~Role 1~0987654321
        Agent 2~agent2@company.com~0176254621~agent~Company 2~Role 2~0445664332
        User 4~user4@company.com~0456789856~user~~~

1. Create the temporary table

We will create a table contain the following fields:

  • name
  • email
  • phone
  • type
  • company_name
  • agent_role
  • company_phone
DROP TABLE IF EXISTS user_tmp;
CREATE TABLE user_tmp (
        name	 VARCHAR(127),
        email VARCHAR(127),
        phone VARCHAR(20),
        TYPE VARCHAR(20),
        company_name	VARCHAR(127),
        agent_role VARCHAR(127),
        company_phone VARCHAR(20),
        id INT(11) NOT NULL AUTO_INCREMENT,
        PRIMARY KEY (`id`),
        UNIQUE KEY `IDX_ATTRIBUTE_VALUE` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

2. Fill the temporary table

Import your CSV file into the temporary table:

    LOAD DATA LOCAL INFILE 'PATH_TO_YOUR_CSV_FILE/users.csv'
        INTO TABLE user_tmp CHARACTER SET 'utf8' FIELDS TERMINATED BY '~' LINES TERMINATED BY '\n';

3. Fill your own tables

Suppose you have the following two tables:

User

  • name
  • phone
  • email

Agent

  • name
  • phone
  • email
  • company_name
  • role
  • company_phone

Insert data with SQL queries:

INSERT INTO USER (name, phone, email)
    SELECT name, phone, email FROM user_tmp WHERE TYPE = 'user';
INSERT INTO agent (name, phone, email, company_name, ROLE, company_phone)
    SELECT name, phone, email, company_name, agent_role, company_phone FROM user_tmp WHERE TYPE = 'agent';

All done! Your tables are complete.

This is a simple example, you can use this method to make more complex data imports (with joins). All you need to do is to adapt your SQL queries.

Here we have seen how we can leverage something fast but apparently limited (LOAD DATA) and make it powerful, by using a temporary table and SQL requests inserting data into the actual tables.

Filed under : Theodo, Tips — Vincent Guillon @ 10 h 53 min