Soon the GDPR will take effect! Are you ready?

As of May 25th, 2018, the European privacy regulation General Data Protection Regulation (GDPR) will take effect. This regulation concerns the ‘protection of natural persons with regard to the processing of personal data and on the free movement of such data’.

Important parts of the regulation are the right to access (and correct) of personal data that’s recorded and stored by organizations. The most profound and fundamental part however is that of the right to remove – under conditions – personal data (the “right to Erasure”, article 17 of the regulation).

Companies that are not compliant with the GDPR risk a substantial fine: a maximum of 20 million euros, or 4% of yearly gross revenue. However, according to research by EY, 33% of global respondents have yet to put a plan in place to realize compliancy. Time to get moving, only three short months remain!

Companies that source their IT-support or systems externally will have to make sure their vendors will be compliant. Even companies that have their own (internal) software development teams face multiple challenges to become compliant. For example, in two earlier blog posts, I researched a number of potential solutions for software applications that use Event Sourcing: here and here.

Are you ready for the GDPR? I’m more than happy to help you make the right choices. I’m not a legal expert, maar but can advise you in terms of (software) implementation or tools & technology.

Binnenkort treedt de AVG (GDPR) in werking!

Vanaf 25 mei 2018 is de Europese privacyverordening Algemene Verordening Gegevensbescherming (AVG, ook bekend als General Data Protection Regulation of GDPR) van toepassing. Deze verordening gaat over de ‘bescherming van natuurlijke personen in verband met de verwerking van persoonsgegevens en betreffende het vrije verkeer van die gegevens’.

Belangrijke onderdelen van de verordening zijn het recht op inzage en correctie van persoonlijke gegevens die door bedrijven worden bijgehouden. Maar het meest fundamentele recht is waarschijnlijk het recht op verwijdering – onder voorwaarden – van persoonlijke gegevens (“Right to Erasure”, artikel 17 van de verordening).

Bedrijven die zich niet aan de AVG houden riskeren een stevige boete: maximaal 20 miljoen euro of 4% van de jaarlijkse omzet. Echter, volgens EY heeft 27% van de Nederlandse organisaties nog geen plan om AVG-compliance te realiseren. Tijd om in actie te komen, er resteren nog slechts een kleine drie maanden!

Organisaties die hun IT-ondersteuning of -pakketten extern sourcen zullen moeten verifiëren dat hun leveranciers zorgdragen voor naleving van de AVG. Ook voor bedrijven die hun IT-ontwikkeling intern hebben belegd zijn er meer dan voldoende uitdagingen om tot naleving van de verordening te komen. In twee eerdere (Engelstalige) blog posts onderzocht ik mogelijke oplossingen voor software-toepassingen die gebruik maken van Event Sourcing: hier en hier.

Uit onderzoek blijkt dat de meerderheid van de Nederlanders nog niet (volledig) op de hoogte is van de regels en mogelijkheden die de AVG biedt. Geconfronteerd met deze rechten zegt 50% ervan gebruik te willen maken; dit kan bedrijven flink op kosten gaan jagen!

Bent u klaar voor de GDPR? Ik help u graag om de juiste stappen te zetten. Ik ben geen juridisch expert, maar kan wel adviseren op het gebied van implementatie of technologiekeuze!

Season’s greetings from Utrecht!

Utrecht – Domtoren in de sneeuw (Chris Heijmans – order a print here)

This amazing photo was taken by Chris Heijmans in 2012. Last year, I ordered a canvas print, which is now proudly displayed in my home. Admittedly, current conditions (partly cloudy, windy, 8 degrees celcius) are nowhere near what’s pictured, but one can hope 😉

2016 has been interesting! I’ve met loads of cool people and been part of some amazing projects. Thank you for reading this blog, attending my talks, following me on Twitter, or listening to me advocate (rant?) in person.

From my cozy, little hometown Utrecht I wish you and yours happy holidays, see you in 2017!

Dutch Web Alliance, leaders in web technology

I’m proud to announce that I’ve joined forces with a group of very talented web developers in the Netherlands and Belgium: the Dutch Web Alliance.

Dutch Web Alliance

More information is available on our website, the full (dutch) press release is posted below:

APELDOORN: Freelance web developers bundelen krachten in de Dutch Web Alliance.
De Dutch Web Alliance (DWA) is een vereniging voor en door freelance
web developers uit Nederland en Belgie. Onze missie is om via kennisdeling, samenwerking en het combineren van resources, de positie van de freelancer en de kwaliteit van web professionals te verbeteren. Een lid van de DWA staat voor kennis en kunde op alle vlakken in web-ontwikkeling en kan zich als zodanig onderscheiden worden door klanten en opdrachtgevers.

De DWA richt zich niet op kwantiteit, maar op kwaliteit. Zo zijn het de leden onderling die bepalen of een nieuw lid ook daadwerkelijk kan toetreden tot de vereniging. Binnen de vereniging houden leden elkaar actief op de hoogte van alle laatste ontwikkelingen, bieden ze hulp en dienen ze als vraagbaak. Maar DWA leden springen ook graag bij andere leden in situaties zoals ziekte, deadlines, specifieke kennis. Zo krijgen afnemers van de DWA de flexibiliteit van freelancers, met de voordelen van full-service internet bureaus.

De DWA is meer dan alleen een keurmerk en kwaliteitswaarborg. Onze leden zijn voor het grootste deel actief in open source communities zoals PHP, dev/ops en system administration. Zij zijn dan ook vaak terug te vinden op nationale en internationale conferenties als spreker waarbij ze als experts en “leaders of the field”, andere ontwikkelaars helpen met nieuwe technieken, best practices en algemene kennis die is opgedaan uit de vele complexe projecten waar ze dagelijks aan werken. De DWA draagt hier nog een extra steentje aan bij door het organiseren van workshops en meetups voor niet alleen ontwikkelaars, maar ook leidinggevenden, project managers en CTO’s/CIO’s. Kortom: onze vereniging zet zich in voor een beter, professioneler en efficiënter klimaat binnen de web-ontwikkeling.

Using conditional build steps to speed up your Jenkins PHP builds

At my client Spotney, we have a pretty solid (and common) build infrastructure for our PHP projects; SVN commits are checked out by Jenkins, tests are run by PHPUnit, Sonar runs static code analysis, and artifacts are built and deployed to a staging environment by Phing. However, some of the code relies pretty heavily on (complex) db queries, adding the need for DbUnit style tests. The nature and quantity of the tests, combined with a slow VM (possibly related to this Xdebug issue) meant that our buildtimes were becoming prohibitively long.

An interesting post by @pascaldevink triggered a conversation and sparked an idea. I started working on our build setup, eventually resulting in a 60-70% decrease of our build times!

Here’s how I did it.

Starting point

Let’s assume we have a fairly standard Jenkins job. The job checks out an SVN repository, and periodically scans that repository for changes, triggering a new build if any are detected.

Each build of the job performs three steps:

  • Run phpunit
  • Run phing (target “build”)
  • Invoke Sonar (using the Jenkins Sonar plugin – this plugin also allows invoking Sonar as a post-build action, but that option requires Maven)

After the build steps, the job publishes the test and code coverage reports, and archives the generated artifacts.

Disabling code coverage and Sonar for regular builds

Two of the most obvious optimizations (also suggested by Pascal) are disabling code coverage on all tests and disabling Sonar runs during regular Jenkins builds. We define regular as either manually started by a user, or by an SCM trigger.

Disabling code coverage generation in PHPUnit is easy, simply remove the “coverage-xxx” elements from the logging section of your phpunit.xml configuration file (see this section of the PHPUnit manual). Disabling Sonar is trivial too, just remove the last build step from the job.

However, this is not an ideal solution: we do want to generate code coverage information and run Sonar at some point, such as during nightly builds, preferably without cloning our existing job. This means that we’ll need to skip code coverage and Sonar invocations on all but the scheduled (nightly) builds.

The Sonar plugin supports excluding SCM triggered builds (“Skip if triggered by SCM Changes”), but that only works if you use the post-build action. Additionally, we need to be able to change the PHPUnit configuration – one file to enable code coverage generation, one file to disable it.

Conditional build steps

The Conditional BuildStep plugin wraps one or more build steps in a conditional execution block. One of the possible conditions is the build cause, i.e. was the build triggered by a user, an upstream project, a timer, an SCM change, etc. etc.

First we define the steps that should be taken for each nightly build of our job. These steps should only be executed when the build is trigger by a timer.

We add a “Conditional steps (multiple)” build step, setting the condition to “Build Cause” and the Build Cause to “TimerTrigger”.

Conditional Sonar Build Config [Jenkins]2

Then we add our three (original) build steps:

Conditional Sonar Build Config [Jenkins]3

As stated before, regular builds are those that are triggered by a user or an SCM change.

We again add a “Conditional steps (multiple)” build step. The condition for this step is a little more interesting, as seen below. We combine two Build Cause conditions (one set to “UserCause”, the other to “SCMTrigger”) using the Or condition.

Conditional Sonar Build Config [Jenkins]4

We then add two build steps: the first will run PHPUnit without code coverage (note that we are specifying a different configuration file here), the second one will run Phing.

Conditional Sonar Build Config [Jenkins]5

Note that in the above build steps we’re invoking Phing from the shell instead of using the Phing plugin. Unfortunately this plugin is currently not supported as a conditional build step (probably because of this JIRA issue).

Build schedule

As a final step we need to update our build schedule.

Conditional Sonar Build Config [Jenkins]1

This will ensure our job runs somewhere after midnight (between 12:00 AM and 2:59 AM to be precise).

The end result:

  • A nightly scheduled build, with all the bells and whistles enabled
  • User and SCM triggered builds run (a lot) faster

Please let me know if you think this post is useful, or if you have any other Jenkins/PHP optimization tips!

Phing development update

The last Phing development update was almost a year ago, so I guess it’s about time for a discussion of what’s new in Phing world.

2.5.0 and Semantic Versioning

The most important news in this update is the brand new 2.5.0 release (pushed out yesterday). We bumped the minor version number because we now (try to) follow the Semantic Versioning spec, and this release introduces new functionality. Our changed roadmap reflects this as well.

This version closes the following issues:

#979 svncommit: invalid switch ignoreexternals
#977 phpunit Task doesn’t support @codeCoverageIgnore[…] comments
#972 SvnCopyTask: remove “force” from documentation
#971 TokenSource does not work
#969 PHPUnit task does not report diffs for failed assertions
#968 Proper handling of STDOUT and STDERR
#963 XSLT task fails with fatal error on PHP 5.4
#962 DbDeploy: infinite loop in case if directory not found
#961 DbDeploy: checkall output isn’t informative
#960 Documentation of Dbdeploy task
#959 Bug in SvnListTask Version 2.4.14
#958 Property wrapped in if/then structure is not substituted by it’s value
#954 Paths becoming part of S3 file names on Windows
#953 Add PHP extension check to Available Task
#952 Properly document how to load environment variables as properties
#951 S3Put throws “Source is not set” exception
#949 SymfonyConsoleTask improvements: checkreturn and output of command

 

Other goodies

The past few months Phing has seen a few interesting additions:

  • Composer packager support (we’re listed on Packagist as well)
  • phploc task
  • apply task
  • improved phpdoc(2) support
  • lots of documentation, bug and other fixes

GitHub

The move to GitHub in early 2012 has been very successful in speeding up Phing’s development. We can definitely still use your help though, so fork our project and submit a pull request. Thanks!