Author Archives: krcmic.com

Stay Up to Date - Continuous Integration Notification System

Stay Up to Date – Continuous Integration Notification System discussion

Leave a reply

Regardless if everything has gone smoothly or if there has occurred an error you want to be timely notified about it. Therefore it is imperative to use a notification system.

Also, you should not miss the previous part of the article series about web integration – Continuous Integration – Cure for Human Error in Deployment. It is a must-read!

Generally, you notifications can be sent to:

  • E-mail,
  • RSS feed,
  • HipChat, Slack,
  • Jabber – Google Talk, AOL Instant Messenger,
  • Google Cloud Messaging (GCM) service – Android devices
  • Skype,
  • MSN Messenger,
  • Custom service in XML or JSON.

Below are sample notification messages in a Hipchat room (advising about successful and unsuccessful deployments of demo projects):

The basic notification option is to send notifications by e-mail; every CI tool supports it. Email notifications allow getting very detailed information. On the other hand, if you get hundreds of e-mails and messages get easily snowed under, you may need another solution. So what is it? Critically important messages can be accompanied by a text message notification, which you should always notice.

Most of the tools allow for a number of communication channels. You can choose from the above, among which you’ll likely find your favourite chat tool. We use Hipchat for common notifications, which we accompanied by e-mails in combination with text messages for projects of critical importance.

Intelligent Notifications

Every project involves a build practically after every commit. Hundreds of messages take some effort to get organised. That’s why we are happy to be able to use “intelligent notifications”. Do you wonder how it works?

We do not use artificial intelligence, but relatively simple logic. A notification is sent if the result of the current build is different than of the previous one.

E.g. It can happen that your last build ended in a failure, even though the previous build was successful. We should find out the cause and remedy it, if we are able to. Of course, it may have happened that the build failed because some cloud service hasn’t been available. In that case we cannot directly influence the situation, but at least the rest of the team and the client will be notified.

Jenkins CI notification setup is minimalist and in most cases sufficient. Administrators can’t go wrong. The setup contains the outlined “intelligent notification”.

If you are after a wider setup options you can go for e.g. Atlassian Bamboo, which allows distinction of who is to get a common notification, and who is to be notified about an important event i.e. by “intelligent notification”.

Tool Selection

Personally, I can give you a thorough comparison of Jenkins CI, which I have been using for several years, along with Atlassian Bamboo as a modern tool, and a bold adept at further system integration.

Jenkins CI is free; including plugins, there is no hidden pricing.

Though, you’ll have to sacrifice some time and effort installing it and setting up configuration tools. Yet, Jenkins has a historical advantage, as it builds on its predecessor named Hudson. Thanks to this long history, it has extensive documentation, and a plenty of plugins for a variety of additional functions.

Job management is clear, everything is on one page; you simply can’t get lost. On the other hand, the user interface could be considered a bit of disadvantage, as it doesn’t look very “cool” at first glance, but rather “old school”. Well, I guess we shouldn’t expect a bunch of frills in the time of flat design.

You can see below what a Jenkins CI job dashboard looks like. On the left is administrator menu, history of builds, including the one in progress; in the middle are quick links, and an overview of recent successful / unsuccessful builds, including times.

For comparison, I also added Atlassian Bamboo. It’s nice, clickable and aimed at less technical users. In addition, if you use Bitbucket as SCM platform and Atlassian JIRA for requirements tracking, you can easily set up the integration and create a compact automated tool ecosystem, which work well together.

However, I see a significant flaw in the complicated deployment process breakdown, which is plan – stages – jobs – tasks. The system reduces the redundancy of settings, for example, when you define a global variable in the plan and will use it in all jobs; one task can be used in more jobs.

I miss an overall overview on one page, as it is Jenkins CI. “Plan” management in Bamboo has tabs with different sub-settings. Jobs are broken down to tasks; so you always work only certain fragments. That’s why I consider Bamboo confusing, and potentially contributing to inattentive mistakes. Like other tools from Atlassian Bamboo is licensed and the price may be, especially for small business, also a limiting factor. You can try the basic version for $ 10, but the real operation starts at $ 800 per year.

To give you a comparison, you can see here what Atlassian Bamboo history of builds looks like (preview including detailed documentation for each screen part).

As it is not quite possible for a common user to compare all the available alternatives.

You can find a systematic comparison of 40 continuous integration tools here Comparison of continuous integration software. You will find there various criteria, from supported build tools to notifications and integration options.

But perhaps none of us has time for that. Yet, we should aim to choose the right tool that will suit you, and which you will be able to operate properly. Jenkins CI proved to be that for me. I can’t imagine continuous integration without it anymore.

Continuous Integration Can’t be Done without the Right Tool

As we have seen, continuous integration involves a number of activities. There is plenty of space for mistakes, if they are to be done by a human user. We can’t also disregard the time it consumes and the knowledge required.

On the top that, if we use human resources we also need to keep in mind possible substitutability.

Continuous integration experts are expensive and not easy to find. It is therefore appropriate, if not necessary, to also engage in the process an automation tool. How we choose it may be influenced by various criteria. Personally, I suggest not taking into account only the cost of the tool itself, but also what it can do, and how well it works – notifications, the possibility of integration with other services, and user interface. Consider what is most important for you, and test the tools accordingly. If you do encounter problems, try to find someone to help you with that. The chances are there are many companies around the world dealing with a similar situation.

Finally, certainly don’t worry about starting with continuous integration and the process setup. It may be a bit of an investment at the beginning, but it will save you both time, nerves (and even reputation). Once you have properly set up the automated process within continuous integration, you can just “replicate it only” in the next projects.

Continuous Integration - Cure for Human Error in Deployment

Continuous Integration – Cure for Human Error in Deployment

Leave a reply

Continuous integration is a necessary development tool. Thanks to automation and standardization, it helps to effectively prevent errors when deploying applications into operation.

Also, you should not miss the previous part of the article series about web integration – Architecture and operating of web applications in a cloud environment (Part 2). It is a must-read!

What is Continuous Integration and What are its Stages?

Man is not a robot, and that’s the reason why there are a number of repeated steps in IT practice that are worth automating. In application development are constantly repeated certain steps. It is boring to do them over and over, people make mistakes, and machines do not. Continuous integration is a cure for such struggle. So what are the possible uses?

I consider continuous integration a necessary addition to software development, often adrenalin, and sometimes even fun. My favourite tool is Jenkins CI, which I have the most experience with. There is a whole range of similar tools used for continuous integration, build automation and deployment. Each has its advantages and disadvantages. Selection criteria can vary depending on the price or user-friendliness, and the possibility of integrating other tools and systems. But before we look at the specific tools, let’s first mention something about continuous integration as such.

Application development goes through various phases, from design and development to testing. Some may run in parallel, and especially in development, it happens that more specialists create more or less interdependent parts of the application.

The phases need to be combined into a functional unit for the customer, which is not quite that simple. Individual team members perform best work possible, which they have tested and would vouch for. But we do not live in an ideal world, and thus the contributions of various people may conflict sometimes, or someone may make a mistake in the project due to his/her carelessness or negligence.

Continuous integration helps prevent the client and the system users from having to deal with errors. It involves using various tools designed for this purpose as well as establishing error rate reducing processes.

Both large and small development teams need to re-build the project over and over.

This means that particular parts, which are developed by individual teams (or individuals), need to be assembled into a functional project or application, where individual modules smoothly communicate with each other. At the same time it is also necessary that responsible personnel are timely advised if an error occurs (the system correctly notifies authorized personnel).

After all, it would be a problem if, say, Internet banking users could not make payments, because some of the developers made a mistake in implementation and none of the responsible personnel knew about it. Continuous integration tools are set up to ensure this doesn’t happen and everything goes smoothly as clockwork.

For this reason continuous integration process should also include automated testing.

Auth. Note: Naturally, an application build and deployment can theoretically also be carried out manually. Though, we could dispute this would be “continuous” integration. Regardless of the possible error rate (and the time consumed) is not desirable to have processes depend on the presence of trained personnel. So it makes more sense to use an existing tool that is already in place.

Such tools are Jenkins CI, Atlassian Bamboo, Teamcity, Travis CI, and others. How to pick one is a matter of personal preference, all listed can deal with sub-tasks outlined in this article.

Stage One of Continuous Integration – Download the Latest Source Code

The first step in continuous integration is to download the latest code from a source code management (SCM) system.

Auth. Note: SCM system addresses the following:

  • Hosting – physical housing of code repositories of code version control system
  • Centralization of developed code – for submission and checks
  • Security setup (individual repositories access setup for particular groups)
  • Code verification
  • Other tools can be connected to SCM (request tracking, documentation,..).

That is why using SCM is an absolute necessity. I could elaborate on it in more detail, but I could go on as long as this article. For this reason, I will mention only the most important details.

I personally use Git version control system; you may be using Mercurial SCM or Apache Subversion. For more details check the article on version control systems.

Exception in versioning should be a specific configuration in runtime environment -credentials to log in into the production database, keys to connect to API, etc. For security reasons, these shouldn’t be available to every developer, neither should they float in the “public space”.

In addition to taking care of security issues, we also need to set a specific debugging level in certain instances independently of the common standard. This is done by employing a versioned local configuration.

Either way, a versioned code, in its “raw form” may not suffice for the application operation, because after downloading the source code, we must transform it into an executable form.

Stage Two of Continuous Integration – Building the Project

Theoretically, a small project can be a small application in itself, built with only a few classes.

In practice, we do not want to write a code entirely from scratch, and so it is smarter to use one of the frameworks as the basis (Zend, Nette, Spring MVC,..) where the basic requirements, such as safety and programme logic separation in comparison with the presentation, are already sorted out. And of course, each framework includes bits we had prepared to reuse instead of having to create the project from scratch.

Indeed, we can use plenty of frameworks and libraries. Yet, we need to address systematically their definition and subsequent downloading. So here come in play composer install / update (PHP), maven install (Java), and gradle build. They are here to help downloading related libraries and thus completing the project.

Stage Three – Build Front-End Code

In addition to program code, applications also have their own front-end (interfaces) for users. Modern front-end coders routinely prepare styles for LESS / SASS preprocessors. As in the previous case, when working with a program code we need to compile LESS / SASS documents to CSS (into a publishable and usable form). Tools such as Grunt (The JavaScript Task Runner), and/or (A package manager for the web) are here to help us with it.

Stage Four – Editing Front-End Code

As I mentioned above, we don’t usually get the code in the desired form straight away. Also, our secondary goal is to save data and CPU power (on the server and on the client’s side), and therefore we try to convert the front-end code (CSS, JavaScript, images) into a compressed form.

The same rules apply both to front-end codes and programme codes. That means we should do syntax checking and potential error checking.

Such operations as minification, compilation and testing are done by tools like Grunt (The JavaScript Task Runner) or Bower (A package manager for the web).

The above tools are here to help us create a ready-made application, which is possible to deploy. Yet, we should test it, prior to publishing it for real users. Even the most conscientious developer can make a mistake. The truth is that when multiple developers work together it can happen that separate pieces produced by each of them are functional on their own, but the joint piece may behave weird or show errors. And that’s the reason why we should pay duly care to checking the code and testing it.

Monitoring and Testing Continuous Integration

The tests can be divided into the following groups depending on the moment of initiation:

  • Pre-commit (prior to handover)
  • Pre-deploy (prior to deployment)
  • Post-deploy (post deployment into the target environment).

The tests can be initiated prior to the integrated change depending on the test type and integration strategy – pre-commit (see e.g. npm pre-commit package, SonarQube pre-commit analysis) or post integration – i.e. post-commit.

Early stage tests are not related to continuous integration tool, as code testing is done already on the developer side. SonarQube and pre-commit test are done directly in developers’ IDE using Eclipse and InteliJ plugins. Pre-commit tests relate to the source code. This means we don’t test the running application. Instead, we do syntax checking; verify compliance with coding conventions, check whether there are no duplications in the proposed classes, and check the overall code quality.

Later on, in pre-deploy are done unit tests, we check functionality and correct implementation of particular system units in the form of classes and functions. Here we touch upon the issue of test-driven development, where tests are written prior to the code itself. They reflect all functionality requirements, and we cannot go further unless they are in compliance. If the tests go right we can finish the deployment, and then move on to post-deploy tests.

Post-deploy tests are initiated following successful deployment:

  • Automated user tests – are created by Selenium IDE and run by CI tool in Selenium server environment.
  • Integration tests verify functionality of the system.

Nature of some projects requires manual testing.

These can be, for instance, electronic banking and applications of similar security, which can be logged into only by human users. Some operations may involve two-factor authentication, which we want to test by real human testers. In addition, there are many components, such as forms and calculators that can be tested automatically. Basic tests checking if an application is working after an update, and behaves as expected, can be machine done.

Automated testing brings in the long run:

  • Repetitiveness – tests are carried out following the same scenario, in more or less the same conditions,
  • Saved time – compared to demanding user testing.

Despite all the efforts of the development team, building or testing can end in a failure. Also, someone always needs to be notified about the negative result (advising about positive outcomes is not always necessary). This is resolved by a notification system, which we will discuss in the second part of the article.

You can now follow to the next part – article called: Stay Up to Date – Continuous Integration Notification System discussion.

Architecture and operating of web applications in a cloud environment (Part 2)

Architecture and operating of web applications in a cloud environment (Part 2)

Leave a reply

In the first part of this article we got acquainted with the conventional models of services provided in a cloud, the principles of development of PHP applications designed for deployment in this environment, and finally principles of economics of cloud operation.

In the second part of the article we will introduce practical experience with deployment of PHP system, which was not originally designed for cloud environment.

Also, you should not miss the previous part of the article series about web integration – Architecture and operating of web applications in the cloud environment (Part 1). It is a must-read!

CMS system

Our CMS system uses a local file system or a relational database as a repository for storing CRM data, it accelerates access to CMS objects by multi-level caching (on the 1st level within one request in memory, on the 2nd level in APC cache and on the 3rd level by reserved section of the file system), for full-text indexing and subsequent searching it uses the Apache Solr service.

Using a local file system in combination with fulltext significantly affects deployment capabilities to the cloud. Interactions of CMS system with CMS model is very intense, in practice, there are dozens to hundreds of small accesses (mtime, fread operations) to objects of CMS model within a single web request.

Thus latency of approach combined with efficient caching fundamentally affects the speed of putting sites together. Within preparation of the system for deployment in Microsoft Azure cloud several implementations of storage were gradually implemented and tested, starting with existing support for the use of relational database and ending with implementation of storage, combining services Azure Table and Blob Storage.

In both cases a relatively high latency in storage access provided under Azure as SaaS service adversely manifested itself (in the case of a relational database in the order of tens of milliseconds, in the case of Table / Blob storage to hundreds).

The solution was to deem the data already loaded into the cache for a certain period of time (in the order of seconds or minutes) to be valid and not to verify their changes in persistent CMS repository. This, however, brings a delay in updating the content in case of deploying in a cluster.

PaaS or IaaS (Infrastructure as a Service)?

A separate chapter was performance tuning in PaaS in the environment of Azure Websites. Unlike the Unix environment in this application container based upon Windows Server technology the WinCache module instead of APC or Zend OpCache is available for the purposes of in-memory caching.

Although in principle this one works the same as the above mentioned modules, however in practice its use brought noticeably worse results within the cache layer for caching PHP bytecode as well as data. For these reasons we preferred the use of laaS for execution of PHP code of web solution based on our own CMS – virtual machines with Linux based operating system, Ubuntu Server, on which it is possible to use proven APC caching or alternatively Zend OpCache extensions.

By switching from PaaS to laaS model we, however, lost a crucial advantage of Azure Websites, that is  auto-scaling and automatic distribution of the application into cluster administered by the platform, therefore we had to solve these function in other ways.

Docker or own PaaS solution

The target solution combines laaS with Docker technology, which allows for preparation of a complete installation of CMS system in the form of an image, which is deployed and run on virtual machines with pre-installed Docker Engine running environment.

In the case of an on-premise solution it is possible to use arbitrary servers with some distribution of Linux. Own management of distribution of the application into the cluster is then carried out by using the Docker Swarm technology, which can be interconnected with a tool of continuous integration responsible for creating and deploying web projects (in our cases Jenkins CI).

Server Docker image contains web server NGINX in the role of reverse cache and proxy server, which is connected through FastCGI interface to Zend PHP Engine with all usually required extensions, optionally then also connected to monolithic PHP engine HHVM, which officially supports CMS from version 2.2.16 (selection of engine is carried out by communication to the relevant port assigned to NGINX).

There is also installation of Apache Solr fulltext engine prepared in the image.This image is then used as a base for creating a complex image of the web solution containing all the application code, which will upload a tool of continuous integration into it, then it may subsequently distribute it into running environment through command line.

Distributed CMS storage

As CMS repository, intended for deployment into dynamically scalable running environment (Azure Cloud Service),  our CMS uses a specific implementation using local file system, which records all the changes in parallel to external (in case of Azure SaaS) relational database server, where at the same time the CMS system database located on this server is extended with application change log.

All changes of CMS objects are recorded into this log within change transaction, then detected by other instances of the same web solution, then retrieved from the database and replicated to the local file storage and cache, while they are indexed by the full-text engine. It is possible to use a temporary disk for cache folder in Azure Virtual Machine, which is connected to the above mentioned directory inside the container of Docker server.

Although this solution is not saving in terms of data storage (all the data in CMS storage are copied to the local file system and full-text of each running container of the given web solution), it brings the speed corresponding with local deployment without cluster.

In the case of deployment to on-premise/statically scaled environment it is possible to connect a selected directory to data directory of the web solution from Docker server and thus to have all data which are stored by our CMS in local file system, or more precisely a complete web folder, outside the actual container of our CMS server.

In case of the cluster the given directory then may be synchronized to other nodes in this cluster using the service GlusterFS, by which replication of change of data across the application cluster is ensured. Such a solution is then generally applicable to arbitrary web applications based on the PHP technology.

Summary

According to the level of specificity of the application environment it is possible to deploy PHP applications in Azure either to the existing PaaS model of Azure Websites, or it is possible to use IaaS (Azure Virtual Machines), preferably then in the combination with Docker technology, which allows for easy packaging and distributing the application with a complete application environment, configured according to specific needs, based on Linux to IaaS cluster using Azure Virtual Machines and Azure Cloud Service technologies. In any case, it is possible to at least use the relational database in the cloud environment as SaaS service.

Useful links

You can now follow to the next part – article called: Continuous Integration – Cure for Human Error in Deployment.

Architecture and operating of web applications in the cloud environment (Part 1)

Architecture and operating of web applications in the cloud environment (Part 1)

Leave a reply

Cloud based platforms catch on more and more as a running environment for web applications. What advantages, compared to the traditional “on-premise” infrastructure or classic web-hosting, do they offer? And what new possibilities (from the point of view of the web integrator) does cloud bring to the architecture of web systems using LAMP technology? These, and many other questions, will answer the first part of the new series.

Also, you should not miss the previous part of the article series about web integration – Putting Continuous Integration (CI) into Practice – Part 2. It is a must-read!

Virtualisation as a foundation of the cloud

In the last decade we have been witnessing gradual but more and more significant advancement of computer virtualisation in server, and later also in desktop environment. First, not very powerful server hypervisors1 based on full virtualisation of the hardware incl. CPU, were followed by revolution based on implementation of the virtualisation support directly on the CPU level.

This significantly improved performance of virtualised systems, whose running costs went from dozens to single digit percentage values of the hosting processor power. Together with improvement of the concept of the peripherals virtualisation in the form of drivers closely cooperating with the host system allowed massive succession of this technology in a broad scale in both on-premise2 environment where companies have their own server infrastructure available, and between infrastructure providers, who offer it on a rental base.

This, originally on-demand3 supplied virtual systems, gradually developed into complex cloud platforms with wide portfolio of services for running the applications. One of the most renowned in this field are Amazon Web Services (AWS) and Microsoft Azure.

1 Hypervisor is a specialised software, or combination of hardware and software, which allows creating and operating of virtualised computers.

2 On-premise is the way of running the solution, where the organisation has it under full control – typically, it’s run on their own infrastructure.

3 On-demand describes the solution operated on the infrastructure of the provider, to which the organisation has access via the Internet and usually does not have full control over it.

Models of operation in the cloud environment

Basic models of service, provided through the cloud environment, are following: IaaS – Infrastructure as a Service features basic directly from virtualisation derived service enabling to create and manage infrastructure consisting of:

  • virtualised servers,
  • private networks (VLAN – Virtual Local Area Network),
  • specialised network components, typically:

— „load balancers“ – application switches, securing high accessibility of the given application service by transmuting requirements into cluster consisting of two or more virtual servers,

— network firewalls (optionally explicitly allow or deny access from public network to selected network services provided by the virtualised servers in the cluster),

— and application firewalls with the possibility of an active detection/prevention of security attacks in given application protocol („Intruder Detection/Prevention Systems“).

In case of IaaS remains the responsibility for the maintenance of the operation environment of the system of virtual servers, creating and maintenance of the application environment within it on the service user.

Configuration of the virtual servers, networks and network services is done via web-based administration console, or there might be API available to allow automation of such administration operations on another system’s level. Within the framework, the IaaS provider usually offers installation of pre-set images of number of different types/versions of operation systems, optimised for running in IaaS cloud (containing drivers for cooperation with hypervisor, used by the provider for server virtualisation – i.e. VMware ESX/ESXi, Xen/Citrix XenSource or Microsoft Hyper-V technologies).

In case of smaller cloud service providers is the IaaS offer often final. Compared to mentioned global players it can, in case of the given cloud is satisfactory for suggested solution, compete in more attractive price and individual local support.

IaaS services are charged on pay-as-you-go basis, where the price is given for computed unit per unit of time (typically combination of CPU and RAM, defined in so called type of instance of the virtual server over the period of one hour) or data unit (transferred or uploaded gigabyte for the network transfer, respectively data warehouse).

Base price can also be influenced by the installed operation system (i.e. in case of Windows Server includes the license fee payment), more powerful CPU architecture (as a standard, cloud service providers’ data centres use processors with the most favourable computing power/power input rate, but those are not the most powerful models) or more powerful repository type (SSDs, sometimes installed locally instead of a shared disk array).

PaaS („Platform as a Service“) offers an environment for operating web-based applications built on a particular application platform (i.e. PHP, .NET, Node.js or Ruby on Rails). In this model, the service provider takes full responsibility for needed infrastructure, consisting of IaaS instruments, operating system and application runtime environment software necessary for running of the application.

Service user obtains in the first place one or more interfaces for uploading the application into the environment (typically FTPS/SFTP, Git SCM, eventually API and/or CLI tool, including support for deployment management).

PasS gives the user reduced options of configuration of the application environment (i.e. language or framework options, insertion of expansion modules and adjustment options of the global configuration of the application environment, which can usually be adjusted through configuration of the application itself, when uploaded into PaaS).

PaaS as a service can also include manual or automated (threshold-parameter controlled) scaling, where, in combination with the load balancer, virtual engines securing processing of the application code depending on the current load are created and disposed of.

Same as in case of IaaS, the price unfolds from actually consumed sources including use of IaaS resources (type and number of instances of the virtual engines, warehouses and data transfers) and other PaaS specific services (i.e. automated scaling support and also preparedness for scaling in greater number of instances). Amongst PaaS platform belong Azure Websites, Google AppEngine, AWS Elastic Beanstalk or cloud platform Heroku.

SaaS – Software as a Service is a cloud layer of the highest level of abstraction and it means the supply of a working server software with clearly defined functions and interface.

In SaaS model are typically supplied data warehouses (SQL/relation and noSQL database systems), directory services (LDAP, Azure Active Directory), CRM or e-commerce systems. Falling into this model are also currently very popular cloud services Google Apps and Office 365 or, for example, on-demand versions of Atlassian company products (JIRA, Confluence). In this category also fall systems for content distribution (CDN – Content Delivery Network).

In SaaS model, the user primarily works with the web interface of the service, which offers the model of secure access to individual functions and data based on the user roles and groups with the possibility of management of the access and optionally also access of other software or cloud services via API, eventually by using of special CLI tools for maintenance and integration.

SaaS platform offers very restricted (or none at all) options of insertion of your own application code. In case of databases, this may be “stored” procedures or functions executed by the database engine, in other cases uploading of the code in the form of package (extension), which the given software allows to load up within the means of its modular architecture. Typically only in case the instance of SaaS is operationally completely separated from instances of other customers, otherwise only integration on different servers via SSI (Server-Side Include – entered on the server side) model or exclusively via code running on the web browser of the user (UI components coded in Javascript that are using accessible server API) is possible.

Architecture and operating of web applications in the cloud environment (Part 1)

 SaaS services are usually charged depending on the on the number of user accounts, size of the data storage or number of parallel client connections or their total number per period of time, or combination of these parameters. Parameters can be usually changed at any time accordingly to the changing needs without having to interrupt accessibility of the service.

In all the models, the cloud platform offers possibilities that are, in traditional on-premise model, either completely missing, or their implementation requires great efforts:

  • high level of security of the access to the network services and active protection against various forms of attack (network firewall, intruder detection/prevention system),
  • high level of the data security (redundant multilayer storage, optionally also geographical),
  • option of pooling cloud instruments in IaaS/PaaS model into virtual private networks and their connection with the network environment of the organisation (site2site VPN), creating of hybrid cloud environment and support while connecting the users into the cloud environment via site2client VPN,
  • load balancer (incl. support of automated grading of cluster sources in accordance with given load parameters),
  • and SSL offloading.

PHP application in the cloud environment

PHP based web applications and portals (or portals still operated in the traditional LAMP model) can be used in the cloud environment utilising all above mentioned models. The easiest is, if considering operational requirements, using PaaS, which offers support for operating of PHP applications, in combination with database in SaaS model. In this case the whole environment is created and further maintained via web administration, deployment of changes is executed using familiar tools (application code via Git SCM push mechanism, or by upload via SSH/SFTP, database changes manually or by script via database console).

Problems can arise, if the PHP application has special requirements on the application environment (i.e. uses specific PHP extensions, which are not available and it’s not possible to upload them into PaaS) or its framework isn’t suitable for deployment in cloud with automated scaling and “failover” mechanisms (typically uses local file system as a persistent storage). In these cases it is necessary to base either part, or the whole solution of operation of the application, base on the IaaS model (thus reduce the utilisation of the cloud platform on the virtualised server infrastructure), or to adjust the system framework of the web solution so that it’s compatible with the PaaS cloud.

While doing this, it’s important to take in an account , that the IaaS model, which is the closest to the traditional on-premise infrastructure, also suffers from infirmities of this approach, in particular it requires resources for the management of the operation system and application running environment, at the same time in case of requirements on high granted accessibility requires conceiving of cluster, by which it puts strain on the application framework equivalent to scaling mechanism, included in PaaS. It follows several recommendations for architecture of PHP applications, which are intended to be deployed in the cloud:

  • Do not mix code and data – do not use file system for saving of persistent data, abstract all the operations with the data storage beyond the general boundary (this will allow to implement one of the available SaaS services) or work directly with the database abstraction of PDO type.
  • Use actively all caching mechanisms that the chosen application platform/framework offers in combination with the chosen cloud environment (in the computing instances there will never be an excess of the computing power or operation memory; for caching you can use in-memory key-value of the database of the Redis if the platform offers this in the SaaS model).
  • Expect serving static source files (stylesheets, pics, videos) from other absolute URLs – within the platform you will be able to utilise accessible solutions of CDN (Content Delivery Network).
  • In case you would like to use relation databases, do not use SQL commands dependent on a particular database engine (with exception of cases, where you know up front, what type of database will be used in cloud); design the queries programmatically by the application code, using the library that will allow or use ORM tool.
  • Do not use PHP extensions that are not available in the original setting in the PaaS model of the given cloud and it’s not possible to upload them with confidence additionally.
  • Do not use libraries and frameworks that are written in conflict with the above mentioned rules.
  • If it’s possible, avoid using server services that cannot be used in the selected cloud platform in the form of SaaS (i.e. exotic key-value databases or full-text indexers).

Costs calculation

Significant change introduced by the cloud platforms in all mentioned models is the pay-as-you-go service accounting system, i.e. interim payments in accordance with used up resources. Planning of the operating costs doesn’t necessarily require execution of the precise sizing for the expected peak attendance in the given period (usually number of years) and consequent purchase or lease of corresponding computing resources (servers and other network elements) not taking in account whether they will be fully utilised (not to mention the case when traffic exceeds expectations, requiring unplanned new infrastructure…).

In case of the cloud, the sizing is only indicative, it determines the estimated operating costs in the given time-frame (typically one month) depending on number of users of the web solution, that are supposed to be served in given response time. We can therefore ultimately calculate the cost of servicing a single user and include it in the entire business transaction, which uses the web-based solution.

This, together with guaranteed accessibility of cloud services (standard in models SaaS and PaaS, or in case of IaaS under condition of generating the cluster) and high level of security, are the main advantages of use of cloud as opposed to the traditional on-premise infrastructure.

In the second part of this article, you will learn about practical experience from deployment and scaling of CMS based on PHP software architecture (which has many specific characteristics)  into cloud environment Microsoft Azure.

You can now follow to the next part – article called: Architecture and operating of web applications in a cloud environment (Part 2).

Putting Continuous Integration (CI) into Practice 2/2

Putting Continuous Integration (CI) into Practice – Part 2

1 Reply

There are a wide variety of task recording, documentation creation, team communication and deployment systems out there. In the second instalment of this article, we shall see how these tools can be turned into an efficient system. By connecting these otherwise independently functioning tools together, you can reap substantial benefits.

Also, you should not miss the previous part of the article series about web integration – Putting Continuous Integration (CI) into Practice – Part 1. It is a must-read!

A project management and issue tracking system

Whether you have a completely new project or a post-sale application, an issue logging and tracking system is a must. Before your developers start cutting code, you need to define the tasks to which you will assign particular programmers, that is, tasks the progress and cost of which can be monitored against targets.

At Krcmic.com and Onlineandweb.com, we used Mantis Bug Tracker in the past, which, however, does not meet the requirements of today’s software project management. It is a good, albeit a little cumbersome issue logging system, but it lacks certain core features that are absolutely necessary for us today.

Modern application development is unthinkable without a flexible bug management system that actively supports the implementation of agile development management methods. We currently use the JIRA issue tracking system which has the following advantages over Mantis:

  • customisable project workflows;
  • combines the features of an issue tracker and time tracker;
  • supports team work (a user access and role management system)
  • can be easily configured to interact with other tools (Bitbucket.org, HipChat IM, Confluence);
  • is user-friendly.

Like other Atlassian products, JIRA is not cheap, but we currently use it as a replacement for two systems (issue tracking and time reporting). JIRA is easier and faster to work with and as such, directly saves us time. Moreover, we have custom-developed one of the systems, so we will be able to save considerable development, servicing and bugfixing costs in the future.

We recommend connecting the JIRA system to a central source code repository (see below), making it possible to drill down for details on the work that has been done and its authors. In addition to the verbal comments left by team members, we have access to the related technical details of the implementations.

Workflow

The choice and use of a suitable workflow are very important considerations. JIRA offers some out-of-the-box workflows that are available as part of the installation, but these do not exactly fit the processes within software development.

The workflows in JIRA are fully customisable, which can be both a blessing and a curse. This freedom may be a certain advantage, but you are faced with the necessity of capturing recurrent situations and processes common to most projects. Creating workflows that are usable for more projects necessarily involves some compromises. The deeper you dig into the configuration options, the more likely you are to accommodate specific requirements and create special-purpose workflows. This has more disadvantages than it has advantages. A smaller number of workflows aid in orientation; the statuses (To do, Done, Accepted…) and other properties ought to be unified across projects, which helps avoid misunderstandings in communication and in managing the processes themselves. Both managers and developers have an easier time of navigating a limited number of universal workflows, rather than having to choose from dozens of specialised workflows. For this reason, we have adopted the use of workflow schema definitions, which can be carefully prepared for the company and which users can choose from when setting up new projects.

User-friendliness may not be the primary consideration, nevertheless, a good GUI saves time and motivates employees to use the system with respect, if not with pleasure. On-site editing, search, autocomplete and keyboard shortcuts are just fantastic features!

Time reporting

We used to have a time reporting system that was set up independently of the issue tracking system. It was a custom-built system that we created ourselves. However, we could not continue developing it forever and to upgrade it to meet our evolving needs would have meant virtually creating a new application.

What are the benefits of merging the two systems into one?

  • directly related information on bug logging and the progress of bug resolution is stored in a single system;
  • less time is spent on logging work, reporting and reassignment of tasks within teams;
  • information on persons, methods and the complexity of solutions is consolidated at one location;
  • the evaluation of the status and success rate of projects is much simpler and more accurate thanks to the aforementioned features.

For more sophisticated development work, we use the JIRA Tempo Timesheets Plugin integration. Already in the basic version, JIRA supports time reporting (worklog). The built-in feature will be sufficient for the needs of a small team, while larger teams will probably consider purchasing an extension that allows sorting by teams, projects, etc. Pricing is based on team size, as is usual for the Atlassian ecosystem.

Versioning system

The tasks have been defined, assigned to individual team members and the project picks up steam. The progress of the work must, however, be tracked and changes made by several people must be incorporated into a whole. To this end, we have for years been using the Git versioning system. If you wanted to purchase a version control system in the past, you didn’t have many choices. Certainly, many of you are familiar with Subversion. Today, we cannot but recommend distributed version control systems (DVCS). The most popular such systems worldwide are Git and Mercurial. The choice is a matter of personal preference. If you don’t have any experience with either, in the next article about versioning systems we are looking at the pros and cons of some of these systems. The article discusses in detail the reasons why DVCS is better than the client-server model.

Your developers will deal with the versioning system on a daily basis, so if you have not decided on a particular one yet, you can organise a vote among your developers.

When it comes to price, versioning systems as such are free, unlike data storage.

A central source code repository

A more complicated topic which builds on the choosing of a versioning system is a central source code repository. A web integrator, being a supplier of client solutions, is working on a much greater number of repositories than there are developers on the team. This disproportion significantly affects the costs that the various providers have.

Repositories can be hosted on an internal server or outsourced to a hosting provider. Should you opt to run your own server, you have the choice of using an open-source solution, which requires specific know-how and can be time-consuming to set up, or you can use an off-the-shelf solution with paid support. Last but not least, there is the option of renting server space, in other words, using a cloud service, in which case you pay by the month.

On-premise solution – “what’s in the house?”

When we started with CI, we had our own git server. When it comes to applications, one option is to use free, open-source technologies, which, however, need to be configured. It goes without saying that time is money. You must also factor in the overhead costs associated with the management of IT resources.

An alternative to open-source solutions are paid solutions, which require a relatively large lump-sum investment. Their advantage is that they work out of the box. We can recommend Atlassian Stash, which offers easy and effective management tools and numerous options for managing user accounts and groups as well as Active Directory integration. Alternatives include GitLab, which comes in both community and enterprise versions, and GitHub Enteprise. Apart from price, you should be looking at user comfort, availability of documentation and the liveliness of the user community. Do not go for tools that you have reservations about: choose tools that are intuitive to use.

Rental – on-demand cloud elasticity

A software cloud can expand and contract just like the real cloud. Typical examples of such elastic clouds include Bitbucket, GitHub and GitLab. The advantages of these are:

  • non-stop availability from anywhere regardless of internal infrastructure;
  • it takes away the worries and hassles of running your own hardware and infrastructure;
  • backup, disk space and security are taken care of by a team of experts;
  • automatic system and add-on updates;
  • you can easily change your provider.

When it comes to price, GitHub is more suited to larger teams that work on a smaller number of projects (products). Such teams appreciate its pricing model based on the number of private repositories. This model would be costly for us.

Bitbucket has a different pricing model based on the number of team users; the number of repositories is not taken into account. For example, to develop an online shop on the Magento e-commerce platform, we would need tens of repositories for the various extensions. We have therefore decided to switch from GitHub to Bitbucket.

A cloud service is more flexible and more comfortable to operate and we can quite easily switch to a different provider of a similar service if such a need arises. However, a cloud solution may turn out to be more costly in the long run.

Deployment of modifications in various environments

The developer has committed his work, i.e., a piece of code versioned in the central repository. Now comes the time to deploy the code in a runtime environment. Application deployment should not be handled manually by the developer; rather, this part of the process should be entrusted to experts on continuous delivery and handled as an automatic deployment task. When a developer commits his code, we call the execution of deployment by a tool that inserts the code in the relevant environment. For more information on deployment, please see the article The advantages of continuous integration and automated deployment.

Project documentation

All projects, however big or small, must be documented. After creating a solution, developers often hand their project over to another team for maintenance. Perhaps we can all agree that without documentation to refer to, work with code that we have not written ourselves can be a headache.

Documentation should always be supplied with any piece of code, although it may not always be sufficient when new developers need to be taken on board quickly. It is advisable that developers create and maintain compact documentation, describing the architecture and implementation of specific functional components of particular solutions, ideally using an on-line system that permits quick viewing and editing of stakeholder groups. We had trialled MediaWiki before switching to Confluence.

One of the reasons for the switch was the option that the latter offered of integrating with other Atlassian systems, not to mention the nice user interface and the flexible security model allowing detailed management of permissions. Unlike Confluence, MediaWiki requires no licence fees, however, the basic version offers only a very simple, open UI with next to no formatting options. As it is a Wiki platform, there is no granular access management. Tens of hours in configuration, extension set-up, etc., are needed and the system may, in the end, come at a steep price.

Which features are important and what makes our life easier?

  • import from MS Word documents;
  • export to MS Word/PDF formats;
  • formatting;
  • inserting links and images;
  • code documentation;
  • support for macros.

Macros are useful tools for the insertion of specific content (e.g. software code) or dynamic content such as child page summaries, etc. Frequently employed and recommendable tools include JIRA integration and filters for viewing task lists.

The filters are dynamic and impervious to the inaccuracy of authors. In the last step, information on activities in Confluence is sent in the form of notifications to topic rooms in HipChat (for more information on the topic of chat see the next chapter).

Real-time communication – chat

You send a group e-mail and the next day you find out that you forgot to include the most important addressee. A client who is unsure whom to address sends an e-mail request to ten people. The message reaches the right addressee, but the other people have received a message that is irrelevant to them.

Not only can this be a bother to some people, but it may be really hard for the employees to stay on top of things if their inboxes are inundated by hundreds of e-mails every day.

Now imagine a chat room to which you can invite a group of people and where everyone can talk without worrying that he or she might have left someone out of the loop. If the ongoing discussion is not relevant to a particular employee, he or she can temporarily log off. If needed, he or she can be invited to re-join later.

As a result, our inboxes are not cluttered with too many internal messages and e-mail, in the end, becomes a tool for external rather than internal communication. A part of our communication with clients also takes place via chat. Chat is useful especially in situations where we need to clarify some details, ideally in real time and involving more people at one time.

We draw information from communication among actual users as well as from other systems we have in place, i.e., JIRA, Bitbucket, Jenkins and Confluence. Unlike the aggregation of information in JIRA, where information is linked to particular tasks, we use HipChat to aggregate information in chat rooms based on teams and projects.

To give you the full picture, here is the cost information: the cost can be calculated quite simply: an organisation running a cloud with unlimited history and message search pays 2 dollars a month per each user.

Integration and standardisation

All Atlassian systems are mutually integrable. This may not be anything special since other systems, including open-source tools, offer this functionality to some extent, however, it is a lot easier to integrate products from one family. It is usually a matter of minutes or hours, at the worst.

Integration is a foundation for an effective mode of work. If you don’t have such integration at your company, we recommend that you take steps in this direction, although the price you will have to pay now and in the future may be a deterrent.

You will spare yourself a headache, save money and improve quality. The transmission and aggregation of information saves a lot of e-mail and personal communication as well as time spent in team meetings. Moreover, you will have traceable history in addition to traditional documentation.

If you have already set about implementing new systems or are preparing to do so, be sure to enlist the help of someone with sufficient experience, e.g. a similar company in the same vertical or an expert consultant. You will spare yourself the headache of learning and implementing a new system and you will also save money.

Last but not least, you will lead by example: the employees of the organisation will be more disposed to accept changes when they see someone who has been through the teething troubles and now reaps the benefits.

You can now follow to the next part – article called: Architecture and operating of web applications in the cloud environment (Part 1).

Putting Continuous Integration (CI) into Practice - part 1

Putting Continuous Integration (CI) into Practice – Part 1

Leave a reply

Practical advantages and pitfalls of introducing continuous integration (CI) into (not only) a web integrator environment, what is easy and what is harder, what requires attention, whether we speak about tools or people.

Also, you should not miss the previous part of the article series about web integration – Front-End Task Automation – Part 2. It is a must-read!

In the current era of strong competition and high pressure on maintaining output quality, one of the essential parts of effective development and web applications is continuous integration (CI). Implementing of a correct and compact CI process is not a matter of one day and may contain obstacles. The last but not least aspect of the process is the actual speed of change distribution into various environments, where continuous integration greatly helps reducing errors while reaching a sufficient speed.

Required tools

The subject of this article is not a detailed analysis from a technical perspective. To give you a better idea, I recommend the previous articles, Advantages of continuous integration and automated deployment from the perspective of web integration and Version control system and web integration projects.

For a general idea, let us at least provide a basic list of tools and information systems that need to be integrated into the production process of a company:

Pros and cons

Implementing new systems means changes in both the infrastructure and the process level and affects operation principles of the whole company. Individual workers’ reactions to those changes within an organization will vary. Generally, significant changes do not cause waves of enthusiasm throughout the whole organization. A part of the staff will believe the existing system is sufficient and will prefer using the established procedures.

Another part will be indecisive and will wait until those changes are proven and have acceptable consequences. Those people will neither vote against changes nor actively support them. Finally, the third part of the workforce, dissatisfied with the current state, will more or less support the changes. The last-mentioned group may also contain future members of a team responsible for the correct setting for the new systems and maintenance.

Apart from the technical support itself, the last group also has a significant psychological impact on their co-workers, which they can naturally convince of the benefits of changes in work procedures. Enthusiastic supporters will gladly learn the new system and will naturally pass the enthusiasm on their co-workers.

Therefore, the changes will not seem as “dictated from above” and will be regarded as actually beneficial.

Do not get me wrong, however. The group of critics is not trying to prevent progress, it only requires us to clearly describe what benefits the change would bring them at the price of their temporary discomfort.

The most common objections against changes that consist of implementing an infrastructure and a continuous integration process:

  1. Learning to use new tools requires extra time.
  2. New tools are lacking functions of the old tools.
  3. New changes cause confusion in the established procedures and lead to errors.
  4. It comes with additional standards and limitations.
  5. Automation reduces the developer’s control over the deployment process and the control of a code deployed into a runtime environment.

Counter-arguments:

  1. Workers increase their qualification and broaden their knowledge.
  2. Modern systems supporting continuous integration feature most of the previously used systems’ functions, and also:
    • Include additions/extensions that can partially or entirely replace functions of the old system.
    • Allow integration into other systems within the continuous integration process.
    • Allow integration into bug & issue tracking systems.
    • Feature new functions that can be used to improve work effectiveness.
  3. Short-term decrease of effectiveness is compensated by a long-term increased level of stability and a better-quality (more easily estimable) output towards the clients.
  4. In a clearly specified system, standards also serve the workers as manuals that always define a repeatable process; bigger changes allow more substantial reworks of the established security models, improving them qualitatively.
  5. Replacing manual work with an automated tool reduces the probability of human error and limits responsibility to a smaller group of people. This also means the developers, after submitting their work, do not have to worry about if and where their work has been deployed.

Where to look for good advice and a helping hand?

Implementing new systems and their setting for the company’s specifics requires a considerable time. Some attempts may end in failure, or a certain direction may prove to be a dead end. In any case, it will provide us with valuable experience, even though it costs us valuable resources. Implementing a compact system may take hundreds of hours of work. Good information can save a significant portion of the worker’s time required to acquire new skills and, last but not least, spare our nervous system.

Available source of information:

  • Internet – on the internet, you can find documentation, manuals, personal experience of individuals and discussion on a given topic.
  • Customer support and community – you can consult the customer support (in case of commercial products) or the developer community (in case of open source).
  • Seminars, training – as in any other field, you can attend to seminars or training.
  • Consultant – a subject with experience.

The aforementioned options are more or less sorted according to their cost and also reflect the quality of received support. Both open source and commercial systems usually have out-of-the-box functionality preset to the most common user requirements. In most cases, you can thus start working almost immediately. A lot of information is free of charge as long as you are able to find it. Time, however, is not free, and hours of experimenting on your own can be avoided by an individual consultation.

To make the transition as painless as possible, it might be worth it to involve an external consultant specialized in continuous integration or to cooperate with a company in your field that has already implemented such system and uses it successfully. A subject that experienced a transition from an uncontrolled system to continuous integration can provide you with:

  • Personal experience that can point you in the right direction.
  • Warning against a number of errors resulting from experimenting with new tools and processes.
  • Help with a basic setting, which you would otherwise have to discover on your own.
  • Recommendations on how to adjust the basic setting to the needs of your organization.

Besides that, it will act as a mentor and role model for your employees, a presence of such person usually makes the team more inclined towards changes.

Implement changes gradually

It is not wise technically or organizationally to turn all systems upside down. For instance, the maintenance of ongoing projects requires the impact to be as small as possible. In this case, we recommend to fine-tune the process on a chosen project and then apply the process on other projects.

Our suitable candidate should not be too small and should generate sufficient amount of change requirements, on which we can test the process and learn it. Besides, this way the participants will be shocked to a lesser degree than they would be otherwise. Despite all efforts for a smooth progress and maximum efficiency, not everything will be perfect since not all risks can be prevented. Unexpected events throughout the development process require a certain level of improvisation and inspiration.

But if you anticipate these risks – especially if you involve an expert consultant – you will overcome the period of implementing changes smoothly and to the satisfaction of developers, managers and clients. In the second part of this article, we will focus on practical experience with implementing of some of the aforementioned tools and information systems in the building process of continuous integration in a web integrator environment.

You can now follow to the next part – article called: Putting Continuous Integration (CI) into Practice – Part 2.

Front-End Task Automation - Part 2

Front-End Task Automation – Part 2

Leave a reply

Work on a web integration project includes a lot of repetitive and not-so-interesting tasks. The following lines describe several very useful tools that can significantly facilitate and accelerate our work. These tools include Bower, Grunt, Gulp and Yeoman.

In the previous part, we explained what Bower, Grunt, Gulp and Yeoman are based on and now we can move on to introducing them further.

Bower

Bower is a package manager from Twitter and, as opposed to npm, which deals with the “server side”, Bower deals with client components, such as jQuery, Angular, Bootstrap, etc. Modules can be browsed at bower.io/search/.

Bower is available as an npm module and, since we will use it in all projects, we install it globally in the following way:

npm install bower -g

The commands are similar to npm and the installation of Bootstrap, for example, will look like this:

bower install bootstrap

Here you can see that the bootstrap module is also dependent on jQuery, and both were downloaded in the bower_components folder. From this folder, we can start using jQuery or Bootstrap in the project.

During the installation, we can again enter either the module names directly or refer to URL, local file, git repo, github, etc.

New project initialization is then executed by the following command:

bower init

After a brief questionnaire, it will create the file bower.json in the project (with a structure similar to package.json from npm), which describes the given project. The content of bower.json file is shown in the picture above.

Therefore, if we receive a new project from someone, it should contain both the package.json and bower.json files, on the basis of which we then perform the given project installation:

npm install
bower install

and everything important will be downloaded into the project. It should be noted, that although the command bower install bootstrap itself, in an already existing project, physically adds bootstrap to this project, it does not update the bower.json file (dependencies section). If we want the given project to include this module, we must add the --save parameter.

Grunt

Grunt “The JavaScript Task Runner” is an automation tool for front-end tasks and thus enables, for example, automatic minification of CSS and JavaScript, conversion of preprocessor codes (LESS, SASS …) to CSS, compilation of Coffee Script into JavaScript, etc. There are thousands of modules for Grunt, enabling a variety of interesting tasks:

The installation of Grunt itself is executed by npm in the following way:

npm install -g grunt-cli

When working with Grunt, there are two key files – package.json and a so called Gruntfile (Gruntfile.js, alternatively Gruntfile.coffee). Both files are already installed in an already existing project, only the dependencies remain to be installed by the npm install command. In a new project, we can create the package.json file either manually by the npm init command, or by the grunt-init template.

Then, using npm, we install Grunt itself between so called devDependencies, together with its modules, such as the uglify module used for the minification of JavaScript, in the following way:

npm install grunt-contrib-uglify --save-dev

Gruntfile

Gruntfile (Gruntfile.js or Gruntfile.coffee) is a file containing the configurations of individual Grunt tasks.

module.exports = function(grunt) {

  //loading package.json
  grunt.initConfig({
    pkg: grunt.file.readJSON('package.json'),

    //uglify module definition - for
JavaScript minification
    uglify: {
    options: {
    banner: '/*! <%= pkg.name %>
<%= grunt.template.today("dd-mm-yyyy") %>
*/\n'
    },
    dist: {
    files: {
      'js/input.min.js': ['js/input.js']
    }
    }
    },

    //watch module definition - this monitors
changes of js files and minifies them
    watch: {
    files: ['js/*.js'],
    tasks: ['uglify']
    }
  });

  //loading the required Grunt modules
  grunt.loadNpmTasks('grunt-contrib-uglify');
  grunt.loadNpmTasks('grunt-contrib-watch');

  //registering of the default task executed
by entering the command grunt into
the command line
  grunt.registerTask('default', ['watch']);

};

The example above shows a created simple Gruntfile with two modules (uglify and watch), where one module (watch) is added to the default Grunt task and is executed by entering the command grunt into the command line. This task monitors changes in JavaScript files and when such change happens, it runs the module uglify, which takes the input.js file and minifies it as input.min.js file. We can also run the uglify task separately by the grunt uglify command in command line.

Gulp

Gulp has the same use as Grunt. It only works in a slightly different way. Because Grunt focuses on configurability, it is more suitable for users preferring to configure code rather than write their own. To write our own code, we use Gulp instead. There is a variety of modules, although the amount is lower than in case of Grunt. Gulp does not need that much anyway. What would otherwise require a module in Grunt, you can write by yourselves using Gulp. Gulp also focuses on speed, although that can be increased in Grunt too, by the module jit-grunt, which ensures that other modules are loaded only when needed.

Another difference between Grunt and Gulp is in their approach to processing files. While in Grunt you can see individual intermediate steps of, for example, JavaScript merging and minification, Gulp works with so called Node.js streams. In this case, there are no intermediate results generated on the drive, but data are transferred between individual tasks via the pipeline instead.

The installation of Gulp itself is executed by npm in the following way:

npm install -g gulp

When working with Gulp, there are two key files – package.json and a so called Gulpfile (Gulpfile.js), it is therefore very similar to Grunt, including the installation of Gulp modules.

Gulpfile

//loading Gulp itself into an object named gulp
var gulp = require('gulp');

//loading watch module
var watch = require('gulp-watch');

//creating a new task "test", which only puts text into the command line
gulp.task('test', function() {
  console.log('Hello, this is test ...');
});

//creating a new "default" task, which monitors the contents of input folder and copies every file created or modified in this folder into the output folder
gulp.task('default', function() {
  gulp.src('input/*')
    .pipe(watch())
    .pipe(gulp.dest('output/'))
});

This simple example shows two tasks, one named test and the other one default. The default task is executed by entering gulp into the command line. The default task uses the gulp-watch module. The “test” task is executed by entering the command gulp test. It is obvious from this code what is the difference between the files Gruntfile and Gulpfile and why Gulp users require less modules.

Yeoman

Yeoman is used to generate code according to previously set templates (scaffolding). The templates in this case are not text files, but a pattern, according to which a base for an application or a part of it can be generated. This way we can generate, for instance, the base of a WordPress website, AngularJS application, etc.

Installation is performed in the following way:

npm install yo -g

We also need a so called generator, which decides what is created, where and how. We can either use a prearranged generator or create it ourselves.

Useful Tools – Alternative to Command Line

If you are not fans of the command line, I have good news for you. Bower, Grunt and Gulp are all natively supported in IDE Webstorm (PhpStorm) and the projects using these tools can thus be managed directly in IDE. The modules can be easily managed, but Gruntfile or Gulpfile still have to be edited manually.
Interesting links with examples of working with these tools:

Conclusion

Using these tools greatly facilitates and accelerates work of a web developer. It facilitates and, more importantly, unifies work in team, since individual configuration files are versioned (as opposed to separate modules). This way the unified configuration is shared by the whole team and one does not have to worry that every developer creates, for example, different CSS from SASS, or that there will be problems in JavaScript if the whole team uses JSHint.

You can now follow to the next part – article called: Putting Continuous Integration (CI) into Practice – Part 1.

Front-End Task Automation - Part 1

Front-End Task Automation – Part 1

Leave a reply

Work on a web integration project includes a lot of repetitive and not-so-interesting tasks. The following articles describe several very useful tools that can significantly facilitate and accelerate our work. Before we start with individual tools such as Bower, Grunt, Gulp and Yeoman, let’s talk about what these tools are based on.

Also, you should not miss the previous part of the article series about web integration – Version control system and web integration projects. It is a must-read!

First, it should be noted that Bower, Grunt, Gulp and Yeoman are modules of Node.js and will be further explained in the following article. Node.js is a powerful event-driven framework for JavaScript, based on the V8 JavaScript engine from Google, which is used in Google Chrome. Therefore, we install these tools using npm (node package manager), which is the default package manager for Node.js.

Node.js and npm

Node.js is an event-driven I/O framework for the V8 JavaScript engine. In other words, it is a V8 engine enriched with functions allowing scripts to access files or network functions. This means we can create a server listening to a certain port almost the same way as we create, for instance, event handlers in a browser.

Npm (node package manager) is a package manager for Node.js and is installed together with Node.js by default. In Node.js, modules (packages) are installed the same way as, for example, software in Linux via APT. Modules can be browsed at npmjs.org.

It is important to say that the operation of most of these tools requires us to be at least partially familiar to the command line, because this is the usual way these tools are operated / installed. There are also alternative ways for some of them which will be discussed in the conclusion.

Npm is operated via the classic command line and all modules are installed by a simple command npm install. For example, if we want to install the module coffee-script, we use in the following command:

npm install coffee-script

A directory named node_modules was created in the project directory and contains locally installed modules used by the given project. If we want to install a module globally, we use the parameter -g. The module than will be installed globally and can be used in multiple projects.

npm install coffee-script -g

The previous example shows that also mkdirp module has been installed together with coffee-script, because the module is dependent on it.

During installation, we can refer directly to a module from the npm registry or install modules from Github, local modules, modules available at URL, etc. Npm offers much more, as can be seen in the project documentation.

Package.json

If you look into a project or module, you will probably find the file package.json. This file contains basic information about a project, such as name, version, description of dependencies on other modules (including version), etc. This very file allows the installation of given module and its dependencies. If we use the command npm install in a project, npm downloads all dependencies into node_modules folder, including other dependencies of modules and so forth.

To create a new project (package.json), we use the command:

npm init

This, after a brief questionnaire, creates a file package.json in the project, which provides a description of the project. Its content is shown in the picture above. The installation of modules in a project is then performed by the command npm install <pkg> –save, as we are advised by a hint during project creation.

The installation of coffee-script module directly into our project is then executed in the following way:

npm install coffee-script --save

And the corresponding package.json:

{
"name": "my-project",
"version": "0.0.0",
"description": "my description",
"main": "index.js",
"scripts": {
"test": "echo \"Error: no test specified\"
&& exit 1"
},
"repository": {
"type": "git",
"url": "https://bitbucket.org/my-project"
},
"author": "",
"license": "BSD-2-Clause",
"dependencies": {
"coffee-script": "~1.8.0"
}
}

One can see that coffee-script module was added to the dependencies section. This way the project is ready to be shared, for instance, by a team, where other members only install the remaining modules used by the project via the npm install command.

Conclusion

We have briefly described what the tools, which we will get to in the next part of this article, are based on and shown the basic principles used by all modules based on Node.js and npm. In the next part, we will finally get to the particular tools that can significantly facilitate work on a web integration project. That includes Bower – a simple package manager for front-end components, Grunt and Gulp – automation tools for front-end tasks and Yeoman – a scaffolding tool for generating complete projects.

You can now follow to the next part – article called: Front-End Task Automation – Part 2.

Version control system and web integration projects

Version control system and web integration projects

Leave a reply

Why does a version control system increase quality levels in a web integration project? What are the advantages of using such a system? And what is its overall influence on development?

Also, you should not miss the previous part of the article series about web integration – Advantages of continuous integration and automated deployment from the perspective of web integration. It is a must-read!

The Version Control System (VCS) plays an important role in the development of a web integration project. When using VCS, the main goal is to record the changes in project development and support for team collaboration, and deploy an application to various environments. Modern versioning systems resolve not only the issue of cooperation and recording changes, but also bring a range of other benefits.

VCS supports tools for continuous integration, such as Jenkins, Hudson, Bamboo. They can help us automatize application testing and deployment. Ultimately, tests and reporting will be followed by reporting in the form of email or other notifications.

Features of the most popular version control systems

Traditionally, Subversion (SVN) is the client-server application, which for a long period was the dominant system in terms of revision management. SVN saves space and uses different revisions. The project is versioned on a central repository on the server. For versioning, a functional connection with the service on the server is vital.

A similar philosophy can be seen in the Git and Mercurial distributed systems. The main difference between these systems and SVN is that all developers can work locally and do not require server connections to do their work. Additionally, they may submit only part of their work on a remote server and retain other parts locally for later adjustments.

Git is very flexible, involves a whole set of single-purpose tools and can easily be upgraded. Among others, we use Gitflow. In contrast, Mercurial is a single, compact library and has shown itself to be rather inflexible. For a detailed comparison of git and Mercurial, please refer to the Git vs. Mercurial article.

Decentralised nature of development, with a central repository

The project evolution results from a basically decentralised scheme. All developers work on their own local copies of the project and in specific teams in separate locations. Different teams use their local environments for developing and testing before a specific batch of changes is shared with other teams. As the number of teams and developers grows, so does the risk that two developers are involved in the same part of the application and the adjustments made will be incompatible. The distributed VCS is a better solution to such problems.

There are many approaches, either centralised or decentralised, which can be dealt with by various types of model (e.g. Workflow).

A unified system and standardisation of development

The settings of a unified form of changes versioning and a system of submitting work creates rules for the developer’s work, and a firm foundation for standardisation of development and distribution. For developers, this means clearly defined processes and, as a result, a set of instructions for dealing with possible situations that could arise.

Updating the application in various environments poses problems in the absence of VCS. In practice, we make use of continuous integration, using tools connected with VCS.

The project launch, and updates in operations in the testing and production environment, is further simplified, reducing the costs of handing over work and inspections.

To hire services or use your own

A versioning system can be operated on your own server, a solution which offers the maximum configurability. In projects where sensitive client data is processed, it is also imperative that the source codes are not made public, with an arrangement that under no circumstances could anybody other than the responsible developer be allowed access to such data.

If we do not want to deal with an operating a versioning system, we can use specialised hosting services supported by a selected versioning system. Github supports use of Git and SVN in parallel, giving the developer the freedom to choose the versioning system.

Migration between services

Everything progresses. In the past here at Krcmic.com or at Onlineandweb.com we operated our own SVN server and Git server. But despite its benefits, we had to move on and hand over to hosting services.

The first migration involves the transfer from our server to GitHub. Later on we moved our projects from GitHub to Bitbucket.

Moving repositories to GitHub

Migration on GitHub involves just a few simple steps.

  1. Create the repository on GitHub.
  2. In the original location on our server we configure as remote the repository that we set up in point 1.
  3. The entire history is sent to the remote repository.
  4. Security is configured for the developer team, and we inform its members about the migration carried out.

That is it. Then, work on the project can continue on without interruption.

Transfer from GitHub to Bitbucket

Thanks to import, the project need not only be migrated between GitHub and Bitbucket. This is a convenient method, where with the help of a single form a repository is created on Bitbucket. In addition, a request is made for information needed for access to the original repository on GitHub.

As can be seen, this form of migration is even easier than a switch to GitHub.

Blending versioned and non-versioned parts of the application

In virtually all web integration projects, there are parts that we do not want to version. Developers need local settings for a variety of environments (for example settings for a database connection). At the user level, we have user-managed data, through a Content Management System (texts, images, video…). Such parts should be separate from a versioning system. There is no need, for both security and purely practical reasons. In addition, the user data has a very large volume. Versioning is problematic at the very least, and in most cases is simply not possible.

In projects we resolve the situation through placing of configuration files and data folders in a list of ignored parts of the project. In Git, such a setting has the file content marked .gitignore, where specific files and whole can be specified and rules can be set for ignoring folders parametrically (for details see GitHub help and others).

Positive impacts of the version control system on the development of a web integration system

Version control system in the life cycle of a web integration project has an immediate impact on several areas:

  • Collaboration of developers, who tend to be in separate locations and focus on different professional areas;
  • responsibility linked with the traceability changes to the application codes,
  • secure deployment – the option to go through the changes history allows us return to a previous, stable version, if updating appears unsuitable for the operating environment;
  • documentation – records from a versioning system complement the traditional developer documentation and also add a time dimension, thanks to which it is possible to retrospectively monitor the speed and robustness of the development. Data export can be turned into graphic format, which will be especially appreciated by non-technical people in the team.
  • audit – after completing the project we turn to the versioning system audit of project,
  • change log – a summary of all changes for a particular period can be produced according to history.

Advantages and disadvantages – a summary

Version brings certain disadvantages, including an undoubted increase in levels of labour-intensiveness. Developers are required to extend their training to cover new tools that they will need for their work. On the other hand, versioning requires minimum effort in comparison with the development of web applications. Practice shows that the overall level of labour intensiveness does not increase in project development. They will therefore cut deployment costs, and development in turn is made more effective.

The increase in quality levels and effectiveness in development as a consequence outweighs the disadvantages. As well as the technical aspects outlined above, mention should be made of the audit option, which is also appreciated by managers.

The above clearly shows that using a versioning system should be included in the development of every web integration project.

You can now follow to the next part – article called: Front-End Task Automation – Part 1.

Advantages of continuous integration and automated deployment from the perspective of web integration

Advantages of continuous integration and automated deployment from the perspective of web integration

Leave a reply

Although the choice of specific tools supporting the implementation of the principle of continuous integration into the organization engaged in development is not associated with continuous integration as such in any way, in another interpretation we will stick to the terminology of Git version system, Git flow version workflow, and principles of semantic versioning.

Concept of continuous integration

In an organization where there is the development of projects or products of larger scale in the team, developers typically work on particular parts of the assignment (“features” or “hotfixes”), which are necessary to be systematically combined into one whole of the project while maintaining the integrity (functionality) of already existing work. Continuous Integration is based on continuous performance of acts of integration of source code, testing, building, and deployment in response to each change to the source code of the project submitted by the developer and for the use of tools for the support of the development and testing by compliance with the established procedure automatedly.

System of control of versions of source code

The cornerstone of deploying continuous integration in an organization is the consistent use of a version system for ALL the code of projects, and ideally incl. runtime configuration (with an exception of configurations specific for a particular target runtime environment) and database configuration as defined by data structures – in case of relational databases of DDL* commands, for other types of repositories than other descriptive files, usually in XML or JSON format).

(* Data Definition Language – a language for describing data structures (typically database tables)

It is also appropriate to version not only the program and configuration code of the projects themselves in the repository, but also meta files, which help to uniformly manage the process of development and integration, e.g. single configuration of Integrated development environment (IDE) of the team, files controlling building of the project (see below) and files configuring this process (project build configuration).

Version systems are generally divided into centralized (CVCS) based on server-client architecture and distributed (DVCS) not requiring server service because they generally suffice with any network-accessible file repository. Among centralized VCS are included CVS, SVN (Subversion) or SourceSafe, among distributed are included e.g. Git or Mercurial, where especially Git has observed a vast expansion not only in the world of open source in recent years.

Central source code repository

Central source code repository represents a network location or better service that solves:

  • hosting – physical location of repositories of code of used version system,
  • centralization of developed code for transmission and control,
  • security settings (restricting access to specific repositories for specific users or better to user groups)
  • code control,
  • connection of tools for cooperation in development of the project.

Utilizing cloud service eliminates overheads associated with maintaining own solutions to a central repository. In contrast, when using own hardware resources the code remains within the organization’s infrastructure and can thus be considered to be stored more safely (let us ignore the fact that statistically the information leakage occurs the most just inside the corporate network infrastructure). The technical aspect for selecting the type of operation of the service may be also the speed of connection, where the availability of the service located in the internal network of the company is higher in orders of magnitude than for external services.

Centralization brings advantages both for developers and manager roles involved in the project. Developers have one destination for takeover of current code of the project and the subsequent submission of work and thus there is a space for them to focus on their core specialization that is production of functional quality code. On the other hand, staff focused managerially have clearly designated place where they can check the progress of the course of development of the project at any time.

After submitting the project code, its control typically follows. Submitting of work may then take the form of a request for acceptance of set of changes (commits) in the project code (so called pull request), when the developer cannot directly affect the version of the project in a remote repository, but a proposed change by him/her must pass the quality control (code review). The actual control of the code has both the dimension of a control by a responsible person and also automated tests are used to check the submitted code. For more details see below in the chapter Automated testing and code control.

It is advantageous if not even necessary to solve the above mentioned parts of the development by means of specialized existing services / tools and in the sense of hosted (on-demand) service or application which you install on your own infrastructure and operate so called on-premise. Hosted services are represented by Github and Bitbucket, where the organization has a web-based user interface for:

  • comfortable management of safety,
  • control of pull requests,
  • code review,
  • history display
  • possibility of integration with other services (for tracking of requests, automated deployment etc.)

A tool for solution on own hardware resources may be Stash for Linux (on-premise similar to Bitbucket cloud service) GitWeb, Gitolite or Bonobo Git Server or GitStack for Windows.

Packaging and dependency management (PDM) tool

Within the development infrastructure in addition to determining a central repository of source codes a need arises to storage and organization of packages of already compiled code, either within the organization or independently outside of it or possibly within packaged source code that does not change – it concerns scripting application environments and / or distribution of code for the needs of code assist feature in the development environment (IDE) and the need to clearly define the dependence of the code of the project on these packages. It is therefore necessary to use a tool that allows you to configurationally define the necessary dependencies of the source code (i.e. different frameworks and libraries needed to run the application) and put the resulting compiled application into the form of a distributable package. While in some application platforms the use of this tool is inherent (e.g. Apache Maven not only for Java-based projects, Ruby Gems for Ruby or NPM Node.js/Javascript based projects) in the world of web development on PHP platform its existence in the form of a Composer tool is still a relative novelty and a number of open source projects still does not consider it, nevertheless its use is increasingly promoted.

Each PDM tool commonly brings the following features into the development

  • Defines the configuration file of the project for definition of dependencies including repositories, in which it is possible to obtain the dependent packages, and other conditions for the formation of application, usually in XML or JSON format
  • Defines format/form of the package of reusable source or compiled code (according to application platform) incl. its meta definition (analogy for project configuration)
  • Usually has one or more own public repositories in which packages with popular open source code are made available (in case of Composer tool it is Packagist service)
  • A tool for command line (CLI) for controlling the life cycle of the project following the chosen archetype, typically thus
    • Creation of directory structure of the project following the chosen application framework
    • download of all dependencies necessary for development (source codes for code assistance in IDE) or running of the project
    • update of downloaded dependencies at change of configuration
    • build (package) with the possibility of loading into runtime environment (application) or repository of packages (libraries); within building then performing a series of operations that depend on each other

Continuous Integration tool – CI

Continuous Integration tool ensures all the automatable tasks within implementation of CI process i.e. using or in cooperation with above mentioned tools for source code management and controlling of dependencies, its building and package. There are many tools available as open-source as well as commercial, let us mention the best known

Some are directed to use in connection with a specific application platform, others are platform-neutral. CI tool can be operated in both traditional on-premise model or use them as a cloud service. On-premise deployment may be a necessity in some cases, typically when the source code repositories and / or target runtime environments for deploying of the application are placed in a private network of the organization and they cannot be made accessible to the cloud service due to security policies.

The basic property of CI tool is the definition of parametrized tasks, which can disintegrate into individual partial steps. The task may be subsequently done either by running through a web interface of CI tool or through its API by calling upon an event in another system (typically). Tasks may also be combined to a higher logical units. Thus CI tools, to a considerable extent, reiterate the principles of separate building command line tools of Ant, Maven or Phing type with which they may also complement (a step in the building task may call a building tool controlled by building configuration file (build config file), which may be a part of version code of the project or is stored on the CI server independently from it).

Automation of building

Building means an activity which by downloading of the source code and configuration files incl. definition of dependencies and a code required by this definition for translation and / or running of the project will create a form of a project executable in runtime environment and usually in the form of a package, but it may be only a workspace directory created on CI server for further use.

Depending on the chosen application technology a translation may be a part of building, in case of PHP it is usually installation or update of built form of the project by Composer tool. Automated building is usually done using CI tool based on a change (a new commit) in one of the main branches in the repository of the source code of the project.

A successful automated building of the project then may be an input (trigger) of subsequent automated deployment. In case of a product (application or library) the CI process usually ends with automated building (finished package is delivered to CI server for further use).

Automation of deployment

Based on successful automated deployment and testing of project the CI tool may immediately or independently in a separate task, proceed to automated deployment into target environment.

Set of environments which are available in the organization for deployment of the project should be clearly defined and the set of environments which are available in the organization for deployment of the project need to be unanimously understood in the context of development. These environments usually stem from the life cycle of the project.

  • Internal test for deployment of current (HEAD) form of the project in the development. In case that shared server environment is used for development of web projects (which is, in case of PHP development, widespread practice in bigger organizations), then the internal test is usually implemented in the same server environment, in which developers develop, by which equivalent running conditions are ensured for it, as a local instance of the project of developers. Development team (incl. testers and internal product owner, delivery manager or project manager continually supervising the process of development) work with the internal test.
  • Preview, sometimes also UAT or simply testing environment. There is usually a deployment into the preview from release or hotfix branch of the source code of the project, possibly a separate preview branch may be defined to which developers merge changes, which they want to build into preview environment by means of automation of project deployment. Primarily the Client (product owner) works with preview environment and he/she checks (accepts) incorporated changes in them.
  • Production or live environment.

Automated deployment reduces if not directly does not need inputs by developers into runtime environments within the process of deployment, which positively contributes to stable behaviour of the applications.

Automated testing and code control

According to the time moment of launch, tests can be divided into groups pre-commit (before submission), pre-deploy (before deployment) and post-deploy (after deployment into the target environment).

Pre-commit tests

Control of syntactic validity and coding conventions (formatting, documentation of the source code)

Pre-deploy tests

Pre-deploy tests relate to the source code and can be implemented before building or within it (thus typically after all the necessary dependencies are prepared by a package tool).

These test may be launched in response to integration of changes in the code in system for administration of the source code into some of the main branches (develop, master) of the main repository of the project. In response to the nature of the test or strategy of integration they can be launched even before the change will be integrated into the branch (so called pre-commit) or after it (post-commit).

  • Launching of unit tests implemented within the project for testing individual features (typically using PHPUnit framework)

Post-deploy tests

Post-deploy tests may be implemented only in response to a successful automated deployment as they require an application running in the target environment. These are typically:

  • Automated user tests created in Selenium IDE tool and launched by CI tool in Selenium server environment
  • Integration tests verifying functionality of the system

Automation of testing makes it obvious and ever present part of the development process. Compared to traditional testing implemented ad-hoc (or neglected) the CI process may require the presence of tests while forcing the team to respond to their failures (be it by a repair of the source code or by an update of the test in response to required implemented functional changes).

Main contributions of deployment of continual integration into the process of making web solutions

Automation of individual repetitive tasks of building to deployment of the application eliminates occurrence of errors associated with manually performing them. In addition, an automated process is always described at least in the form of configuration files or configuration settings of the tools that perform it, when these configurations are managed by limited group of persons (role CI/deployment manager only) and in addition it is usually possible to apply version control system to them as well and thus have a control in the whole team over when and who changed such a meta definition of the process.

Automated implementation of tests ensures continually running testing of the project in response to ongoing change at the level of the source code (typically thus integration of commit into some of the main branches of the project) or even independently on it (post-deploy tests may continuously verify the integrity of running application in the chosen target runtime environment)

Continually running process of building and deployment ensures that in response to the work submitted by the developers and workflow of the project mapping the individual development phases to target runtime environments there are always current tested versions of the project in these environments.

At the same time automated deployment facilitates extraordinary remedies, e.g. allows to easily deploy the previous version in case that despite implemented automatized or possibly manual tests the project shows problems in the given environment.

A concept of continual integration is naturally conveniently complemented by agile, interactive methodologies of the software development (Scrum).

Continual integration tool gathers and according to the needs / rules of the organization further distributes information on the result of automated task via available information channels (e-mail, instant messaging, possibly SMS messages). Thus it contributes to the awareness of each member of the development team as well as persons of interest standing outside the team (development manager, project management, product owner) about the course of the process of controlled change in the web project.

Useful links

You can now follow to the next part – article called: Version control system and web integration projects.