Not a long time ago, I decided to switch and start my newly assigned project with Gulp, rather than Grunt.

It worked out well. I was able to do what my grunt configuration was doing, but this time via gulp. This was not possible without the help of all those community plugins, which I actually just glued together.

There was only one big problem left, which I had to solve again, but this time in a gulp way. Gulpfile.js looked like copy-pasted patchwork. There was plenty of repeatable code and most importantly no structure.

I had to find a new solution to the problem that I had with my grunt configuration. I was using load-grunt-config, but there was no such option for doing the same in gulp.

So this is what I did :

Separate configuration from build scripts

First things, first. I always hated to copy paste these "./.build/" and "./src" strings all-over my Gulp file. It doesn't feel good and it meant project search & replace whenever I needed to change these.

That's why, you can usually find a ./configuration.yaml file in most of my projects that also contain the project structure. It looks like this :

#
#  Configuration for MyAwesomeProject
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
#
#  The `paths` section is used by Gulp to provide fine grain control and DRY
#  principle in the configuration files
#

paths:
  source:
    root: '.'
    fonts: '/fonts'
    server: '/src'
    images: '/images'
    scripts: '/scripts'
    templates: '/templates'
    stylesheets: '/stylesheets'
  build:
    root: './.build'
    server: '/'
    templates: '/templates'
    package: '/../.packages'
    www:
      root: '/www'
      fonts: '/fonts'
      images: '/images'
      vendors: '/vendors'
      scripts: '/scripts'
      stylesheets: '/stylesheets'

# Other notable variables also can exist here

port: 8080

Doing this gives you a couple of benefits :

1. There will be an easily-maintainable variable, instead of a hard-coded string for all directories in your project ;
2. Everyone can see what is stored where, by opening this file ;
3. You can reuse this configuration file in other parts of your application that needs them. ( e.g. ExpressJS middleware for static files, templates rendering, etc. )

Let's see how can we use this with Gulp.

Treat Gulpfile.js as nothing more than a NodeJS module

Well, this is one of the main benefits of using gulp. It's not a declarative configuration file, it's just a module that you can ( should ) consider as a separate application.

So let's do what we do with any application that we want to scale... Let's separate it.

My Gulpfile.js looks like this :

/**
 *
 * ### Overview
 *
 * We are using [gulp](http://gulpjs.com/) as a build system. Gulp in
 * MyAwesomeProject is responsible for a couple of things :
 *
 * 1. Compiles the project ( written in TypeScript ) to Javascript ;
 * 2. Helps during the development by watching for changes and
 *    compiles automatically.
 *
 * ### Structure
 *
 * Our gulp configuration starts in the root `./Gulpfile.js`, which
 * loads all tasks in the directory `./gulp`.
 *
 * The gulp-task files itself are written according to JSDoc specs
 * to make generating the future documentation flawless.
 *
 * There is another special directory, called `./gulp/lib`, which
 * purpose is to store all non-gulptask files that have helpers
 * for the tasks ( e.g. configuration options )
 *
 * ### External Configuration
 *
 * Gulp uses configuration variables stored in `./configuration.yaml`
 *
 * @name gulp
 * @module
 *
 */

require( "./gulp/development" );
require( "./gulp/build" );
// ... more, such as "./gulp/package", "./gulp/deploy", etc.

That's right. I'm including this long JSDoc comment ( and a lot more as you will see next ), because I want to make the life of my fellow team easier.

Also making nice JSDoc comments in our Gulp configuration will lay down the foundation of making a beautiful living documentation that explains how to build the project and what that means, without opening any files.

Separating Gulpfile.js, means that not everything will be stored in one single file, so documenting the build process becomes more and more important.

Create a scalable tasks structure

If you've read carefully that JSDoc in the root Gulpfile.js, you already know the structure and the purpose of the files in the whole gulp app.

Let's make it more visual :

./gulp/
./gulp/lib/helpers.js
./gulp/lib/paths.js
./gulp/build.js
./gulp/development.js
./Gulpfile.js
./configuration.yaml

So what's the reasoning behind this.

In the root of our ./gulp directory we position our main tasks. Those are usually tasks that are composite and define inside their building blocks. The task structure looks something like this

development          - Watch files and executes "build-[task]" accordingly
build                - It's a container for the whole build process
| build-clean        - Cleans the build folder
| build-compile      - Server-side files manipulation (e.g. TypeScript->JS)
| build-scripts      - Client-side javascripts manipulation (e.g. Browserify)
| build-stylesheets  - Client-side stylesheets ( e.g. autoprefixer )
| build-assets       - Just copies fonts, images and other static files
package              - Container for packaging task executes package-*
| package-assets     - Copy other files that we want to add to our package
| package-tar-gz     - Creates a zip file
documentation        - Container for documentation task. Exec. documentation-*
| documentation-app  - Generates documentation for our app
| documentation-gulp - Generates docs for the build system ( parsing JSDocs )

... more

Helpers and other shared code between the tasks that we can put inside /lib. Actually let's see what we have there so far.

Separate execution helpers separate from the task files

First, let's see what's inside gulp/lib/helpers.js

/**
 * @module lib:helpers
 * @private
 */

var gulp = require( "gulp" );

module.exports = {

    /**
     * Import tasks provided as an object into gulp
     *
     * @param tasks {object}
     */
    importTasks : function( tasks ) {
        Object
          .keys( tasks )
          .forEach( function( key ) { gulp.task( key, tasks[key] ); }
    }

}

We export importTask which basically replaces gulp.task('task', function() { ... }) with our own helper. We will use that in our gulp task files.

Our gulp/lib/paths.js is a bit more complicated and looks like this :

/**
 * @module lib:paths
 * @private
 */

var yaml = require( "js-yaml" ),
    fs   = require( "fs" ),
    _    = require( "lodash" );

/**
 * Parse all paths and populate the directories with their roots if any.
 *
 * @private
 */
var normalizePaths = function( paths ) {

    var result  = {},
        recurse = function( cursor, adder, prop ) {

            if ( _.isString( cursor ) ) {
                if ( prop !== "root" ) {
                    result[prop] = adder + cursor;
                }
            } else if ( _.isArray( cursor ) ) {
                result[prop] = cursor.map( function( item ) {
                    return adder + item;
                } );
            } else {

                var isEmpty = true;

                if ( cursor["root"] ) {
                    adder += cursor["root"];
                    result[prop ? prop : "root"] = adder;
                }

                Object.keys( cursor ).forEach( function( key ) {
                    isEmpty = false;
                    recurse( cursor[key], adder, key );
                } );

                if ( isEmpty && prop ) {
                    result[prop] = {};
                }

            }

        };

    recurse( paths, "", "" );

    return result;

};

/**
 * Parse paths for all modules
 *
 * @private
 */
var paths     = yaml.safeLoad(
      fs.readFileSync( __dirname + "/../../configuration.yaml", "utf8" )
    )["paths"],
    source = normalizePaths( paths['source'] ),
    build  = normalizePaths( paths['build'] );

/**
 * Export helpers
 */
module.exports = {

    /**
     * Exports all paths
     */
    for : paths,

    /**
     * Returns path for source
     *
     * @param [path] {string}
     * @returns {string}
     */
    forSource : function( path ) { return source[path ? path : "root"]; },

    /**
     * Returns path for build
     *
     * @param [path] {string}
     * @returns {string}
     */
    forBuild : function( path ) { return build[path ? path : "root"]; }

};

So what we have as an outcome is friendly functions that can return the proper nested path from our configuration.yaml. Doing this will keep us out from any code duplications that may occur for our directories.

Actually since we already know that Gulpfile.js is an actual application, we should also not worry about using popular libraries and speed. Most of the libraries out there use lodash, so it's not an issue if we also use it every time we run gulp.

Let's see how these two files can make our build system scalable.

Keep your task files readable

The infrastructure that we created gives huge benefits on how the task files can look.

A typical build file, using the principles above should look like :

/**
 * @module build
 */

var gulp         = require( 'gulp' ),
    sequence     = require( 'gulp-sequence' ),
    gutil        = require( 'gulp-util' ),
    del          = require( 'del' ),
    es           = require( 'event-stream' ),
    sass         = require( 'gulp-sass' ),
    ...
    paths        = require( './lib/paths' ),
    helpers      = require( './lib/helpers' );

/**
 * ### Overview
 *
 * Tasks used for building the NodeJS application
 *
 * @namespace tasks
 */
var tasks = {

    /**
     * @task build
     * @namespace tasks
     */
    'build' : function( callback ) {

        /**
         * It is a composite task that runs the following tasks in sequence
         *
         * 1. `build-clean`
         * 2. `build-compile`
         * 3. `build-scripts`
         * 4. `build-stylesheets`
         * 5. `build-assets`
         *
         * The different tasks are found below :
         *
         * @namespace tasks:build
         */
        sequence(
            'build-clean',
            'build-compile',
            'build-scripts',
            'build-stylesheets',
            'build-assets',
            callback );
    },

    /**
     * #### Cleans the build target folder
     *
     * Cleans the folder, which is the root of the compiled app ( `./.build` )
     *
     * @task build-clean
     * @namespace tasks
     */
    'build-clean' : function() {
        return del( [paths.forBuild() + '/**'] );
    },

    /**
     * #### Compiles the app
     *
     * Compiles the source application directory to the build directory
     *
     * @task build-compile
     * @namespace tasks
     */
    'build-compile' : function() {
        return gulp
            .src( [paths.forSource( 'server' ) + '/**/*.*'] )
            .pipe( /* ... */ )
            .pipe( gulp.dest( paths.forBuild( 'server' ) ) );
    },

    /**
     * #### Compiles front-end scripts
     *
     * Compiles all front-end scripts into bundles using ...
     *
     * @task build-scripts
     * @namespace tasks
     */
    'build-scripts' : function( callback ) {
        return gulp
            .src( [paths.forSource( 'scripts' ) + '/**/*.*'] )
            .pipe( /* ... */ )
            .pipe( gulp.dest( paths.forBuild( 'scripts' ) ) );
    },

    /**
     * #### Compiles stylesheets
     *
     * Compiles source stylesheets via [Sass](http://sass-lang.com/) with
     * SCSS syntax
     *
     * @task build-stylesheets
     * @namespace tasks
     */
    'build-stylesheets' : function() {
        return gulp.src( paths.forSource( 'stylesheets' ) + '/**/*.scss' )
            .pipe( sass().on( 'error', sass.logError ) )
            .pipe( gulp.dest( paths.forBuild( 'stylesheets' ) ) )
            .on( 'error', gutil.log );
    },

    /**
     * #### Copy assets
     *
     * Copies the static assets that will be used by the server :
     *
     * - templates
     * - images
     * - fonts
     *
     * @task build-assets
     * @namespace tasks
     */
    'build-assets' : function() {
        return es.merge(
            gulp.src( paths.forSource( 'templates' ) + '/**/*' )
                .pipe( gulp.dest( paths.forBuild( 'templates' ) ) ),
            gulp.src( paths.forSource( 'images' ) + '/**/*' )
                .pipe( gulp.dest( paths.forBuild( 'images' ) ) ),
            gulp.src( paths.forSource( 'fonts' ) + '/**/*' )
                .pipe( gulp.dest( paths.forBuild( 'fonts' ) ) )
        );
    }

};

//
// Registering Tasks
//

helpers.importTasks( tasks );

As you can see gulp/build.js file looks totally different than most of the Gulpfile.js that you can find. The main differences are :

1. Using an object ( var tasks = { /* ... */ } ) that stores all the tasks as keys ;
2. Use of JSDocs to annotate and summarise what a task is actually doing ;
3. Use a configuration file as a storage for source and destination paths.

The tasks object separates what the local gulp command can execute in a very prominent way. Since it lives in a separate object any non gulp task functions that may exist as helpers ( e.g. custom logging function ) lives outside the variable and is strictly separated from the task itself.

With a little bit of help from gulp-jsdoc-to-markdown you could generate a documentation out of the whole gulp application. I'm using some custom annotation tags ( @task ) that I later filter and render with the proper html markup. Teams benefit a lot from docs like this.

The way that the source and destination paths are specified, gives us the opportunity to reuse the same strings multiple times, without worrying when we need to refactor and change them. Imagine a commit message stating "Changing build folder" and 50 changed lines. Using this method the modifications will be only one row.

What do you think?