roll-right

A tool that helps generate HTML pages, or other types of pages such as PWA components.

Overview

roll-right operations begin with the tool reading a configuration file that lists output groups. Each output group defines targets, which include a list of concerns and formulas for output directories, indicating where the files for the concerns should go. Each output group also contains a table mapping output file names to skeletons that should be used to generate them.

roll-right compiles the skeletons, .skel files, which are description of web pages in a skeleton language. The language allows for describing repetitions of sections, data driven section replication and expansion, sub-skeleton inclusion with conditional selection based on configuration variables and simple calculations.

install it

npm install -g roll-right

run it

Here is an example:

roll-right --phase prepare --sources [websites]/template-configs/  --generator generate.json --structure parsed.json

roll-right --phase template --sources [websites]/template-configs/  --generator generate.json --structure parsed.json

roll-right --phase page --sources [websites]/template-configs/ --values assignments.json

In the examples, there is a parameter sources that is a directory. The other parameters that take files as arguments are expected to be offsets from the sources parameter. The documentation that follows will include a description of these parameters.

purpose

This tool helps create a flow in creating and maintaining web pages and PWAs in a few phases.

Each phase runs against a configuration file that helps select web page skeletons for a number of concerns (different URLs and web applications) and operates to fill out the skeletons with the custom content through a few steps until outputting a final pre-release page for testing, prior to release.

The web pages (apps) in the pre-release directories can undergo further transformation before they are deposited into a final staging directory, where the pages can reside in their compressed form with bundles in subdirectories structured for delivery to directories accessed by a web server. Other tools may take on this final step, for example copious-software-dev-manager.

three phases:

There are three phases:

1. Skeletons to skeleton parsing data and a named section database for each page belonging to a concern.
2. Parsing data along with finalized named section databases to Web Page Templates.
3. Web Page Templates to pre-staging HTML files (or app files)

Once pages are in staging, delivery to server directories or other uses are left to other tools. But, the files in pre-staging may be observed in the browser for testing and display quality.

The following provide a few more hints about the phase:

1. prepare

   In the preparation phase (prepare), the roll-right reads in skeleton files, creates a parse tree from them for the concerns, and produces a database of visual regions for each file generated for a concern. The tool puts the parse tree into a file in the project's sources directory. Then. in a directory set aside for a concern. the tool writes a named section db for each file being generated for the concern. The content creator may edit the named databases.

2. template

   During the template phase, the tool reads the files generated by the prepare phase along with the edited versions of the named databases. In this phase, the tool finalizes any skeleton compilation left given the named databases reflect the content choices provided by a content editor. The tool generates visual template files, e.g. HTML files with variable forms and file import forms. Besides generating templates, the variable valuation files are generated. A content editor would put values into the valuation file, such things as dates, company names, etc. The tool attempts to merge in value files that may have previously supplied values to the variables.

3. pages

   pages is the tool's final phase. During the pages phase, the tool reads in the templates and the value files and generates final HTML files and manages their JavaScript bundles. The files are placed into pre-staging directories for each concern, where each page file belonging to a concern has all the values provide for the concern for the page

initial setup

Prior to the use of the tool, a directory or set of directories, containing skeletons should be created. roll-right does not create these skeletons. That is up to the manager of a website concern or may be created automatically by scraping sites for themes. roll-right will read from the directory specified in its configuration file. In my own setup, final skeletons are placed into a pre-template directory, and there are others, such as those closer to the PWA projects. The final skeletons are called "final" because they start, originally, as files listing all the JavaScript file and visual components that needed for a number of projects. But, among a number of sites, much of the code can be replicated. So, there are other tools that edit the original skeletons by removing some file inclusions which get moved into bundles, CSS files that are included may be turned into links to CSS bundles are deferred loading files. The roll-right configuration can be used to make the tool aware of bundles and link files that should be moved into the directories used by the concerns' pages.

Background

For more on the evolution of this tools you may be interested in the following: roll-right background

Differences Compared to Bundlers

The main process used by this package is substitution and expansion. The output is mainly a self contained file or a small collection of files. The output is not bundled. Also, there is a dependency on Handlebars, which handles conditional substitution.

This tool does put script and markup into a file, given a repository of commonly occurring template snippets and code files. The files an output page will use and include must be mentioned in the original skeleton file in some way, directly or through decision logic or code. This tool leaves compression and a certain amount of tree shaking to other tools.

Basic Use Process

Here are steps of a generation process for site maintainers:

1. Store commonly used, well-tested code in files in selected directories
2. Provide a collection of files each describing families of pages called skeletons
3. Provide a JSON description, the configuration, that selects which skeleton parts are to be used along with source directories
4. Run roll-right in the prepare phase with the JSON configuration to generate intermediate compilation files and prepare databases of sections for each output.
5. Decide on further template customization (or select defaults) by editing component descriptions from the prepare phase.
6. For each concern, provide static html components, pictures, etc. to be used to populate templates; keep them in directories per website, app, etc. (usually a static directory within the concern's directory )
7. Run the template phase to generate .tmplt files, one for each output page/file being generated. This also generates .subst files.
8. Edit .subst files for each file being output (or possibly one master .subst file per concern) in order to specify content.
9. Run roll-right in the page phase with .subst files specifying the template instantiation values.
10. Use other tools to deploy files that have been dropped into a pre-staging directory.

It is possible that command line call of the tool can be done once if the user is willing to accept defaults and to have all the resource directories ready. Generally, however, the command line tool will be called once for each phase for each initial configuration file.

HTML Generation Steps

1. For generation of HTML, a project should begin first with a skeleton The developer will either make one of these, or use one from a repository of them. The skeleton will not contain HTML. If it being used to generate HTML, the skeleton may lines indicating that HTML tags should be included at certain points. Between those tags it may have lines indicating the import of partial template files which may include HTML markup as well as substitution variable forms.
2. The developer needs to make a configuration file. roll-right reads configuration file, enabling roll-right to find the pre-template and the set of files that will be use to replace link forms. The configuration file has its own set of variables. It contains a set of directory abbreviations, some global variable assignments, and a list of output groups. Each output group indicates the targets that should be generated and the skeletons that should be used to generate the outputs for a set of concerns indicated within the output group structure.
3. roll-right compiles the skeletons given the variable assignments in the configuration file. The parsing output will be stored in the sources directory. The parsing output will include intermediate parsing information for every skeleton in use for a run. A number of section database file, called calc_<something>.dbwill be generated, one for each output file being generated. The calc_<something>.dbfile may be edited by the user. It is called "calc" because it may either make a link reference to a file in a static directory belonging to a concern, or it may be a calculation. In some cases, the calc file indicates how to make replications of a section which is again a calculation used in skeleton processing.
4. roll-right reads the parsing information and the edited calc_<something>.dbfiles and generates template files. roll-right places the template files for a concern in a template directory. It also places files containing maps of substitution variables to values, one for each template being generated. It also produces a master substitution map for the entire website belonging to each concern.
5. During the template phase roll-right attempts to merge previously determined substitutions in with the new substitution maps being generated. It might be the case that the site maintainer does not have to change any values in the substitution maps, but he may. In either case, roll-right will read in the templates and the substitution maps and used them to generate complete HTML files. The HTML files will be deposited in the pre-staging directories for each concern.

An example

Here is a full skeleton for a kind of page that may be used by a concern.

$$defs:{
    "top_level" : true,
    "path_abreviations" : {
        "[alpha-copious]" : "[github]/alphas/alpha-copious",
        "[github]" : "~/GitHub",
        "[websites]" : "[alphas]/websites"
    },
    "ext_default_dir" : {
        "tmplt" : "[alpha-copious]/html",
        "js" : "[alpha-copious]/client",
        "svg" : "[alpha-copious]/icons",
        "css" : "[alpha-copious]/css"
    },
    "top_dir_location" : {
        "script" : "[alpha-copious]/client",
        "for-humans" : "[alpha-copious]/for-humans",
        "files" : "[alpha-copious]/html"
    }
}
//  ----
$$html:start_doc_head<<
$$files::header.tmplt<<
$$html:end_head<<

$$html:start_style<<
$$css::styles1.css<<
$$html:end_style<<

$$html:start_script<<
$$files<js>::top_vars<<
$$html:end_script<<

$$html:start_body<<

$$files::params::nav_bar_V.smplt<< {
    $@{lr_div}$files::params::left-right-div.smplt<<    {
        $@{logo}$files::logo.tmplt<<
        $@{spacer}$files::spacer.tmplt<<
        $@{menu}$files::shroom.tmplt<<{
             $@{mushroom}$icons::mushroom-menu-icon.svg<<
        }
    }
    $@{logout}$files::logout.tmplt<<
}

$$files::flexy_items_A-main.tmplt<<

$$files::footer_A.tmplt<<

$$verbatim::{
<!-- start dynamic content -->
}

$$files::squashable_menu_A.tmplt<<

$$files::calc::contact_box<<${box_i=100}

$$files::calc::about_box<<${box_i++}

$$files::calc::thankyou_box<<${box_i++}

$$files::loop::calc::topicBox_<1,3><<${box_i++}

$$files::calc::register<<${box_i++}

$$files::calc::login<<${box_i++}

$$files::intergalactic-explain.tmplt<<

$$template::{
{{{advertPopups.content}}}
}

$$html:end_body_html<<
//
//
$$html:start_script<<

// global defs of some static topology
$$script::[alt-script]/flexy_items_A-top.js<<

$$script::common.js<<

$$script::field_checks.js<<

$$script::https_checks.js<<

$$script::post_fetch.js<<

$$script::uploader_class.js<<

$$script::validation_class.js<<

$$script::captcha_service.js<<
//
$$script::login-logout.js<<

$$script::one_table_db.js<<

$$script::file_ops.js<<

$$script::[alt-script]/flexy_items_A-layout-and-menu.js<<

$$script::[alt-script]/flexy_items_A-animation.js<<


// CAPTCHA PERFORMANT

$$script::captcha_app.js<<

$$verbatim::{
//  === ---------------------------------------  === ---------------------------------------  === --------------------------------------
// APPLICATIONS
// -- -- -- -- -- -- -- -- -- -- -- -- --  -- -- -- -- -- -- -- -- -- -- -- -- --  -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
}

$$script::[for-humans]/shared_constants.js<<

$$script::[for-humans]/msg_vars.js<<

$$script::[for-humans]/external-id-intake.js<<

$$script::[for-humans]/human_frame_client.js<<

$$script::[for-humans]/frame_page_opener.js<<

$$script::[for-humans]/site_page_tab_com.js<<

$$script::user_db.js<<

$$script::[app<scripts>]/site_app.js<<

$$script::[alt-script]/flexy_items_A-app-finalize.js<<

$$html:end_script<<

roll-right can work on this skeleton, but by using some tools pull out files that an be in bundles or linked, roll-right can work on a smaller file. Here is the file transformed after the use of some tools that pre-process the skeletons.

$$defs:{
    "top_level" : true,
    "path_abreviations" : {
        "[alpha-copious]" : "[github]/alphas/alpha-copious",
        "[github]" : "~/GitHub",
        "[websites]" : "[alphas]/websites"
    },
    "ext_default_dir" : {
        "tmplt" : "[alpha-copious]/html",
        "js" : "[alpha-copious]/client",
        "svg" : "[alpha-copious]/icons",
        "css" : "[alpha-copious]/css"
    },
    "top_dir_location" : {
        "script" : "[alpha-copious]/client",
        "for-humans" : "[alpha-copious]/for-humans",
        "files" : "[alpha-copious]/html"
    }
}
//  ----
$$html:start_doc_head<<
$$files::header.tmplt<<
$$bundle::bundle_13.js
$$bundle::web3-mixed-boxy_bundle.js
$$link<css>::shared_styles.css<<
$$html:end_head<<

$$html:start_style<<
$$css::quick_view.css<<
$$html:end_style<<

$$html:start_script<<
$$files<js>::top_vars<<
$$html:end_script<<

$$html:start_body<<

$$files::params::nav_bar_V.smplt<< {
    $@{lr_div}$files::params::left-right-div.smplt<<    {
        $@{logo}$files::logo.tmplt<<
        $@{spacer}$files::spacer.tmplt<<
        $@{menu}$files::shroom.tmplt<<{
             $@{mushroom}$icons::mushroom-menu-icon.svg<<
        }
    }
    $@{logout}$files::logout.tmplt<<
}

$$files::flexy_items_A-main.tmplt<<

$$files::footer_A.tmplt<<

$$verbatim::{
<!-- start dynamic content -->
}

$$files::squashable_menu_A.tmplt<<

$$files::calc::contact_box<<${box_i=100}

$$files::calc::about_box<<${box_i++}

$$files::calc::thankyou_box<<${box_i++}

$$files::loop::calc::topicBox_<1,3><<${box_i++}

$$files::calc::register<<${box_i++}

$$files::calc::login<<${box_i++}

$$files::intergalactic-explain.tmplt<<

$$template::{
{{{advertPopups.content}}}
}

$$html:end_body_html<<
//
//
$$html:start_script<<

// global defs of some static topology
$$script::[alt-script]/flexy_items_A-top.js<<

//

$$script::[alt-script]/flexy_items_A-layout-and-menu.js<<

$$script::[alt-script]/flexy_items_A-animation.js<<
// CAPTCHA PERFORMANT
$$verbatim::{
//  === ---------------------------------------  === ---------------------------------------  === --------------------------------------
// APPLICATIONS
// -- -- -- -- -- -- -- -- -- -- -- -- --  -- -- -- -- -- -- -- -- -- -- -- -- --  -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
}
$$script::[app<scripts>]/site_app.js<<

$$script::[alt-script]/flexy_items_A-app-finalize.js<<

$$html:end_script<<

The tools that make this change are not part of roll-right. But, roll-right does pay attention to the bundling statements:

$$bundle::bundle_13.js
$$bundle::web3-mixed-boxy_bundle.js

Also, there is a CSS link, whereas before the file was included. In the first version of the skeleton, roll-right would have included the CSS in the output file. In the edited version, a link to the file will be included in the HTML header.

$$link<css>::shared_styles.css<<

configuration

The configuration for the first phase is used to pick a general structure by picking skeletons for concerns. The configuration will specify a common source for the files providing skeletal forms for all the concerns listed in that particular configuration. One configuration may specify a number of skeleton uses and output specializations for a number of concerns. Different configurations can be used all the same for collecting outputs into other targeted directories.

{
    "@target" : "templates/",
    "inputs" : {
        "path_abreviations" : {
            "[websites]" : "[alphas]/websites",
            "[PWA]" : "[alphas]/PWA-apps",
            "[alphas]" : "[github]/alphas",
            "[alpha-copious]" : "[alphas]/alpha-copious",
            "[github]" : "~/GitHub/",
            "[skeletons]" : "[alpha-copious]/pre-template",
            "[names]" : "[alpha-copious]/name-drops",
            "[PWA-lib]" : "[PWA]/lib",
            "[PWA-skeletons]" : "[PWA]/pre-template",
            "[client]" : "[alpha-copious]/client",
            "[template-configs]" : "[websites]/template-configs"
        }
    },
    "ext_default_dirs" : {
        "tmplt" : "[alpha-copious]/html",
        "js" : "[alpha-copious]/client",
        "svg" : "[alpha-copious]/icons"
    },
    "top_dir_locations" : {
        "script" : "[client]",
        "client" : "[client]",
        "pwa-lib" : "[PWA-lib]",
        "databases" : "[alpha-copious]/databases",
        "alt-script" : "[alpha-copious]/script",
        "for-humans" : "[alpha-copious]/for-humans",
        "html" : "[alpha-copious]/html",
        "css" : "[alpha-copious]/css",
        "icons" : "[alpha-copious]/icons",
        "names" : "[alpha-copious]/name-drops"
    },
    "use_case<[targets.dir]>" : {
        "app<scripts>" : "[targets.dir]",
        "app<static>" : "[targets.dir]",
        "app<images>" : "[targets.dir]"
    },
    "app_skel_vars" : {
        "boxy" : {
            "topic_count" : 3
        },
        "mixed" : {
            "topic_count" : 3,
            "groups" : [
                { "group_name" : "docs", "SOURCE-LINK" : "{{{shop_docs}}}", "FRAME-ACTIONS" : "onclick=''" },
                { "group_name" : "blog", "SOURCE-LINK" :  "{{{shop_blog}}}", "FRAME-ACTIONS" : "onclick=''" },
                { "group_name" : "search", "SOURCE-LINK" : "{{{shop_search}}}", "FRAME-ACTIONS" : "onclick=''" }
            ]
        },
        "shops" : {
            "topic_count" : 3,
            "groups" : [
                { "group_name" : "docs", "SOURCE-LINK" : "{{{shop_docs}}}", "FRAME-ACTIONS" : "onclick=''" },
                { "group_name" : "blog", "SOURCE-LINK" :  "{{{shop_blog}}}", "FRAME-ACTIONS" : "onclick=''" },
                { "group_name" : "search", "SOURCE-LINK" : "{{{shop_search}}}", "FRAME-ACTIONS" : "onclick=''" }
            ]
        }
    },
    "global_variable_values" : {
        "var_to_value" : {
            "AUTHOR" : "R. Leddy"
        },
        "by_concern" : {
            "copious" : {},
            "popsong" : {},
            "villa-family" : {},
            "bakersfield-robots": {},
            "docs.copious.world": {},
            "shops.copious.world": {},
            "shops.for-humans.net": {}
        }
    },
    "outputs" : [
        {
            "targets" : {
                "dir" : "[websites]/@concern/@kernel",
                "dir_form" : "[websites]/@concern/@target",
                "concerns" : ["copious","popsong","villa-family"],
                "uses_config_vars" : "boxy"

            },
            "skeletons" : {
                "index.tmplt" : "[skeletons]/web3-boxy.skel",
                "login.tmplt" : "[skeletons]/login-boxy.skel",
                "blog/index.tmplt" : "[skeletons]/galactic-blog.skel",
                "demos/index.tmplt" : "[skeletons]/galactic-blog.skel",
                "streams/index.tmplt" : "[skeletons]/galactic-blog.skel"
            },
            "name_parameters" : {
                "db" : "[names]/name-drop.db",
                "parameter_values" : "[websites]/@concern/@target/name-drop.json",
                "index.tmplt" : [ "contact_box", "about_box", "topicBox_<1,3>", "thankyou_box", "register" ],
                "login.tmplt" : [ "contact_box", "thankyou_box", "login" ]
            }
        },
        {
            "targets" : {
                "dir" : "[websites]/@concern/@kernel",
                "dir_form" : "[websites]/@concern/@target",
                "concerns" : ["bakersfield-robots","docs.copious.world"],
                "uses_config_vars" : "mixed"
            },
            "skeletons" : {
                "index.tmplt" : "[skeletons]/web3-mixed-boxy.skel",
                "login.tmplt" : "[skeletons]/login-boxy.skel",
                "blog/index.tmplt" : "[skeletons]/galactic-blog.skel",
                "demos/index.tmplt" : "[skeletons]/galactic-blog.skel"
            },
            "name_parameters" : {
                "db" : "[names]/name-drop.db",
                "parameter_values" : "[websites]/@concern/@target/name-drop.json",
                "index.tmplt" : [ "contact_box", "about_box", "topicBox_<1,3>", "thankyou_box", "register" ],
                "login.tmplt" : [ "contact_box", "thankyou_box", "login" ]
            }
        },
        {
            "targets" : {
                "dir" : "[websites]/@concern/@kernel",
                "dir_form" : "[websites]/@concern/@target",
                "concerns" : ["of-this.world","for-humans.net"]
            },
            "skeletons" : {
                "index.tmplt" : "[skeletons]/galactic.skel"
            },
            "name_parameters" : {
                "db" : "[names]/name-drop.db",
                "parameter_values" : "[websites]/@concern/@target/name-drop.json",
                "index.tmplt" : [ "contact_box", "about_box", "topicBox_<1,3>", "thankyou_box", "register" ],
                "login.tmplt" : [ "contact_box", "thankyou_box", "login" ]
            }
        },
        {
            "targets" : {
                "dir" : "[websites]/@concern/@kernel",
                "dir_form" : "[websites]/@concern/@target",
                "concerns" : ["shops.copious.world","shops.for-humans.net"],
                "uses_config_vars" : "shops"
            },
            "skeletons" : {
                "index.tmplt" : "[skeletons]/shop-kiosk.skel"
            },
            "name_parameters" : {
                "db" : "[names]/name-drop.db",
                "parameter_values" : "[websites]/@concern/@target/name-drop.json",
                "index.tmplt" : [ "contact_box", "about_box", "topicBox_<1,3>", "thankyou_box", "register" ],
                "login.tmplt" : [ "contact_box", "thankyou_box", "login" ]
            }
        },
        {
            "targets" : {
                "dir" : "[PWA]/@concern/@kernel",
                "dir_form" : "[PWA]/@concern/@target",
                "concerns" : ["safe-recorder"]
            },
            "skeletons" : {
                "index.tmplt" : "[skeletons]/PWA.skel",
                "recorder-PWA.tmplt" : "[PWA-skeletons]/recorder-PWA.skel",
                "ownership.tmplt" : "[PWA-skeletons]/ownership.skel"
            },
            "name_parameters" : {
                "db" : "[names]/name-drop.db"
            }
        }
    ]
}

How it Helps

Once the snippets, sub templates, skeletons, and code modules are in place, this tool generates entire sites with a carriage return. The project copious-software-dev-manager can call this tool. The software dev manager is web app that helps prepare the skeletons, fill out substitution maps, and finally release the software, doing some bundle management. This is all a work in progress.