UI library gadget-ui updated with ES6 module support

gadget-ui, JavaScript, Open Source, Web Development

I just published gadget-ui v. 6.0.0. There are no significant changes or additions to the library itself, but I have added ES6 modules exports to support importing the library, in whole or in part, using the ES6 import command.


tasklist - Client Application for a NodeJS REST API- Part I

JavaScript, NodeJS

A few days ago, I posted a NodeJS application built around a REST API, and some further thoughts around architecture with the NodeJS mysql package. I have pushed the complete tasklist application to GitHub. If you want to grab it and see it run, clone the repo:

$ git clone https://github.com/robertdmunn/tasklist

Today, I am going to start a short series of posts going into some detail about the client application and how it interacts with the REST API.

Warning

Before I get started, let me be clear about what I am showing. The client application is an example implementation of a JavaScript framework I wrote last year (now almost two years ago) as an academic exercise, general exploration of some features of JavaScript, and bringing together of some things I had written in the past. That framework, called altseven, is now available on GitHub:

$ git clone https://github.com/robertdmunn/altseven

and can be installed through npm:

$ npm install altseven

altseven relies on several external packages, including an experimental UI+ library called gadget-ui I started a few years ago and have worked on sporadically ever since. Also available separately on GitHub and npm, if you are interested.

There might be an element or two of gadget-ui being run in production, but mostly it's just a playground for ideas that I have toyed with. You can see my posts about the gadgetui.input.FileUploader component, which is one of the things that might actually be in production use. 

Enough with the warnings, let's look at some code.

Serving Static Content with NodeJS

In my previous posts about the REST API in NodeJS,  I left out a piece of code in the index.js that serves static content for the client application, since it wasn't relevant to the topic. Here is the index file with the missing code:

index.js
----------


const express = require( 'express' );

const app = express();

// map our client-side libraries
app.use( express.static( 'client' ) );
app.use( "/lib/gadget-ui", express.static( 'node_modules/gadget-ui' ) );
app.use( "/lib/altseven", express.static( 'node_modules/altseven' ) );
app.use( "/lib/modlazy", express.static( 'node_modules/modlazy' ) );
app.use( "/lib/velocity", express.static( 'bower_components/velocity' ) );
app.use( "/lib/mustache.js", express.static( 'bower_components/mustache.js' ) );
app.use( "/lib/open-iconic", express.static( 'node_modules/open-iconic' ) );

// routes for the API
require( './routes/tasks.js' )(app);

// set our listener
var server = app.listen( 4000, function(){

});

Let's examine what's going on here.

First, you can see we're using Express in the Node application. Fortunately, Express includes the app.use() and express.static() functions to make it easy. To serve static content to the root of the application, we see the first line:

app.use( express.static( 'client' ) );

By passing the express.static( 'client' ) argument without a corresponding path to app.use(), we set the 'client' folder as the root of the static content that will be served by NodeJS.

Following the root folder, we specify both a path from the root and a static folder to serve on that path:

app.use( "/lib/gadget-ui", express.static( 'node_modules/gadget-ui' ) );

But why not use Browserify or Webpack? That's certainly something that can be done, and I'll cover it in a future post. For now, app.use() with express.static() is an easy way to serve static assets from NodeJS without a Web server.

Now let's look at the "old school" index.html, complete with dependency loader.

index.html
--------------

<html>
  <head>
    <title>
      NodeJS Task List
    </title>
    <script src="/lib/modlazy/dist/modlazy.min.js"></script>
    <script>
      modlazy.load(["/js/app.components.js < /js/app.remote.js < /js/app.utils.js < /js/app.main.js < /js/app.events.js < /lib/altseven/dist/a7.js < /lib/gadget-ui/dist/gadget-ui.js < /lib/velocity/velocity.js ",
        "/lib/mustache.js/mustache.js", "/css/styles.css", "/lib/altseven/dist/a7.css",
        "/lib/gadget-ui/dist/gadget-ui.css",
        "/lib/open-iconic/font/css/open-iconic.css"
      ], function() {
        var
          options = {
            auth: { // sessionTimeout: ( 60 * 15 * 1000 ) // default time in  milliseconds to refresh system auth
            },
            console: {
              enabled: true,
              wsServer: 'ws://127.0.0.1:8000',
              top: 100,
              left: 800,
              height: 300,
              width: 500
            },
            logging: {
              logLevel: "INFO,ERROR,FATAL,TRACE"
            },
            remote: {
              modules: app.remote,
              useTokens: false // defaults to true for the auth system
            },
            ui: { // renderer: // renderer is implicitly set by existence of the templating library, currently Mustache or Handlebars
              templates: "/templates.html"
            }
          };
        var p = new Promise(function(resolve,
          reject) {
          a7.init(options, resolve, reject);
        });
        p.then(function(state) {
          app.main.init(state);
        });
        p['catch'](function(message) {
          console.log(
            "Something went wrong.");
        });
      });
    </script>
  </head>

  <body>
    <div name="main">
  		<div name="app" >

  		</div>

  	</div>

  </body>
</html>

We don't need to spend too much time on the loading of the assets. modlazy is a dependency loader that guarantees loading of your JS dependencies in the correct order. The arrows "<" indicate dependency, so assets to the left of the arrow only load once assets to the right of the arrow have loaded. For assets like CSS files that have no dependencies, you can simply specify them in a quoted list, as shown.

More importantly, let's focus on defining the options for our application and the a7.init() function. As you can see in the options{} definition, altseven is divided into a number of packages- components, console, error, events, logging, model, remote, security, ui, and util. (auth is technically not a package but a function in the remote package, but it has a configuration option for session timeout).

Console is a package that renders as a floating window pane in the application. It displays a running list of entries from the log package. Optionally, it can also be configured to display a running list of log entries pushed from a web socket server. You can see a simple NodeJS socket server in my post Real-time server debug info using NodeJS, jQuery, Websockets and Logbox. I plan on coming back to this topic later, so feel free to ignore it for now. The gist of it is that you can include server-side logging information - or anything that can be pushed to a socket connection - in the console.

The logging package does just what it says- handles log statements in the application code. logLevel specifies what log levels from the code will be parsed and sent to the console.

The model package isn't listed in the options for this application, but it has a configuration option that can be used to set the model to any model package of your choosing. By default it uses the model in the gadget-ui library.

The remote package handles remote calls to the server. It has a built-in auth function that can be used with the built-in components.User object to provide authentication services against a server and set the application into secure mode. We're not using it in the tasklist application, but I will re-visit it later. Specify the application's remote modules using the modules option. The useTokens option can be set to true if you need an authenticated session with the server. It handles auth tokens that can be set by the server and passed back and forth for secure sessions.

The ui package handles ui templating and objects. It currently supports Mustache and Handlebars, but might include React support soon (hint hint). Load the Mustache library as part of your application to use Mustache, load Handlebars to use Handlebars.

Lastly for this post, the a7.init() call for altseven needs to be made inside a Promise. The a7.init() call resolves or rejects the promise and allows you to then proceed with running your application.

That's it for now. Next, I'll cover the client application code in more detail.


Exploring Architecture with the NodeJS mysql package

JavaScript, MySQL, NodeJS

In software design, architectural choices have a significant influence on the maintainability of your code base. Choosing the proper toolkit, while important, is not enough to guarantee a maintainable solution. Frameworks attempt to solve this problem by imposing an architecture on your code base that (hopefully) uses best practice patterns in software design to help you build a maintainable code base. I will return to the issue of frameworks (and their strengths and weaknesses) another day. 

Meanwhile, let's explore some architectural design choices we can make with the NodeJS mysql package when building CRUD applications. You can refer to my post on Creating a NodeJS REST API with Express and MySQL as a base design blueprint for building a CRUD application using the MySQL package. The application in the post makes several design choices that are not explained in any detail. Let's explore them now.


Creating a NodeJS REST API with Express and MySQL

JavaScript, NodeJS, Open Source, Web Development

NodeJS offers the ability to build Web apps using JavaScript. Today, I will show you how to build a NodeJS REST API for a task list using Express and MySQL.


gadgetui.input.FileUploader example - Part III - database and CFML engine

CFML, CommandBox, JavaScript

In parts I and II of my FileUploader example code, I showed you the client-side and server-side code for the FileUploader component and some example code for a CFML-based server-component to handle the uploaded data. What we haven't covered yet is the database component. There isn't very much to it - just a single table - and it doesn't really even need to be in a database, I just built it that way to enable persistence in broken uploads in case your browse crashes during the upload. I haven't actually finished that piece of development yet, but the underpinnings are there to make it work.

So, getting right to the point, here is a SQL create statement for a table in MySQL/MariaDB  to handle the file upload metadata. Note that this table is only used during the upload process to track the individual fileparts before they are stitched back together into the resulting uploaded file. 

CREATE TABLE `filepart` (
  `fileId` varchar(50) NOT NULL,
  `filepath` varchar(500) NOT NULL,
  `filepart` int(11) NOT NULL,
  `parts` int(11) NOT NULL,
  PRIMARY KEY (`fileId`,`filepart`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

I'm using UTF-8 and InnoDB, as seen here. You can use whatever SQL back-end you choose. You could even use a NoSQL back-end, but you would need to write your own server-side code for it. I have written a set of server-side components (in CFML) to store uploaded files (not the fileparts, the actual files) and file metadata in MongoDB. That code is available for licensing for anyone who is interested in a more scalable solution than uploading to the file system.

To run the FileUploader example, you will need to create this table in a database and add a datasource to it using the Admin application for your CFML engine. If you are new to CFML engines, the easiest way to get the example running is to install CommandBox and run it that way. Some useful links to get you started:

Installing CommandBox

Running local development services with CommandBox

Logging into Lucee Admin of a New Instance

 


So I created a JavaScript framework ...

JavaScript, Open Source

Last year, and largely as an academic exercise, I created a JavaScript framework.


gadgetui.input.FileUploader example, Part II - the model.FileService server component

CFML, CLI, JavaScript

A FileUploader component does no good by itself. In order to provide upload capability, you need something on the server that can receive the uploaded files and process them. In Part I, I demonstrated a CFML page that handles the initial capture of the uploaded data and passes it to a component, model.FileService, for further processing. model.FileService isn't all that complicated, but it helps to walk through the process so you understand what is happening.

model.FileService.cfc


component output="false"{
	public function init( filePath, FilePartDAO ){
		variables.filePath = arguments.filePath;
		variables.viewFilePath = "/test/upload/";
		variables.FilePartDAO = arguments.FilePartDAO;
		variables.separator = iif(expandpath("./") contains "/",de("/"),de("\"));
		return this;
	}
	public function upload( required string id,
		required any temp_file,
		required numeric part,
		required numeric parts,
		required string filename,
		required string filesize ){

		local.result = {};
		local.addedPart = variables.FilePartDAO.create( fileid = arguments.id, filepath = arguments.temp_file, 
                                             filepart = arguments.part, parts = arguments.parts );

		if( arguments.part eq arguments.parts ){
			if( server.OS.name contains 'Windows' ){
				local.str = " copy /b ";
				local.files = variables.FileParDAO.readByFileId( fileid = arguments.id );
				local.str = local.str & local.files.filepath[ 1 ];
				for( local.ix = 2; local.ix lte local.files.recordcount; local.ix++ ){
					local.str = local.str & "+" & local.files.filepath[ local.ix ];
				}
				local.str = local.str & " ""#variables.filePath##arguments.filename#""";

				local.runtime = createObject("java", "java.lang.Runtime");
				local.process_runtime = local.runtime.getRuntime();
				local.process_exec = local.process_runtime.exec( javacast( "string[]", ["cmd.exe", "/c", local.str]) );
				local.exitCode = local.process_exec.waitFor();
			}else{
				local.str = " cat ";
				local.files = variables.FilePartDAO.readByFileId( fileid = arguments.id );
				for( local.ix = 1; local.ix lte local.files.recordcount; local.ix++ ){
					local.str = local.str & " " & local.files.filepath[ local.ix ];

				}
				local.str = local.str & " > ""#variables.filePath##arguments.filename#""";
				//writelog( file="upload", text="Concatenating: " & local.str );
				local.runtime = createObject("java", "java.lang.Runtime");
				local.process_runtime = local.runtime.getRuntime();
				local.process_exec = local.process_runtime.exec( javacast( "string[]", ["bash", "-c", local.str]) );
				local.exitCode = local.process_exec.waitFor();
			}
			//clean up
			for( local.ix = 1; local.ix lte local.files.recordcount; local.ix++ ){
				if( fileExists( local.files.filepath[ local.ix ] ) ){
					fileDelete( local.files.filepath[ local.ix ] );
				}
			}
			//clean up
			variables.FilePartDAO.delete( fileid = arguments.id );
		}

		local.result.path = variables.viewFilePath;
		//local.result.tags = arguments.tags;
		local.result.filename = arguments.filename;
		local.result.disabled = 0;
		local.result.filesize = arguments.filesize;
		local.result.mimetype = "application/octet-stream";
		local.result.created = now();

		return local.result;
	}
}

First let's check on the init() method that instantiates the component. Mostly it is fairly obvious what is going on- we configure the component with upload and view paths according to our server configuration. The only non-obvious configuration is FilePartDAO, which is just what its name implies - a DAO that manages persistence of filepart information to the database. Technically, we don't need a database to store this information, we could just store it in a session variable. However, persisting the data provides the underpinning for a broken upload resume feature that I am planning to add soon.

Next, let's review the upload() method. First, the method signature:

	public function upload( required string id,
		required any temp_file,
		required numeric part,
		required numeric parts,
		required string filename,
		required string filesize )

The signature shows what we are dealing with as input. An id (in this case, a generated UUID for each filepart), a path to the filepart on disk, the index and total number of parts, the original filename and filesize. Nothing too complicated there.

What you will notice next is that the method does almost nothing until you get to the last part of the fileparts in the upload. It basically just persists the filepart information to the database. Once you get to the last part, though, the upload method checks whether you are in a Windows or *nix environment, loops over the records of fileparts and assembles a CLI string to use to concatenate the fileparts into a single file. It then instantiates a Java Runtime object and invokes a CLI interpreter (cmd or bash for Windows or *nix) with the concatenate statement as an argument. 

What does that all mean? It means the page invokes a command line statement to put the file back together in the final upload destination. Personally, I have found this method to be an order of magnitude faster than trying to use native file commands in CFML to re-assemble the uploaded file. 

Once the file is re-assembled, the method goes through the temporary file parts and deletes them, then deletes the record of them in the database, then returns a struct with the final uploaded file's metadata. That information is then sent back to the browser as the final reply to the upload function

To be clear, this is a vanilla implementation of the upload function, meant mainly to demonstrate a basic way of capturing and storing the uploaded file and its metadata. I am available for consulting engagements if you would like a more complex implementation, such as storing the file in a NoSQL store like MongoDB.

 


gadget-ui.input.FileUploader example

CFML, CLI, JavaScript

I'm going to demonstrate how to use the new gadget-ui.input.FileUploader component in the gadget-ui library. First, we need an HTML page to host the component. This code comes directly from the /test folder in the gadget-ui repo:

fileuploader.htm

<!doctype html>
<html>

<head>
	<title>gadget-ui File Uploader Test</title>
	<script src='../bower_components/modlazy/dist/modlazy.min.js'></script>
	<script>
		modlazy.load(
			[
				"fileuploader.js < ../dist/gadget-ui.js",
				"../dist/gadget-ui.css"
			],
			function() {
				console.log("All files have been loaded");
			}
		);
	</script>

	<style>
		body {
			font-size: 1em;
		}

		input,
		select,
		select option {
			font-size: 1em;
		}

		#fileUploadDiv {
			height: 500px;
			width: 500px;
			text-align: center;
			margin-top: auto;
			margin-bottom: auto;
		}
	</style>

</head>

<body>
	<p>Test the FileUploader control.</p>

	<div id="fileUploadDiv"></div>
</body>

</html>

I am using a small dependency loader called modlazy to load my assets. This page shows a div called fileUploadDiv that will host the component. Let's look at the fileuploader.js file:

fileuploader.js


var options = {
  uploadURI: "/test/fileuploader.upload.cfm",
  tags: "file upload",
  willGenerateThumbnails: true,
  title: "Upload Files",
  showUploadButton: false
};

filedialog = gadgetui.objects.Constructor( gadgetui.input.FileUploader, [ document.querySelector("#fileUploadDiv"), options ]);

The uploadURI specifies a CFML page  that should be compatible with Lucee and Adobe ColdFusion, although I am testing it with Lucee, so there could be bugs in the ACF implementation right now. We'll dig into that page in a minute.

Right now, what I want to show is how the component is being instantiated. Rather than use the new keyword, I have ported all of he gadget-ui code to use the built-in gadgetui.objects.Constructor method, which internally uses Object.create(). I have also included a new option, showUploadButton, so you can either have a drop zone, or have a drop zone and a file upload button.

On drop  or file select, the component cuts the file up into 1MB chunks and uploads the chunks one at a time. Let's have a look at the CFML receiver file:

fileuploader.upload.cfm

<cfscript>
	filePath = "#expandpath("./")#upload\";
	separator = iif(expandpath("./") contains "/",de("/"),de("\"));
	filePath = expandpath("./") & "upload" & separator;
	chunkPath = filePath & "temp" & separator & createUUID();
	inputStream = getPageContext().getRequest().getOriginalRequest().getInputStream();
	ioutil = createObject( "java", "org.apache.commons.io.IOUtils" );
	outputStream = createObject( "java", "java.io.FileOutputStream" ).init( chunkPath );
	ioutil.copy( inputStream, outputStream );

	contentLength = gethttprequestdata().headers['Content-Length'];
	args.id = gethttprequestdata().headers['x-id'];
	args.filename = gethttprequestdata().headers['x-filename'];
	args.filesize = gethttprequestdata().headers['x-filesize'];
	args.part = gethttprequestdata().headers['x-filepart'];
	args.parts = gethttprequestdata().headers['x-parts'];

	if( args.part eq 1 ){
		fileId = createUUID();
	}else{
		fileId = args.id;
	}
	args.temp_file = chunkPath;

	args.id = fileId;
	FilePartDAO = new model.FilePartDAO();
	FileService = new model.FileService( filePath = filePath, FilePartDAO = FilePartDAO );
	file = FileService.upload( argumentcollection = args );

	if( args.part eq args.parts ){
		writeoutput( SerializeJSON( file ) );
	}else{
		pc = getpagecontext();
		pc.setHeader("X-Id", fileId );
	}

</cfscript>

There is a lot going on in this code. Look at the use of Java input and output streams to copy the chunked data. I use Java classes here to guarantee good data coming through from the binary data POST. Note that I am using UUIDs to identify the chunks and store them in a temporary folder on the server, where they stay until they are re-assembled into the original file.

Also see the use of gethttprequestdata() to pull the headers for the chunked data. The x- headers are all custom headers created by the FileUploader component to track the file upload. x-filepart tells the server which part of the file is being uploaded, and x-parts tells the server how many total parts there are.

Next, note that this code takes place in a loop controlled by the FileUploader component. When the end of the loop is reached, the file upload is completed and a response is sent back to the client. 

The real work of the upload is being done in the FileService CFML component. This example uses a baseline implementation that uploads the file(s) to a destination folder on the server. Other implementations may store the file in MongoDB or other NoSQL stores. I will dig into the FileService component next time.


gadget-ui JavaScript library Updated

JavaScript

I have been working on an update of the gadget-ui JavaScript library that includes a new feature - a multi-file upload component called gadgetui.input.FileUploader. The component is based on some earlier work of mine, and originally used jQuery, so the jQuery version was written first and sticks to the original. 

The plain JavaScript version is a port of the jQuery version, and I am moving ahead with development on the plain JS version. The original component instantiated inside a dialog box with a close button, and I have eliminated those features in the plain JS version, opting to allow the developer to instantiate the component wherever needed, and managing its lifecycle as required. You can clone the library from my gadget-ui GitHub repository.


Creating a Blog site with ContentBox on Ubuntu

CFML, CLI, CommandBox, ContentBox, Linux, Ubuntu

[ This post is part of my ongoing instructional series on setting up some baseline IT infrastructure for the fictional startup Shoestring Lab. Shoestring has committed to using Open Source wherever possible. Shoestring Lab has standardized on Ubuntu for its server and desktop/laptop computer systems.

Today's lesson

Shoestring Lab has decided to set up several website for its operations, and you have been tasked with using ContentBox to build the first site, an internal content and blog site.]


Categories


Recent Entries

Entries Search