Category Filtering: 'cli'

gadgetui.input.FileUploader example, Part II - the model.FileService server component

CFML, CLI, JavaScript

A FileUploader component does no good by itself. In order to provide upload capability, you need something on the server that can receive the uploaded files and process them. In Part I, I demonstrated a CFML page that handles the initial capture of the uploaded data and passes it to a component, model.FileService, for further processing. model.FileService isn't all that complicated, but it helps to walk through the process so you understand what is happening.

model.FileService.cfc


component output="false"{
	public function init( filePath, FilePartDAO ){
		variables.filePath = arguments.filePath;
		variables.viewFilePath = "/test/upload/";
		variables.FilePartDAO = arguments.FilePartDAO;
		variables.separator = iif(expandpath("./") contains "/",de("/"),de("\"));
		return this;
	}
	public function upload( required string id,
		required any temp_file,
		required numeric part,
		required numeric parts,
		required string filename,
		required string filesize ){

		local.result = {};
		local.addedPart = variables.FilePartDAO.create( fileid = arguments.id, filepath = arguments.temp_file, 
                                             filepart = arguments.part, parts = arguments.parts );

		if( arguments.part eq arguments.parts ){
			if( server.OS.name contains 'Windows' ){
				local.str = " copy /b ";
				local.files = variables.FileParDAO.readByFileId( fileid = arguments.id );
				local.str = local.str & local.files.filepath[ 1 ];
				for( local.ix = 2; local.ix lte local.files.recordcount; local.ix++ ){
					local.str = local.str & "+" & local.files.filepath[ local.ix ];
				}
				local.str = local.str & " ""#variables.filePath##arguments.filename#""";

				local.runtime = createObject("java", "java.lang.Runtime");
				local.process_runtime = local.runtime.getRuntime();
				local.process_exec = local.process_runtime.exec( javacast( "string[]", ["cmd.exe", "/c", local.str]) );
				local.exitCode = local.process_exec.waitFor();
			}else{
				local.str = " cat ";
				local.files = variables.FilePartDAO.readByFileId( fileid = arguments.id );
				for( local.ix = 1; local.ix lte local.files.recordcount; local.ix++ ){
					local.str = local.str & " " & local.files.filepath[ local.ix ];

				}
				local.str = local.str & " > ""#variables.filePath##arguments.filename#""";
				//writelog( file="upload", text="Concatenating: " & local.str );
				local.runtime = createObject("java", "java.lang.Runtime");
				local.process_runtime = local.runtime.getRuntime();
				local.process_exec = local.process_runtime.exec( javacast( "string[]", ["bash", "-c", local.str]) );
				local.exitCode = local.process_exec.waitFor();
			}
			//clean up
			for( local.ix = 1; local.ix lte local.files.recordcount; local.ix++ ){
				if( fileExists( local.files.filepath[ local.ix ] ) ){
					fileDelete( local.files.filepath[ local.ix ] );
				}
			}
			//clean up
			variables.FilePartDAO.delete( fileid = arguments.id );
		}

		local.result.path = variables.viewFilePath;
		//local.result.tags = arguments.tags;
		local.result.filename = arguments.filename;
		local.result.disabled = 0;
		local.result.filesize = arguments.filesize;
		local.result.mimetype = "application/octet-stream";
		local.result.created = now();

		return local.result;
	}
}

First let's check on the init() method that instantiates the component. Mostly it is fairly obvious what is going on- we configure the component with upload and view paths according to our server configuration. The only non-obvious configuration is FilePartDAO, which is just what its name implies - a DAO that manages persistence of filepart information to the database. Technically, we don't need a database to store this information, we could just store it in a session variable. However, persisting the data provides the underpinning for a broken upload resume feature that I am planning to add soon.

Next, let's review the upload() method. First, the method signature:

	public function upload( required string id,
		required any temp_file,
		required numeric part,
		required numeric parts,
		required string filename,
		required string filesize )

The signature shows what we are dealing with as input. An id (in this case, a generated UUID for each filepart), a path to the filepart on disk, the index and total number of parts, the original filename and filesize. Nothing too complicated there.

What you will notice next is that the method does almost nothing until you get to the last part of the fileparts in the upload. It basically just persists the filepart information to the database. Once you get to the last part, though, the upload method checks whether you are in a Windows or *nix environment, loops over the records of fileparts and assembles a CLI string to use to concatenate the fileparts into a single file. It then instantiates a Java Runtime object and invokes a CLI interpreter (cmd or bash for Windows or *nix) with the concatenate statement as an argument. 

What does that all mean? It means the page invokes a command line statement to put the file back together in the final upload destination. Personally, I have found this method to be an order of magnitude faster than trying to use native file commands in CFML to re-assemble the uploaded file. 

Once the file is re-assembled, the method goes through the temporary file parts and deletes them, then deletes the record of them in the database, then returns a struct with the final uploaded file's metadata. That information is then sent back to the browser as the final reply to the upload function

To be clear, this is a vanilla implementation of the upload function, meant mainly to demonstrate a basic way of capturing and storing the uploaded file and its metadata. I am available for consulting engagements if you would like a more complex implementation, such as storing the file in a NoSQL store like MongoDB.

 


gadget-ui.input.FileUploader example

CFML, CLI, JavaScript

I'm going to demonstrate how to use the new gadget-ui.input.FileUploader component in the gadget-ui library. First, we need an HTML page to host the component. This code comes directly from the /test folder in the gadget-ui repo:

fileuploader.htm

<!doctype html>
<html>

<head>
	<title>gadget-ui File Uploader Test</title>
	<script src='../bower_components/modlazy/dist/modlazy.min.js'></script>
	<script>
		modlazy.load(
			[
				"fileuploader.js < ../dist/gadget-ui.js",
				"../dist/gadget-ui.css"
			],
			function() {
				console.log("All files have been loaded");
			}
		);
	</script>

	<style>
		body {
			font-size: 1em;
		}

		input,
		select,
		select option {
			font-size: 1em;
		}

		#fileUploadDiv {
			height: 500px;
			width: 500px;
			text-align: center;
			margin-top: auto;
			margin-bottom: auto;
		}
	</style>

</head>

<body>
	<p>Test the FileUploader control.</p>

	<div id="fileUploadDiv"></div>
</body>

</html>

I am using a small dependency loader called modlazy to load my assets. This page shows a div called fileUploadDiv that will host the component. Let's look at the fileuploader.js file:

fileuploader.js


var options = {
  uploadURI: "/test/fileuploader.upload.cfm",
  tags: "file upload",
  willGenerateThumbnails: true,
  title: "Upload Files",
  showUploadButton: false
};

filedialog = gadgetui.objects.Constructor( gadgetui.input.FileUploader, [ document.querySelector("#fileUploadDiv"), options ]);

The uploadURI specifies a CFML page  that should be compatible with Lucee and Adobe ColdFusion, although I am testing it with Lucee, so there could be bugs in the ACF implementation right now. We'll dig into that page in a minute.

Right now, what I want to show is how the component is being instantiated. Rather than use the new keyword, I have ported all of he gadget-ui code to use the built-in gadgetui.objects.Constructor method, which internally uses Object.create(). I have also included a new option, showUploadButton, so you can either have a drop zone, or have a drop zone and a file upload button.

On drop  or file select, the component cuts the file up into 1MB chunks and uploads the chunks one at a time. Let's have a look at the CFML receiver file:

fileuploader.upload.cfm

<cfscript>
	filePath = "#expandpath("./")#upload\";
	separator = iif(expandpath("./") contains "/",de("/"),de("\"));
	filePath = expandpath("./") & "upload" & separator;
	chunkPath = filePath & "temp" & separator & createUUID();
	inputStream = getPageContext().getRequest().getOriginalRequest().getInputStream();
	ioutil = createObject( "java", "org.apache.commons.io.IOUtils" );
	outputStream = createObject( "java", "java.io.FileOutputStream" ).init( chunkPath );
	ioutil.copy( inputStream, outputStream );

	contentLength = gethttprequestdata().headers['Content-Length'];
	args.id = gethttprequestdata().headers['x-id'];
	args.filename = gethttprequestdata().headers['x-filename'];
	args.filesize = gethttprequestdata().headers['x-filesize'];
	args.part = gethttprequestdata().headers['x-filepart'];
	args.parts = gethttprequestdata().headers['x-parts'];

	if( args.part eq 1 ){
		fileId = createUUID();
	}else{
		fileId = args.id;
	}
	args.temp_file = chunkPath;

	args.id = fileId;
	FilePartDAO = new model.FilePartDAO();
	FileService = new model.FileService( filePath = filePath, FilePartDAO = FilePartDAO );
	file = FileService.upload( argumentcollection = args );

	if( args.part eq args.parts ){
		writeoutput( SerializeJSON( file ) );
	}else{
		pc = getpagecontext();
		pc.setHeader("X-Id", fileId );
	}

</cfscript>

There is a lot going on in this code. Look at the use of Java input and output streams to copy the chunked data. I use Java classes here to guarantee good data coming through from the binary data POST. Note that I am using UUIDs to identify the chunks and store them in a temporary folder on the server, where they stay until they are re-assembled into the original file.

Also see the use of gethttprequestdata() to pull the headers for the chunked data. The x- headers are all custom headers created by the FileUploader component to track the file upload. x-filepart tells the server which part of the file is being uploaded, and x-parts tells the server how many total parts there are.

Next, note that this code takes place in a loop controlled by the FileUploader component. When the end of the loop is reached, the file upload is completed and a response is sent back to the client. 

The real work of the upload is being done in the FileService CFML component. This example uses a baseline implementation that uploads the file(s) to a destination folder on the server. Other implementations may store the file in MongoDB or other NoSQL stores. I will dig into the FileService component next time.


Creating a Blog site with ContentBox on Ubuntu

CFML, CLI, CommandBox, ContentBox, Linux, Ubuntu

[ This post is part of my ongoing instructional series on setting up some baseline IT infrastructure for the fictional startup Shoestring Lab. Shoestring has committed to using Open Source wherever possible. Shoestring Lab has standardized on Ubuntu for its server and desktop/laptop computer systems.

Today's lesson

Shoestring Lab has decided to set up several website for its operations, and you have been tasked with using ContentBox to build the first site, an internal content and blog site.]


Installing CommandBox on Ubuntu 18.04 to manage CFML-based Web app development

CLI, CommandBox, Lucee, Open Source, Ubuntu

[ This post is part of my ongoing instructional series on setting up some baseline IT infrastructure for the fictional startup Shoestring Lab. Shoestring has committed to using Open Source wherever possible. Shoestring Lab has standardized on Ubuntu for its server and desktop/laptop computer systems.

Today's lesson

Part of your job is setting up and managing internal and external web sites for the company. Your task is to install and configure CommandBox to manage the Lucee-based sites you are building.]


Install and Configure OpenLDAP on Ubuntu 18.04

CLI, Linux, OpenLDAP, Ubuntu

[ This post is part of my ongoing instructional series on setting up some baseline IT infrastructure for the fictional startup Shoestring Lab. Shoestring has committed to using Open Source wherever possible. Shoestring Lab has standardized on Ubuntu for its server and desktop/laptop computer systems.

Today's lesson

As the admin for Shoestring Lab, you need to install OpenLDAP server to manage users and groups for various network services.]


Verifying Checksums via CLI

CLI

When you download files from the Internet, particularly source files for programs, or compiled binaries, you may often see checksums next to the download links or contained in files in an FTP server. Open source projects, for instance, often include checksums for compressed versions of their source code for particular releases. 


Compiling Nginx from source on Ubuntu - April 2016 Update

CLI, Nginx, Ubuntu

Compiling Nginx, like other open source projects, tends to be a bit of a moving target. New versions are regularly released, dependent libraries are updated, and new modules become available over time. SPDY has been retired in favor of HTTP v2, and that changes is reflected in the latest build target for Nginx.


JavaScript Automation, Part III - Configuring Grunt Tasks

CLI, JavaScript

Having installed and configured Grunt in Part I and Part II of our series on automating JavaScript builds, we proceed to the heart of the matter - configuring Grunt tasks to automate away the tedium of building our JavaScript client app from source files.


JavaScript Automation, Part II - Configuring Grunt

CLI, JavaScript

Building complex JavaScript front-ends is complicated. Grunt can help you improve your code/debug lifecycle by automating a lot of repetitive tasks that otherwise take up a lot of your time. If you are not familiar with Grunt yet, read Part I - Setting up Grunt. Once you have Grunt installed in your project, proceed to the next step - Grunt configuration.


JavaScript Automation, Part I - Setting Up Grunt

CLI, JavaScript

if you work on single-page apps and other JavaScript-centric Web development, you quickly learn that the code-debug cycle in client-side JavaScript is anything but automatic. In these kinds of environments, you end up with enough code that several things happen ...


Categories


Recent Entries

Entries Search