Category Filtering: 'cfml'

gadgetui.input.FileUploader example - Part III - database and CFML engine

CFML, CommandBox, JavaScript

In parts I and II of my FileUploader example code, I showed you the client-side and server-side code for the FileUploader component and some example code for a CFML-based server-component to handle the uploaded data. What we haven't covered yet is the database component. There isn't very much to it - just a single table - and it doesn't really even need to be in a database, I just built it that way to enable persistence in broken uploads in case your browse crashes during the upload. I haven't actually finished that piece of development yet, but the underpinnings are there to make it work.

So, getting right to the point, here is a SQL create statement for a table in MySQL/MariaDB  to handle the file upload metadata. Note that this table is only used during the upload process to track the individual fileparts before they are stitched back together into the resulting uploaded file. 

CREATE TABLE `filepart` (
  `fileId` varchar(50) NOT NULL,
  `filepath` varchar(500) NOT NULL,
  `filepart` int(11) NOT NULL,
  `parts` int(11) NOT NULL,
  PRIMARY KEY (`fileId`,`filepart`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

I'm using UTF-8 and InnoDB, as seen here. You can use whatever SQL back-end you choose. You could even use a NoSQL back-end, but you would need to write your own server-side code for it. I have written a set of server-side components (in CFML) to store uploaded files (not the fileparts, the actual files) and file metadata in MongoDB. That code is available for licensing for anyone who is interested in a more scalable solution than uploading to the file system.

To run the FileUploader example, you will need to create this table in a database and add a datasource to it using the Admin application for your CFML engine. If you are new to CFML engines, the easiest way to get the example running is to install CommandBox and run it that way. Some useful links to get you started:

Installing CommandBox

Running local development services with CommandBox

Logging into Lucee Admin of a New Instance

 


gadgetui.input.FileUploader example, Part II - the model.FileService server component

CFML, CLI, JavaScript

A FileUploader component does no good by itself. In order to provide upload capability, you need something on the server that can receive the uploaded files and process them. In Part I, I demonstrated a CFML page that handles the initial capture of the uploaded data and passes it to a component, model.FileService, for further processing. model.FileService isn't all that complicated, but it helps to walk through the process so you understand what is happening.

model.FileService.cfc


component output="false"{
	public function init( filePath, FilePartDAO ){
		variables.filePath = arguments.filePath;
		variables.viewFilePath = "/test/upload/";
		variables.FilePartDAO = arguments.FilePartDAO;
		variables.separator = iif(expandpath("./") contains "/",de("/"),de("\"));
		return this;
	}
	public function upload( required string id,
		required any temp_file,
		required numeric part,
		required numeric parts,
		required string filename,
		required string filesize ){

		local.result = {};
		local.addedPart = variables.FilePartDAO.create( fileid = arguments.id, filepath = arguments.temp_file, 
                                             filepart = arguments.part, parts = arguments.parts );

		if( arguments.part eq arguments.parts ){
			if( server.OS.name contains 'Windows' ){
				local.str = " copy /b ";
				local.files = variables.FileParDAO.readByFileId( fileid = arguments.id );
				local.str = local.str & local.files.filepath[ 1 ];
				for( local.ix = 2; local.ix lte local.files.recordcount; local.ix++ ){
					local.str = local.str & "+" & local.files.filepath[ local.ix ];
				}
				local.str = local.str & " ""#variables.filePath##arguments.filename#""";

				local.runtime = createObject("java", "java.lang.Runtime");
				local.process_runtime = local.runtime.getRuntime();
				local.process_exec = local.process_runtime.exec( javacast( "string[]", ["cmd.exe", "/c", local.str]) );
				local.exitCode = local.process_exec.waitFor();
			}else{
				local.str = " cat ";
				local.files = variables.FilePartDAO.readByFileId( fileid = arguments.id );
				for( local.ix = 1; local.ix lte local.files.recordcount; local.ix++ ){
					local.str = local.str & " " & local.files.filepath[ local.ix ];

				}
				local.str = local.str & " > ""#variables.filePath##arguments.filename#""";
				//writelog( file="upload", text="Concatenating: " & local.str );
				local.runtime = createObject("java", "java.lang.Runtime");
				local.process_runtime = local.runtime.getRuntime();
				local.process_exec = local.process_runtime.exec( javacast( "string[]", ["bash", "-c", local.str]) );
				local.exitCode = local.process_exec.waitFor();
			}
			//clean up
			for( local.ix = 1; local.ix lte local.files.recordcount; local.ix++ ){
				if( fileExists( local.files.filepath[ local.ix ] ) ){
					fileDelete( local.files.filepath[ local.ix ] );
				}
			}
			//clean up
			variables.FilePartDAO.delete( fileid = arguments.id );
		}

		local.result.path = variables.viewFilePath;
		//local.result.tags = arguments.tags;
		local.result.filename = arguments.filename;
		local.result.disabled = 0;
		local.result.filesize = arguments.filesize;
		local.result.mimetype = "application/octet-stream";
		local.result.created = now();

		return local.result;
	}
}

First let's check on the init() method that instantiates the component. Mostly it is fairly obvious what is going on- we configure the component with upload and view paths according to our server configuration. The only non-obvious configuration is FilePartDAO, which is just what its name implies - a DAO that manages persistence of filepart information to the database. Technically, we don't need a database to store this information, we could just store it in a session variable. However, persisting the data provides the underpinning for a broken upload resume feature that I am planning to add soon.

Next, let's review the upload() method. First, the method signature:

	public function upload( required string id,
		required any temp_file,
		required numeric part,
		required numeric parts,
		required string filename,
		required string filesize )

The signature shows what we are dealing with as input. An id (in this case, a generated UUID for each filepart), a path to the filepart on disk, the index and total number of parts, the original filename and filesize. Nothing too complicated there.

What you will notice next is that the method does almost nothing until you get to the last part of the fileparts in the upload. It basically just persists the filepart information to the database. Once you get to the last part, though, the upload method checks whether you are in a Windows or *nix environment, loops over the records of fileparts and assembles a CLI string to use to concatenate the fileparts into a single file. It then instantiates a Java Runtime object and invokes a CLI interpreter (cmd or bash for Windows or *nix) with the concatenate statement as an argument. 

What does that all mean? It means the page invokes a command line statement to put the file back together in the final upload destination. Personally, I have found this method to be an order of magnitude faster than trying to use native file commands in CFML to re-assemble the uploaded file. 

Once the file is re-assembled, the method goes through the temporary file parts and deletes them, then deletes the record of them in the database, then returns a struct with the final uploaded file's metadata. That information is then sent back to the browser as the final reply to the upload function

To be clear, this is a vanilla implementation of the upload function, meant mainly to demonstrate a basic way of capturing and storing the uploaded file and its metadata. I am available for consulting engagements if you would like a more complex implementation, such as storing the file in a NoSQL store like MongoDB.

 


gadget-ui.input.FileUploader example

CFML, CLI, JavaScript

I'm going to demonstrate how to use the new gadget-ui.input.FileUploader component in the gadget-ui library. First, we need an HTML page to host the component. This code comes directly from the /test folder in the gadget-ui repo:

fileuploader.htm

<!doctype html>
<html>

<head>
	<title>gadget-ui File Uploader Test</title>
	<script src='../bower_components/modlazy/dist/modlazy.min.js'></script>
	<script>
		modlazy.load(
			[
				"fileuploader.js < ../dist/gadget-ui.js",
				"../dist/gadget-ui.css"
			],
			function() {
				console.log("All files have been loaded");
			}
		);
	</script>

	<style>
		body {
			font-size: 1em;
		}

		input,
		select,
		select option {
			font-size: 1em;
		}

		#fileUploadDiv {
			height: 500px;
			width: 500px;
			text-align: center;
			margin-top: auto;
			margin-bottom: auto;
		}
	</style>

</head>

<body>
	<p>Test the FileUploader control.</p>

	<div id="fileUploadDiv"></div>
</body>

</html>

I am using a small dependency loader called modlazy to load my assets. This page shows a div called fileUploadDiv that will host the component. Let's look at the fileuploader.js file:

fileuploader.js


var options = {
  uploadURI: "/test/fileuploader.upload.cfm",
  tags: "file upload",
  willGenerateThumbnails: true,
  title: "Upload Files",
  showUploadButton: false
};

filedialog = gadgetui.objects.Constructor( gadgetui.input.FileUploader, [ document.querySelector("#fileUploadDiv"), options ]);

The uploadURI specifies a CFML page  that should be compatible with Lucee and Adobe ColdFusion, although I am testing it with Lucee, so there could be bugs in the ACF implementation right now. We'll dig into that page in a minute.

Right now, what I want to show is how the component is being instantiated. Rather than use the new keyword, I have ported all of he gadget-ui code to use the built-in gadgetui.objects.Constructor method, which internally uses Object.create(). I have also included a new option, showUploadButton, so you can either have a drop zone, or have a drop zone and a file upload button.

On drop  or file select, the component cuts the file up into 1MB chunks and uploads the chunks one at a time. Let's have a look at the CFML receiver file:

fileuploader.upload.cfm

<cfscript>
	filePath = "#expandpath("./")#upload\";
	separator = iif(expandpath("./") contains "/",de("/"),de("\"));
	filePath = expandpath("./") & "upload" & separator;
	chunkPath = filePath & "temp" & separator & createUUID();
	inputStream = getPageContext().getRequest().getOriginalRequest().getInputStream();
	ioutil = createObject( "java", "org.apache.commons.io.IOUtils" );
	outputStream = createObject( "java", "java.io.FileOutputStream" ).init( chunkPath );
	ioutil.copy( inputStream, outputStream );

	contentLength = gethttprequestdata().headers['Content-Length'];
	args.id = gethttprequestdata().headers['x-id'];
	args.filename = gethttprequestdata().headers['x-filename'];
	args.filesize = gethttprequestdata().headers['x-filesize'];
	args.part = gethttprequestdata().headers['x-filepart'];
	args.parts = gethttprequestdata().headers['x-parts'];

	if( args.part eq 1 ){
		fileId = createUUID();
	}else{
		fileId = args.id;
	}
	args.temp_file = chunkPath;

	args.id = fileId;
	FilePartDAO = new model.FilePartDAO();
	FileService = new model.FileService( filePath = filePath, FilePartDAO = FilePartDAO );
	file = FileService.upload( argumentcollection = args );

	if( args.part eq args.parts ){
		writeoutput( SerializeJSON( file ) );
	}else{
		pc = getpagecontext();
		pc.setHeader("X-Id", fileId );
	}

</cfscript>

There is a lot going on in this code. Look at the use of Java input and output streams to copy the chunked data. I use Java classes here to guarantee good data coming through from the binary data POST. Note that I am using UUIDs to identify the chunks and store them in a temporary folder on the server, where they stay until they are re-assembled into the original file.

Also see the use of gethttprequestdata() to pull the headers for the chunked data. The x- headers are all custom headers created by the FileUploader component to track the file upload. x-filepart tells the server which part of the file is being uploaded, and x-parts tells the server how many total parts there are.

Next, note that this code takes place in a loop controlled by the FileUploader component. When the end of the loop is reached, the file upload is completed and a response is sent back to the client. 

The real work of the upload is being done in the FileService CFML component. This example uses a baseline implementation that uploads the file(s) to a destination folder on the server. Other implementations may store the file in MongoDB or other NoSQL stores. I will dig into the FileService component next time.


Creating a Blog site with ContentBox on Ubuntu

CFML, CLI, CommandBox, ContentBox, Linux, Ubuntu

[ This post is part of my ongoing instructional series on setting up some baseline IT infrastructure for the fictional startup Shoestring Lab. Shoestring has committed to using Open Source wherever possible. Shoestring Lab has standardized on Ubuntu for its server and desktop/laptop computer systems.

Today's lesson

Shoestring Lab has decided to set up several website for its operations, and you have been tasked with using ContentBox to build the first site, an internal content and blog site.]


Using the provider namespace in ColdBox/WireBox with JavaLoader

CFML, ColdBox, ColdFusion, Lucee

Injecting classes in ColdBox via WireBox is generally pretty easy, but it can become complicated depending on what you need to inject, and where it needs to be injected. Sometimes you need the advanced functionality of the provider namespace to make it work. Sound interesting? Read on...


A Proper IDE for the new Lucee scripting language

CFML, Lucee

IntelliJ is losing CFML support. What does that mean for CFML programmers, and what might a good next-generation CFML/Lucee IDE look like? Read on ...


Real-time server debug info using NodeJS, jQuery, Websockets and Logbox

CFML, Java, JavaScript

Have you ever been working on a web site and wished you could get the server-side debug information without all the tedium of browsing log files or dumping debug information in the UI? Using Logbox and Websockets, you can do just that.


New Open Source CFML Server - Lucee

CFML, Lucee

There is a new kid on the block in the land of open source CFML engines. Lucee 4.5 rolled out yesterday and generated immediate buzz in the CFML community. Lucee is a fork of the popular Railo CFML engine, brought to life by many of the same people that created and maintained Railo over the last several years. 


Categories


Recent Entries

Entries Search