QUESTION: Working with the IFS
milesje opened this issue · 35 comments
Really wish for some better documentation. Especially around the IFS and Program calls.
How would you go about writing a file from the local system (where the Node app is running) to the IFS?
Or just how to write data to the a file on the IFS?
Have you tried piping output from the local system (via normal node.js fs methods) to ifs.createWriteStream('/foo/bar2.txt') ?
That will create the file on the IFS at '/foo/bar2.txt' how to I write data to that file?
I have tried the following and it creates the file, but it doesn't put any data into the file.
const ifs = pool.ifs();
const ws = ifs.createWriteStream('/eFile/test.txt', { append: false });
fs.readFile('./test.txt', 'utf8', function (err, data) {
console.log(data);
ws.write(data);
});
Following the above code if I replace 'ws.write(data)' with 'ws.pipe(data)' which is what I assume you mean to do by stream.pipe as ws is the created writeStream (ifs.createWriteStream), I get the following error.
events.js:167
throw er; // Unhandled 'error' event
^
Error [ERR_STREAM_CANNOT_PIPE]: Cannot pipe, not readable
at IfsWriteStream.Writable.pipe (_stream_writable.js:238:22)
at C:\Users\jmiles\code\nodeJS\as400Test\index.js:24:8
close connectionpool. at FSReqWrap.readFileAfterClose [as oncomplete] (internal/fs/read_file_context.js:53:3)
Emitted 'error' event at:
at IfsWriteStream.FlushWritable.emit (C:\Users\jmiles\code\nodeJS\as400Test\node_modules\flushwritable\lib\FlushWritable.js:37:31)
at IfsWriteStream.Writable.pipe (_stream_writable.js:238:8)
at C:\Users\jmiles\code\nodeJS\as400Test\index.js:24:8
at FSReqWrap.readFileAfterClose [as oncomplete] (internal/fs/read_file_context.js:53:3)
Hi @milesje
Sorry to hear about the documentation.
When writing a file to the IFS you want to create a readstream, that reads your file from nodejs, upload, etc and then pipes it to the writeStream.
Imagine us taking bricks from a box A and moving it to box B. I am gonna stand at box A (readStream) and pass it to you (writeStream) and you will put it in box B. That's piping. Move one brick at a time from box A to box B. So when you do ws.pipe you are piping from the wrong direction. You need to pipe the readstream to the writestream.
I put together some examples. I haven't tested them but hopefully this will help you in the right direction. Let me know if you need any further assistance :)
To create a file on the IFS you can do:
const writeStream = jt400.ifs().createWriteStream(fullFileName)This is a basic writeStream so you can pipe whatever you want into it. Basically empty box B ready to have something put into it.
- For example from nodejs app:
const fs = require('fs') //nodejs filesystem
const path = require('path')
const filename = path·join(__dirname, 'test.txt')
const readStream = fs.createReadStream(filename) // Reading file from the nodejs app as a stream
readStream.pipe(writeStream)- If you're uploading a file in an express app. You could use busboy as a middlewere and pipe the file directly. (see examples: https://github.com/mscdex/busboy) - just use the writeStream given in my example above)
busboy.on('file', function(fieldname, file, filename, encoding, mimetype) {
const saveTo = path.join(os.tmpDir(), path.basename(fieldname));
file.pipe(writeStream); // writesStream is our IFS writestream defined above
});If you have a buffer you could use something like streamifer or other libraries that take your buffer and change it to stream.
const createReadStream = require('streamifer').createReadStream
const path = require('path')
const filename = path·join(__dirname, 'test.txt')
const file = fs.readFileSync(filename) // Reading file from the nodejs app sync buffer
const readStream = createReadStream(file)
readStream.pipe(writeStream)Note that this is not really pure streaming since you are synchronously reading the file into a buffer (holding the whole process) and then changing it to a stream, which finally gets piped. So in our example moving from box A to box B. We're gonna first dump everything from box A to the floor and then you're gonna move it one-by-one to box B. Not as efficient.
Let me know if that helps :)
Here is my test code. doing this creates the file on the IFS but does not write any content into the created file. Am I missing something?
var fs = require('fs');
const config = require('./config.json');
const pool = require('node-jt400').pool(config);
const ifs = pool.ifs();
const path = require('path');
const filename = path.join(__dirname, 'test.txt');
const readStream = fs.createReadStream(filename);
const ws = ifs.createWriteStream('/eFile/test.txt');
readStream.pipe(ws);
This would be so much simpler if it also had writeFile function like 'fs' does. Just declare the file you want and the data to be written to the file and your done.
example:
fs.writeFile('./myFile.text', data, function(err){
if (err) throw err;
console.log('saved!');
})
Doing the readStream & writeStream purly with 'fs' works.
const readStream = fs.createReadStream('./test.txt');
const writeStream = fs.createWriteStream('./test2.txt');
readStream.pipe(writeStream);
But replacing the fs.createWriteStream with ifs.createWriteStream doesn't actually write the data into the file on the IFS.
Could it have anything to do with text encoding? 'utf8' vs whatever is used on the IBM i (AS/400)
This would be so much simpler if it also had writeFile function like 'fs' does. Just declare the file you want and the data to be written to the file and your done.
This is an open source project, you should add it!
Could it have anything to do with text encoding? 'utf8' vs whatever is used on the IBM i (AS/400)
If there's no data in the file at all, I doubt it...
The data in the file on the Windows PC has text. "This is a test file for the NodeJS integration."
The file when viewing it on the AS/400 doesn't appear to have any text in it.
And reading a file from the IFS and saving it locally works.
const stream = ifs.createReadStream('/eFile/app.config');
let data = '';
stream.on('data', chunk => {
data += chunk;
});
stream.on('end', () => {
fs.writeFile('./app.config', data, function(err){
if (err) throw err;
console.log('saved!');
})
});
This is an open source project, you should add it!
If I knew how I would be happy to create a Pull Request to add this functionality. But as I can't get the writeStream to work....
Not to mention that I barley know JavaScript and have never worked with TypeScript which is how the module is written I will have a bit of a learning curve before I could contribute, but I am willing to help where and when I can.
If I read the file off of the IFS and then pipe it into another file on the IFS it works, my issue appear to only be when trying to pipe a local file (Windows FileSystem) to a file on the IFS.
const writeStream = ifs.createWriteStream('/eFile/test2.txt');
const stream = ifs.createReadStream('/eFile/test.txt');
stream.pipe(writeStream);
This would be so much simpler if it also had writeFile function like 'fs' does. Just declare the file you want and the data to be written to the file and your done.
Yes that is a nice method but not very efficient since you are writing the whole buffer instead of streaming it.
This module was developed for Tryggingamidstodin (an Icelandic insurance company) that has AS400 legacy systems. We kinda figured that other companies would be dealing with the same problem so we made it open source.
This module is basically a node version of the IBM Java Toolbox and choosing what methods would be available in this module was marely based on the needs of Tryggingamidstodin. We needed to save files to the IFS and we like streams so we implemented that feature.
The writeStream is mapping IFSFileOutputStream and has been working in production here.
It is possible to map other functions from the com.ibm.as400.access package
Doing the readStream & writeStream purly with 'fs' works.
const readStream = fs.createReadStream('./test.txt'); const writeStream = fs.createWriteStream('./test2.txt'); readStream.pipe(writeStream);But replacing the fs.createWriteStream with ifs.createWriteStream doesn't actually write the data into the file on the IFS.
I'm not sure what I can tell you about that. I can stream with 'fs' and the 'ifs'. The original example the docs was bascially reading a file from IFS and saving it as another file
ifs.createReadStream('/foo/bar.txt').pipe(ifs.createWriteStream('/foo/bar2.txt'));
And reading a file from the IFS and saving it locally works.
const stream = ifs.createReadStream('/eFile/app.config'); let data = ''; stream.on('data', chunk => { data += chunk; }); stream.on('end', () => { fs.writeFile('./app.config', data, function(err){ if (err) throw err; console.log('saved!'); }) });
Ok its good good to hear that reading works. Just note that your second example where you pipe directly is a tad better/faster since you're not waiting for the whole file and then writing.
If I read the file off of the IFS and then pipe it into another file on the IFS it works, my issue appear to only be when trying to pipe a local file (Windows FileSystem) to a file on the IFS.
const writeStream = ifs.createWriteStream('/eFile/test2.txt'); const stream = ifs.createReadStream('/eFile/test.txt'); stream.pipe(writeStream);
Do you get any errors when you write to the ifs? What happens if you do on('error') does it show anything? Are you sure that the path is correct? Do you have any other systems writing to that path? I'm just thinking out loud.
If you clone the project and build it you should be able to mess with some tests.
Checkout: /lib/test/ifs-spec.ts - line 55 or just ind "should write file"
- There is a test that reads a file on our system called hello world and pipes it into a new file. You should be able to mess with that test with your paths and see how it goes.
There's a another test for binary data, same file, line 83 "should pipe image" where we pipe image.
Maybe it helps if you mess around with these tests.
OK, I'm wondering if it has something to do with text encoding ('utf8', ...)
Here is an example I wrote, and when it hits the stream.on('end', () => { console.log(data)} it appears to be reading the correct data in which I'm writing to the file on the IFS. But If I use a 5250 terminal on the IBM i and I open the file it appears to be empty.
var fs = require('fs');
const config = require('./config.json');
const pool = require('node-jt400').pool(config);
const ifs = pool.ifs();
const readStream = fs.createReadStream('./test.txt');
const writeStream = ifs.createWriteStream('/eFile/test.txt');
readStream.pipe(writeStream).on('finish', () => {
console.log('pipe of data is finished');
const stream = ifs.createReadStream('/eFile/test.txt');
let data = '';
stream.on('data', chunk => {
data += chunk;
});
stream.on('end', () => {
console.log('read end');
console.log(data);
});
stream.on('error', (err) => {
console.log(err)
});
});
Results from running...
pipe of data is finished
read end
This is a test file for the NodeJS integration.
close connectionpool.
@milesje Ehm I can't really help with the 5250 terminal since I don't have it installed and I never work with it.
It might be that the file is there with all the data and the terminal is just not showing it. That would explain alot, but I don't have any ways of really testing it.
Can you map the IFS drive in windows and check if you see the file there? If you open it in notepad or some other editor.
If I map the IFS folder to my windows PC I'm able to view the file, and it does have data in it. It is a little weird that when viewing the file in the 5250 terminal it doesn't show any data. I have a few IBM i experts here where I work I will have them look over the file that is being created and see if they can figure out why its not view-able in the 5250 terminal.
Awesome, good to hear that it's working.
I'm closing this issue since the module is writing content to the file and the issue seems more related to the terminal.
If I try to open the file in a 5250 terminal using QSH (QShell) with the cat command I get an error.
cat: 001-2104 Error found reading from file test.txt. Conversion error
This leads me to believe that it is a text encoding issue. I am able to use the cat command on files that I created on my Windows PC and then transferred to the IFS by mapping the IFS folder as a network folder.
@bergur I disagree that the issue is with the terminal.
Is there a way to tell the createWriteStream to create the file with a specific encoding such as UTF8?
There are several ways of checking encoding of a file For example: https://stackoverflow.com/questions/3710374/get-encoding-of-a-file-in-windows
I am open to the thought that the module is doing something wrong and I've reopened the issue but I'm afraid that you have to give use a little more that relates to module itself and how it is incorrectly writing files.
One solution would be to try using the IBM Java Toolbox to see if you are getting the same behavour.
https://www.ibm.com/support/knowledgecenter/ssw_ibm_i_71/rzahh/javadoc/com/ibm/as400/access/IFSFileOutputStream.html
That is: What happens if you do the exact same thing through the java code this module is based on. If its the same behaviour then there's not much we can do. If it behaves differently then we know that the mapping from node-to-java is doing something wrong and we can try to isolate that.
Other food for thoughts: Is there a standard encoding on IFS? What encoding does the terminal show?
If I open the file in the 5250 terminal and tell it to Display as HEX I get the this.
3F3F3F3F 3F3F3F3F 3F3F
and the text I sent was
This is just a test!
I will build a small Java app that use the jt400.jar and see what I get. Java is something I do know and it shouldn't take me long to put together a test for this.
Here is a sample Java program using the jt400.jar that writes the file and it is readable via the 5250 terminal.
public class Main {
public static void main(String ... args){
System.out.println("This is a test");
try {
AS400 as400 = new AS400("172.16.1.13", "JMILES", "myPassWrd1");
IFSFileOutputStream file = new IFSFileOutputStream(as400, "/eFile/test3.txt", IFSFileOutputStream.SHARE_ALL, false, 1252 );
file.write("This is a test - Java!".getBytes());
// Close the file.
file.close();
} catch (Exception e){
e.printStackTrace();
}
}
}
I think the key in the above program is the last parameter of the IFSFileOutputStream which is the text encoding.
In this case 1252 creates a file that is Windows, Latin1 encoding.
So if this library would allow for the CCSID (text encoding) to be set, that would fix all of my problems... so far anyway!
For my Java test code was I am using jt400 version 9.4 in case that matters.
This is the line that appears to be your issue line in the code.
This is a PR I made really quickly that is a similar type modification to the node-jt400 code to allow the library path to be supplied to the PGM call, maybe you can use that to guide you if you're up to modifying this library.
I have already forked the project and I made the changes, but when I run the test "npm run test" it errors out. I'm not sure if the test worked before my changes or not. I will submit a pull request as soon as I get it working.
I couldn't get all the tests to work either, so I only ran my new test when I submitted my PR.