xml-stream is collecting less number of items than in the actual xml

[relentless-coder]

Hi,

I have implemented xml-stream in the following manner.

class FromStreamableXML extends Parser {
  constructor() {
    super('xmlStream');
  }

  setupRequestStream(link) {
    return new Promise((resolve, reject) => {
      const req = request(link, res => {
        resolve(res);
      });
      req.on('error', err => {
        reject(err);
      });
      req.end();
    });
  }

  setupFileStream(path) {
    return Promise.resolve(createReadStream(path));
  }

  async toJs({ path, itemToSelect, identifier }) {
    let streamSource = this.setupRequestStream;
    const itemsMap = {};
    const items = [];
    const stream = await streamSource(path);
    const myXML = new this.xmlStream(stream);

    return new Promise((resolve, reject) => {
      myXML.on(`endElement: ${itemToSelect}`, item => {
        if (identifier) {
          if (!itemsMap[identifier]) {
            itemsMap[identifier] = [];
          }
          itemsMap[identifier].push(item);
        } else {
          items.push(item);
        }
      });
      myXML.on('end', () => {
        if (identifier) {
          resolve(itemsMap);
        } else {
          resolve(items);
        }
      })
    });
  }
}

I am passing a request stream, however the situation is that it is not collecting all the items. I tested with an rss feed url that contains 200 items, however, I get random number of items, 15, 53, 13.

Any ideas how to resolve it?

[relentless-coder]

So this was combination of two issues, 1). Missing Connection: keep-alive header, 2) promise resolved streams weren't working with xml-stream.

I don't know why 2nd is an issue, I lack knowledge here, so if someone has anything to add to it then please do so.

Here's my updated code:

class FromStreamableXML extends Parser {
  constructor() {
    super('xmlStream');
  }

  processXml({ stream, itemToSelect, identifier }) {
    const xml = new xmlStream(stream);
    const itemsMap = {};
    const items = [];
    return new Promise((resolve, reject) => {
      xml.on(`updateElement: ${itemToSelect}`, item => {
        if (identifier) {
          if (!itemsMap[identifier]) {
            itemsMap[identifier] = [];
          }
          itemsMap[identifier].push(item);
        } else {
          items.push(item);
        }
      });
      xml.on('end', () => {
        if (identifier) {
          resolve(itemsMap);
        } else {
          resolve(items);
        }
      });
      xml.on('error', (err) => {
        reject(err);
      })
    });
  }

  fromURL({ path, itemToSelect, identifier }) {
    return new Promise((resolve, reject) => {
      request({
        url: path,
        headers: {
          Connection: 'keep-alive',
        },
      })
        .on('response', res => {
          res.setEncoding('utf-8');
          resolve(this.processXml({ stream: res, itemToSelect, identifier }));
        })
        .on('error', err => {
          reject(err);
        });
    });
  }

  fromFile({ path, itemToSelect, identifier }) {
    try {
      return this.processXml({
        stream: createReadStream(path),
        itemToSelect,
        identifier,
      });
    } catch (err) {
      throw err;
    }
  }

  toJs({ path, itemToSelect, identifier, type }) {
    if (type === 'url') {
      return this.fromURL({ path, itemToSelect, identifier });
    } else if (type === 'file') {
      return this.fromFile({ path });
    } else {
      return identifier ? {} : [];
    }
  }
}