tencentyun/hadoop-cos

CosN的实现里面是不是要覆盖获取默认端口的方法

shangwen opened this issue · 1 comments

我按照你们的配置,提交任务出现了

org.apache.hadoop.fs.InvalidPathException: Invalid path name Wrong FS: cosn://xxxx/emr/hadoop-yarn/staging/hadoop/.staging/job_1555056072016_0008, expected: cosn://xxxx/ 

通过分析可以发现, 问题处在了AbstractFileSystem这行thisPort等于-1,thatPort等于0

public void checkPath(Path path) {
...
    // Ports must match, unless this FS instance is using the default port, in
    // which case the port may be omitted from the given URI
    int thisPort = this.getUri().getPort();
    int thatPort = uri.getPort();
    if (thatPort == -1) { // -1 => defaultPort of Uri scheme
      thatPort = this.getUriDefaultPort();
    }
    if (thisPort != thatPort) {
      throw new InvalidPathException("Wrong FS: " + path + ", expected: "
          + this.getUri());
    }
}


看了逻辑,我们是不是需要在CosN类里面覆盖默认端口getUriDefaultPort?经过修改好像没了这个问题,不知道是我配置不正确还是,你们的代码有bug

public class CosN extends DelegateToFileSystem {
    public CosN(URI theUri, Configuration conf) throws IOException, URISyntaxException {
        super(theUri, new CosFileSystem(), conf, CosFileSystem.SCHEME, false);
    }

    @Override
    public int getUriDefaultPort() {
        return -1;
    }
}

我按照你们的配置,提交任务出现了

org.apache.hadoop.fs.InvalidPathException: Invalid path name Wrong FS: cosn://xxxx/emr/hadoop-yarn/staging/hadoop/.staging/job_1555056072016_0008, expected: cosn://xxxx/ 

通过分析可以发现, 问题处在了AbstractFileSystem这行thisPort等于-1,thatPort等于0

public void checkPath(Path path) {
...
    // Ports must match, unless this FS instance is using the default port, in
    // which case the port may be omitted from the given URI
    int thisPort = this.getUri().getPort();
    int thatPort = uri.getPort();
    if (thatPort == -1) { // -1 => defaultPort of Uri scheme
      thatPort = this.getUriDefaultPort();
    }
    if (thisPort != thatPort) {
      throw new InvalidPathException("Wrong FS: " + path + ", expected: "
          + this.getUri());
    }
}

看了逻辑,我们是不是需要在CosN类里面覆盖默认端口getUriDefaultPort?经过修改好像没了这个问题,不知道是我配置不正确还是,你们的代码有bug

public class CosN extends DelegateToFileSystem {
    public CosN(URI theUri, Configuration conf) throws IOException, URISyntaxException {
        super(theUri, new CosFileSystem(), conf, CosFileSystem.SCHEME, false);
    }

    @Override
    public int getUriDefaultPort() {
        return -1;
    }
}

谢谢反馈,这里确实需要覆盖getUriDefaultPort方法。已修复。