CosN的实现里面是不是要覆盖获取默认端口的方法
shangwen opened this issue · 1 comments
shangwen commented
我按照你们的配置,提交任务出现了
org.apache.hadoop.fs.InvalidPathException: Invalid path name Wrong FS: cosn://xxxx/emr/hadoop-yarn/staging/hadoop/.staging/job_1555056072016_0008, expected: cosn://xxxx/
通过分析可以发现, 问题处在了AbstractFileSystem这行thisPort等于-1,thatPort等于0
public void checkPath(Path path) {
...
// Ports must match, unless this FS instance is using the default port, in
// which case the port may be omitted from the given URI
int thisPort = this.getUri().getPort();
int thatPort = uri.getPort();
if (thatPort == -1) { // -1 => defaultPort of Uri scheme
thatPort = this.getUriDefaultPort();
}
if (thisPort != thatPort) {
throw new InvalidPathException("Wrong FS: " + path + ", expected: "
+ this.getUri());
}
}
看了逻辑,我们是不是需要在CosN类里面覆盖默认端口getUriDefaultPort?经过修改好像没了这个问题,不知道是我配置不正确还是,你们的代码有bug
public class CosN extends DelegateToFileSystem {
public CosN(URI theUri, Configuration conf) throws IOException, URISyntaxException {
super(theUri, new CosFileSystem(), conf, CosFileSystem.SCHEME, false);
}
@Override
public int getUriDefaultPort() {
return -1;
}
}
yuyang733 commented
我按照你们的配置,提交任务出现了
org.apache.hadoop.fs.InvalidPathException: Invalid path name Wrong FS: cosn://xxxx/emr/hadoop-yarn/staging/hadoop/.staging/job_1555056072016_0008, expected: cosn://xxxx/
通过分析可以发现, 问题处在了AbstractFileSystem这行thisPort等于-1,thatPort等于0
public void checkPath(Path path) { ... // Ports must match, unless this FS instance is using the default port, in // which case the port may be omitted from the given URI int thisPort = this.getUri().getPort(); int thatPort = uri.getPort(); if (thatPort == -1) { // -1 => defaultPort of Uri scheme thatPort = this.getUriDefaultPort(); } if (thisPort != thatPort) { throw new InvalidPathException("Wrong FS: " + path + ", expected: " + this.getUri()); } }
看了逻辑,我们是不是需要在CosN类里面覆盖默认端口getUriDefaultPort?经过修改好像没了这个问题,不知道是我配置不正确还是,你们的代码有bug
public class CosN extends DelegateToFileSystem { public CosN(URI theUri, Configuration conf) throws IOException, URISyntaxException { super(theUri, new CosFileSystem(), conf, CosFileSystem.SCHEME, false); } @Override public int getUriDefaultPort() { return -1; } }
谢谢反馈,这里确实需要覆盖getUriDefaultPort方法。已修复。