Upload 0Kb, does api need to continue to configure on the server?
CNBroderick opened this issue · 23 comments
upload 0 Kb. Flow is my test code. wanna to help me.
package org.bklab.wot.warmonger;
import org.aarboard.nextcloud.api.NextcloudConnector;
import java.io.*;
import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
import java.util.Arrays;
import java.util.List;
import java.util.concurrent.atomic.AtomicInteger;
@SuppressWarnings("unused")
public class UploadToBlueHawkCloud {
private NextcloudConnector lanyingCloud = getConnection();
private static final String base = "warmongers";
private String s = initString();
public static UploadToBlueHawkCloud create() {
return new UploadToBlueHawkCloud();
}
public NextcloudConnector getConnection() {
return new NextcloudConnector("example.com", true, 443, "admin", "password");
}
public UploadToBlueHawkCloud doUpload() {
File upLoadDir = new File("E:\\Broderick Resources\\mods\\");
List<File> modsList = Arrays.asList(upLoadDir.listFiles());
System.out.println("start upload");
InputStream stream = new ByteArrayInputStream(s.getBytes(StandardCharsets.UTF_8));
lanyingCloud.uploadFile(stream, base + "/a.txt");
modsList.forEach(file -> {
if(file.getName().endsWith(".wbp") || file.getName().endsWith(".txt")) {
try {
System.out.println("uploading " + file.getName());
InputStream stream1 = new FileInputStream(file);
String remotePath = base + "/" + getEncodeUrl(file.getName());
lanyingCloud.uploadFile(stream1, remotePath);
} catch (Exception e) {
e.printStackTrace();
}
}
});
return this;
}
private String getEncodeUrl(String s) {
try {
return URLEncoder.encode(s, "utf-8");
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
}
return "";
}
public static void main(String[] args) {
UploadToBlueHawkCloud.create().doUpload();
}
private String initString() {
return "test string, about 4KB";
}
}Uploaded WebSites:
Shared problem linked: https://xn--9kq.xn--fx1at73c.com/index.php/s/2KbrqgBK4peKA8t
@CNBroderick What is the issue you see here? I see code, but no problem or what is expected to be done
after upload file by run these code, the nextcloud show 1.txt size is 0kb, and can not open it
String TEST1_FILE_CONTENT = "Content of Test 1 file"; InputStream inputStream = new ByteArrayInputStream(TEST1_FILE_CONTENT.getBytes()); n.uploadFile(inputStream, "/1.txt");
Hey there,
we ran into the same issue after our nextcloud was updated to Version 18.03 (by our administration team without a notice to our softwaredevelopment team ;) ). After uploading files through the nextcloudapi, they all had a size of 0 Kb. After some research (and trial and error) we found out, that if we add an additional parameter in the sardine.put method, it works for our nextcloud installation. I'm not shure if thats just the problem with our installation or if it's a general problem with the newer nextcloud version.
I overloaded the uploadFile method to accept an additional boolean Parameter for the continue header, if it's ok and there are other people with the same problem, I can send a merge Request.
Yes, please send a merge request, I will then review it
@TobiWineKenobi What does work in your environment? continue = false, oder continue = true?
Oh, sorry I forgot. We need to set the continue header to false. Then it works for us.
I have merged your patch request.
But so far I have not been able to reproducte the problem, also not on 18.0.4
What sardine.jar version are you using?
I have 5.9 and it works with the setting true and false, both ways...
Thanks, for your fast response.
We are using all libraries as in your pom, so it's also the 5.9 version of sardine. The only change that I made was the modification of these methods.
I think my error could be the chunked transfer encoding problem. As the problem suddenly appears when we updated to Version 18 after years of usage without any problems, It could also be a configuration issue on the nextcloud or it's webserver. Sadly our nextcloud is hosted by our Networkadministration-department and they were not able to change configuration to make it work again.
But these are just assumtions from my side. The only thing I can say for sure is, that if we use this parameter it works and if we leave it or set it "true" we get files with 0 Kb.
I am seeing the same effect using Sardine 5.9, release 11.2 with Nextcloud 18.04. Upload methods return without exception - both with the header parameter set to true and false, but the uploaded file is 0KB in both cases.
Is there any other information that could help me track down the issue, @a-schild ?
Just for the records, it seems to be related to chunking as @TobiWineKenobi already mentioned. I found quite a few users having this problem, and while there are workarounds by means of configuration for NGINX, there is nothing for Apache.
Here is an alternative upload using Apache Commons HTTP client and Jackrabbit WebDAV:
import java.io.ByteArrayInputStream;
import java.io.File;
import java.io.FileInputStream;
import java.util.List;
import java.util.logging.Level;
import java.util.logging.Logger;
import org.aarboard.nextcloud.api.NextcloudConnector;
import org.apache.commons.httpclient.Credentials;
import org.apache.commons.httpclient.methods.InputStreamRequestEntity;
import org.apache.commons.httpclient.methods.RequestEntity;
import org.apache.commons.httpclient.HttpClient;
import org.apache.commons.httpclient.HttpException;
import org.apache.commons.httpclient.UsernamePasswordCredentials;
import org.apache.commons.httpclient.auth.AuthScope;
import org.apache.jackrabbit.webdav.client.methods.PutMethod;
public class NextcloudTest {
public static void main(String[] args) {
try {
final String baseUrl = "https://some.server.com/remote.php/webdav";
try {
HttpClient client = new HttpClient();
Credentials creds = new UsernamePasswordCredentials("user", "pwd");
client.getState().setCredentials(AuthScope.ANY, creds);
PutMethod method = new PutMethod(baseUrl + "/" + "temp/briefkopf.odt");
RequestEntity requestEntity = new InputStreamRequestEntity(
new FileInputStream(new File("/home/jens/briefkopf.odt")));
method.setRequestEntity(requestEntity);
client.executeMethod(method);
System.out.println(method.getStatusCode() + " " + method.getStatusText());
} catch (HttpException ex) {
}
} catch (Throwable ex) {
Logger.getLogger(NextcloudTest.class.getName()).log(Level.SEVERE, null, ex);
}
}
}
So far, this seems to run fine with small and large files, independent of the webserver serving Nextcloud.
Would it make sense to make it the default for file uploads in nextcloud-java-api?
If you can give me a not working configuration of an webserver we could do real tests with it.
Have you tested the uploadFile method with continueHeader true/false?
Does one of these work for you?
I don't wish to integrate yet another webdav library, so fixing the current one is more important for me.
I don't wish to integrate yet another webdav library, so fixing the current one is more important for me.
I totally agree.
Have you tested the uploadFile method with continueHeader true/false? Does one of these work for you?
Yes, I have tried all combinations and they make no difference for me. I verified this behaviour using two self-hosted Nextcloud instances and one Nextcloud instance provided by a hosting company. All show the same issue.
If you can give me a not working configuration of an webserver we could do real tests with it.
What specifically do you need? I can setup an account for you on my instance and send you the credentials - is that what you mean? I am going to send the credentials to your email address ae@s.ws. (this is not a regex, just trying to not put your address here :-) )
Just for the records, it seems to be related to chunking as @TobiWineKenobi already mentioned. I found quite a few users having this problem, and while there are workarounds by means of configuration for NGINX, there is nothing for Apache.
Here is an alternative upload using Apache Commons HTTP client and Jackrabbit WebDAV:
import java.io.ByteArrayInputStream; import java.io.File; import java.io.FileInputStream; import java.util.List; import java.util.logging.Level; import java.util.logging.Logger; import org.aarboard.nextcloud.api.NextcloudConnector; import org.apache.commons.httpclient.Credentials; import org.apache.commons.httpclient.methods.InputStreamRequestEntity; import org.apache.commons.httpclient.methods.RequestEntity; import org.apache.commons.httpclient.HttpClient; import org.apache.commons.httpclient.HttpException; import org.apache.commons.httpclient.UsernamePasswordCredentials; import org.apache.commons.httpclient.auth.AuthScope; import org.apache.jackrabbit.webdav.client.methods.PutMethod; public class NextcloudTest { public static void main(String[] args) { try { final String baseUrl = "https://some.server.com/remote.php/webdav"; try { HttpClient client = new HttpClient(); Credentials creds = new UsernamePasswordCredentials("user", "pwd"); client.getState().setCredentials(AuthScope.ANY, creds); PutMethod method = new PutMethod(baseUrl + "/" + "temp/briefkopf.odt"); RequestEntity requestEntity = new InputStreamRequestEntity( new FileInputStream(new File("/home/jens/briefkopf.odt"))); method.setRequestEntity(requestEntity); client.executeMethod(method); System.out.println(method.getStatusCode() + " " + method.getStatusText()); } catch (HttpException ex) { } } catch (Throwable ex) { Logger.getLogger(NextcloudTest.class.getName()).log(Level.SEVERE, null, ex); } } }So far, this seems to run fine with small and large files, independent of the webserver serving Nextcloud.
Would it make sense to make it the default for file uploads in nextcloud-java-api?
How to download files from NextCloud
And some more
nextcloud/server#7995
Basically it's a server problem, where the webserver talks with fpm to php and the content length is not set on the put operation.
Since InputStream is a stream of unknown size there seems to not realy exist a client side solution for a server side problem.
I wonder if your workarround also would work with an normal InputStream and not a FileInputStream.
The FileInputStream knows about the stream length and can probably send the content-length to the webserver
@a-schild thanks for looking into this.
Yes, I agree it seems like a server problem, because the nextcloud-java-api implementation seems to work for some, and not work for others. I think having a clientside workaround would help more users and would be less invasive, but I also totally understand that adding additional libraries (that even serve the same purpose) is not ideal.
The only valid way forward might be to really understand / track down what the workaround I posted above is doing different, maybe by doing some analysis on exchanged requests / reponse headers.
@a-schild what exactly do you mean by "normal InputStream"? I just tried the workaround with a ByteArrayInputStream instead of a FileInputStream, with both a small 5KB file and a larger 22MB file. Both were uploaded fine.
The jackrabbit library uses not chunked uploads, with a known length. not sure where the length is known, since an inpurtstream can be "indefinite" length
The sardine uses chunked uploads, without length information.
I see. So, I think it's just "wait till the Nextcloud guys solve the problem, and leave nextcloud-java-api unchanged" for now, to not clutter it with competing / similar libraries. Anyone with an upload requirement can use the workaround. It's just that the library has an API with a somewhat unpredictable behaviour.
The problem is not on NextCloud side, but rather the webserver<->PHP connection.
As soon as it's using fpm or fcgi and the data is sent in chunked mode, any client will get this problem.
The only possible thing to correct this on server side, would be to configure the fpm/fcgi connection so it does buffer all chunks until it has all parts.
Unfortunally this opens an potetial hole for attacks on webservers, since the size it not know, it could eat up all server resources.
If you set a safety limit on the server, all uploads for files larger than that limit will also fail.
The woraaround you mentions does also not work in all cases, as soon as you have a streaming input source, then the data get sent as chunks and we have the same problem as with sardine.
In the Apache Http Client implementation is a workarround with using retryable put, but in that case the InputStream must be readable multiple times....
The woraaround you mentions does also not work in all cases, as soon as you have a streaming input source, then the data get sent as chunks and we have the same problem as with sardine.
I think this could be documented as a "known limitation", and users can deal with it, e.g. by fully retrieving the source and then doing the upload. Not ideal, but workable.
My knowledge on Sardine is limited, might dig into the sources to figure out whether they have some conditional logic for deciding on chunking / non-chunking PUTs. If that were configurable / can be controlled by a parameter, I'd vote for not doing chunking at all, which would make the workaround (and need for additional libraries) unnecessary.
I have added a method to upload a Fileinsteand of InputStream
This works fine, also when the server is having problems with chunked uploads.
You can use the 11.3.0-SNAPSHOT to test this.
Can you please also do some tests with very large (650MB, 2GB) uploadfiles and report back if they also work?
@a-schild thanks for the change. I locally built 11.3.0-SNAPSHOT and ran some tests, basically uploading files of various sizes and taking MD5 checksums locally and on the server once they were uploaded.
All of the uploads were successful, checksums always matched. The largest file I tried was around 900megs. Is there a need for me to really try 2gigs? No issue with doing it, but will take a while given my bandwidth.
Thanks for all your support! Is there a projected date on when 11.3.0 will be released?
The 11.3.0 release deprecates the uploadFile(Inputstream)methods and provides an uploadFile(File)method which solves this problem.
If someone has another idea how to work arround the server problem, feel free to post it here or provide a patch for it