Checking for external changes - Suspended, high cpu usage
arsi-apli opened this issue · 8 comments
I spent several hours debugging and the problem is caused by Unix domain socket file in /tmp directory.
Directory watch service is calling BaseFileObj.refresh(final boolean expected, boolean fire) and the test on this file:
final boolean isDir = file.isDirectory();
final boolean isFile = file.isFile();
if (isDir == isFile || isFolder() != isDir || isData() != isFile) {
invalidateFO(fire, expected, true);
}isDir == isFile -> false == false
And it's called over and over again...
I was unable to find the originator why the /tmp is added to the watch for changes. But the reason is somewhere around Gradle and NB version control history.
A simple solution is to change java tmp directory location int the netbeans/etc/netbeans.conf:
-J-Djava.io.tmpdir=/custom/tmp
Ideally, mount a new tmpfs..
/etc/fstab
none /media/ramdisk tmpfs defaults,user,size=20G,mode=0777 0 0I also have a netbeans cache on RAMdisk, so it's so big. And the acceleration of Netbeans is brutal.. ;)
And it's back..
Netbeans folder watching service has positively a problem with Unix domain socket files and broken Symbolic Links on linux. It always ends with creating recursive file events.
Problems with Symbolic Links causes Gradle:
https://github.com/gradle/gradle/blob/master/subprojects/native/src/main/java/org/gradle/internal/nativeintegration/filesystem/jdk7/Jdk7Symlink.java
private static boolean doesSystemSupportSymlinks() {
Path sourceFile = null;
Path linkFile = null;
try {
sourceFile = Files.createTempFile("symlink", "test");
linkFile = Files.createTempFile("symlink", "test_link");
Files.delete(linkFile);
Files.createSymbolicLink(linkFile, sourceFile); //Symbolic Link is only created, not deleted
return true;
} catch (InternalError e) {I added a class to delete them org.nbandroid.netbeans.gradle.symlink.GradleSymLinkRemover
public class GradleSymLinkRemover implements Runnable {
private final File tempDir;
public GradleSymLinkRemover() {
String tmpdir = System.getProperty("java.io.tmpdir");
tempDir = new File(tmpdir);
}
@Override
public void run() {
File[] listFiles = tempDir.listFiles(new FilenameFilter() {
@Override
public boolean accept(File dir, String name) {
return name.startsWith("symlink") && name.endsWith("test_link");
}
});
for (File file : listFiles) {
if (Files.isSymbolicLink(file.toPath())) {
try {
Files.deleteIfExists(file.toPath());
} catch (IOException ex) {
Exceptions.printStackTrace(ex);
}
}
}
}
}And is scheduled from org.nbandroid.netbeans.gradle.v2.Installer
POOL.scheduleWithFixedDelay(new GradleSymLinkRemover(), 30, 60, TimeUnit.SECONDS);Can this smth similar to this problem: https://issues.apache.org/jira/browse/NETBEANS-168
Netbeans Linux killer
Usage:
First run and stop the killer, to create some broken links. It does not work without it.
Start Netbeans and open some file from /tmp, to enable directory watch.
Run the killer, and commonly use netbeans, compile some project, etc.
btw: on my i7-8750H is 500ms infinity :)
public static void main(String[] args) {
for (int i = 0; i < 10; i++) {
Path sourceFile = null;
Path linkFile = null;
try {
sourceFile = Files.createTempFile("symlink", "test");
linkFile = Files.createTempFile("symlink", "test_link");
System.out.println(linkFile);
Files.delete(linkFile);
Files.createSymbolicLink(linkFile, sourceFile);
Files.delete(sourceFile);
} catch (InternalError e) {
} catch (IOException e) {
} catch (UnsupportedOperationException e) {
}
}
while (true) {
Path sourceFile = null;
Path linkFile = null;
try {
sourceFile = Files.createTempFile("symlink", "test");
linkFile = Files.createTempFile("symlink", "test_link");
System.out.println(linkFile);
Files.delete(linkFile);
Files.createSymbolicLink(linkFile, sourceFile);
} catch (InternalError e) {
} catch (IOException e) {
} catch (UnsupportedOperationException e) {
} finally {
try {
if (sourceFile != null && sourceFile.toFile().exists()) {
Files.delete(sourceFile);
}
if (linkFile != null) {
Files.deleteIfExists(linkFile);
}
} catch (IOException e) {
// We don't really need to handle this.
}
}
try {
Thread.sleep(500);
//500ms high cpu usage - after some time Suspended, 100ms Suspended
} catch (InterruptedException ex) {
Logger.getLogger(TmpTest.class.getName()).log(Level.SEVERE, null, ex);
}
}So I'm debugging again and I got some progress:
After delete is created remove event and this is first problematic code which I found:
org.netbeans.modules.versioning.masterfs.FilesystemInterceptor.deletedExternally()
@Override
public void deletedExternally(FileObject fo) {
VCSFilesystemInterceptor.deletedExternally(VCSFileProxy.createFileProxy(fo));
}org.netbeans.modules.versioning.core.api.(VCSFileProxy.createFileProxy(fo)
public static VCSFileProxy createFileProxy(final FileObject fileObject) {
try {
VCSFileProxyOperations fileProxyOperations = getFileProxyOperations(fileObject.getFileSystem());
if (fileProxyOperations == null) {
File file = FileUtil.toFile(fileObject);
if(file != null) {
final VCSFileProxy p = createFileProxy(file);
p.isDirectory = fileObject.isFolder();
p.fileChangeListener = new FileChangeListener() {
@Override public void fileDeleted(FileEvent fe) {
p.isDirectory = null;
}
@Override public void fileFolderCreated(FileEvent fe) { }
@Override public void fileDataCreated(FileEvent fe) { }
@Override public void fileChanged(FileEvent fe) { }
@Override public void fileRenamed(FileRenameEvent fe) { }
@Override public void fileAttributeChanged(FileAttributeEvent fe) { }
};
Runnable run = new Runnable() {
@Override
public void run () {
fileObject.addFileChangeListener(WeakListeners.create(FileChangeListener.class, p.fileChangeListener, fileObject));
// I think this is the problem, listener is registered after remove event
// this creates a lot of calls to FileObject.removeFileChangeListener()
}
};
if (EventQueue.isDispatchThread()) {
Utils.postParallel(run);
} else {
run.run();
}
return p;
} else {
return null; // e.g. FileObject from a jar filesystem
}
} else {
return new VCSFileProxy(fileObject.getPath(), fileProxyOperations);
}
} catch (FileStateInvalidException ex) {
Logger.getLogger(VCSFileProxy.class.getName()).log(Level.SEVERE, null, ex);
}
return new VCSFileProxy(fileObject.getPath(), null);
}And finally the second problem:
org.netbeans.modules.masterfs.filebasedfs.fileobjects.FileObjectFactory.issueIfExist(...)
if (parent != null && parent.isValid()) {
if (child != null) {
if (foForFile == null) {
exist = (realExists == -1) ? true : touchExists(file, realExists);
if (fcb.impeachExistence(file, exist)) {
exist = touchExists(file, realExists);
if (!exist) {
refreshFromGetter(parent,asyncFire);
}
}
assert checkCacheState(true, file, caller);
} else if (foForFile.isValid()) {
exist = (realExists == -1) ? true : touchExists(file, realExists);
if (fcb.impeachExistence(file, exist)) {
exist = touchExists(file, realExists);
if (!exist) {
refreshFromGetter(parent,asyncFire);
}
}
assert checkCacheState(exist, file, caller);
} else {
//!!!!!!!!!!!!!!!!! inconsistence
exist = touchExists(file, realExists);
if (!exist) {
//Problematic test, for broken link is false
refreshFromGetter(parent,asyncFire);
}
}
} else {if (!exist) { changed to if (!exist && !Files.isSymbolicLink(file.toPath())) {
And CPU has been quiet since then..
Attention change just this one line, I tried to move it into the method touchExists, but again it was the result Suspended
I created plugins with patches for Netbeans 8.2
https://github.com/NBANDROIDTEAM/org-netbeans-modules-masterfs-patches
Add new update center http://server.arsi.sk/masterfs/updates.xml and install modules:
org-netbeans-modules-masterfs-patch-module
org-netbeans-modules-versioning-masterfs-patch-module
After installation restart Netbeans..
And the Unix domain socket problem
org.netbeans.modules.masterfs.filebasedfs.fileobjects.BaseFileObj.refresh(boolean , boolean ) to:
public final void refresh(final boolean expected, boolean fire) {
Statistics.StopWatch stopWatch = Statistics.getStopWatch(Statistics.REFRESH_FILE);
stopWatch.start();
try {
if (isValid()) {
refreshImpl(expected, fire);
if (isValid()) {
final File file = getFileName().getFile();
final boolean isDir = file.isDirectory();
final boolean isFile = file.isFile();
final boolean isOther = isOther(file);
if ((isDir == isFile || isFolder() != isDir || isData() != isFile) && !isOther) {
invalidateFO(fire, expected, true);
}
} else if (isData()) {
refreshExistingParent(expected, fire);
}
}
} catch (Error e) { // #249301
LOG.log(Level.INFO, "Cannot refresh file {0}", getPath()); //NOI18N
throw e;
} finally {
stopWatch.stop();
}
}
private static boolean isOther(File file) {
boolean other = false;
try {
other = Files.readAttributes(file.toPath(), BasicFileAttributes.class).isOther();
} catch (IOException ex) {
Exceptions.printStackTrace(ex);
}
return other;
}