ATSPM + SFTP Support
markKHA opened this issue · 16 comments
Has anyone used SFTP to get data off controllers? We are working with some Commander controllers running Trafficware that have FTP disabled, but SFTP enabled. We understand that ATSPM doesn't support STFP right now, but we're curious if any agencies have developed a workaround.
You can try changing the controller types to 9 (EOS). I believe there is something in that that will use FTPs if set to those. There is also an update to the FluentFTP library that can auto detect connection settings for either FTP or FTPs. Doing so slightly increases each connection time and therefore overall logging time. When I get a chance I will try the logging out using auto connect and see how much of a time difference it makes
I'll give that a shot. These controllers claim to only support SFTP though, not FTP or FTPS.
hmmm.... Yeah, I haven't seen anyone use that yet. I can see if I can find an easy library to use but I would have no easy way to test it. Would be interesting to see who else needs to add it.
Yes! SCP, SFTP, FTP is all the same! This is just done through a windows scheduled task with a script that pulls the data from the controller and dumps/copies it to a local storage location (ie. on the server that pulls the data).
@markKHA just so you know, we're working on a new logging service that will make it much easier to implement different protocols like SFTP
@runneals I don't know what you mean by those are all the same but currently there is only a logger for FTP and HTTP (for a specific manufacturer)
@markKHA FDOT has some experiences integrating with cubic controllers using SFTP. There are two components I added to the original UDOT solution to make SFTP work.
Step 0: Install SSH.NET from Nuget package, I am using the library written by Renci, which has 33.8M downloads.
Step 1: Under MOE.Common application, add two SFTP handler functions: (you can name the function as you fit)
public void GetCubicFilesAsync()
{
// run the sftp fetch operation async
Thread sftpFetch = new Thread(delegate () {
// to-do: replace with common data access to access signal IP in batch from ATSPM DB
string host = "signal IP";
string username = "SFTP signal user name";
string password = "SFTP signal password";
string remoteDirectory = "SFTP remote directory";
string localDirectory = "local path to store dat files from remote directory";
using (SftpClient sftp = new SftpClient(host, username, password))
{
try
{
sftp.Connect();
var files = sftp.ListDirectory(remoteDirectory);
var cubicFiles = files.Where(x => x.FullName.Contains(".dat")).ToList();
//download current files, remove files from remote directory
TransferCubicFiles(cubicFiles, localDirectory, sftp);
sftp.Disconnect();
}
catch (Exception ex)
{
//to-do: add some custom error handling as fit
throw ex;
}
}
});
sftpFetch.Start();
}
private void TransferCubicFiles(List<SftpFile> receivedFiles, string directory, SftpClient client)
{
var errorRepository = ApplicationEventRepositoryFactory.Create();
try
{
foreach (var ret in receivedFiles)
{
// FullName will include full path in sFTP
// Name will just include the file name without path
string fileName = ret.Name;
string remoteFileName = ret.FullName;
using (Stream fileStream = File.OpenWrite(Path.Combine(directory, fileName)))
{
Console.WriteLine("Downloading {0}", fileName);
//copy file and get to local diretory
client.DownloadFile(remoteFileName, fileStream);
//remove file in remote directory
Console.WriteLine("deleting {0} in sFTP instance", remoteFileName);
//delete file in remote sFTP directory
if (client.Exists(remoteFileName))
{
client.DeleteFile(remoteFileName);
}
}
string newFileName = RenameDatFiles(fileName, Signal.SignalID);
File.Move(Path.Combine(directory, fileName), Path.Combine(directory, newFileName));
}
}
catch (FTPException ex)
{
//capture if there is any sFTP related exception
errorRepository.QuickAdd("sFTPFromControllers", "SignalFtp", "TransferCubicFiles", ApplicationEvent.SeverityLevels.Medium, Signal.SignalID + " @ " + Signal.IPAddress + " - " + ex.Message);
Console.WriteLine(Signal.SignalID + " @ " + Signal.IPAddress + " - " + ex.Message);
}
catch (IOException ex)
{
//capture if there is any file IO exception
errorRepository.QuickAdd("sFTPFromControllers", "SignalFtp", "TransferCubicFiles", ApplicationEvent.SeverityLevels.Medium, Signal.SignalID + " @ " + Signal.IPAddress + " - " + ex.Message);
Console.WriteLine(Signal.SignalID + " @ " + Signal.IPAddress + " - " + ex.Message);
}
}
Step 2: Create another console application called SFTPFromControllers, you can reuse most of the code from FTPFromAllControllers, but make sure to call the public function defined in step 1.
Step 3: After compile, deploy the package to your server and run this in a scheduled windows task.
Hope it helps.
Yi
The new logger will be a service and will run any protocols at once. no need to have a bunch of little scripts running on a task scheduler and overlapping each other. When it is complete I'll post a how to in the wiki. There is a base class for a downloader and a decoder. anyone can easily implement these and override two methods that return a DirectoryInfo object that then passes that directory to the decoder.
@ychehntb This is fantastic - thank you for your functions. We'll integrate it into our deployment and see how it does with Trafficware.
@ChesireFreeThrow should this issue remain open? I think there's very valuable conversation and code in here that could be beneficial to agencies until the new logger is complete. It's harder to find when it's closed.
@ChesireFreeThrow Will do and I will make sure it formats correctly. It seems like I dont have the edit privilege, is there any specific role required for enable editing the wiki page?
@ChesireFreeThrow It seems like I am not invited to the ATSPM git repository, do you mind adding me in the repository so that I can push related changes in the future if needed. For your reference, I posted the link below with updated md file of SFTP example, I really appreciate if you are able to grab and updated the master branch of this wiki page.
Yi
Hi Yi,
Can you tell me what the 'RenameDatFiles' function does?
Just wanted to follow up on this and get some clarity on what the 'RenameDatFiles' function does.
@markKHA Hi Mark, sorry for get back to you late. The "RenameDatFiles" function is some customization FDOT does as we deal with some controller cabinet files which are not named in ".dat" coming out of SFTP. I believe you can comment out this function as it doesn't affect any of the SFTP polling function. Hope it helps.
Yi