1 (quasi)issue + a few Q's
Closed this issue ยท 13 comments
Hi ๐ THANK YOU for copyparty ๐ it's really great!
First, the issue. This very probably is not your issue, which is precisely why I didn't create a Bug Report to bring it to your attention and elected to go with "Something Else," but about 2 weeks or so ago, completely out of nowhere, copyparty "broke." I run a script at boot which invokes copyparty so there's very little handling on my part once arguments have been settled on, so I'm not really sure what I did but I would not be the only instance of someone mentioning this if it were on your end so I'm pretty confident it was something that either me or my machine did.
Anyway, from this point onward (and still to this day), any copyparty commands are met with:
Traceback (most recent call last):
File "/usr/bin/copyparty", line 5, in <module>
from copyparty.__main__ import main
ModuleNotFoundError: No module named 'copyparty'
so the binary is still present. I alluded to this only being a quasi-issue because I was able to install a second (functioning, thankfully) instance of copyparty via pip (or rather, pipx). I honestly don't recall how or why I ended up doing this but assume it must've been the shortest path to a working copyparty server (fully functioning, anyway. I know of the emergency backups and nearly had to go that route). The strange thing is that when looking at the binary file which runs without a hitch, located at /~/.local/bin/ and comparing it with the one that fails, located at /usr/bin/, they're the same, verbatim. The only difference is that the former executes as expected and the latter fails with the above output Re: missing module. I wish I knew enough about Linux binaries to provide more info but that's about all I've got on hand. However, I'm happy to provide anything that might be helpful, as needed. Anyway, as mentioned I'm up and running again but I don't love having a broken instance installed or needing to include /home/user_name_here/.local/bin before my copyparty commands, even if it is the case that they're issued via script 9x out of 10.
Any clue what I might've done and/or more importantly, what I can do to right the ship and get the party going again?
Now the couple of questions, if I may. They should be pretty easy compared to the mess above. First, is there any way to get copyparty to recognize and play TS files(.ts) as videos rather than offer to open them as (longAF) text files as is the default behavior? I poured over the README and supplemental documentation and couldn't find any plugins or anything that appeared to extend video codec support. I guess I just assumed that copyparty would support TS files if the browser in use did but that doesn't seem to be the case.
Thing 2: I know I saw this in the documentation somewhere, but I can't for the life of me find it now so please forgive me for asking but how do I essentially give myself "A" permissions and anonymous users no perms, like not even read? Would I need to resort to some of the trickery you mention, like nesting within a hidden folder or some such?
Last thing: I run copyparty with both -e2dsa and -e2ts (this isn't the question, but is that necessary or redundant? I couldn't tell from the documentation, but it seems the latter is for metadata only so I think it's ๐ also, iirc I only did this as a means to try to account for the issue I'm about to explain) but when I attempt to use the search function, nothing is ever returned. On a few occasions, I've noticed upon clicking the ๐, suddenly all of my many files disappear and only a handful of files are left, which also disappear as soon as I enter my query (assuming the query doesn't match with one of the 5 or so files, which it wouldn't because they're right there ) , which leads me to believe that something is off with indexing. Possibly relevant, in the early days before I added a crap ton of arguments (I've copy-pasta'd the command in its entirety below, in case a poorly chosen/conflicting argument, etc. could be causing any of my troubles), there was A LOT of indexing going on, which always prevented uploads even hours and iirc, possibly even after a full day or more. So, since every time I actually looked at what was being indexed it seemed to be indexing dot files, none of which I need to be indexed, I appended the no index argument, like so --no-idx \..+
This seemed to do the job but maybe too well, as now I see no indexing happening. Did I eff up? I don't recall if I was able to search prior to adding that argument, though I suppose I could easily find out but as I'm already this far and nearly finished, I'm going to hit submit rather than wait to test it out only to inevitably get distracted and fail to finish this monstrosity of an issue.
/home/user_name_here/.local/bin/copyparty -e2dsa -e2ts --no-robots -a user_name_here:pw -v .::r:A,user_name_here --no-idx ..+ --dbd=wal -i 1.2.3.4 -p 3923 --no-dupe --xlink --db-act 30 --no-athumb --no-acode --nos-dup --grid --favico ๐พ --theme 6
(I know I should probably go the config file route...)
PS - sorry for my long-windedness, but really - thanks again for diligently maintaining the project!
Hey, and thanks for using my software! Sounds like you're giving it a proper workout too ๐
And awesome work digging through that issue -- welcome to the world of fun that is python packaging ๐ ๐
When you install a python package with pip install --user, it'll install to ~/.local like you've already seen, but it doesn't plonk everything into one single place -- instead it kinda sprays all over in there. You get the binaries in ~/.local/bin/, maybe some docs in ~/.local/share, and then the actual python code goes in ~/.local/lib/pythonX.Y/site-packages/copyparty. That's how it works for most python software in general too
And here's the kicker; this sounds a lot like the kind of thing that would happen if you install the package locally, and then later something like one of these things happens:
- python is upgraded, from say 3.11 to 3.12 or such, so it goes looking in
~/.local/lib/python3.12/instead of~/.local/lib/python3.11/ - or maybe the packages in
~/.local/lib/pythonX.Y/site-packagesare deleted
As for why the launcher binary that's in /usr/bin works while the one in ~/.local/bin doesn't, I would have guessed that the very first line, the #! one would be different. That line (called a shebang) tells Linux what interpreter to run the script with, so for example, if it says /usr/bin/python3 then you're actually running /usr/bin/python3 copyparty foo every time you run copyparty foo. So, if one of the launchers says /usr/bin/python3.11 and the other is like /usr/bin/env python3 then that's it, we figured it out! But if those are the same too, then we'll have to keep digging :>
..and since I mentioned /usr/bin/env python3 briefly in the example, that's telling linux to grab the first python3 binary it can find (according to $PATH) and to use that
one thing to note is that the launcher binary is pretty much just a fancy alternative to python3 -m copyparty, or more specifically /usr/bin/python3 -m copyparty if that's what the shebang says. So that's another way to check.
but I'm curious if the following command would fix it? and if nothing else, it might throw us a hint in the output: python3 -m pip install --user -U copyparty
as for the questions,
is there any way to get copyparty to recognize and play TS files(.ts) as videos rather than offer to open them as (longAF) text files as is the default behavior?
if they do appear as text in your browser, then my first hunch is that either copyparty is sending the wrong mimetype to the browser, or the browser doesn't understand it... But maybe I'm missing some details here; is this something that happens if you click the filename to download it, or are you clicking the txt link next to the filename, or maybe something else? Oh and, I'm guessing your copyparty is running on a Linux distro, but what browser/OS are you accessing it from? At any rate, we can at least verify the mimetype suspicion with curl: curl -vo /dev/null http://127.0.0.1:3923/some.ts should mention Content-Type: video/mp2t near the end, if that's working as intended.
how do I essentially give myself "A" permissions and anonymous users no perms, like not even read?
that comes down to this part of your command, -v .::r:A,user_name_here -- to make it slightly less confusing, this is exactly the same as -v ./:/:r:A,user_name_here, or in other words, share the folder you're launching copyparty from (./) at the server root (/), with read-access for everyone (since there's no usernames before the next :), and A for yourself. So you'd get what you want with -v .::A,user_name_here instead :>
but just to mention what this could all look like in a config file instead,
[global]
e2dsa, e2ts, xlink, grid, no-dupe
no-athumb, no-acode, no-robots, nos-dup
no-idx: /\.
i: 1.2.3.4
db-act: 30
favico: ๐พ
theme: 6
[accounts]
user_name_here: the_pw
[/]
/home/dave/Music
accs:
A: user_name_hereboth -e2dsa and -e2ts (this isn't the question, but is that necessary or redundant?
almost, but not quite -- -e2ts does enable -e2ds, but the a in -e2dsa makes it scan through absolutely all volumes, and not just the ones where someone has write-access.
there was A LOT of indexing going on, which always prevented uploads even hours and iirc, possibly even after a full day or more
yeah, this is kind of a bummer... but I haven't found a solid way to avoid it yet. The biggest practical issue is that it would mess with the deduplication if someone's allowed to upload a dupe which just hasn't been discovered locally yet, but it'd also require some proper cleanup in the indexing/database code before that would be safe to even consider! so maybe one day hehe
I appended the no index argument, like so
--no-idx \..+
yup, there we go -- that's gonna skip any file that has a file extension ๐ this one should work better: --no-idx '/\.' to catch any files with a dot immediately following a directory separator (and note the single-apostrophes to prevent the shell from eating the backslash, just in case)
that's all I can think of though, your setup looks alright aside from that ๐ ๐
PS - sorry for my long-windedness
not at all! thx for the effort, happy to help :>
EDIT: the idea was good, but this doesn't really work...
okay, think I have an idea how to avoid having to wait for indexing, if you're alright with possibly accepting duplicate uploads while it's indexing...
if you change the -e2dsa -e2ts to just -e2d -e2t (to keep the database enabled but without filesystem scanning) and start it like normal, but then, while that is up and running, you start a second instance of copyparty with the exact same arguments plus these extra ones at the end: -e2dsa -e2ts -p 0 --ign-ebind-all --exit=idx
the second copyparty doesn't accept any connections, and then runs through the indexing before it exits on its own once it's finished. After that you can change back the main copyparty to -e2dsa -e2ts like before so it'll catch any new files added to the filesystem the next time you start it.
I think this should be safe, since the actual database backend (SQLite) is real solid stuff and should prevent any unintended competition between the copyparties. Let me know how it goes if you decide to give this a try! ๐
Wow, that might well be a new record for timeliest dev reply, at least as far as personal experience goes. So thank you for that but more so for the comprehensive and thoughtful response therein!
I generally respond to text-based communication in a linear fashion, but I'm sorry... I can't help but still be laughing at myself for effectively instructing copyparty not to index anything with a file extension, so essentially everything and anything, yet still have the audacity to wonder why searches we're coming back empty-handed ๐ ๐คฃ Though, it does make me wonder about the 5 or so files that regularly did turn up prior to executing the search but after clicking on the ๐ ยฏ\_(ใ)_/ยฏ
Either way, thank you! Despite being what I closed out the issue with this was the one that cropped up most often and was the most bothersome of the issues I mentioned by a substantial margin, including the quasi-issue itself.
Speaking of, you were exactly right regarding your suspicion about differing shebangs. I guess I've grown accustomed to looking at everything following but not the shebang itself. Sorry for the oversight. Also I'm not sure this is of any consequence but it's actually the other way around, the binary at ~/user_name_here/.local/bin/ is the working one and the one at /usr/bin/ fails with the error I'd given. Further, because I use an Arch-based distro and therefore pacman, I get the externally managed environment error when trying to use pip without venv and to my knowledge, you can't use the --user -U portion of python3 -m pip install --user -U copyparty with venv. Or at least I wasn't able to and am not well versed enough w/ pip to know otherwise. That said, the command without that chunk did work to install or reinstall copyparty and I thought everything was going to be golden as far as the quasi issue was concerned, but unfortunately that particular error regarding the module being missing persists. So at least for the time being I'll have to make do with the alternate installation, which is not much of an inconvenience at all. I appreciate your input and am open to any further suggestions you might have in this regard but don't feel obligated by any means. Also thank you for the refresher course on Python packaging, I genuinely needed it.
And here's the kicker; this sounds a lot like the kind of thing that would happen if you install the package locally, and then later something like one of these things happens:
- python is upgraded, from say 3.11 to 3.12 or such, so it goes looking in ~/.local/lib/python3.12/ instead of ~/.local/lib/python3.11/
- or maybe the packages in ~/.local/lib/pythonX.Y/site-packages are deleted
Are you wondering aloud if one of these might be what's happened in my case or just commenting on the somewhat disjointed nature of python packaging? Or perhaps "door โ.3?"
as for the questions,
is there any way to get copyparty to recognize and play TS files(.ts) as videos rather than offer to open them as (longAF) text files as is the default behavior?
if they do appear as text in your browser, then my first hunch is that either copyparty is sending the wrong mimetype to the browser, or the browser doesn't understand it... But maybe I'm missing some details here;
is this something that happens if you click the filename to download it, or are you clicking the txt link next to the filename, or maybe something else?
So, if I click on the thumbnail in grid view, I get e.g., "this file is 355.4 MiB large -- really view as text?" However, if I click on the file name in list view then it tries to open the video straight away but the video fails to open and has the "broken media" symbol within the player. Interestingly(sort of), when I click a TS file located among search results (because search is working, yay!!), depending on the browser it either prompts a download or automatically starts it.
Oh and, I'm guessing your copyparty is running on a Linux distro, but what browser/OS are you accessing it from?
You guessed correctly! it's an Arch-based distro called Garuda Linux. However, the browser(s) I've most often experienced this TS files issue with is on my Android tablet and phone on which it occurs in any browser, but I just checked on my Linux machine and the same behavior is seen in any browser there as well. Or at least the few that I tried, I assume it will persist across any others.
At any rate, we can at least verify the mimetype suspicion with curl:
I generally respond to text-based communication in a linear fashion, but I'm sorry... I can't help but still be laughing at myself for effectively instructing copyparty not to index anything with a file extension, so essentially everything and anything, yet still have the audacity to wonder why searches we're coming back empty-handed ๐ ๐คฃ Though, it does make me wonder about the 5 or so files that regularly did turn up prior to executing the search but after clicking on the ๐ยฏ\_(ใ)_/ยฏ
Either way, thank you! Despite being what I closed out the issue with this was the one that cropped up most often and was the most bothersome of the issues I mentioned by a substantial margin, including the quasi-issue itself.
Speaking of, you were exactly right regarding your suspicion about differing shebangs. I guess I've grown accustomed to looking at everything following but not the shebang itself. Sorry for the oversight. Also I'm not sure this is of any consequence but it's actually the other way around, the binary at ~/user_name_here/.local/bin/ is the working one and the one at /usr/bin/ fails with the error I'd given. Further, because I use an Arch-based distro and therefore pacman, I get the externally managed environment error when trying to use pip without venv and to my knowledge, you can't use the --user -U portion of python3 -m pip install --user -U copyparty with venv. Or at least I wasn't able to and am not well versed enough w/ pip to know otherwise. That said, the command without that chunk did work to install or reinstall copyparty and I thought everything was going to be golden as far as the quasi issue was concerned, but unfortunately that particular error regarding the module being missing persists. So at least for the time being I'll have to make do with the alternate installation, which is not much of an inconvenience at all. I appreciate your input and am open to any further suggestions you might have in this regard but don't feel obligated by any means. Also thank you for the refresher course on Python packaging, I genuinely needed it.
And here's the kicker; this sounds a lot like the kind of thing that would happen if you install the package locally, and then later something like one of these things happens:
- python is upgraded, from say 3.11 to 3.12 or such, so it goes looking in ~/.local/lib/python3.12/ instead of ~/.local/lib/python3.11/
- or maybe the packages in ~/.local/lib/pythonX.Y/site-packages are deleted
Are you wondering aloud if one of these might be what's happened in my case or just commenting on the somewhat disjointed nature of python packaging? Or perhaps "door โ.3?"
as for the questions,
is there any way to get copyparty to recognize and play TS files(.ts) as videos rather than offer to open them as (longAF) text files as is the default behavior?
if they do appear as text in your browser, then my first hunch is that either copyparty is sending the wrong mimetype to the browser, or the browser doesn't understand it... But maybe I'm missing some details here;
is this something that happens if you click the filename to download it, or are you clicking the txt link next to the filename, or maybe something else?
So, if I click on the thumbnail in grid view, I get e.g., "this file is 355.4 MiB large -- really view as text?" However, if I click on the file name in list view then it tries to open the video straight away but the video fails to open and has the "broken media" symbol within the player. Interestingly(sort of), when I click a TS file located among search results (because search is working, yay!!), depending on the browser it either prompts a download or automatically starts it.
Oh and, I'm guessing your copyparty is running on a Linux distro, but what browser/OS are you accessing it from?
You guessed correctly! it's an Arch-based distro called Garuda Linux. However, the browser(s) I've most often experienced this TS files issue with is on my Android tablet and phone on which it occurs in any browser, but I just checked on my Linux machine and the same behavior is seen in any browser there as well. Or at least the few that I tried, I assume it will persist across any others.
At any rate, we can at least verify the mimetype suspicion with curl: curl -vo /dev/null http://127.0.0.1:3923/some.ts should mention Content-Type: video/mp2t near the end, if that's working as intended.
Ding ding ding, you nailed it again! You're pretty good at this ๐
* Request completely sent off
< HTTP/1.1 403 Forbidden
< Content-Length: 62
< Connection: Keep-Alive
< Date: Tue, 07 May 2024 10:03:03 GMT
< Vary: Origin, PW, Cookie
< Cache-Control: no-store, max-age=0
< X-Robots-Tag: noindex, nofollow
< โญContent-Type: text/html; charset=utf-8โญ
<
{ [62 bytes data]
100 62 100 62 0 0 58106 0 --:--:-- --:--:-- --:--:-- 62000
* Connection #0 to host left intact
I'd be lying if I said I know where to go from here, though, if I'm being honest.
(see my edit /post script)
both -e2dsa and -e2ts (this isn't the question, but is that necessary or redundant?
almost, but not quite -- -e2ts does enable -e2ds, but the a in -e2dsa makes it scan through absolutely all volumes, and not just the ones where someone has write-access.
I'm thinking that perhaps this is a bit extreme for my needs but I'll continue to mull it over for a bit before I make any changes.
there was A LOT of indexing going on, which always prevented uploads even hours and iirc, possibly even after a full day or more
yeah, this is kind of a bummer... but I haven't found a solid way to avoid it yet. The biggest practical issue is that it would mess with the deduplication if someone's allowed to upload a dupe which just hasn't been discovered locally yet, but it'd also require some proper cleanup in the indexing/database code before that would be safe to even consider! so maybe one day hehe
well, it's still indexing at the moment but it also hasn't got a chance to index anything in weeks or months even so we'll see if/when that chills out and take it from there but fingers crossed for one day, if for no other reason then to make your life easier, or the lives of your users, which for a developer like yourself, I would imagine is effectively the same thing.
I appended the no index argument, like so
--no-idx \..+
yup, there we go -- that's gonna skip any file that has a file extension ๐ this one should work better:
--no-idx '/\.'to catch any files with a dot immediately following a directory separator (and note the single-apostrophes to prevent the shell from eating the backslash, just in case)
Still laughing at this ๐๐ ๐๐คฃ๐คก
that's all I can think of though, your setup looks alright aside from that ๐ ๐
Now if I could just figure out that pesky TS file issue. That one really has me stumped especially now that I find it's the browsers which seem to be the issue despite them supporting the file format or do I have what curl returned backwards?
(edit - I wish GitHub had a strikethrough text styler, I would have used it here but sufficed to say that I was wrong and had it exactly backwards)
Regardless, words really can't express how grateful I am for both your feedback to my low priority issues and the promptness of it and generally for your dedication to the software upkeep/maintenance. I'm sure you get it fairly often because it's very abundantly clear but you really are an exemplary developer/programmer and the software shows it!
I'll give you an opportunity to respond but it's pretty safe to say that this could be closed out whenever you think appropriate or upon my next response whichever comes sooner. The TS file thing is fairly low priority.
edit - is there a way to manually set MIME types for specific file extensions w/ copyparty?
edit 2 - can you not move files while indexing is ongoing as well as upload?
okay, think I have an idea how to avoid having to wait for indexing, if you're alright with possibly accepting duplicate uploads while it's indexing...
if you change the
-e2dsa -e2tsto just-e2d -e2t(to keep the database enabled but without filesystem scanning) and start it like normal, but then, while that is up and running, you start a second instance of copyparty with the exact same arguments plus these extra ones at the end:-e2dsa -e2ts -p 0 --ign-ebind-all --exit=idxthe second copyparty doesn't accept any connections, and then runs through the indexing before it exits on its own once it's finished. After that you can change back the main copyparty to
-e2dsa -e2tslike before so it'll catch any new files added to the filesystem the next time you start it.I think this should be safe, since the actual database backend (SQLite) is real solid stuff and should prevent any unintended competition between the copyparties. Let me know how it goes if you decide to give this a try! ๐
Holy crap, you're a machine! I was sitting here rereading my response and editing as needed and looked up and you had responded again. Words fail me other than to say thank you. Although I have to ask, I'm sure you wouldn't suggest it if it didn't work but in the past when I've tried to run a second instance of copyparty, I always get:
Exception: port 3923 is busy on interface blah.di.blah.1
Or, by
the second copyparty doesn't accept any connections, and then runs through the indexing before it exits on its own once it's finished.
Are you essentially saying that it will do just that but not before performing whatever indexing it needs to perform according to the command plus arguments? If so that's a pretty creative approach!
Still, assuming you are correct, which I think is a pretty safe assumption, this is a perfectly workable solution for me. As 99.9% of the time I'm going to be the only one accessing/uploading files from/to the copyparty server and I should be able to manage not duplicating uploads. Although I've screwed up simpler things, so I guess we'll see lol
Seriously man, thank you! I can't test this out at the moment but I should be able to before the day is out and I'll be sure to let you know once I have & how/if it worked!
Are you wondering aloud if one of these might be what's happened in my case
mostly this, but I assumed it was the /usr/bin launcher that was working and the ~/.local one that didn't, so ignore that part :p
though since you mentioned you're on an arch-based distro, are you also using the aur package to install copyparty?
to my knowledge, you can't use the
--user -Uportion ofpython3 -m pip install --user -U copypartywith venv.
ah yep that's right, while in a venv the correct thing would be just pip install -U copyparty , so not even the python3 -m
the video fails to open and has the "broken media" symbol within the player
just going by this, it sounds like your browser doesn't support playing .ts files - come to think of it, are there any browsers which do? I'm not seeing it listed on mdn, so my assumption would have been no; https://developer.mozilla.org/en-US/docs/Web/Media/Formats
if I click on the thumbnail in grid view, I get e.g., "this file is 355.4 MiB large -- really view as text?"
heh, yeah that's pretty pointless behavior -- assuming there's no browsers able to play .ts files, it should just start a download instead ๐
edit - is there a way to manually set MIME types for specific file extensions w/ copyparty?
not yet! that would be super useful, so I'll get that added
edit 2 - can you not move files while indexing is ongoing as well as upload?
nope ... ๐
when I've tried to run a second instance of copyparty, I always get:"port busy"
yup! and usually that makes sense, because you can't have multiple things listening on the same tcp port, so you'd have to give it a different port with -p, but in this case we don't want the 2nd instance to be able to respond to any requests at all, just handle the indexing for us without getting too much in the way of the 1st instance... so we tell it to listen on port 0 (which makes no sense), and then to ignore the fact that it's not able to listen for connections with --ign-ebind-all
but.... here's the bad news -- it turns out the idea didn't really work :(
the 2nd instance grabs the database so hard that the regular copyparty isn't able to do much of anything at all. So unfortunately you'll still need to wait for indexing to finish before uploads/moves/renames becomes possible. Ahh...
at least it's only the initial indexing that takes forever, so once all the files are in the database it'll be MUCH faster the next time you restart it :>
and thanks for the nice words! best payment really ๐
Well it's a bummer that it doesn't work but at least that gets me off the hook for not finding the time to test it out yesterday ๐ฅธ
the video fails to open and has the "broken media" symbol within the player
just going by this, it sounds like your browser doesn't support playing .ts files - come to think of it, are there any browsers which do? I'm not seeing it listed on mdn, so my assumption would have been no; https://developer.mozilla.org/en-US/docs/Web/Media/Formats
If this is the case then I feel like a first-rate jerk for even bothering you with it and now that you mention it I don't know that I've ever expressly been able to play a TS file in browser via the browsers media player but I know that different players hosted on X, Y or Z site is capable of playing them. I only know this because almost every single TS file I have was ripped from one of these sites. Sadly it seems like it's a very popular format these days so while most of my video files are in MP4 format, probably upwards of a good ยผ of them now which were ripped from an online source as opposed to downloaded in a prepared download if that makes sense are in the TS format. I'm sure I can find a SFW example if it would be of any help but it sounds like I might be SOL anyway. It just makes little sense to me that the browser would be able to play it in one context but not in another but that shows you how little I know about the inner workings of web browsers and things of that nature. Anyway I'm going to close this issue out with this comment because you have gone above and beyond to help me with everything that seems helpable but if you would be so kind as to let me know if and when you are able to incorporate some sort of manual override for MIME type, I would be very interested to know about it!
Thanks again for everything, my friend! It's been a pleasure chatting with you and while I don't wish for any issues to crop up with my copyparty server, I can rest easy knowing that if they should it'll be a pleasurable experience dealing with them if it means having to consult with you on the matter. If there's ever anything I can do on my end in terms of testing out any new features or features that you're considering but not sure about implementing in the release or anything of that nature, please don't hesitate to ask! It's the very least I could do!
It looks like your suspicion was/is correct and I may very well be completely SOL
However, maybe there is room for some hope, after all ๐ค
It looks like it's possible but perhaps not advisable or easy or some combination thereof
According to Co-Pilot it is indeed possible but I'll let you decide how feasible or worthwhile it may be. Here's a copy of its response when directly asked the question about playing local TS files in one's web browser:
Hello! Playing local transport stream files (
.ts) directly in a web browser can be challenging because most browsers do not support the.tsformat natively. However, there are workarounds to play these files:
HTML5 Video Tag: You can use the HTML5
<video>tag if the.tsfiles are in a format compatible with HTTP Live Streaming (HLS). This method works well if you have an.m3u8playlist file pointing to your.tsfilesยน.Media Source Extensions (MSE): This is a W3C specification that allows JavaScript to send byte streams to media codecs within web browsers that support HTML5 video. The streams can be in the form of MPEG-2 Transport Stream (MPEG-TS) or ISO Base Media File Format (MP4). Libraries like
hls.jscan be used to transmux.tsfiles to a compatible format on the flyยฒ.Browser Extensions: Some browser extensions can play
.tsfiles by implementing HLS. For example, HLS Player is a Chrome extension that can play.m3u8files and relies on HTML5 video and MediaSource Extensions for playbackโด.Conversion Tools: Convert
.tsfiles to a more widely supported format like.mp4using tools likeffmpeg. This conversion can be done server-side or client-side, although it might introduce a slight delay in live streaming scenariosยฒ.Third-Party Players: Use third-party players like JWPlayer or Flowplayer that can handle
.tsfiles. These players often have their own methods for dealing with streaming and compatibility issuesยน.Remember, while these methods can help you play
.tsfiles in a browser, the exact approach may vary based on your specific use case and the browser's capabilities. Always ensure you're using legal and compliant methods to play and stream content.Source: Conversation with Bing, 5/8/2024
(1) How to play .ts files (video/MP2T media type) in Browser?. https://stackoverflow.com/questions/16386510/how-to-play-ts-files-video-mp2t-media-type-in-browser.
(2) How to stream live TS (transport stream) files on HTML player?. https://stackoverflow.com/questions/65095047/how-to-stream-live-ts-transport-stream-files-on-html-player.
(3) HLSPlayer. https://www.hlsplayer.org/.
(4) Local Video Player - Online Video Player. https://onlinevideoplayer.net/local-video-player/.
(5) ffmpeg - View MPEG-TS stream in browser - Stack Overflow. https://stackoverflow.com/questions/48662572/view-mpeg-ts-stream-in-browser.
(6) undefined. http://osmfhls.kutu.ru/.
PS - for the record, I despise all things M$, but Co-Pilot allows one to use GPT-4 for free so ยฏ\_(ใ)_/ยฏ
If there was a way to share these files as videos to VLC or MX Player or whatever video player one might have on the device that would probably be the easiest fix by far but I couldn't figure out how to accomplish this. I can share mp4 files to a local player but sharing a TS file doesn't seem to shake the text/html MIME type. Something as simple as highlighting/selecting a particular file and having a share button might accomplish it but IDK what all that involves on your end and how difficult it would be to implement. It sounds simple from an amateur's perspective but that's easy to say from my armchair lol
I can share mp4 files to a local player but sharing a TS file doesn't seem to shake the text/html MIME type
funny you should mention, I just noticed a small detail from earlier -- when we tried to check the mimetype copyparty was returning for ts files, the response was a HTTP 403 Forbidden, so it was getting the login page instead of the ts files... so the mimetype was correct, since the login page is html hehe
I'm guessing the ts files are in a volume that is password-protected, so VLC is also getting that login page instead of the file.
there's two ways to solve that; either you can add your password to the end of the url, like ?pw=hunter2 -- but that's a bit of a hassle, so instead you might want to enable filekeys, which makes the server generate URLs which can be shared to access a file without having to log in.
Something as simple as highlighting/selecting a particular file and having a share button might accomplish it
so I think filekeys comes pretty close to this :>
you can enable filekeys by changing your -v .::A,yourname into this: -v .::A,yourname:g:c,fk -- this gives unauthenticated users get-access, meaning they can download files if they know the correct URL but not see directory contents, and then it locks it down a bit by also enabling fk (filekeys) so just knowing the URL isn't enough, but the url must also contain the server-generated per-file "password" -- so if you copy a link from the web-UI it'll have the correct filekey added to it already.
that said, a completely different workaround is to convert ("remux") the ts files into mp4 or mkv files, and that way you also save some disk space (around 15%) -- ts is very wasteful, as it was designed for unreliable networks, so it keeps repeating redundant data (headers). I think mp4 supports all the important features that ts files might use, and I think more players support mp4 than mkv, so that's probably(?) the better choice.
if you want to convert your ts files to mp4 files without any loss in quality, then something like this should work (copypaste all of it into a terminal in the folder with ts files):
for infile in *.ts;
do outfile="${infile%.*}.mp4";
[ -e "$outfile" ] && echo "outfile exists, skipping $outfile" && continue ;
if ffmpeg -nostdin -i "$infile" -map 0:v -map 0:a? -c copy -movflags +faststart "./$outfile" ;
then echo "converted OK; deleting $infile ..."; rm -- "$infile";
else echo "ERROR: failed to convert $infile"; break;
fi; donethe -movflags +faststart is only necessary for mp4 files and can be removed if you change it to mkv instead; what it does is move the file header from the end of the file to the start instead, so it doesn't have to do a bunch of seeking on the hdd before it can start playing
NOTE: the command deletes the input files if FFmpeg thinks they converted successfully! So if you have enough disk space and want to actually make sure they did, then remove the rm -- "$infile"; part :>
wait... does your .ts files have the file extension in uppercase? because I just noticed some code that assumes them to be lowercase ๐
Thanks for looking into this and providing all this helpful info! To answer your last question, no - to my knowledge every instance of a .ts file is in lower case, I also overestimated the number of them contained within my library. It's probably closer to 10% than 25%. As such and since I came to the decision prior to reading your responses here, I elected to go the conversion route. Although, I'm never really sure I completely understand the differences between re-encooding and remuxing. Also, I sure hope the answer is no because the process is already underway (but it just started so...), do you see anything wrong with the following?
#!/bin/bash
# Iterate
for file in *.ts; do
# Get the filename without extension
filename=$(basename -- "$file")
filename_no_ext="${filename%.*}"
# convert
ffmpeg -i "$file" -c copy "${filename_no_ext}.mp4"
done
Thanks for the other info, too. Despite electing to go the conversion route, it will be handy to know how to proceed in a manner that doesn't require converting everything every time. Especially considering this is going to take a lot longer than I bargained for. I was like, "Oh, it's only like 800 some files..." But forgot that many of them are in 4K format and a good chunk of them are feature-length movies. D'oh!
Edit - I think I may go the file key route anyway as that seems like the smart way to do things. Also, I too noticed the fact that the curl command returned a http 403 error and was curious about that....
the differences between re-encooding and remuxing.
as long as you have the -c copy in there, you are doing remuxing and not reencoding, which is good! remuxing is fast (barely uses any cpu, so is only limited by your HDD speed) and it retains the original quality exactly as it was -- the only difference is the container (the wrapping, so to say). Moving the video and audio streams from a box and onto a shelf or something like that hehe
but reencoding actually converts the video stream itself, which causes a quality loss and uses a LOT of cpu, so it also takes forever - but sometimes it is a necessary evil, if you need to play a file on a device that doesn't understand some of the codecs within. Usually the issue is the audio, which is much faster to reencode than video, but many automated tools reencode both, which is a huge waste...
I sure hope the answer is no because the process is already underway (but it just started so...), do you see anything wrong with the following?
nope, that looks good! I was being overly careful with my suggestion (-map 0:v -map 0:a?), which is more eager to intentionally crash if it hits a file that isn't entirely compatible with converting to mp4, but honestly I can't think of any realistic examples where that could happen. And copyparty is entirely fine with not having the -movflags +faststart in there, but some other web servers might not be! And finally, not having -nostdin is fine because you're doing iteration with a for-loop, instead of something recursive like find -iname '*.ts' | while read filename; do ffmpeg -i "$filename" ...
EDIT: just one thing to look out for is that your converter doesn't abort if one of the files fail to convert, so worst-case you might have some files which stopped midways, but that's also not likely to happen in practice :>
This is entirely unrelated to any of the issues we discussed, but I probably should have gone with your script after all. My computer has been acting up lately and my external hard drive has recently developed a habit of unmounting and remounting itself once every few hours or so for some reason and obviously when this happens, anything that's using files on it crashes or at the very least is interrupted. This happened while the process of converting .ts files was underway. The script I posted here would have been fine in this case, as it didn't delete anything but because it effectively equates to double the number of files until deleted, before I knew it I had like 30GB left on my 10TB HDD and it was crawling at a snail's pace. So I deleted the unnecessary files and adjusted the script so that it would delete the converted files but only in the case of a successful full conversion. Obviously it didn't catch that last part and when the hard drive unmounted itself I found that the file that it must have been in the process of converting was both incomplete and the original gone. So because now I didn't trust myself to make the necessary adjustments since I had considered this eventuality and thought I planned for it but failed, I consulted the nearest LLM chatbot. It provided a further modified script, only when the hard drive did its thing again, it turned out that this script also deleted not 1, not 2 but 3 files. It was partially my fault because I tried to quit the script midway (since I had to pipe "no" through it bc any previously started but not finished files would also be skipped entirely if "yes," Plus manually sitting there and answering "no" like thousands of times is for idiots. I made this realization while the script was underway ๐ฎโ๐จ๐คฆ๐ผโโ๏ธ) but I thought that there would be no harm in doing so since it was in process and wouldn't delete the file it was processing until after it was done at the earliest, and in which case it would be fine because it was done. Silly me. When I hit Ctrl+C, it just quit converting file, not the entire script and moved on to the next and idiot that I am, I just hit Ctrl+C again (I didn't yet realize that a file had already been deleted and I just deleted another) and it did the same thing. I finally quit in a way that actually quit the parent program/terminal window but not before deleting one more file in the process. And unfortunately as luck would have it, these files didn't just have some significance but also are apparently impossible to replace. Long story short I've since moved back to the script I posted here which involves the most amount of manual intervention but is also safe in that it won't delete anything. And of course the LLM chatbot shows no remorse ๐๐ญ