Issues with sending H.264 post conversion to the bot, please help!
Closed this issue · 3 comments
bengeek2 commented
Hello, i am trying to stream a raw video source on discord. to do this, I am taking the rawvideo source and encoding it to h264, then sending the output to the bot. Joining, streaming, and disconnecting all function as intended, and properly start and stop ffmpeg. if I modify the ffmpeg output to send to an .mp4, i can verify the encoding is working.
the ffmpeg command i am generating is this:
-f v4l2 -f v4l2 -pix_fmt yuyv422 -r 30 -i /dev/video0 -vcodec rawvideo -vcodec libx264 -filter:v scale=1080:720 -loglevel info -fflags nobuffer -analyzeduration 0 -preset ultrafast -tune zerolatency -f flv unix:./2.sock
the command is a bit scuffed as it calls format twice, but does work when you change output from unix:./2.sock to output.mp4
The issue I have is the bot's stream has no content. I believe the output for the ffmpeg command is incorrect, or it is integrating with H264NalSplitter properly. Can you please let me know what I am missing, or any direction to start moving in? I'm not very good with JS so I am stumped. Here is the index.ts file I am using:
import { MediaUdp, Streamer, setStreamOpts } from "@dank074/discord-video-stream";
import config from "./config.json";
import { Client, StageChannel } from "discord.js-selfbot-v13";
import ffmpeg from "fluent-ffmpeg";
import { StreamOutput } from "@dank074/fluent-ffmpeg-multistream-ts";
import { VideoStream, H264NalSplitter, streamOpts } from "@dank074/discord-video-stream";
const streamer = new Streamer(new Client());
setStreamOpts({
bitrateKbps: config.streamOpts.bitrateKbps,
maxBitrateKbps: config.streamOpts.maxBitrateKbps,
video_codec: 'H264'
});
let customFfmpegCommand: ffmpeg.FfmpegCommand | null = null;
streamer.client.on("ready", () => {
console.log(`--- ${streamer.client.user.tag} is ready ---`);
});
streamer.client.on("messageCreate", async (msg) => {
if (msg.author.bot) return;
if (!config.acceptedAuthors.includes(msg.author.id)) return;
if (!msg.content) return;
if (msg.content.startsWith(`$play-live`)) {
const args = msg.content.split(" ");
if (args.length < 2) return;
const url = args[1];
if (!url) return;
const channel = msg.author.voice.channel;
if (!channel) return;
console.log(`Attempting to join voice channel ${msg.guildId}/${channel.id}`);
await streamer.joinVoice(msg.guildId, channel.id);
if (channel instanceof StageChannel) {
await streamer.client.user.voice.setSuppressed(false);
}
const streamUdpConn = await streamer.createStream();
try {
await playVideo(url, streamUdpConn);
} catch (e) {
console.error("Error playing video:", e.message);
// Stop the stream if an error occurs during video playback
streamer.stopStream();
}
} else if (msg.content.startsWith("$disconnect")) {
if (customFfmpegCommand) {
customFfmpegCommand.kill("SIGINT");
customFfmpegCommand = null;
}
// Leave the voice channel and stop the stream when the user requests disconnection
streamer.leaveVoice();
streamer.stopStream();
}
});
streamer.client.login(config.token);
async function playVideo(video: string, udpConn: MediaUdp) {
try {
console.log("Starting customStreamVideo function...");
console.log("Video URL:", video);
const videoStream: VideoStream = new VideoStream(udpConn, streamOpts.fps);
const videoOutput = new H264NalSplitter();
const ffmpegCommand = ffmpeg(video)
.addOption("-loglevel", "info")
.addOption("-fflags", "nobuffer")
.addOption("-analyzeduration", "0")
.inputFormat('v4l2')
.inputOptions(['-f v4l2', '-pix_fmt yuyv422'])
.inputFps(streamOpts.fps)
.videoCodec('rawvideo')
.outputOptions(['-preset ultrafast', '-tune zerolatency'])
.videoFilters(`scale=${streamOpts.width}:${streamOpts.height}`)
.format('flv')
.videoCodec('libx264')
.output(StreamOutput(videoOutput).url, { end: false });
console.log("FFmpeg Command:", ffmpegCommand._getArguments().join(" "));
customFfmpegCommand = ffmpegCommand
.on("end", () => {
console.log("Video streaming ended.");
customFfmpegCommand = null;
streamer.stopStream();
})
.on("error", (err, stdout, stderr) => {
console.error("Error occurred:", err);
customFfmpegCommand = null;
streamer.stopStream();
})
.on("stderr", console.error);
customFfmpegCommand.run();
console.log("Video streaming completed successfully");
udpConn.mediaConnection.setSpeaking(true);
udpConn.mediaConnection.setVideoStatus(true);
} catch (e) {
console.error("Error playing video:", e.message);
throw new Error("cannot play video " + e.message);
}
}
bengeek2 commented
this is the console output when i say $play-live /dev/video0
Attempting to join voice channel xxxxxxx/xxxxxxx
Starting customStreamVideo function...
Video URL: /dev/video0
FFmpeg Command: -f v4l2 -f v4l2 -pix_fmt yuyv422 -r 30 -i /dev/video0 -vcodec rawvideo -vcodec libx264 -filter:v scale=1080:720 -loglevel info -fflags nobuffer -analyzeduration 0 -preset ultrafast -tune zerolatency -f flv unix:./2.sock
Video streaming completed successfully
ffmpeg version 4.4.2-0ubuntu0.22.04.1 Copyright (c) 2000-2021 the FFmpeg developers
built with gcc 11 (Ubuntu 11.2.0-19ubuntu1)
configuration: --prefix=/usr --extra-version=0ubuntu0.22.04.1 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-librsvg --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared
libavutil 56. 70.100 / 56. 70.100
libavcodec 58.134.100 / 58.134.100
libavformat 58. 76.100 / 58. 76.100
libavdevice 58. 13.100 / 58. 13.100
libavfilter 7.110.100 / 7.110.100
libswscale 5. 9.100 / 5. 9.100
libswresample 3. 9.100 / 3. 9.100
libpostproc 55. 9.100 / 55. 9.100
[video4linux2,v4l2 @ 0x558cb2665900] The driver changed the time per frame from 1/30 to 117/7013
Input #0, video4linux2,v4l2, from '/dev/video0':
Duration: N/A, start: 38315.786159, bitrate: 1988671 kb/s
Stream #0:0: Video: rawvideo (YUY2 / 0x32595559), yuyv422, 1920x1080, 1988671 kb/s, 59.94 fps, 59.94 tbr, 1000k tbn, 1000k tbc
Multiple -c, -codec, -acodec, -vcodec, -scodec or -dcodec options specified for stream 0, only the last option '-c:v libx264' will be used.
Stream mapping:
Stream #0:0 -> #0:0 (rawvideo (native) -> h264 (libx264))
Press [q] to stop, [?] for help
[libx264 @ 0x558cb266b540] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX FMA3 BMI2 AVX2
[libx264 @ 0x558cb266b540] profile High 4:2:2, level 3.1, 4:2:2, 8-bit
[libx264 @ 0x558cb266b540] 264 - core 163 r3060 5db6aa6 - H.264/MPEG-4 AVC codec - Copyleft 2003-2021 - http://www.videolan.org/x264.html - options: cabac=0 ref=1 deblock=0:0:0 analyse=0:0 me=dia subme=0 psy=1 psy_rd=1.00:0.00 mixed_ref=0 me_range=16 chroma_me=1 trellis=0 8x8dct=0 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=0 threads=4 lookahead_threads=4 sliced_threads=1 slices=4 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=0 weightp=0 keyint=250 keyint_min=25 scenecut=0 intra_refresh=0 rc=crf mbtree=0 crf=23.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=0
Output #0, flv, to 'unix:./2.sock':
Metadata:
encoder : Lavf58.76.100
Stream #0:0: Video: h264 ([7][0][0][0] / 0x0007), yuv422p(tv, progressive), 1080x720, q=2-31, 30 fps, 1k tbn
Metadata:
encoder : Lavc58.134.100 libx264
Side data:
cpb: bitrate max/min/avg: 0/0/0 buffer size: 0 vbv_delay: N/A
frame= 1 fps=0.0 q=20.0 size= 3kB time=00:00:00.00 bitrate=26608.0kbits/s speed=N/A
frame= 22 fps=0.0 q=13.0 size= 5kB time=00:00:00.70 bitrate= 55.2kbits/s speed=1.37x
frame= 44 fps= 43 q=13.0 size= 6kB time=00:00:01.43 bitrate= 35.8kbits/s speed=1.41x
frame= 69 fps= 45 q=13.0 size= 8kB time=00:00:02.26 bitrate= 29.0kbits/s speed=1.49x
frame= 92 fps= 45 q=13.0 size= 10kB time=00:00:03.03 bitrate= 26.0kbits/s speed= 1.5x
frame= 115 fps= 45 q=13.0 size= 11kB time=00:00:03.80 bitrate= 24.3kbits/s speed= 1.5x
frame= 132 fps= 43 q=13.0 size= 12kB time=00:00:04.36 bitrate= 23.4kbits/s speed=1.44x
frame= 150 fps= 42 q=13.0 size= 14kB time=00:00:04.96 bitrate= 22.6kbits/s speed= 1.4x
frame= 173 fps= 43 q=13.0 size= 15kB time=00:00:05.73 bitrate= 21.9kbits/s speed=1.41x
frame= 190 fps= 42 q=13.0 size= 17kB time=00:00:06.30 bitrate= 21.5kbits/s speed=1.38x
frame= 211 fps= 42 q=13.0 size= 18kB time=00:00:07.00 bitrate= 21.1kbits/s speed=1.38x
frame= 234 fps= 42 q=13.0 size= 20kB time=00:00:07.76 bitrate= 20.7kbits/s speed=1.39x
frame= 253 fps= 42 q=14.0 size= 23kB time=00:00:08.40 bitrate= 22.7kbits/s speed=1.38x
frame= 274 fps= 42 q=13.0 size= 25kB time=00:00:09.10 bitrate= 22.3kbits/s speed=1.38x
frame= 295 fps= 42 q=13.0 size= 26kB time=00:00:09.80 bitrate= 21.9kbits/s speed=1.38x
frame= 316 fps= 42 q=13.0 size= 28kB time=00:00:10.50 bitrate= 21.6kbits/s speed=1.38x
frame= 335 fps= 41 q=13.0 size= 29kB time=00:00:11.13 bitrate= 21.4kbits/s speed=1.37x
frame= 358 fps= 41 q=13.0 size= 31kB time=00:00:11.90 bitrate= 21.1kbits/s speed=1.38x
frame= 378 fps= 41 q=13.0 size= 32kB time=00:00:12.56 bitrate= 20.9kbits/s speed=1.38x
ludwig7685 commented
you need to put a video URL after $play-live !
look into the example of README.md!
dank074 commented
-f needs to be h264, not flv
Also it's missing the bitstream filter for adding AUD into the h264 byte stream because that's how the h264nalsplitter splits the video frames (known as access units in h264). Under outputOptions add -bsf:v h264_metadata=aud=insert