The User Manual for Scene Weaver. A toolkit that allows you to turn non-visual works into dynamic visual accompaniment using generative artwork.
Scene Weaver is a toolkit that allows you to turn non-visual works into dynamic visual accompaniment using (primarily) generative artwork.
Beginning with something as simple as a Scene Weaver Script file and a Project Parameters file, you can create a full video complete with optional subtitles and frame-based camera movements.
To do this, Scene Weaver runs on top of an Image Generator like the AUTOMATIC1111/stable-diffusion-webui Web API.
But, you're not limited to Stable Diffusion.
By creating a subclass of the ImageGenerationInterface class, you can implement custom generation sources targeting anything from manual hand-drawn input to any other generation platform that supports Text2Image and Image2Image requests.
- Plain text using a Scene Weaver Script file.
- Additions included raw music input are planned.
- All images generated during the render process.
- Based on the structure of your input and configuration within your Project Parameters file.
- Videos of the generated images.
- Based on configuration within your Project Parameters file.
In the Hull was generated from 287 Scene Segments at 24 FPS with a Render Ratio of 4.
Images were generated at a size of (768, 512) using the SDWebApiImageGenerator on an RTX 3080ti (12GB) with xformers enabled paired with an AMD 12 Core CPU.
The operation outputted two videos at (1920, 1080) and (1080, 1920).
In this configuration, the complete render of all images and the two output videos took around 11 hours. This time will vary greatly depending on the hardware you have access to.
On the other end of the spectrum, the Simple Example included in this repo can be rendered with the same parameters in under 12 minutes.
The short answer is that at this time there is no existing software license available that will provide the protection and expressiveness desired for this repository.
The intention, at the time of writing, is to release Scene Weaver in a semi-open format once an adequate software license is created and vetted. In particular, the intention of this license will be to open the use of Scene Weaver up to artists and creators so that they can craft their own amazing works. I want to leave full commercialization of the videos created with Scene Weaver in the hands of the end user (as limited by whatever license is present for the generative image solution that was used).
However, it is also desired to deny commercialization of the Scene Weaver code for products like Software as a Service (SaaS) business models and extortionist mobile applications that feed off ignorance about the real inner workings of machine learning models ("ai"). The reasoning behind this is because of the technology in use by Scene Weaver. Scene Weaver is a program that requires a large amount of compute and time to create an output as we are limited by the technology of our time. As such, there are limited options for consumer hardware that can run the image generation models associated with Scene Weaver. This means that large, already solidified businesses have a huge advantage over any competitors in this field. An advantage that comes from the simple fact that they possess a larger pocket and have the ability to purchase or operate much larger amounts of compute than any average consumer. And this presents the problem, that with current technology, creating a service for Scene Weaver would require a heavy amount of available computational power on standby.
I have plans for this project as an artistic tool but open source has a history of being subverted by those with more funding. For the time being, only the documentation will be released while a good solution to this issue is worked on with Illegal Astronauts.
While the licensing issues for the code base are being handled, commisions will be considered on a case by case basis. Review the User Manuals below and feel free to contact me through Instagram or Twitter to discuss.
- Project Structure Reference: Reference documentation on the
parameters.jsonandscript.sptfiles that compose a Scene Weaver project. - Running Project Tests: Documentation on how to run Scene Weaver's unit tests.
- Using Cameras with Scene Weaver: Documentation on how to use existing cameras for Scene Weaver and how to create your own.
- Using Image Generators with Scene Weaver: Documentation on how to setup existing image generation interfaces for Scene Weaver and how to create your own.
While I get things rolling and measure engagement, the best way to support the project is by following me on your favorite social media platform to see more of Scene Weaver in action:
Or by subscribing to the MBM newsletter on the website.