/Graphics-vNext

The future of server-side graphics in ASP.NET

Primary LanguageC#

Server-side graphics in ASP.NET - the present and future.

Update: March 9 2017

Imageflow is in public alpha.

Update: June 1 2016

Help Imageflow bring uncompromising security and image quality to the entire web: Support the Kickstarter Today! 10x faster than ImageMagick.

Update: May 26 2016

I'm launching a Kickstarter to make this happen

Check out imageflow.io and leave us your e-mail.

What is imageflow?

libimageflow is the image library you wish your stack shipped with. It's correct, fast, safe, and has an evolvable API. It creates compact and sharp files. Its stateless C ABI can work with even the most troublesome multi-tenant host languages. Initial bindings will include Ruby and (?). Support the Kickstarter and vote to prioritize your favorite language.

libimageflow is tested daily on Linux, Mac, and Windows (x86 and x86_64 architectures).

imageflow-server exposes a REST API – an image URL followed by a simple querystring of commands, like http://server/prefix/image.jpg?width=200&sharpen=30 You can map prefixes to different backend storage locations, like S3 or other HTTP servers. If you've been putting off moving to responsive images, imageflow-server will help make the transition painless. ImageResizer already solves this for Windows servers, but is highly coupled to Windows APIs..

imageflow-server offers a JSON API – Post JSON operation lists or graphs along with multiple inputs and outputs, and the results are returned to you.

What's the problem?

Security: Image toolkits and codecs are notorious. Until a few weeks ago, ImageMagick would run any shell scripts it found in .svg files. ImageMagick is intended to be used in a sandbox. In practice, it is run by privileged server accounts. A recent string of vulnerabilities in ImageMagick were given the ImageTragick moniker to raise awareness.

Quality: Most visual artefacts you see in images today are entirely avoidable. Decades of hacky approximations, bad mathematical and color space reasoning, and legacy compression behaviors combine to establish a very low bar for image quality. Imageflow will set the bar for default quality and correctness.

Speed: There's no valid reason your web server can't deliver image quality on par with Adobe Lightroom, and do so in 8 to 200 milliseconds. Imageflow enables that scenario. We don't need to shy away from on-the-fly image processing; we just need to focus on it and invest in our tools.

Update: Oct 26 2015 I posted a roadmap update regarding ImageResizer 5.

Update: July 20 2015

Microsoft has closed the proposal with no plan of action, roadmap, or solution.

Update: June 24 2015

Microsoft has created a proposal for a Cross-plat Server-side Image Manipulation Library.

Update: March 3 2015

We hope that someone steps up to complete libgd.net — but it will probably not be Imazen. Our focus needs to remain on providing scalable, efficient, image server sofware.

While libgd.net is very neccessary, it is not perfectly aligned with our goals. We would not mind the detour, but after 8 months, our time to find a sponsor has run out. We must return our focus to ImageResizer and its successors.

By focusing specifcally on a low-level imaging API for ImageResizer, we can reach our goals faster. We will also be free to take advantage of newer revisions of C and C++, and modern languages like Rust, Go, and Lua. These tools (and some more specialized) can let us build a faster, more flexible pipeline than if we were to limit ourselves to C89 and C# (as needed by a libgd, or any core platform API). We can greatly improve the security, performance, and compatibility of our software by decreasing our use of C#, although .NET will remain our primary user API platform.

Thank you to all who shared this on Twitter. If you find something below that sparks your interest, feel free to reach out, and we'll help get you started.

— Nathanael Jones

TLDR; ASP.NET developers presently use the closed-source System.Drawing and WPF libraries, which (for excellent reasons, like global locks and GC issues) are unsupported for sever-side use.

Neither will work in .NET Core, let alone cross-platform. System.Drawing is a GDI+ wrapper, and WPF is a WIC/DirectX 9 wrapper with a bit of glue. There are no plans to open-source or port WPF.

.NET lacks a server-friendly graphics/imaging library, especially on Windows. Let's build one.

Unless we start now, there will be NO image-processing story ready when ASP.NET vNext and .NET Core reach stable status. Given the popularity of images inside webapps, this might affect adoption.

We're actively looking for companies to help sponsor the replacement, libgd.net. We've invested as much time and money as we've had available for the last two years (and got a ton of stuff done, see end of page), but it's not enough. We're still 6 aggressive months away from beta and 12 months away from stable, and we've had to stop development for funding reasons.

Among the improvements we can bring to the table - better quality and 8-22x higher throughput compared to System.Drawing.Bitmap.DrawImage. If you spend a lot on server infrastructure, or have a business model (like e-commerce) that is affected by image quality, you should contact us. Our algorithms can process 1.9 gigapixels per second on a laptop VM; if moved to the YCbCr colorspace and flattened with the decoder, we suspect 3+ gigapixels/second to be possible. For a CDN or search engine, we would expect this to provide immediate ROI.

The .NET Future is open-source and cross-platform

.NET is going open-source and cross-platform, and already has multi-platform intellisense.

We are building a .NET Core CLR for Windows, Mac and Linux and it will be both open source and it will be supported by Microsoft. It'll all happen at https://github.com/dotnet.

ASP.NET 5 will work everywhere. ASP.NET 5 will be available for Windows, Mac, and Linux. Mac and Linux support will come soon and it's all going to happen in the open on GitHub at https://github.com/aspnet.

Server-side graphics - the present

We've all seen this:

"Classes within the System.Drawing namespace are not supported for use within a Windows or ASP.NET service. Attempting to use these classes from within one of these application types may produce unexpected problems, such as diminished service performance and run-time exceptions. For a supported alternative, see Windows Imaging Components." http://msdn.microsoft.com/en-us/library/system.drawing(v=vs.110).aspx

And ignored it. It might have been more effective with some detail, namely "GDI+ (and therefore System.Drawing) has process-wide locks everywhere".

We might have ignored it because Windows Imaging Components (WIC) lacked MTA support until Windows Server 2008 R2.

In 2007, Microsoft published a minimal set of (broken, memory leaking) interop classes for using WIC from .NET. It's now a broken link, as the gallery was later retired. As far as I can tell, that has been the extent of 'using WIC from .NET' documentation. [insert meme here].

WPF, which wraps WIC and DirectX, is also unsupported for use in ASP.NET applications. It's certainly more popular with ASP.NET developers (via DynamicImage, notably) than WIC, but System.Drawing usage dwarfs them both. Most major memory leaks were finally patched in 2012, so WPF has fewer gotchas than System.Drawing.

WPF's key flaw is that it lacks high-quality image scaling. If it was to expose the high-quality scaling provided in recent versions of Direct2D, it would be far slower than System.Drawing - regardless of threading configuration. Currently, it only exposes the inaccurate averaging (or cubic partial sampling) algorithms from WIC, which are fast but provide unacceptable quality for product photos.

Client-side graphics libraries (like WIC, WPF, Cairo, System.Drawing, etc) will always be making the wrong trade-offs for a server-side context. They favor response-time over throughput, features over security (usually), and make dozens of invalid assumptions (such as output color space and the threading context).

Will anyone pick up the ball?

I've found that developers (both inside and outside of MSFT) will wait until something actually breaks on their machine to care. There's just too much other stuff to worry about if it 'works for me'.

System.Drawing uses a process wide lock - not AppDoman, worker-process-wide. If you're doing imaging, you now have 1 CPU core. That's a hosting density killer. System.Drawing has always been unsupported on ASP.NET, but it has always been widely used - because it could be kind of sort of made to work for some things, and because there is NO competitive alternative. Adding more sticks of RAM, turning up the Web Garden count, and taking the massive cache miss and I/O hit is apparently par for the course. I'm desperately hoping that now Microsoft is in the hosting business, that they'll eventually decide this waste is unacceptable.

However, the current level of pain hasn't been enough to provoke an actual solution yet. So until the house is on fire (I.e, System.Drawing and WPF throw NotSupportedExceptions when called from ASP.NET), I don't anticipate community-driven progress to a solution.

What my team has done so far

Over the last 7 years @nathanaeljones (and recently, others contracting for @Imazen, including @ecerta, @suetanvil, @tostercx, @ddobrev, @avasp, and @ydanila) have tried to improve the state of server-side imaging for .NET, writing safer (OSS) wrappers for System.Drawing & WIC, creating (or updating) interop layers for other native libraries (WebP, FreeImage), contributing to underlying libraries, and trying to educate developers about common mistakes and memory leaks in ASP.NET imaging.

Here are some of our efforts:p

All of our income @imazen goes back into open-source development, but we can't do this alone.

What is still needed

We looked at nearly 100 libraries, and found LibGD to be the only imaging library specifically designed for server-side use. It has the added benefit of already being used on millions of servers as a core part of PHP, and being written in simple and approachable ANSI C.

We've put a lot of work into both libgd and the .NET wrapper over the last 2 years, but there's a long way to go before the the combination could be considered production ready.

In libgd itself:

  • Integrate our high-quality, high-performance image scaling algorithm from ImageResizer. This will yield visual quality better than System.Drawing, while providing 430-2200% better performance (yes, you read that right). We figured out a pretty cool technique for memory structure pivoting that solves memory locality. No SIMD or assembly required after all, although we do hand-unroll loops.
  • Upgrade from 7 to 8-bit alpha component
  • Allow enforced contiguous memory bitmaps
  • Implement format detection based on magic bytes instead of file extensions.
  • Extend API to offer lower-level integration with libjpeg/libjpeg-turbo; we can further reduce RAM requirements by performing some operations during decoding.
  • Make error handling consistent and easily map to both .NET and PHP wrappers.
  • Spend some quality time with valgrind. PHP is forgiving of memory leaks. .NET is not.
  • Improve test coverage

In the wrapper

  • Design intuitive abstractions and classes, then map the several hundred operations to them. Translate error conditions on every one.
  • Increase test coverage from < 20% to 100%.
  • Make both automated and explicit memory management seamless
  • Fully support .NET streams (vs. fopen/fclose/pathname)

In the ecosystem/tooling

Native binaries are prohibitively painful to work with when dealing with more than one architecture or platform. We must make this better, or LibGD.NET and everything like it will have limited adoption.

Ideally?

  • Conditional/platform-architecture-specific references, in projects, .NET assemblies, and in NuGet packages. Preferable without hand-editing XML.
  • Conditional nuget references, in particular, are essential. Building from source on Windows is a non-starter. Fat packages (or binaries) are insufficient on their own. Imaging distributing something huge like OpenCV. Packaging 3 architectures x 3 platforms x 180MB would make for a 1.6GB nuget package.
  • Architecture-aware assembly loaders. Skip assemblies if they're not in the right format; don't fail with BadFormatException. We can't prevent all runtime incompatibilities, but 99% are easily solved.
  • First-class native dependencies for managed assemblies.
  • A standard bin subfolder convention, respected by Visual Studio, NuGet, the ASP.NET loader, testing tools, and the .NET probe path. We should be able to take an immutable directory tree and execute tests under a variety of runtime modes - WoW64, emulation, etc. We should be able to switch between 32 and 64 bit application pools without fear. .NET set the expectation of architecture-agnosticism, and it should be upheld consistently.
  • Flexible hooks for overriding search paths for both managed and native DLLs, or injecting runtime binary fetching. We could fix most of this ourselves with permissive APIs.

Practically?

There are a lot of issues that make it hard to monkey-patch around, although we're trying - very hard.

  • We need an open-source, cross-platform assembly parser [built it, check].
  • We need an injectable replacement loader for ASP.NET websites, so that we can load the right version of any arch-specific managed assemblies. This will uglify Web.config, but even PreApplicationStart is too late to help. [in progress]
  • We need a replacement assembly loader for non-web projects. This is more difficult, as AssemblyResolve is only called when there is no matching dll. Do we ask for an empty bin folder?
  • Come help us and track progress at Imazen.NativeDependencyManager.

Also, some standard point/rectangle data structures in the BCL would be nice.

What about Cairo and Mono System.Drawing?

Client-side graphics libraries make the wrong tradeoffs for our context. This means

  • Increased overhead
  • increased attack surface area
  • Integration with the GPU, device, and display layers makes unit testing harder and less likely. We only need byte[].
  • Focus on the wrong performance metrics

Compatibility with System.Drawing will be important for porting apps to vNext, but System.Drawing is a poor end goal.

We intend to contribute to Mono System.Drawing. There is much from ImageResizer and libgd that could help fill gaps in the API. We hope Mono System.Drawing has a well-supported future in vNext. We're tryng to help fix the native dependency nightmare so that it will be more practical for it to appear on NuGet.

But the scope of both Cairo and System.Drawing overlaps very poorly with what is most needed in a server context. LibGD is of managable size, scope, and complexity, and would seem to be a better use of our resources.