Apple and Flash again

[VectraMan]

Steve Jobs comment on Flash:

Thoughts on Flash

El Reg:

Jobsian Vendetta - Stiletto Steve knifes Flash

"We know from painful experience that letting a third party layer of software come between the platform and the developer ultimately results in sub-standard apps and hinders the enhancement and progress of the platform," he writes. "This becomes even worse if the third party is supplying a cross platform development tool. The third party may not adopt enhancements from one platform unless they are available on all of their supported platforms.

"Hence developers only have access to the lowest common denominator set of features. Again, we cannot accept an outcome where developers are blocked from using our innovations and enhancements because they are not available on our competitor’s platforms."

Perhaps what he says about a lowest common denomonator isn't far from the truth (although I don't think the common denomonator amongst smartphones is that low - the iPhone isn't anything that special), but computing has been moving towards cross platform'ness for a while now and it's hard to see this as anything other than a huge step backwards. He also talks up HTML5, but that's a whole new layer of third party software in effect, and because it has to be downloaded as script, interpreted, compiled, and then executed, has a much bigger overhead than Flash. And of course HTML5 is platform neutral, which means it has to be designed for the lowest common denomonator in the same way.

Apple has many proprietary products too. Though the operating system for the iPhone, iPod and iPad is proprietary, we strongly believe that all standards pertaining to the web should be open.

That's very interesting because it shows he makes a clear distinction between "apps", and "web". One of the reasons they're so terrified of Flash is not just because of the cross platform'ness, but "apps" can exist anywhere, as part of any website and beyond their control. Which makes you question where they really stand on HTML5. Once the web can do anything that an app can, then where does it leave them and their AppStore? The only option Apple have is to hold back the abilities of HTML5, or at least their implementation of it, to try to protect their apps business.

Or am I wrong?

[administrator]

Part of me thinks Jobs is right. Another site I work on had a Flash MPU on it. It was only 50KB in size but if you opened more than 5 or 6 tabs containing pages from the site it ground the browser to a halt. Tried on various machines and multiple browsers.

Suggested to me that the Flash was badly made and it does have the potential to cripple machines. If the Flash developers CBA to get it right then why should it be automatically be allowed on a system?

[VectraMan]

Part of me thinks Jobs is right. Another site I work on had a Flash MPU on it. It was only 50KB in size but if you opened more than 5 or 6 tabs containing pages from the site it ground the browser to a halt. Tried on various machines and multiple browsers.

Suggested to me that the Flash was badly made and it does have the potential to cripple machines. If the Flash developers CBA to get it right then why should it be automatically be allowed on a system?

What's an MPU?

That probably says more about the way browsers handle plugins than the plugin. What happens if Javascript gets into an endless loop? Does it not "cripple" the machine in the same way? I.e. use all the CPU until something stops it.

So if you're going to block flash because it can be used in a way that might use lots of CPU, then you also have to block JavaScript. And Apple want us to use their Javascript runtime drawing Canvas system as the alternative to Flash. Imagine how many inneficient, badly written websites there'll be then.

[Sysman]

Part of me thinks Jobs is right. Another site I work on had a Flash MPU on it. It was only 50KB in size but if you opened more than 5 or 6 tabs containing pages from the site it ground the browser to a halt. Tried on various machines and multiple browsers.

The problem with porting apps to another platform is that assumptions which are valid for the original development environment are no longer valid on a target environment. This can make some ports an absolute pig to get right.

As I read it, the new requirement for iPhone/iPad apps to be developed using Apple's recommended tools is that at worst a recompile and link should be sufficient to take advantage of new O/S versions. Reading between the lines this may have something to do with the innards of how multitasking and other future features will work.

[NickFitz]

What's an MPU?

That probably says more about the way browsers handle plugins than the plugin. What happens if Javascript gets into an endless loop? Does it not "cripple" the machine in the same way? I.e. use all the CPU until something stops it.

So if you're going to block flash because it can be used in a way that might use lots of CPU, then you also have to block JavaScript. And Apple want us to use their Javascript runtime drawing Canvas system as the alternative to Flash. Imagine how many inneficient, badly written websites there'll be then.

An MPU is a 300*250 medium rectangle web ad, usually containing a Flash ad - often one of those annoying video things that starts shouting at you when you accidentally roll the mouse over it.

Browsers detect when a script engine has gone into an infinite loop, or has been running for a long time (more than a few seconds), and will prompt the user asking if they want to kill the script. As Flash is a plugin (using the Netscape Plugin API) the browser doesn't have any control over its processor usage, so Flash can indeed consume endless resources until eventually it crashes the browser.

Over the last couple of years the browser manufacturers have been putting an enormous amount of effort into optimising their JS engines. The performance of something like Safari or Chrome with HTML5 canvas is easily reaching the point where Flash is no longer necessary - have a look at something like this Asteroids game (although his collision detection is a bit iffy, that's down to his algorithms, not the platform), or this neat demo of HTML5 video being mapped onto canvas blown into little pieces (works best on Safari or Chrome).

The problem with Flash on the Mac is that it is extremely buggy. It seems quite obvious that the Flash engine developers program for Windows first, and put limited effort into optimisation and testing on other platforms.

The latest version of Safari on x64-based systems under Snow Leopard runs the Flash plugin in a separate process so that, when Flash crashes, it doesn't bring the browser down. I haven't heard any reports of people complaining that their browsing experience has been degraded when some shoddily-written advertisement ceases to function

[Durbs]

or this neat demo of HTML5 video being mapped onto canvas blown into little pieces (works best on Safari or Chrome).

Nice!

[VectraMan]

Browsers detect when a script engine has gone into an infinite loop, or has been running for a long time (more than a few seconds), and will prompt the user asking if they want to kill the script.

As does Flash.

As Flash is a plugin (using the Netscape Plugin API) the browser doesn't have any control over its processor usage, so Flash can indeed consume endless resources until eventually it crashes the browser.

The browser doesn't have to run the plugin in the same thread as everything else it's doing, as Administrator's (catchy name) experience suggests. Of course it can consume endless resources, but we've heard a lot about browser running individual tabs in seperate processes so that no one page can bring down the entire system, not to mention the fact that the whole OS shouldn't be able to be brought down by any one process. What does it say about the OS and browser if one plugin in one tab can bring down the system? That's not a 21st century OS, that's Windows 3.1!

And on something closed like an iPad, the OS definitely could restrict one browser instance or tab from consuming too many resources.

And of course Adobe have no choice but to stick with the incredibly old creaky NPAPI from yesteryear. If browser makers wanted plugins to behave better, they should create a better interface as Microsoft did (though I'm not sure ActiveX really behaves better, it is at least a more grown up interface).

Over the last couple of years the browser manufacturers have been putting an enormous amount of effort into optimising their JS engines. The performance of something like Safari or Chrome with HTML5 canvas is easily reaching the point where Flash is no longer necessary.

It's interesting you say that, because you're saying that with years and years of development, by big names like Google and Apple, the "built in" Javascript based systems are as good as Flash was 10 years ago, which is what we're talking about for simple games like the examples you've given. Which is great, but it's hardly a ringing endorsement. Why should we reject something perfectly good in favour of the people in the slow lane? Why are we not striving for something that's 10 years ahead of where we are now, rather than all agreeing to make do with something not that bad because computers and technology have evolved that even something fundamentally slow and clunky is acceptable?

The problem with Flash on the Mac is that it is extremely buggy. It seems quite obvious that the Flash engine developers program for Windows first, and put limited effort into optimisation and testing on other platforms.

I'll take your word for that. But what's the technical explanation? The Flash player is x86 code. The bulk of it is the vector rendering engine, which is x86 code, which renders to a bitmap in memory somewhere, nothing to do with the OS. The next most important bit is the script engine, which is also x86 code, which does things internally, also nothing to do with the OS. It's only round the edges where it needs to interact with the OS, stuff like user input and getting the final bitmap to the screen, which means the Mac player is 99% identical to the Windows player. So it's hard to see why one would be buggy (this doesn't apply to the iPhone and iPhad which is different of course).

The only issue I've ever managed to find a real technical explanation for is the difference in performance of full screen video, and that's because Apple don't provide an API for hardware stretching (which Microsoft do). I suspect 99% of what we hear about Flash being a resource hog on the Mac is down to this one issue, something which is 100% Apple's fault.

It seems to me that a large part of this issue is ads. The ordinary user thinks Flash is for ads and video, and they'd quite happily lose the ads if some alternate way was provided for video. But they're deluding themselves. Ads aren't going to go away, and if they're not done with Flash they'll be done with something else, and possibly in a more intrusive way.

[NickFitz]

As does Flash.

Sometimes, but it isn't any good at preventing a Flash app from hogging the CPU.

The browser doesn't have to run the plugin in the same thread as everything else it's doing, as Administrator's (catchy name) experience suggests. Of course it can consume endless resources, but we've heard a lot about browser running individual tabs in seperate processes so that no one page can bring down the entire system, not to mention the fact that the whole OS shouldn't be able to be brought down by any one process. What does it say about the OS and browser if one plugin in one tab can bring down the system? That's not a 21st century OS, that's Windows 3.1!

Which is why that doesn't happen. Flash crashes the browser, not the OS (although Jobs's wording doesn't make that very clear). As I said, the current version of Safari sandboxes Flash in a separate process on 64-bit Snow Leopard, and the next version is going to add tab process isolation as used in Chrome.

And on something closed like an iPad, the OS definitely could restrict one browser instance or tab from consuming too many resources.

The OS on a device shouldn't have to pander to the needs of badly-written software. The software should be written to work correctly in the first place. The main reason Flash is excluded from Apple's devices is that Adobe have resolutely refused to improve the quality of their code.

And of course Adobe have no choice but to stick with the incredibly old creaky NPAPI from yesteryear. If browser makers wanted plugins to behave better, they should create a better interface as Microsoft did (though I'm not sure ActiveX really behaves better, it is at least a more grown up interface).

The NPAPI was extended with the NPRuntime in 2004 as a collaborative project by all the major browser manufacturers (except MS). Google are adding a technique for running plugins out-of-process, which will presumably be contributed back to the WebKit project, thereby benefiting all such browsers.

It's interesting you say that, because you're saying that with years and years of development, by big names like Google and Apple, the "built in" Javascript based systems are as good as Flash was 10 years ago, which is what we're talking about for simple games like the examples you've given. Which is great, but it's hardly a ringing endorsement. Why should we reject something perfectly good in favour of the people in the slow lane? Why are we not striving for something that's 10 years ahead of where we are now, rather than all agreeing to make do with something not that bad because computers and technology have evolved that even something fundamentally slow and clunky is acceptable?

Did you look at the video demo? Bear in mind that's just a demo: a few lines of code showing what the system is capable of. I suspect few Flash programmers would be competent to do that in ActionScript, assuming Flash is capable of it. To say that the HTML5/CSS/JS combination is "fundamentally slow and clunky" is to ignore the point I've made about the massive improvements that have been made in the last few years. To be honest, Firefox isn't very good at the moment (although it's still better than IE8) so if you're looking at this stuff on there you're not getting a proper impression of it: try a WebKit-based browser. Or you could have a look at the IE9 Platform Preview (MS's new name for "Beta") - except that IE haven't yet committed to supporting canvas, probably due to internal politics concerning Silverlight.

I'll take your word for that. But what's the technical explanation? The Flash player is x86 code. The bulk of it is the vector rendering engine, which is x86 code, which renders to a bitmap in memory somewhere, nothing to do with the OS. The next most important bit is the script engine, which is also x86 code, which does things internally, also nothing to do with the OS. It's only round the edges where it needs to interact with the OS, stuff like user input and getting the final bitmap to the screen, which means the Mac player is 99% identical to the Windows player. So it's hard to see why one would be buggy (this doesn't apply to the iPhone and iPhad which is different of course).

Rendering to a bitmap is in itself a process that depends on the OS - just have a look at the Win32 API calls for obtaining a bitmap from a device context and rendering to it. In fact here's an example of some simple drawing to a device-independent bitmap on Win32, courtesy of Raymond Chen:

BITMAP bm;
  if (GetObject(g_hbm, sizeof(bm), &bm) == sizeof(bm) &&
                bm.bmBits != NULL &&
                bm.bmPlanes * bm.bmBitsPixel <= 8) {
   struct BITMAPINFO256 {
    BITMAPINFOHEADER bmiHeader;
    RGBQUAD bmiColors[256];
   } bmiGray;
   ZeroMemory(&bmiGray, sizeof(bmiGray));
   HDC hdc = CreateCompatibleDC(NULL);
   if (hdc) {
    HBITMAP hbmPrev = SelectBitmap(hdc, g_hbm);
    UINT cColors = GetDIBColorTable(hdc, 0, 256, bmiGray.bmiColors);
    for (UINT iColor = 0; iColor < cColors; iColor++) {
     BYTE b = (BYTE)((30 * bmiGray.bmiColors[iColor].rgbRed +
                      59 * bmiGray.bmiColors[iColor].rgbGreen +
                      11 * bmiGray.bmiColors[iColor].rgbBlue) / 100);
     bmiGray.bmiColors[iColor].rgbRed   = b;
     bmiGray.bmiColors[iColor].rgbGreen = b;
     bmiGray.bmiColors[iColor].rgbBlue  = b;
    }
    bmiGray.bmiHeader.biSize        = sizeof(bmiGray.bmiHeader);
    bmiGray.bmiHeader.biWidth       = bm.bmWidth;
    bmiGray.bmiHeader.biHeight      = bm.bmHeight;
    bmiGray.bmiHeader.biPlanes      = bm.bmPlanes;
    bmiGray.bmiHeader.biBitCount    = bm.bmBitsPixel;
    bmiGray.bmiHeader.biCompression = BI_RGB;
    bmiGray.bmiHeader.biClrUsed     = cColors;
    SetDIBitsToDevice(pps->hdc, 0, 0,
                      bmiGray.bmiHeader.biWidth,
                      bmiGray.bmiHeader.biHeight, 0, 0,
                      0, bmiGray.bmiHeader.biHeight,
                      bm.bmBits,
                     (BITMAPINFO*)&bmiGray, DIB_RGB_COLORS);

    BitBlt(pps->hdc, bm.bmWidth, 0, bm.bmWidth, bm.bmHeight,
           hdc, 0, 0, SRCCOPY);
    SelectBitmap(hdc, hbmPrev);
    DeleteDC(hdc);
}

That code isn't going to map to the APIs used for the same functionality on OS X or Linux, so the underlying rendering code must be different. The script engine has to use the OS's memory management, threading support, and so forth, so again its runtime must be OS-specific in many ways, even though things like the lexer and bytecode interpreter may be the same.

The only issue I've ever managed to find a real technical explanation for is the difference in performance of full screen video, and that's because Apple don't provide an API for hardware stretching (which Microsoft do). I suspect 99% of what we hear about Flash being a resource hog on the Mac is down to this one issue, something which is 100% Apple's fault.

Apple have now opened that API, and Adobe are using it in a forthcoming update to Flash. HTML5 video will still work better, though, because it doesn't rely on a buggy plugin.

It seems to me that a large part of this issue is ads. The ordinary user thinks Flash is for ads and video, and they'd quite happily lose the ads if some alternate way was provided for video. But they're deluding themselves. Ads aren't going to go away, and if they're not done with Flash they'll be done with something else, and possibly in a more intrusive way.

What could be more intrusive than Flash? Something that literally, rather than figuratively, reaches out of the screen and pokes you in the eye?

[VectraMan]

The OS on a device shouldn't have to pander to the needs of badly-written software. The software should be written to work correctly in the first place. The main reason Flash is excluded from Apple's devices is that Adobe have resolutely refused to improve the quality of their code.

But listen to what you're saying. For 20-30 years, even 40 years if you count UNIX, OSs have been designed to protect themselves against badly written software, and in the PC world this started with the 386 in 1985. Using your logic we should all be using something like Windows 3.0 in standard mode, because after all, OSs shouldn't have to pander to the needs of badly written software.

Flash is a layer between the application and the OS, much like .NET or Java. Is it the layer itself that is bad? I don't believe so; the crticism levelled by many is that it's some Flash apps that cause problems, which is down to the the developer of the Flash applicaiton, not the runtime. If the main reason really is the quality of the Flash runtime, why are Apple not working with Adobe to sort it out? Why when they've made such a big deal about the iPhad as an internet platform are they not taking these steps to make it support one of the most popular formats used on the internet?

To do so would be a win-win situation for Apple and Adobe if that were the case, but clearly Jobs has a different motive. I can't speak for Apple devices, but Flash works perfectly well on both my PCs, as in fact it did on the PCs I was using 10 years ago. It also runs perfectly well on my el-cheapo Nokia phone (which doesn't do video, but does do everything else). Are Apple devices not as good? Do they not have the power of a £100 PAYG Nokia from 4 years ago?

Rendering to a bitmap is in itself a process that depends on the OS - just have a look at the Win32 API calls for obtaining a bitmap from a device context and rendering to it. In fact here's an example of some simple drawing to a device-independent bitmap on Win32, courtesy of Raymond Chen:

Now you're showing your ignorance I'm afraid. Your example code is rendering a 256 colour bitmap using GDI, which is almost certainly not what Flash would do, and not what decent graphics software would do this side of 1995. It'll use DirectX, and yes the final blit with DirectX is OS dependent, but the point was that 99.9% of what Flash would do was before that single operation to copy a bitmap to the screen. That isn't buggy, or at least if it was buggy, they would have found out very quickly.

What could be more intrusive than Flash? Something that literally, rather than figuratively, reaches out of the screen and pokes you in the eye?

Flash generally speaking is in small areas of the screen, and the really obtrusive popups (which many people think are down to Flash) are actually done by HTML anyway. Take away those sorts of ads, and you could have, for example ads which The new iPad from Apple. Almost 1/4 as fast as a Dell laptop and only £1499 are inserted in the middle of interesting and intelligently written text, which would be far more distracting. I'd rather have a couple of footballs animating away at the side of the scrfeen using <1% CPU (as is happening right now) thankyou.

[NickFitz]

But listen to what you're saying. For 20-30 years, even 40 years if you count UNIX, OSs have been designed to protect themselves against badly written software, and in the PC world this started with the 386 in 1985. Using your logic we should all be using something like Windows 3.0 in standard mode, because after all, OSs shouldn't have to pander to the needs of badly written software.

iPhone OS is Unix.

What you suggested is that the OS ought to have the responsibility for performing a real-time analysis of resource consumption (including processor cycles) and step in if an app behaves badly. What OS does that? And if desktop systems, with no worries about battery life and minimal resources such as memory, don't accept responsibility for policing shoddily-written software, why should mobile devices be expected to do so? It's more sensible to simply not allow such garbage on the system in the first place.

Flash is a layer between the application and the OS, much like .NET or Java. Is it the layer itself that is bad? I don't believe so; the crticism levelled by many is that it's some Flash apps that cause problems, which is down to the the developer of the Flash applicaiton, not the runtime. If the main reason really is the quality of the Flash runtime, why are Apple not working with Adobe to sort it out? Why when they've made such a big deal about the iPhad as an internet platform are they not taking these steps to make it support one of the most popular formats used on the internet?

To do so would be a win-win situation for Apple and Adobe if that were the case, but clearly Jobs has a different motive. I can't speak for Apple devices, but Flash works perfectly well on both my PCs, as in fact it did on the PCs I was using 10 years ago. It also runs perfectly well on my el-cheapo Nokia phone (which doesn't do video, but does do everything else). Are Apple devices not as good? Do they not have the power of a £100 PAYG Nokia from 4 years ago?

However, Flash runs grossly inefficiently on OS X, even though Apple have been trying to get Adobe to do something about it for years, and it is indeed the Flash implementation itself that is a resource hog. Adobe thought they could rest on their laurels because they had dominance in the market. As a result Flash will soon be dead, and good riddance.

As Adobe themselves don't claim to have a stable release implementation of the full Flash engine on any mobile device yet, I don't see why they should expect Apple to support it on their devices. Maybe when Adobe actually come up with a working product it will be time to reconsider, but not before.

Now you're showing your ignorance I'm afraid. Your example code is rendering a 256 colour bitmap using GDI, which is almost certainly not what Flash would do, and not what decent graphics software would do this side of 1995. It'll use DirectX, and yes the final blit with DirectX is OS dependent, but the point was that 99.9% of what Flash would do was before that single operation to copy a bitmap to the screen. That isn't buggy, or at least if it was buggy, they would have found out very quickly.

Raymond Chen's example code, actually. Until you can prove that Flash doesn't use the native graphics capabilities of the OS, which would be the only sane approach to take, it seems reasonable to assume that it does. After all, it can't use architecture-independent bitmaps without at some point translating them into the native device-independent bitmap structure of the platform on which it's running, which can then translate them to device-dependent bitmaps during final rendering; why would they want the performance hit of adding that extra layer of abstraction, and an additional step translating from their internal structures to the platform's own structures, rather than taking advantage of the platform's own graphics capabilities, which exist for that very purpose?

Flash generally speaking is in small areas of the screen, and the really obtrusive popups (which many people think are down to Flash) are actually done by HTML anyway. Take away those sorts of ads, and you could have, for example ads which The new iPad from Apple. Almost 1/4 as fast as a Dell laptop and only £1499 are inserted in the middle of interesting and intelligently written text, which would be far more distracting. I'd rather have a couple of footballs animating away at the side of the scrfeen using <1% CPU (as is happening right now) thankyou.

You're only talking about ads there. Furthermore, not all obtrusive popups (leaving aside windowed popups, which are a dying breed) are done using HTML.

Anyway, your talk about lack of Flash leading to even more intrusive advertising is a straw man: there is no reason to suppose that the demise of an unlamented technology will cause the online advertising industry to suddenly lose its marbles, abandon all its established advertising formatting standards (such as MPUs, banners, and skyscrapers), and start doing stuff that would inevitably alienate consumers. (Most Flash ads are backed up by an animated GIF just in case, and most of them would have been better left that way.)