Google Finally Shifting To "Upstream First" Linux Kernel Approach For Android Features

[pal666]

You're confused. I'm talking about Android devices manufacturers there, not Google. Google doesn't make the drivers so it can't upstream them.

phone manufacturers don't write drivers either. chip manufacturers for phones are not different from chip manufacturers for computers. if ones can make drivers, others can too.

Android doesn't need to support a lot of filesystems nor a lot of architectures,

android has to support any filesystem its users have and every phone board is like new architecture(it can have same cpu, but it won't work automatically on your kernel, that's why "windows on arm" only works on rpi and 1.5 snapdragons)

and Windows successfully competes with the Linux kernel on the desktop.

windows kernel never competed with linux kernel on desktop. actually, windows automatically runs linux kernel in vm(wsl2) because windows users need real kernel. windows preinstalls successfully competed with linux preinstalls, but kernel has nothing to do with that. and windows works only on x86 desktops which have some common architecture, there are many kinds of desktops where only linux kernel works. and still on this very constrained "windows desktop", microsoft has not enough man power for writing kernel. on linux most drivers come with linux, on windows most drivers have to be downloaded from internet. i.e. any author of unicorn os has to convince every hardware vendor to write drivers for his unicorn os. good luck. also desktop sales were falling for many years, except for one subsegment: chromebooks, which is linux desktop

But more than that, it's a successful OS that runs on millions of devices and was developed by a single company.

and it can't compete with linux. it lost to linux in total and in every segment except on desktop, where it just abuses monopoly(not because it has even remotely comparable kernel). actually "can't compete with linux kernel" wasn't my idea, it was words of ms people in relation to how shitty their ntfs codebase is. they don't have manpower to make competitive general purpose kernel, they are busy supporting linux on azure

So a company definitely has the manpower to write a commercially usable kernel, and no goalpost moving will change that.

instead of posting bullshit you should check sales numbers of losephone

The smartphone may very well be a generic computer in small form factor, but Android is a constrained purpose OS for those devices, and is certainly not designed for the user to play around with settings, support multiple filesystems, implement software RAID, use arbitrarily varied socket types, do high performance routing, handle thousands of processors, support hot plug of processors and memory. I can make an even larger "nonsense" list, but like it or not, all those features are bloatware when it comes specifically to Android, it doesn't need them so the replacement doesn't need to implement them.

your android is constrained, my android can be turned into desktop by inserting it into dock

[oiaohm]

That makes sense. I never thought about binning for completely different purposes. I knew about it for processor tiers only.

Exactly and its that binning that means ODMs and SOC makers want to use the same kernel and driver development in multi different markets.

ODM that designs a bit of silicon area may be selling that design to multi SOC vendors who are making SOC for multi different markets as well. So this is ODM motivation to support a OS kernel that used in as many markets as possible.

Its a little trick to have to think of ODM and SOC makers having different motivations. Those making silicon making 90%+ design and the drivers to run that designs are most of the time no the same ones ordering the SOC itself to made. So this is a lot more horrible fragmented than people first presume.

Android taking off successfully got SoC vendors and ODMs on the same page about wanting a single OS to develop for instead of the history unique OS per usage. Like it or not there has been a side effect to google success. Google wanting to make a non Linux kernel as core product has to go head to head with this side effect.

Yes before android it was nothing uncommon to see a spec sheet of SoC chip list 5 to 6 support operating systems it was uncommon to less than than 4 and only 1 OS listed as support was basically unheard of. Today most SoC chips(over 90%) list only 1 support OS type with rare ones list 2 OS types and insanely rare listing 3 or more(less than 1 percent).

The market has absolutely changed.

[sinepgib]

That is not as straight forwards.

https://www.netgear.com/support/product/MR2100.aspx

Some routers do have screens. Some routers doe use SoC chips that were intended of the phone but has some silicon defect so GPU was disabled and they are then sold as 3-5g to wifi routers. Lot are their own custom variants.

So yes some routers do use the same soc chips as phones. Think about this you have made a soc chip for a phone the GPU fails validation but everything else passes do you want to loss this chip into the scrap bin to be recycled or would you like to send this down another production line with the GPU fused out for good to come a router or something else. Remember you are a ODM you are not wanting to spend more time on development to sell off broken chips either.

ODM have a serous reason to favour having their developers support a OS(as in single OS) that can go out more than 1 channel. If a bit of silicon gets recycled the ODM like the wifi.... does not get paid their royalty payment for that unit but if the soc chip sold into any market as long as their part of the silicon chip is functional they get paid their royalty payment for that unit. Yes SoC vendors who by different IP from different ODM its in their advantage as well to be able to avoid recycling silicon as its not a cheap process. Higher the yield per wafer does work out to better profits all round and to get higher yield is helped by by being able to send defective silicon out different doors where they defect will never be a problem.

There is a money reason basically why a OS kernel for mobile and a OS kernel for router being the same kernel is a good thing to ODM and SoC makers. This is something google will have to argue with attempting to change the OS kernel. SoC vendors and the ODM want 1 or close to 1 OS kernel for everything this point of view of theirs is a new thing majorly caused by Android winning as much market share as it has.

Yes Allwinner in some ways is a lousy company at different parts of their history for driver development. Allwinner had some of the worst numbers some drivers Allwinner got as bad as 1/365 of a developer as in 1 day per year and that was the total budget for driver development for that part of the soc and it was like the critical power on the hardware driver resulting in people having to do boot and pray if the hardware would start up with some Allwinner soc chips. Please note I said best is 1/12 of a developer these days. There is way worse than that.

There are a few companies that manage to make Allwinner even in their worst time frame look great. Yes these companies were releasing soc chips with basically a developer allocation of 0 after product release. The fly by night android phone vendors with integrated spy ware commonly used these socs what made these phones even worse.

That makes sense. I never thought about binning for completely different purposes. I knew about it for processor tiers only.

[oiaohm]

On another topic, do routers use the same SoCs as phones or different variants? It sounds wasteful to e.g. include a GPU there. And in terms of development, _if_ I had to implement support for two OSes, one only for routers, I'd go without implementing some stuff (bluetooth, GPU, SD, etc).

That is not as straight forwards.

https://www.netgear.com/support/product/MR2100.aspx

Some routers do have screens. Some routers doe use SoC chips that were intended of the phone but has some silicon defect so GPU was disabled and they are then sold as 3-5g to wifi routers. Lot are their own custom variants.

So yes some routers do use the same soc chips as phones. Think about this you have made a soc chip for a phone the GPU fails validation but everything else passes do you want to loss this chip into the scrap bin to be recycled or would you like to send this down another production line with the GPU fused out for good to come a router or something else. Remember you are a ODM you are not wanting to spend more time on development to sell off broken chips either.

ODM have a serous reason to favour having their developers support a OS(as in single OS) that can go out more than 1 channel. If a bit of silicon gets recycled the ODM like the wifi.... does not get paid their royalty payment for that unit but if the soc chip sold into any market as long as their part of the silicon chip is functional they get paid their royalty payment for that unit. Yes SoC vendors who by different IP from different ODM its in their advantage as well to be able to avoid recycling silicon as its not a cheap process. Higher the yield per wafer does work out to better profits all round and to get higher yield is helped by by being able to send defective silicon out different doors where they defect will never be a problem.

There is a money reason basically why a OS kernel for mobile and a OS kernel for router being the same kernel is a good thing to ODM and SoC makers. This is something google will have to argue with attempting to change the OS kernel. SoC vendors and the ODM want 1 or close to 1 OS kernel for everything this point of view of theirs is a new thing majorly caused by Android winning as much market share as it has.

Yes Allwinner in some ways is a lousy company at different parts of their history for driver development. Allwinner had some of the worst numbers some drivers Allwinner got as bad as 1/365 of a developer as in 1 day per year and that was the total budget for driver development for that part of the soc and it was like the critical power on the hardware driver resulting in people having to do boot and pray if the hardware would start up with some Allwinner soc chips. Please note I said best is 1/12 of a developer these days. There is way worse than that.

There are a few companies that manage to make Allwinner even in their worst time frame look great. Yes these companies were releasing soc chips with basically a developer allocation of 0 after product release. The fly by night android phone vendors with integrated spy ware commonly used these socs what made these phones even worse.

[sinepgib]

Really you have missed effect on ODMs of google rise. Before android in routers and phones was a massive mix of operation systems. The hard reality is the number of unique operating systems in devices has dropped. This means ODMs don't have as large programming teams to make drivers any more. Worse a lot of ODM are no longer making generic software code or documentation on how their hardware works. Yes the Linux kernel driver source code for their hardware is the operational documentation from some ODMs.

Lot of ways Google attempting to change the OS kernel now will basically be a victim of their success. Will be facing more a up hill battle than when Microsoft attempted to get in.

So think about when google says to the current ODM that they need to support a new OS kernel the response is what do we really have to employ more staff.

fuchsia you can look at the source repository you have almost zero ODM on board. Got samsung and AMLogic had google had to pay money and its been a dump and run code delivery the same problem Microsoft run into with Qualcomm. Dump and run that is where delivered code is a starting point then its left to google/Microsoft staff to fix up the problems because the ODM is not interested in helping that much.

sinepgib here is a good question most people don't ask. How many people do you think ODM making parts for mobile phones have employed per silicon part to maintain the drivers is? At best 1/12 of a person(1 month full time a year at best). This is down from 3 people employed from the time Android released and that was at worst at that time. Yes this problem was mentioned in the recent Linux conference over graphical drivers but it is not just graphical drivers in soc chips with lack of developers the shocking reality that the drivers for arm chips in phones, routers are in reality just part time developers there are no fun time driver developers any more in that section of the market any more.

You do raise a critical point here. I once had to work with some Allwinner SoCs and the understaffing really shows in the code, very poorly thought out decisions, buggy drivers, etc. At the time I assumed it was just a lousy company tho.

The number of developers working on arm hardware in drivers have massively dropped to cut costs and part of cutting costs has cut away this also means to support multi independent operating systems from the ODMs is not really there. Google wants todo this they will have to be reaching quite deep into their pocket for many years possible decades to turn this problem around.

Yes google in this case is a total victim of their success because Android success has reduced the amount of staff ODM's have employed.

So, the same thing that gives them the influence to change the ecosystem makes it expensive for them. Interesting.

On another topic, do routers use the same SoCs as phones or different variants? It sounds wasteful to e.g. include a GPU there. And in terms of development, _if_ I had to implement support for two OSes, one only for routers, I'd go without implementing some stuff (bluetooth, GPU, SD, etc).

[oiaohm]

ODM maker will decide it's worth their time if the money is good enough. Android is certainly big enough in the market for that to be a non-trouble for them, and Google can't care less about the router if they don't sell ads through it. Remember we're talking about whether Google can pull something off, not whether or not is good for everyone else.
Android needed Linux to start, that's true, but it's the dominant player now. The application ecosystem is the winning part now. Windows failed because they entered the market too late and nobody would develop for it when there already were two big players that actually made money.
So I still think Google can get away with it.

Really you have missed effect on ODMs of google rise. Before android in routers and phones was a massive mix of operation systems. The hard reality is the number of unique operating systems in devices has dropped. This means ODMs don't have as large programming teams to make drivers any more. Worse a lot of ODM are no longer making generic software code or documentation on how their hardware works. Yes the Linux kernel driver source code for their hardware is the operational documentation from some ODMs.

Lot of ways Google attempting to change the OS kernel now will basically be a victim of their success. Will be facing more a up hill battle than when Microsoft attempted to get in.

So think about when google says to the current ODM that they need to support a new OS kernel the response is what do we really have to employ more staff.

fuchsia you can look at the source repository you have almost zero ODM on board. Got samsung and AMLogic had google had to pay money and its been a dump and run code delivery the same problem Microsoft run into with Qualcomm. Dump and run that is where delivered code is a starting point then its left to google/Microsoft staff to fix up the problems because the ODM is not interested in helping that much.

sinepgib here is a good question most people don't ask. How many people do you think ODM making parts for mobile phones have employed per silicon part to maintain the drivers is? At best 1/12 of a person(1 month full time a year at best). This is down from 3 people employed from the time Android released and that was at worst at that time. Yes this problem was mentioned in the recent Linux conference over graphical drivers but it is not just graphical drivers in soc chips with lack of developers the shocking reality that the drivers for arm chips in phones, routers are in reality just part time developers there are no fun time driver developers any more in that section of the market any more.

The number of developers working on arm hardware in drivers have massively dropped to cut costs and part of cutting costs has cut away this also means to support multi independent operating systems from the ODMs is not really there. Google wants todo this they will have to be reaching quite deep into their pocket for many years possible decades to turn this problem around.

Yes google in this case is a total victim of their success because Android success has reduced the amount of staff ODM's have employed.

[sinepgib]

This skips over major issue. Remember where Microsoft end up doing a deal with Qualcomm where Microsoft is having to money for drivers. Yes there is a bigger story here.

Most people are clueless to how many different ways a arm soc chip starts up. OEM vendors of android phones don't make soc chips. Yes this is even true with samsung because the OEM part of samsung that makes android phones and the part of samsung that makes SOC chips is two different companies.

Android documentation uses the term Original design manufacturer(ODM) this is the party that makes the silicon in the device, LCD.... OEM at most designs a circuit board and a case. Majority of your drivers need to come from ODMs not OEMs.

Lets take like a Samsung Soc chip you find in their phones. Questions how many ODM is that single chip. Most people incorrectly presume 1. The correct answer is at least 20 with only 1 of those being samsung. One of the worse soc chips in a mobile phone that Linux has been able to be made run on in fact has 150 individual ODM for the soc chip. There is most likely soc chips out there with more ODMs.

This is start of problem. Remember where I said Microsoft to have windows support on Qualcomm chips was paying Qualcomm money for drivers the next step is Qualcomm paying the ODMs they used in the soc chip to write drivers. With android phones you see examples where two phones have same soc chip one gets a update Android image the other does not interesting enough this can that the maker of the phone being the OEM one of them had the money to pay for new drivers and the other did not.

OEM if they provide closed source drivers they have normally had to spend money to get license to distribute the drivers. This means the ODM can refuse at any point to give the OEM new driver versions. Yes some cases ODM is refusing to provide new drivers because for some reason they went bankrupt. If the driver is a key driver the SoC chip will not function correctly without it. Please note a key driver can be a stupid reason to be a key driver like one of the Intel atom chips if you don't power up a particular bit of hardware todo nothing the SoC chip goes over voltage and shuts down. So when I say key driver not as in some key feature the OS need as in key feature that soc chip runs and is useful.

Ideal world you can do business with ODM to get all drivers open source and bundled with your kernel.

First, thanks for a very detailed answer. I acknowledged I oversimplified a lot of it, because I don't think it's too relevant. There's a simple solution here, which lines up with phones vendors IMO, which is simply not ship the latest non-Linux Android with newer SoCs. They want you to buy new hardware anyway.
Regarding ideal worlds, that will depend on who you ask. No vendor wants you to have a device you can freely keep up to date, because their sales will suffer. So, the ones with the power to make this change are the ones least likely to want to do it.

EDIT: I just noticed the proposed solution is described backwards. It's not with newer SoCs you wouldn't ship non-Linux, but with older SoCs. Hopefully context made that clear, but just in case.

This here is absolutely not as simple.

Router being linux based and android phone being Linux based and both being arm means the ODM making the common part there has more than 1 reason to make a Linux driver. Now you make a OS that is phone only. ODM maker could decide its not worth their time.

ODM are horrible big picture people. At times they can class phones are small part of there business. Linux advantage with ODMs is due to how many places it used so it way likely to have over 60% market share of the parties the ODM is supplying to. The other 40% can be mostly made up of Windows. Now as new smart phone OS not Linux based you could be less than 4% of the ODM sales space welcome to uphill battle getting driver support.

Yes Android starting with a Linux kernel did start with existing market space in routers and other devices for ODMs that were using Linux, This is quite a hurdle to get over.

ODM maker will decide it's worth their time if the money is good enough. Android is certainly big enough in the market for that to be a non-trouble for them, and Google can't care less about the router if they don't sell ads through it. Remember we're talking about whether Google can pull something off, not whether or not is good for everyone else.
Android needed Linux to start, that's true, but it's the dominant player now. The application ecosystem is the winning part now. Windows failed because they entered the market too late and nobody would develop for it when there already were two big players that actually made money.
So I still think Google can get away with it.

[sinepgib]

Android handsets in the wild today already use numerous filesystems. rootfs is supported by aosp on both ext4 and f2fs, and most modern devices can mount usb keys with fat32, exfat, and in some cases even ntfs.

The question is not what they do today, but what they can get away with doing tomorrow. They need not support both ext4 and f2fs, for example, f2fs will do just fine. NTFS is an oddity in pendrives and is not considered interoperable. FAT32 and exFAT are reasonably similar. But the point is, you named 5 filesystems. Linux supports like 10 partition table types and dozens of filesystems. You see the huuuuuge cut in filesystem drivers you can make if you restrict your scope? It's development 101, you can ship faster if you do less, so you trim off everything you don't need.
Note I'm not attacking Linux here. But Linux is supposed to work in many more places than Android, and that will show in the maintenance and development effort. Android is meant for consumers running a smartphone or a tablet and that's it, and in a very closed system.

[royce]

I don't ever want to see Google's spyware mainlined into the kernel so I say it should cost them more to conduct espionage on the citizens. The more it cost them, the better.

Do you realise how many eyes look at patches before they're integrated into the kernel? Especially after the University of Minesota fiasco a few months ago.

[oiaohm]

I forgot Windows is a myth.
Plus, when the purpose of your kernel is constrained you need a lot less manpower than what you need for something that's expected to be useful in many scenarios.
Think this:
- You don't need SysV IPC, you use Binder for everything;
- You don't need a myriad of architectures, you can for example restrict to little endian, which covers pretty much all consumer hardware other than routers;
- You can go further and only implement ARM support (even if you need the design to be flexible enough to add other architectures later), because it'll run on cellphones only and x86 failed there;
- Drivers? Most of them are up to the OEM to provide;
- Filesystems? One for internal storage, FAT for interoperability;
- No memory hotplug;
- No huge number of network protocols, just Unix, TCP and UDP probably;
- No BPF or only the classic version in case you need to debug packet flow;
- No floppy, ATA, CD-ROM, KVM, etc...

Essentially, making a kernel for Android only means you no longer need to support a lot of stuff, and also that you can break a lot of things safely because you control the userspace too. That makes it much cheaper to develop it than you would think.

This skips over major issue. Remember where Microsoft end up doing a deal with Qualcomm where Microsoft is having to money for drivers. Yes there is a bigger story here.

Most people are clueless to how many different ways a arm soc chip starts up. OEM vendors of android phones don't make soc chips. Yes this is even true with samsung because the OEM part of samsung that makes android phones and the part of samsung that makes SOC chips is two different companies.

Android documentation uses the term Original design manufacturer(ODM) this is the party that makes the silicon in the device, LCD.... OEM at most designs a circuit board and a case. Majority of your drivers need to come from ODMs not OEMs.

Lets take like a Samsung Soc chip you find in their phones. Questions how many ODM is that single chip. Most people incorrectly presume 1. The correct answer is at least 20 with only 1 of those being samsung. One of the worse soc chips in a mobile phone that Linux has been able to be made run on in fact has 150 individual ODM for the soc chip. There is most likely soc chips out there with more ODMs.

This is start of problem. Remember where I said Microsoft to have windows support on Qualcomm chips was paying Qualcomm money for drivers the next step is Qualcomm paying the ODMs they used in the soc chip to write drivers. With android phones you see examples where two phones have same soc chip one gets a update Android image the other does not interesting enough this can that the maker of the phone being the OEM one of them had the money to pay for new drivers and the other did not.

OEM if they provide closed source drivers they have normally had to spend money to get license to distribute the drivers. This means the ODM can refuse at any point to give the OEM new driver versions. Yes some cases ODM is refusing to provide new drivers because for some reason they went bankrupt. If the driver is a key driver the SoC chip will not function correctly without it. Please note a key driver can be a stupid reason to be a key driver like one of the Intel atom chips if you don't power up a particular bit of hardware todo nothing the SoC chip goes over voltage and shuts down. So when I say key driver not as in some key feature the OS need as in key feature that soc chip runs and is useful.

Ideal world you can do business with ODM to get all drivers open source and bundled with your kernel.

you can for example restrict to little endian, which covers pretty much all consumer hardware other than routers;

This here is absolutely not as simple.

Router being linux based and android phone being Linux based and both being arm means the ODM making the common part there has more than 1 reason to make a Linux driver. Now you make a OS that is phone only. ODM maker could decide its not worth their time.

ODM are horrible big picture people. At times they can class phones are small part of there business. Linux advantage with ODMs is due to how many places it used so it way likely to have over 60% market share of the parties the ODM is supplying to. The other 40% can be mostly made up of Windows. Now as new smart phone OS not Linux based you could be less than 4% of the ODM sales space welcome to uphill battle getting driver support.

Yes Android starting with a Linux kernel did start with existing market space in routers and other devices for ODMs that were using Linux, This is quite a hurdle to get over.