Hi,
during the last developer meeting we had some discussions about the next steps for xpcc.
In my opinion one of the main problems in using xpcc for bigger projects is its build system. Although I like SCons and and implemented its beginning in xpcc, it has grown to much into a system builder. That is fine as long as xpcc is your primary platform, but makes it really awkward in cases where it is not. It also makes it difficult to full integrate it into IDEs like Eclipse, because the indexer has some problems deciding which file is used and which isn't.
Another problem off xpcc is its size in combination with the release management. A fresh checkout is 77 MB at the moment, without the Git history its still 43 MB. In general that is not that much, but becomes annoying when you have a lot of small projects which you need to support in with different versions. Without a useful version schema, each project needs its own separate copy of the library. It is not possible/to much work to keep all projects always up-to-date to the most recent version of xpcc. Yet still new features are only added to newer versions, therefore it is also not useful to freeze an older version for all projects.
To solve these issues a possible solution is extract the system builder part of the SCons system (basically everything in /architecture) into a separate tool and make the remaining parts more modular. One approach for this would be the library builder [1]. With this, a project would no longer be build against the complete library, but a subset of the library would be bundled together, containing only the needed and applicable parts for the project/architecture. E.g. a LPC11 application does not need all the STM32F.. library code. This greatly reduces the complexity of the generated source tree, which allows for an easy integration with IDEs and build systems other than SCons.
The disadvantage mainly comes for the library developer, because the generation of the library becomes a separate step. For most projects you do this once at the beginning, after you selected your target architecture, but for library development you generate the library all the time. It may require a different workflow, by developing a new feature within a project, and only after it is finished moving it into the library.
So how to start the work on this: Niklas already reserved modm-io on GitHub (modm stands for "Modular Object-Oriented Development for Microcontrollers") to resolve the naming issues between xpcc (the protocol) and xpcc (the library). I would start to migrating code from the xpcc repository over, separating it into independent modules. This requires a lot of structural changes which makes it difficult to automatically merge changes from xpcc to modm. The device driver generation part of SCons would be extracted into a separate tool which generates the content of the architecture folder. This will be the biggest amount of work. This tool will then be used by the library builder to combine the architecture stuff with the other parts of the library.
So far the idea, any comments/other suggestions on that?
Cheers, Fabian
Hi Fabian,
thank you for your efforts.
On 04/02/16 14:49, Fabian Greif wrote:
To solve these issues a possible solution is extract the system builder part of the SCons system (basically everything in /architecture) into a separate tool and make the remaining parts more modular. One approach for this would be the library builder [1].
I have one question about this: How fine grained do you want to make the individual modules? If we make them to big, we loose flexibility, if we make them to small, we could end up with a dependency hell.
Generally, I think, we need to define a core system, that is available on all platforms and that all other modules can depend on. This could be a subset of the STL library and a module that contains the core xpcc functionality like the timer and the clock. When we do that, I would like to move our Container API to a STL compatible version.
But how about the other modules? Do we add one math module? Or do we put e.g. the Quaternion stuff into its own module?
Maybe you could suggest a list of modules, to start a discussion about the amount of modularity.
The disadvantage mainly comes for the library developer, because the generation of the library becomes a separate step. For most projects you do this once at the beginning, after you selected your target architecture, but for library development you generate the library all the time. It may require a different workflow, by developing a new feature within a project, and only after it is finished moving it into the library.
I would like to add scons support to the library builder. Could you point be to a small example project to prototype that? Is there a plugin API for build systems? And btw. what build systems do you want to support? CMake? Make?
So how to start the work on this: Niklas already reserved modm-io on GitHub (modm stands for "Modular Object-Oriented Development for Microcontrollers") to resolve the naming issues between xpcc (the protocol) and xpcc (the library). I would start to migrating code from the xpcc repository over, separating it into independent modules. This requires a lot of structural changes which makes it difficult to automatically merge changes from xpcc to modm. The device driver generation part of SCons would be extracted into a separate tool which generates the content of the architecture folder. This will be the biggest amount of work. This tool will then be used by the library builder to combine the architecture stuff with the other parts of the library.
Is every device driver going to be its own module? Or do you plan to make a big `architecture` module? We need to be careful not to have two different build systems: The library builder for the modules and the device file code for the `architecture` module. This would imho be redundant.
In general I would recommend you go ahead and start migrating code for a single platform to the mbed repository so that we can start playing around with your library builder solution and so that we have concrete examples to discuss.
Best regards,
Kevin
Fabian is back!
On 04/02/16 14:49, Fabian Greif wrote:
To solve these issues a possible solution is extract the system builder part of the SCons system (basically everything in /architecture) into a separate tool and make the remaining parts more modular. One approach for this would be the library builder [1].
[…] Maybe you could suggest a list of modules, to start a discussion about the amount of modularity.
Let’s do this step by step and build a prototype first. The means doing xpcc/architecture or xpcc/architecture/platform first, then worrying about the rest later. At the very least, driver/core, driver/gpio and perhaps driver/uart.
The complicated stuff is the code generation especially with parameters. If that works well, we can apply the lessons learnt to other modules.
I would like to add scons support to the library builder. Could you point be to a small example project to prototype that? Is there a plugin API for build systems? And btw. what build systems do you want to support? CMake? Make?
If I understand the library builder source code correctly, it will only copy files and templates, it will not actually build them. So lbuild is completely build system agnostic, right?
I would start to migrating code from the xpcc repository over, separating it into independent modules.
+1. Anything we can review and learn from.
The device driver generation part of SCons would be extracted into a separate tool which generates the content of the architecture folder. This will be the biggest amount of work. This tool will then be used by the library builder to combine the architecture stuff with the other parts of the library.
Is every device driver going to be its own module? Or do you plan to make a big `architecture` module? We need to be careful not to have two different build systems: The library builder for the modules and the device file code for the `architecture` module. This would imho be redundant.
+1. The device files and driver files are essentially already modules. I don’t know enough of lbuild to say whether or not it can be generalized.
In general I would recommend you go ahead and start migrating code for a single platform to the mbed repository so that we can start playing around with your library builder solution and so that we have concrete examples to discuss.
+1. May I suggest STM32F407? It is the most well tested, most understood platform we have. Also not a simple as an AVR, so we might spot design problems earlier.
start migrating code […] to the mbed repository
Thewhatnow?!? I think I’m way ahead of you on that one ;-)
Cheers, Niklas
Hi,
Fabian is back!
\o/
On 04/02/16 14:49, Fabian Greif wrote:
To solve these issues a possible solution is extract the system
builder
part of the SCons system (basically everything in /architecture) into
a
separate tool and make the remaining parts more modular. One approach for this would be the library builder [1].
[…] Maybe you could suggest a list of modules, to start a discussion about the amount of modularity.
Let’s do this step by step and build a prototype first. The means doing xpcc/architecture or xpcc/architecture/platform first, then worrying about the rest later. At the very least, driver/core, driver/gpio and perhaps driver/uart.
The complicated stuff is the code generation especially with parameters. If that works well, we can apply the lessons learnt to other modules.
Agreed.
Nonetheless I think it is useful to have the code generation part as a separate bundle of python functions/classes. Those can be then used by the library builder to create the actual library. But it should make testing easier if it is possible to invoke them separately.
I would like to add scons support to the library builder. Could you point be to a small example project to prototype that? Is there a plugin API for build systems? And btw. what build systems do you want to support? CMake? Make?
If I understand the library builder source code correctly, it will only copy files and templates, it will not actually build them. So lbuild is completely build system agnostic, right?
Yes. The build process should later on be as simple as:
env = Environment(...) env.Program('project', env.Glob('src/*.cpp'))
I already extracted the SCons part in xpcc and decoupled it from the system building part for a different (work) project. This should become its own library, independent from xpcc and modm.
The device driver generation part of SCons would be extracted into a separate tool which generates the content of the architecture folder. This will be the biggest amount of work. This tool will then be used by the library builder to combine the architecture stuff with the other parts of the library.
Is every device driver going to be its own module? Or do you plan to make a big `architecture` module? We need to be careful not to have two different build systems: The library builder for the modules and the device file code for the `architecture` module. This would imho be redundant.
+1. The device files and driver files are essentially already modules. I don’t know enough of lbuild to say whether or not it can be
generalized.
Should be doable. If it is not I'll talk to the guy that wrote lbuild ... oh, wait ;-)
In general I would recommend you go ahead and start migrating code for
a
single platform to the mbed repository so that we can start playing around with your library builder solution and so that we have concrete examples to discuss.
+1. May I suggest STM32F407? It is the most well tested, most understood platform we have. Also not a simple as an AVR, so we might spot design problems earlier.
Ok.
start migrating code […] to the mbed repository
Thewhatnow?!? I think I’m way ahead of you on that one ;-)
Lets just use the modm repository for now :-)
Cheers, Fabian
-
Fabian Greif -
Kevin Laeufer -
Niklas Hauser