Discussion:
[ublas] CI setup
Stefan Seefeld via ublas
2018-04-27 14:24:54 UTC
Permalink
Hi all,

as we have three students working on Boost.uBLAS projects this summer, I
think it would be useful to have some CI coverage to allow new code to
be tested across platforms.

I'm thus going to set up some basic CI coverage (Travis-CI and AppVeyor)
over the next couple of days. Once that's done, all the GSoC repos can
rebase onto that and do whatever else is needed to use these services.
I'll send a note once it's ready.

Regards,


Stefan

--

...ich hab' noch einen Koffer in Berlin...
Cem Bassoy via ublas
2018-04-29 19:36:21 UTC
Permalink
Hi,

thanks Stefan. I somehow cannot push local commits to the remote. Are we
supposed to fork from GSOC/UBLAS ?

Stefan, do we agree on my project proposal?
If yes, I would like to start designing the tensor and helper template
classes.
I would choose the rank/order (number of dimensions), the dimensions and
the layout to be runtime variables.
So the only two template parameters would be data type and the storage
array type (unbounded/bounded array).

Regards,
Cem
Post by Stefan Seefeld via ublas
Hi all,
as we have three students working on Boost.uBLAS projects this summer, I
think it would be useful to have some CI coverage to allow new code to be
tested across platforms.
I'm thus going to set up some basic CI coverage (Travis-CI and AppVeyor)
over the next couple of days. Once that's done, all the GSoC repos can
rebase onto that and do whatever else is needed to use these services. I'll
send a note once it's ready.
Regards,
[image: Stefan]
--
...ich hab' noch einen Koffer in Berlin...
_______________________________________________
ublas mailing list
https://lists.boost.org/mailman/listinfo.cgi/ublas
Stefan Seefeld via ublas
2018-04-29 19:48:47 UTC
Permalink
Post by Cem Bassoy via ublas
Hi,
thanks Stefan. I somehow cannot push local commits to the remote. Are
we supposed to fork from GSOC/UBLAS ?
That would be redundant. I think you should push directly into the GSoC
repo, and if you don't have write permission that's a bug we need to
fix. (I don't have the admin rights to that repo to fix this myself. I
suspect David can help with that, though.)
Post by Cem Bassoy via ublas
Stefan, do we agree on my project proposal?
If yes, I would like to start designing the tensor and helper template
classes.
I would choose the rank/order (number of dimensions), the dimensions
and the layout to be runtime variables.
So the only two template parameters would be data type and the storage
array type (unbounded/bounded array).
The point that still needs elaboration is the relationship between the
new and the existing code. You mentioned some possible refactoring that
would make integration easier. Can you please outline what refactoring
you have in mind ? We can then discuss the right timeframe to work on
the refactoring PRs, and decide whether it's indeed best to work on your
tensor code as something mostly independent of the existing ublas
codebase or not.
As I mentioned, I'd still much favour the work being done in-tree rather
than out-of-tree, but I'm ready to be convinced that that's not a good idea.


Stefan

--

...ich hab' noch einen Koffer in Berlin...
David Bellot via ublas
2018-04-30 08:05:01 UTC
Permalink
You mentioned some possible refactoring that would make integration
easier. Can you please outline what refactoring you
​Super open to refactoring of ublas.
Curious to know what you have in mind.
Cem Bassoy via ublas
2018-04-30 13:20:38 UTC
Permalink
On Mon, Apr 30, 2018 at 10:05 AM, David Bellot via ublas <
You mentioned some possible refactoring that would make integration
easier. Can you please outline what refactoring you
​Super open to refactoring of ublas.
Curious to know what you have in mind.
Yes, thx. In general I would have like to compare the current boost
implementation with blaze and eigen.
I think there are several features or also neglect which we might add to
uBLAS.

- Should we integrate smart expression templates? I think there was a gsoc
project but I am not sure. What was the output?
- Are (smart) expression templates really required?
- How often do expressions like A = B*C + D*D - ... occur in numerical
applications?
- Should we provide a fast gemm implementation of the Goto-Algorithm like
in Eigen?

And regarding the code infrastructure:

- Do we need iterators within matrix and vector template classes? Or can we
generalize the concepts?
- Can we maybe simplify/replace the projection function with overloaded
brackets?

General questions:

- Shall we build uBLAS a high-performance library?
- Could it also be a convenient wrapper library with good interfaces to
high-performance libraries such as OpenBLAS?

Cem
_______________________________________________
ublas mailing list
https://lists.boost.org/mailman/listinfo.cgi/ublas
Stefan Seefeld via ublas
2018-04-30 13:58:17 UTC
Permalink
Post by Cem Bassoy via ublas
On Mon, Apr 30, 2018 at 10:05 AM, David Bellot via ublas
You mentioned some possible refactoring that would make
integration easier. Can you please outline what refactoring you
​Super open to refactoring of ublas.
Curious to know what you have in mind.
Yes, thx. In general I would have like to compare the current boost
implementation with blaze and eigen.
I think there are several features or also neglect which we might add
to uBLAS.
- Should we integrate smart expression templates? I think there was a
gsoc project but I am not sure. What was the output?
- Are (smart) expression templates really required?
- How often do expressions like A = B*C + D*D - ... occur in numerical
applications?
- Should we provide a fast gemm implementation of the Goto-Algorithm
like in Eigen?
Not sure about the specific questions, but I definitely think
Boost.uBLAS should provide the means to handle arbitrary expressions,
and allow for optimised backends to be matched (example: recognize `D =
A * B + C` as a statement that can be vectorized and evaluated using
fused multiply-add instructions, wherever available).
Post by Cem Bassoy via ublas
- Do we need iterators within matrix and vector template classes? Or
can we generalize the concepts?
- Can we maybe simplify/replace the projection function with
overloaded brackets?
I don't have an opinion on that question yet.
Post by Cem Bassoy via ublas
- Shall we build uBLAS a high-performance library?
Not quite sure what you mean by that question.
Post by Cem Bassoy via ublas
- Could it also be a convenient wrapper library with good interfaces
to high-performance libraries such as OpenBLAS?
Yes, definitely. One of the GSoC projects is right now looking into
using OpenCL backends to provide GPU support. The infrastructure for
that should be easily extensible, so we could have additional
high-performance backends.
The trick then is to find ways to map specific (smart ?) expression
types to available backend "kernels", and figure out (part compile-time
part runtime) which backend to use.
I have worked on such an infrastructure in OpenVSIP (see the "Dispatch
Engine" part in http://openvsip.org/doc/architecture.html) and think
that something like that would be useful in Boost.uBLAS as well.



Stefan

--

...ich hab' noch einen Koffer in Berlin...
David Bellot via ublas
2018-05-02 20:11:04 UTC
Permalink
Post by Cem Bassoy via ublas
- Should we integrate smart expression templates? I think there was a gsoc
project but I am not sure. What was the output?
​it was really good
​
Post by Cem Bassoy via ublas
- Are (smart) expression templates really required?
​but after second thought, I wonder like you.
​
Post by Cem Bassoy via ublas
- How often do expressions like A = B*C + D*D - ... occur in numerical
applications?
- Should we provide a fast gemm implementation of the Goto-Algorithm like
in Eigen?
​why not.
​
Post by Cem Bassoy via ublas
- Do we need iterators within matrix and vector template classes? Or can
we generalize the concepts?
​once there's been a discussion about that. Can we factorize all this code
into one place, one generic concept ?
This would make things so simple and efficient in the end.
​
Post by Cem Bassoy via ublas
- Can we maybe simplify/replace the projection function with overloaded
brackets?
​Can we do that ? That would be awesome !​
Post by Cem Bassoy via ublas
- Shall we build uBLAS a high-performance library?
​Yes, I suppose.
What do you mean exactly by "high-performance" ?​

- Could it also be a convenient wrapper library with good interfaces to
Post by Cem Bassoy via ublas
high-performance libraries such as OpenBLAS?
​There's been a lot of them written over the years. They're old now and
having a simple way for the user to choose a "backend" would be great.

David
Cem Bassoy via ublas
2018-05-03 13:16:01 UTC
Permalink
On Wed, May 2, 2018 at 10:11 PM, David Bellot via ublas <
Post by Cem Bassoy via ublas
- Should we integrate smart expression templates? I think there was a gsoc
Post by Cem Bassoy via ublas
project but I am not sure. What was the output?
​it was really good
okay. I cannot see it in the development branch. Is it intended to be
integrated into uBLAS?
Post by Cem Bassoy via ublas
​
Post by Cem Bassoy via ublas
- Are (smart) expression templates really required?
​but after second thought, I wonder like you.
I think smart expression templates could be beneficial in terms of
selecting and executing high-performance kernels!
Expression templates seem to be outdated. See
https://epubs.siam.org/doi/abs/10.1137/110830125

​
Post by Cem Bassoy via ublas
Post by Cem Bassoy via ublas
- How often do expressions like A = B*C + D*D - ... occur in numerical
applications?
- Should we provide a fast gemm implementation of the Goto-Algorithm like
in Eigen?
​why not.
Because, tuning algorithms and getting them to nearly peak performance for
standard processors is a nontrivial task to my mind. I am not sure but we
could try to integrate and link to existing highly optimized kernels.
Post by Cem Bassoy via ublas
​
Post by Cem Bassoy via ublas
- Do we need iterators within matrix and vector template classes? Or can
we generalize the concepts?
​once there's been a discussion about that. Can we factorize all this code
into one place, one generic concept ?
This would make things so simple and efficient in the end.
Yes, I will try to built iterators for tensors so we can discuss this by
investigating my code.
Post by Cem Bassoy via ublas
​
Post by Cem Bassoy via ublas
- Can we maybe simplify/replace the projection function with overloaded
brackets?
​Can we do that ? That would be awesome !​
Will try to show that it is possible.
Post by Cem Bassoy via ublas
Post by Cem Bassoy via ublas
- Shall we build uBLAS a high-performance library?
​Yes, I suppose.
What do you mean exactly by "high-performance" ?​
I wanted to say that uBLAS could serve as an optimizer and dispatcher
between different types of existing high-performance libraries instead of
providing high-performance functions.
Post by Cem Bassoy via ublas
- Could it also be a convenient wrapper library with good interfaces to
Post by Cem Bassoy via ublas
high-performance libraries such as OpenBLAS?
​There's been a lot of them written over the years. They're old now and
having a simple way for the user to choose a "backend" would be great.
Yes I think so too.
Post by Cem Bassoy via ublas
David
_______________________________________________
ublas mailing list
https://lists.boost.org/mailman/listinfo.cgi/ublas
Cem
Stefan Seefeld via ublas
2018-05-03 13:40:43 UTC
Permalink
Post by Cem Bassoy via ublas
On Wed, May 2, 2018 at 10:11 PM, David Bellot via ublas
- Should we integrate smart expression templates? I think
there was a gsoc project but I am not sure. What was the output?
​it was really good
okay. I cannot see it in the development branch. Is it intended to be
integrated into uBLAS?
 
​
- Are (smart) expression templates really required?
​but after second thought, I wonder like you.
I think smart expression templates could be beneficial in terms of
selecting and executing high-performance kernels!
Expression templates seem to be outdated. See
https://epubs.siam.org/doi/abs/10.1137/110830125
One of the GSoC projects is adding GPU support to uBLAS. And while it
may be useful to have an API layer that lets users explicitly request
the GPU backend to be used (for the supported subset of BLAS functions
for which there are GPU kernels), we also may want to offer full
integration, whereby a user just uses generic uBLAS expressions, and
leaves the selection of the appropriate backend (GPU or other) to the
library itself.
But to be able to implement this selection mechanism, we need some
advanced dispatching technique. I'm not sure I understand enough about
"smart expression templates", but I did implement such dispatching
infrastructure in the past (http://openvsip.org/), which scales well to
a high number of backends (GPU, SIMD vectorization, TBB-style
parallelization, etc.).
I hope we can manage to get to the point where we can discuss what
techniques are most appropriate for Boost.uBLAS, and perhaps even start
to implement them over this summer. We shall see...
Post by Cem Bassoy via ublas
​
- How often do expressions like A = B*C + D*D - ... occur in
numerical applications?
- Should we provide a fast gemm implementation of the
Goto-Algorithm like in Eigen?
​why not.
Because, tuning algorithms and getting them to nearly peak performance
for standard processors is a nontrivial task to my mind. I am not sure
but we could try to integrate and link to existing highly optimized
kernels.
As with the above, the problem really is to pick the right backend(s)
for the current platform, as the optimal choice depends on many things
(available hardware, data layout, problem size, etc.). Coming up with a
good optimization strategy (and an architecture that supports it) is
non-trivial.
Post by Cem Bassoy via ublas
 
​
- Do we need iterators within matrix and vector template
classes? Or can we generalize the concepts?
​once there's been a discussion about that. Can we factorize all
this code into one place, one generic concept ?
This would make things so simple and efficient in the end.
Yes, I will try to built iterators for tensors so we can discuss this
by investigating my code.
Sounds good.
Post by Cem Bassoy via ublas
 
​
- Can we maybe simplify/replace the projection function with
overloaded brackets?
​Can we do that ? That would be awesome !​
 
Will try to show that it is possible.
 
- Shall we build uBLAS a high-performance library?
​Yes, I suppose.
What do you mean exactly by "high-performance" ?​
I wanted to say that uBLAS could serve as an optimizer and dispatcher
between different types of existing high-performance libraries instead
of providing high-performance functions.
I agree. The project I referred to above (OpenVSIP) started out as a
library implementing operations itself, until we realized how foolish an
idea that was, at which point it turned more and more into "middleware",
i.e. something like an "algorithm abstraction layer", which makes it
easy to plug in new backends (to support new hardware, say), without the
need for applications to change any code.

Boost has long prided itself reimplementing wheels. I hope we can
overcome this NIH syndrom and demonstrate how beneficial it is to reuse
existing know-how / technology. Focusing on C++ APIs  should be the goal
of Boost, while good optimizations are certainly helpful to increase the
rate of adoption.


Stefan
--
...ich hab' noch einen Koffer in Berlin...
David Bellot via ublas
2018-04-30 08:03:48 UTC
Permalink
Yes try to fork from https://github.com/uBLAS/ublas
We have our own repository for ublas development.

Silly questions: can we have right from the beginning a fixed dimensions
tensors on top of the regular tensor ?
I'm not sure my questions is relevant in the tensor world, but what I mean
is, if we can either have the rank/order,etc... as a template parameter we
can maybe benefit from extra optimization from the compiler ?
Post by Cem Bassoy via ublas
Hi,
thanks Stefan. I somehow cannot push local commits to the remote. Are we
supposed to fork from GSOC/UBLAS ?
Stefan, do we agree on my project proposal?
If yes, I would like to start designing the tensor and helper template
classes.
I would choose the rank/order (number of dimensions), the dimensions and
the layout to be runtime variables.
So the only two template parameters would be data type and the storage
array type (unbounded/bounded array).
Regards,
Cem
Post by Stefan Seefeld via ublas
Hi all,
as we have three students working on Boost.uBLAS projects this summer, I
think it would be useful to have some CI coverage to allow new code to be
tested across platforms.
I'm thus going to set up some basic CI coverage (Travis-CI and AppVeyor)
over the next couple of days. Once that's done, all the GSoC repos can
rebase onto that and do whatever else is needed to use these services. I'll
send a note once it's ready.
Regards,
[image: Stefan]
--
...ich hab' noch einen Koffer in Berlin...
_______________________________________________
ublas mailing list
https://lists.boost.org/mailman/listinfo.cgi/ublas
_______________________________________________
ublas mailing list
https://lists.boost.org/mailman/listinfo.cgi/ublas
Cem Bassoy via ublas
2018-04-30 09:26:57 UTC
Permalink
Post by David Bellot via ublas
Yes try to fork from https://github.com/uBLAS/ublas
We have our own repository for ublas development.
But what about https://github.com/BoostGSoC18
Silly questions: can we have right from the beginning a fixed dimensions
tensors on top of the regular tensor ?
Yes we can have. I would like to start with runtime variable parameters
though. On top of that we might add a tensor template class with static
dimensions for static memory allocation. There is one library supporting
tensors with static rank/order (number of dimensions) and static dimenisons
https://github.com/romeric/Fastor with optimizations.
If I recall correctly, Eigen, Blitz and other libraries set the order/rank
as a compile time parameter.
However, some applications with graphical interfaces creating / invoking
tensors at runtime might need the order to be a dynamic parameter.
Boost could be one of the few libraries to support these type of
applications.


I'm not sure my questions is relevant in the tensor world, but what I mean
Post by David Bellot via ublas
is, if we can either have the rank/order,etc... as a template parameter we
can maybe benefit from extra optimization from the compiler ?
Yes for small tensor sizes there is a benefit. The compiler is able to
optimize much better. However, for larger tensor sizes designing tensor
algorithms with high spatial and temporal data locality is the key design
criteria to my mind.
Supporting two or even three versions would be the best option. If we do
not focus on small tensors, I would first start with the most flexible
version. Once that is finished ( GSOC) we could continue on turning runtime
variable parameters into static ones.

Regards
Cem
Post by David Bellot via ublas
On Sun, Apr 29, 2018 at 8:36 PM, Cem Bassoy via ublas <
Post by Cem Bassoy via ublas
Hi,
thanks Stefan. I somehow cannot push local commits to the remote. Are we
supposed to fork from GSOC/UBLAS ?
Stefan, do we agree on my project proposal?
If yes, I would like to start designing the tensor and helper template
classes.
I would choose the rank/order (number of dimensions), the dimensions and
the layout to be runtime variables.
So the only two template parameters would be data type and the storage
array type (unbounded/bounded array).
Regards,
Cem
Post by Stefan Seefeld via ublas
Hi all,
as we have three students working on Boost.uBLAS projects this summer, I
think it would be useful to have some CI coverage to allow new code to be
tested across platforms.
I'm thus going to set up some basic CI coverage (Travis-CI and AppVeyor)
over the next couple of days. Once that's done, all the GSoC repos can
rebase onto that and do whatever else is needed to use these services. I'll
send a note once it's ready.
Regards,
[image: Stefan]
--
...ich hab' noch einen Koffer in Berlin...
_______________________________________________
ublas mailing list
https://lists.boost.org/mailman/listinfo.cgi/ublas
_______________________________________________
ublas mailing list
https://lists.boost.org/mailman/listinfo.cgi/ublas
Stefan Seefeld via ublas
2018-04-30 11:09:25 UTC
Permalink
Yes try to fork from https://github.com/uBLAS/ublas
<https://github.com/uBLAS/ublas>
We have our own repository for ublas development.
But what about https://github.com/BoostGSoC18
I *suppose* what David means is that the `BoostGSoC18/ublas` repo should
actually be a fork of `ublas/ublas`. However, I have to admit that I
find that not very useful. `ublas/ublas` describes itself as
"Development version of Boost.uBLAS". But neither it nor
`boostorg/ublas` has seen any new code in over three years, so one has
to wonder how much sense such a split actually has. As far as I can
tell, this just creates more work to get new code (eventually) into a
release.


Stefan
--
...ich hab' noch einen Koffer in Berlin...
David Bellot via ublas
2018-05-02 20:07:34 UTC
Permalink
Post by Stefan Seefeld via ublas
I *suppose* what David means is that the `BoostGSoC18/ublas` repo should
actually be a fork of `ublas/ublas`. However, I have to admit that I find
that not very useful. `ublas/ublas` describes itself as "Development
version of Boost.uBLAS". But neither it nor `boostorg/ublas` has seen any
new code in over three years, so one has to wonder how much sense such a
split actually has. As far as I can tell, this just creates more work to
get new code (eventually) into a release.
​Agree.

Let's make things simple and efficient.
One ring to rule... sorry, I mean, one repo only
David Bellot via ublas
2018-05-02 20:06:05 UTC
Permalink
Post by David Bellot via ublas
Yes try to fork from https://github.com/uBLAS/ublas
Post by David Bellot via ublas
We have our own repository for ublas development.
But what about https://github.com/BoostGSoC18
​This one is for the students to push their code.
To be honest, after talking with Stefan, we are going to do the development
of ublas directly from the boostorg repository instead of having multiple
github repositories. I think it will simplify the development.
​
Post by David Bellot via ublas
Silly questions: can we have right from the beginning a fixed dimensions
Post by David Bellot via ublas
tensors on top of the regular tensor ?
Yes we can have. I would like to start with runtime variable parameters
though. On top of that we might add a tensor template class with static
dimensions for static memory allocation. There is one library supporting
tensors with static rank/order (number of dimensions) and static dimenisons
https://github.com/romeric/Fastor with optimizations.
If I recall correctly, Eigen, Blitz and other libraries set the order/rank
as a compile time parameter.
However, some applications with graphical interfaces creating / invoking
tensors at runtime might need the order to be a dynamic parameter.
Boost could be one of the few libraries to support these type of
applications.
​This sounds excellent to me.
​
Post by David Bellot via ublas
Yes for small tensor sizes there is a benefit. The compiler is able to
optimize much better. However, for larger tensor sizes designing tensor
algorithms with high spatial and temporal data locality is the key design
criteria to my mind.
Supporting two or even three versions would be the best option. If we do
not focus on small tensors, I would first start with the most flexible
version. Once that is finished ( GSOC) we could continue on turning runtime
variable parameters into static ones.
​Awesome !
Stefan Seefeld via ublas
2018-05-01 23:55:36 UTC
Permalink
Post by Stefan Seefeld via ublas
Hi all,
as we have three students working on Boost.uBLAS projects this summer,
I think it would be useful to have some CI coverage to allow new code
to be tested across platforms.
I'm thus going to set up some basic CI coverage (Travis-CI and
AppVeyor) over the next couple of days. Once that's done, all the GSoC
repos can rebase onto that and do whatever else is needed to use these
services. I'll send a note once it's ready.
This is now done (see
https://github.com/boostorg/ublas/commit/565dd1f5d08a5a2a40912dec3450163362285f7c),
and we have working CI builds for Linux, OSX, and Windows.
For the GSoC projects I suggest that you resync (read: rebase) your
working repos, so you can enable CI services for your own forks to be
able to cross-test any code ahead of pull requests you might want to submit.

Best,

Stefan

--

...ich hab' noch einen Koffer in Berlin...
Loading...