Discussion:
New packaging guidelines for Ruby
Vít Ondruch
2012-02-28 11:39:14 UTC
Permalink
Hi,

since I was not subscribed to this mailing list, I starting new thread
in Hope to move forward with the packaging guidelines. There are still
some outstanding issues.

= Mandatory rebuilding gems =

Yes, Ruby SIG is still against it, since there is known just one gem ATM
which needs such treatment. Now I list several pros/cons:

Pros:
* It would allow ruby packages to follow the same steps as other packages.

Cons:
* More overhead for maintainers.
* More confusion for new-commers, since this approach is not know in
Ruby community and there is no best way how to achieve it.
* There is only one known gem in Fedora, which needs this treatment ATM.
* If you need patch binary part of gem, it is sign that the gem is not
well maintained by upstream, otherwise it would not be needed.

= Vendorlib =

It is not good idea to move vendorlib out of the Ruby directory
structure. Actually it is pair of directories, vendorlib and
vendorarchlib. These directories are typically used by libraries Ruby
bindings, such as geos, subversion, etc. This platform dependent
bindings has no meaning to other Ruby implementations, such as JRuby.

= ruby(name) vs rubygems(name) =

Although we want to see as much libraries as possible provided in a gem
form, there still be some libraries which are not gemified. However,
Gemifies library *should not* always provide also ruby(name) virtual
provide, since these are not always simply interchangeable. Gem caries
with itself its metadata. When gem is loaded to Ruby's library path,
this metadata are processed as well and it might put some other
dependent libraries to Ruby's library path as well. In contrary, the
ruby(name) provide will mean that it is not gemified library, so I would
prefer to stay with distinction between ruby(name) and rubygem(name).
Gem would provide the ruby(name) only in case it obsoletes previously
available ruby library.

These are 3 concerns I remember was the most controversial. Please feel
free to share your thoughts.


Vít
Bohuslav Kabrda
2012-02-28 12:47:45 UTC
Permalink
----- Original Message -----
Post by Vít Ondruch
= ruby(name) vs rubygems(name) =
Here is one important reason why Gems should not provide ruby(name):
The ruby(name) provides are supposed to be provided by the libraries, that are meant to be directly loadable with "require 'name'" even without Rubygems library. The Rubygems are loaded by default in Ruby 1.9.3, but the users may choose not to load them by passing "--disable-gems" option to the interpreter (either directly or via environment variable RUBYOPT).
For a user, who turns of his Rubygems, a packaged Gem providing ruby(name) wouldn't load, while some other non-gem package providing ruby(name) would load, which is obviously a puzzling behaviour.

Therefore, we should distinct these two cases for libraries, that are:
1) Loadable without Rubygems - these should provide ruby(name)
2) Loadable with Rubygems - these should provide rubygem(name)
--
Regards,
Bohuslav "Slavek" Kabrda.
Vít Ondruch
2012-02-28 15:29:54 UTC
Permalink
Post by Vít Ondruch
Yes, Ruby SIG is still against it, since there is known just one gem ATM
* It would allow ruby packages to follow the same steps as other packages.
* More overhead for maintainers.
* More confusion for new-commers, since this approach is not know in
Ruby community and there is no best way how to achieve it.
If this notion of building from source is not known in the ruby
community, I'd highly recommend everyone (fpc, the ruby sig, etc...)
help make them aware of how important that is.
Pleas do not be mistaken. We are not speaking about building gems from
sources. We are speaking about building from package manager output,
i.e. build gem from gem. It is like trying to rebuild RPM with some
applied patch from RPM (yes, I am not speaking about SRPM but about RPM
and that is not mistake). How will you do it? Yes, the RPM contains the
same metadata as there were in original spec + SRPM but how will you
reconstruct them? Would you suggest somebody to use this approach?



Vít
Vít Ondruch
2012-02-28 16:13:16 UTC
Permalink
Post by Vít Ondruch
Pleas do not be mistaken. We are not speaking about building gems from
sources.
OK, so... is there some other better way to do it then? Please do
suggest any better alternative if you do.
Better way is not doing it. Only in exceptional cases. And the case
happened just once as far as I know (and I did rebuild of all gems in
Fedora, so I should know).

This was the original guidelines proposal [1] which builds upon
experiences with previous guidelines and extend/fixed them where it was
appropriate. I found it very sensitive, where mandatory gem rebuild is
definitely not.


Vit



[1]
https://fedoraproject.org/w/index.php?title=PackagingDrafts/Ruby&oldid=270494#Applying_Patches
-- rex
--
packaging mailing list
packaging at lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/packaging
Vít Ondruch
2012-02-28 16:36:43 UTC
Permalink
Post by Vít Ondruch
Post by Vít Ondruch
Pleas do not be mistaken. We are not speaking about building gems from
sources.
OK, so... is there some other better way to do it then? Please do
suggest any better alternative if you do.
Better way is not doing it.
I guess we may have to agree to disagree then. Guidelines that allow
for not building from (preferred form) source is deal-breaker for me
at least.
May be you could elaborate a bit what do you want to achieve and what is
the reasoning. What do you mean by "not building from (preferred form)
source" anyway?


Vit
Vít Ondruch
2012-02-28 16:56:54 UTC
Permalink
Post by Vít Ondruch
May be you could elaborate a bit what do you want to achieve and what is
the reasoning. What do you mean by "not building from (preferred form)
source" anyway?
So, perhaps I'm showing some ignorance here. Do these guidelines
allow for packaging pre-built gems (similar to pre-built java .jar
files) or are they genuinely being built completely from source?
Maybe I'm confusing that with gems being built and generated in one
step, rather than the rpm notion of extracting source/patching
(%prep), building stuff (%build), and installing into buildroot
(%install).
-- rex
--
packaging mailing list
packaging at lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/packaging
Ok, I'll give you 3 examples:

= Old guidelines, used from the time RubyGems were packaged for Fedora =

%prep

%build

%install
mkdir -p %{buildroot}%{gem_dir}
gem install --local --install-dir %{buildroot}%{gem_dir} \
--force %{SOURCE0}




= What we are proposing =

%prep
%setup -q -c -T
mkdir -p .%{gem_dir}
gem install --local --install-dir .%{gem_dir} \
--force %{SOURCE0}

%build

%install
mkdir -p %{buildroot}%{gem_dir}
cp -a .%{gem_dir}/* \
%{buildroot}%{gem_dir}/




= What FPC is proposing =

%prep
%setup -q -c -T
pushd ..
gem unpack %{SOURCE0}

pushd %{gem_name}-%{version}
gem spec %{SOURCE0} -l --ruby > %{gem_name}.gemspec

gem build %{gem_name}.gemspec
popd
popd

%build
mkdir -p ./%{gem_dir}
gem install --local --install-dir ./%{gem_dir} \
--force ../%{gem_name}-%{version}/%{gem_name}-%{version}.gem

%install
mkdir -p %{buildroot}%{gem_dir}
cp -a .%{gem_dir}/* \
%{buildroot}%{gem_dir}/


All three versions provide the same output, unless I did some mistake,
since I did not tested it (actually the middle one was taken directly
from rubygem-POpen4.spec). So which version you prefer? Please note that
the "gem install" will always "unpack" the gem with some additional, for
our case unimportant, steps. We do not distribute the .gem file anywhere.


Vit
Vít Ondruch
2012-02-28 17:25:22 UTC
Permalink
Post by Vít Ondruch
All three versions provide the same output, unless I did some mistake,
since I did not tested it (actually the middle one was taken directly
from rubygem-POpen4.spec). So which version you prefer? Please note that
the "gem install" will always "unpack" the gem with some additional, for
our case unimportant, steps. We do not distribute the .gem file anywhere.
You didn't exactly (directly) answer my question. Pretend I don't
know much about ruby... (not far from the truth).
So, for rubygem packages that include native (C or otherwise code),
how and when is this compiled? If it is always done so during either
version of these guidelines, please do accept my apologies for being
ignorant.
It is compiled during the "gem install" step. So "gem install" is doing
%prep, %build, %install in one step. So yes, it is always done for
either version of guidelines.
However, you do say you do not distribute the .gem, though I'm curious
rpm -qlp rubygem-POpen4-0.1.4-3.fc17.noarch.rpm
/usr/share/gems/cache/POpen4-0.1.4.gem
...
I knew you will point it out :).

It is cached version of the original gem, which RubyGems may use to
restore the gem into its original state (and may be other unknown
purposes). However, it is not used in runtime, nor it is good idea to
restore the gems maintained by RPM by gem command. Moreover, even though
the gem would not be available in the cache dir, RubyGems will download
it. Hence we add new clause into the packaging guidelines:

Since the Gem is installed using RPM, you *must* exclude the .gem file.


Vit


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.fedoraproject.org/pipermail/ruby-sig/attachments/20120228/2f118fa0/attachment.html>
Michal Fojtik
2012-02-28 17:36:22 UTC
Permalink
Post by Vít Ondruch
All three versions provide the same output, unless I did some mistake,
since I did not tested it (actually the middle one was taken directly
from rubygem-POpen4.spec). So which version you prefer? Please note that
the "gem install" will always "unpack" the gem with some additional, for
our case unimportant, steps. We do not distribute the .gem file anywhere.
You didn't exactly (directly) answer my question. Pretend I don't know much about ruby... (not far from the truth).
So, for rubygem packages that include native (C or otherwise code), how and when is this compiled? If it is always done so during either version of these guidelines, please do accept my apologies for being ignorant.
It is compiled during the "gem install" step. So "gem install" is doing %prep, %build, %install in one step. So yes, it is always done for either version of guidelines.
rpm -qlp rubygem-POpen4-0.1.4-3.fc17.noarch.rpm
/usr/share/gems/cache/POpen4-0.1.4.gem
...
I knew you will point it out :).
gem help commands

pristine Restores installed gems to pristine condition from files
located in the gem cache
unpack Unpack an installed gem to the current directory

AFAIK this is the reason gem files are kept in cache dir. I barely understand how 'pristine' command
could be dangerous to RPM (personally I think it can be useful when you make some dirty modification
to installed gem directly as root (trying to debug/solve some issue)).

If the case is that we don't want to keep these files on filesystem, then these two commands should
be 'patched' out from the 'gem' command or some decent warning to user should be provided.

-- Michal
Since the Gem is installed using RPM, you must exclude the .gem file.
Vit
_______________________________________________
ruby-sig mailing list
ruby-sig at lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/ruby-sig
Vít Ondruch
2012-02-28 18:23:34 UTC
Permalink
Post by Michal Fojtik
Post by Vít Ondruch
All three versions provide the same output, unless I did some mistake,
since I did not tested it (actually the middle one was taken directly
from rubygem-POpen4.spec). So which version you prefer? Please note that
the "gem install" will always "unpack" the gem with some additional, for
our case unimportant, steps. We do not distribute the .gem file anywhere.
You didn't exactly (directly) answer my question. Pretend I don't know much about ruby... (not far from the truth).
So, for rubygem packages that include native (C or otherwise code), how and when is this compiled? If it is always done so during either version of these guidelines, please do accept my apologies for being ignorant.
It is compiled during the "gem install" step. So "gem install" is doing %prep, %build, %install in one step. So yes, it is always done for either version of guidelines.
rpm -qlp rubygem-POpen4-0.1.4-3.fc17.noarch.rpm
/usr/share/gems/cache/POpen4-0.1.4.gem
...
I knew you will point it out :).
gem help commands
pristine Restores installed gems to pristine condition from files
located in the gem cache
unpack Unpack an installed gem to the current directory
AFAIK this is the reason gem files are kept in cache dir. I barely understand how 'pristine' command
could be dangerous to RPM (personally I think it can be useful when you make some dirty modification
to installed gem directly as root (trying to debug/solve some issue)).
If the case is that we don't want to keep these files on filesystem, then these two commands should
be 'patched' out from the 'gem' command or some decent warning to user should be provided.
Since the gems managed by RPM are no longer in GEM_HOME, neither the
"gem pristine" nor "gem unpack" can touch them. If you want to
reinstall RPM nanaged gem, you should use something like "yum reinstall
rubygem-foo" or alternatively "rpm -i --force rubygem-foo.rpm".

"gem pristine" and "gem unpack" will work no matter if the .gem file is
kept in cache or not. In the rare case you need them, the .gem files
will be downloaded from rubygems.org. That is a beauty of cache, you
can loose it and it will not hurt.


Vit
Mo Morsi
2012-02-29 11:43:14 UTC
Permalink
Post by Vít Ondruch
= Old guidelines, used from the time RubyGems were packaged for Fedora =
%prep
%build
%install
mkdir -p %{buildroot}%{gem_dir}
gem install --local --install-dir %{buildroot}%{gem_dir} \
--force %{SOURCE0}
= What we are proposing =
%prep
%setup -q -c -T
mkdir -p .%{gem_dir}
gem install --local --install-dir .%{gem_dir} \
--force %{SOURCE0}
%build
%install
mkdir -p %{buildroot}%{gem_dir}
cp -a .%{gem_dir}/* \
%{buildroot}%{gem_dir}/
= What FPC is proposing =
%prep
%setup -q -c -T
pushd ..
gem unpack %{SOURCE0}
pushd %{gem_name}-%{version}
gem spec %{SOURCE0} -l --ruby > %{gem_name}.gemspec
gem build %{gem_name}.gemspec
popd
popd
%build
mkdir -p ./%{gem_dir}
gem install --local --install-dir ./%{gem_dir} \
--force ../%{gem_name}-%{version}/%{gem_name}-%{version}.gem
%install
mkdir -p %{buildroot}%{gem_dir}
cp -a .%{gem_dir}/* \
%{buildroot}%{gem_dir}/
All three versions provide the same output, unless I did some mistake,
since I did not tested it (actually the middle one was taken directly
from rubygem-POpen4.spec). So which version you prefer? Please note
that the "gem install" will always "unpack" the gem with some
additional, for our case unimportant, steps. We do not distribute the
.gem file anywhere.
Alternatively we can go with a hybrid of solutions 2 and 3 (your and
FPC's proposals) where the 'gem unpack' and 'gem spec' steps are optional.

The majority of gems do not need an additional modification or patching
to be converted into a RPM. Yes these steps bring things more inline w/
other packages, but at the expense of unnecessary additional work.

If the solution is to suggest gem unpack / gem spec is used while
allowing for it to be omitted (still need to determine if gem install
should still be run in the %build or %install sections), package
maintainers will have a bit more flexibility to run the steps necessary
to build their package w/out any additional work, while at the same time
still being more compliant and in-line w/ other Fedora practices.

Thoughts?

As a side node, if at all possible, please make sure to cc' both lists
(packaging and ruby-sig) on replies as this discussion is relevant to
both communities. Noticed alot of discussion only on the packaging list
meaning the Fedora ruby community is missing out on alot of this.

Appreciate it,

-Mo
Vít Ondruch
2012-02-29 12:21:55 UTC
Permalink
Post by Mo Morsi
Post by Vít Ondruch
= Old guidelines, used from the time RubyGems were packaged for Fedora =
%prep
%build
%install
mkdir -p %{buildroot}%{gem_dir}
gem install --local --install-dir %{buildroot}%{gem_dir} \
--force %{SOURCE0}
= What we are proposing =
%prep
%setup -q -c -T
mkdir -p .%{gem_dir}
gem install --local --install-dir .%{gem_dir} \
--force %{SOURCE0}
%build
%install
mkdir -p %{buildroot}%{gem_dir}
cp -a .%{gem_dir}/* \
%{buildroot}%{gem_dir}/
= What FPC is proposing =
%prep
%setup -q -c -T
pushd ..
gem unpack %{SOURCE0}
pushd %{gem_name}-%{version}
gem spec %{SOURCE0} -l --ruby> %{gem_name}.gemspec
gem build %{gem_name}.gemspec
popd
popd
%build
mkdir -p ./%{gem_dir}
gem install --local --install-dir ./%{gem_dir} \
--force ../%{gem_name}-%{version}/%{gem_name}-%{version}.gem
%install
mkdir -p %{buildroot}%{gem_dir}
cp -a .%{gem_dir}/* \
%{buildroot}%{gem_dir}/
All three versions provide the same output, unless I did some mistake,
since I did not tested it (actually the middle one was taken directly
from rubygem-POpen4.spec). So which version you prefer? Please note
that the "gem install" will always "unpack" the gem with some
additional, for our case unimportant, steps. We do not distribute the
.gem file anywhere.
Alternatively we can go with a hybrid of solutions 2 and 3 (your and
FPC's proposals) where the 'gem unpack' and 'gem spec' steps are optional.
Yes, that was always part of the proposal. Apply the 3rd step only if
necessary.

Vit
Post by Mo Morsi
The majority of gems do not need an additional modification or patching
to be converted into a RPM. Yes these steps bring things more inline w/
other packages, but at the expense of unnecessary additional work.
If the solution is to suggest gem unpack / gem spec is used while
allowing for it to be omitted (still need to determine if gem install
should still be run in the %build or %install sections), package
maintainers will have a bit more flexibility to run the steps necessary
to build their package w/out any additional work, while at the same time
still being more compliant and in-line w/ other Fedora practices.
Thoughts?
As a side node, if at all possible, please make sure to cc' both lists
(packaging and ruby-sig) on replies as this discussion is relevant to
both communities. Noticed alot of discussion only on the packaging list
meaning the Fedora ruby community is missing out on alot of this.
Appreciate it,
-Mo
Vít Ondruch
2012-03-01 09:05:58 UTC
Permalink
Post by Mo Morsi
Post by Vít Ondruch
= Old guidelines, used from the time RubyGems were packaged for Fedora =
%prep
%build
%install
mkdir -p %{buildroot}%{gem_dir}
gem install --local --install-dir %{buildroot}%{gem_dir} \
--force %{SOURCE0}
= What we are proposing =
%prep
%setup -q -c -T
mkdir -p .%{gem_dir}
gem install --local --install-dir .%{gem_dir} \
--force %{SOURCE0}
%build
%install
mkdir -p %{buildroot}%{gem_dir}
cp -a .%{gem_dir}/* \
%{buildroot}%{gem_dir}/
= What FPC is proposing =
%prep
%setup -q -c -T
pushd ..
gem unpack %{SOURCE0}
pushd %{gem_name}-%{version}
gem spec %{SOURCE0} -l --ruby> %{gem_name}.gemspec
gem build %{gem_name}.gemspec
popd
popd
%build
mkdir -p ./%{gem_dir}
gem install --local --install-dir ./%{gem_dir} \
--force ../%{gem_name}-%{version}/%{gem_name}-%{version}.gem
%install
mkdir -p %{buildroot}%{gem_dir}
cp -a .%{gem_dir}/* \
%{buildroot}%{gem_dir}/
All three versions provide the same output, unless I did some mistake,
since I did not tested it (actually the middle one was taken directly
from rubygem-POpen4.spec). So which version you prefer? Please note
that the "gem install" will always "unpack" the gem with some
additional, for our case unimportant, steps. We do not distribute the
.gem file anywhere.
Alternatively we can go with a hybrid of solutions 2 and 3 (your and
FPC's proposals) where the 'gem unpack' and 'gem spec' steps are optional.
As Vit points out, there is no way to implement #2 alone -- it must be
combined with some form of #3 otherwise you just can't patch C Extensions.
A hybrid approach, though, could debate when you decide to implement #3
instead of #2. (ie: all patching, or only when patching C extensions?)
Post by Mo Morsi
The majority of gems do not need an additional modification or patching
to be converted into a RPM. Yes these steps bring things more inline w/
other packages, but at the expense of unnecessary additional work.
If the solution is to suggest gem unpack / gem spec is used while
allowing for it to be omitted (still need to determine if gem install
should still be run in the %build or %install sections), package
maintainers will have a bit more flexibility to run the steps necessary
to build their package w/out any additional work, while at the same time
still being more compliant and in-line w/ other Fedora practices.
Thoughts?
I've been thinking about this for a week and also waiting for people to
bring forth packages that would be broken by it to see if there's any actual
problems with the strategy. No packages beyond the original one which is
a bug from upstream has been presented. Without that sort of example, I'm
going to vote for #3.
Well it is hard to bring you some example, because there are non. There
are no experiences with the #3 except with rubygem-idn, and in this case
the simple approach described in guidelines fails. Try to remove the
following lines from the spec:

# Avoid "cert_chain must not be nil" error.
sed -i -e "10d" %{gem_name}.gemspec

and you'll see what I am talking about. Please do not ignore these
custom lines I had to come up with to let the #3 work.
#2 is more complex as people have to keep in mind two sets of instructions.
When a bug fix is needed, it will not be simple to apply it as you'd have to
change the whole structure of the spec file in order to apply the patch and
then revert those changes when the patch is no longer needed. The
temptation for the packager in that situation will be to not apply bug fixes
until after they've made it into an upstream gem release.
Yes, it will not be simple to apply changes, therefore we are moving
from approach #1 which was valid up until now and proposing the approach
#2, which works well for all packages except rubygem-idn
#2 does not fix the problem with rpmbuild --short-circuit.
To be honest, I do not understand what is the point with the
--short-circuit. According to documentation "--short-circuit skip
straight to specified stage (only for c,i)". I do not know how are you
using it, but it implies that you have to do "rpmbuild -bp" to be able
to do "rpmbuild -bc --short-circuit". In that case, what does not work
for you? What do you want to see in this step if it is empty?
#3 was compared in the ticket with requiring "Fedora users to always rebuild
each RPM locally, because there might be sometimes broken dependency or
other error." This is a wrong comparison. A packager's job is to deal with
broken dependencies and other errors, not a users. #3 requires packagers to
deal with these issues, not users.
You should understand that RubyGems is packaging system and if I am
packaging the gem for Fedora, I am user of the packaging system from
point of view of RubyGems and I am package maintainer form point of RPM.
No need to argue about "packager's job". I just wanted to find something
what would give you better feeling what is it about and I gave better,
more detailed, example along the lines somewhere in this thread to Rex.
I'm also seeing the argument made for #2 that it's less work because
a single command exists to do all the steps. Therefore we should use it.
This is not a compelling argument because the same can be said of other
%setup -n %{srcname}-%{version}
python setup.py install --root %{buildroot}
even "make install" is valid under this argument.
I am not familiar with python, but you cannot compare "make install"
with "gem install". It is something really different. Moreover, "make
install" doesn't necessarily mean that the "configure && make" will be
run before.
#3 is also (barring someone giving us examples where packages are actually
broken) only a small one-time cost. Changing the rubygem spec files that
are using the old guidelines to use the new guidelines. As stated earlier,
the FPC has always known that the Rubygem guidelines needed to be changed
but was unaware that the time had come when the old guidelines were no
longer needed (lutter hasn't been on the FPC for years). Since you're
proposing changes to the guidelines anyway, you can hardly complain about
a one-time cost. In terms of an ongoing cost, until you show that there's
a non-bug reason that unpacking the source and rebuilding the gem is wrong,
there's no extra cost in maintaining this that package maintainers shouldn't
be responsible for anyhow.
Yes, you see just script which do conversion, but you do not count the
exceptions. Once more, I used #3 for rubygem-idn, the only gem which
required it and in this one case it immediately failed. So for me it is
a *100% failure* of your conversion script.

From last versions of packaging guidelines we moved from #1 to #2 which
solves practical problems. Move from #2 to #3 solves some artificial
problems.

Yes, I agree with you that if differs a bit from other packages. Yes, I
agree with you that it could be better. Yes, I proposed that I will work
with upstream to allow better way of patching C extensions, without need
to move to #3. Neither of this seems to satisfy you.


Please note that this [1] was the last proposal coming from me and
Ruby-SIG. I am not against formal changes of the guidelines, as was done
by Toshio, if FPC believes they will be better aligned with other
guidelines. I never said nothing against. I totally support that effort.
When I was asked to come up with the way how to patch the C extensions
and I did that. However I am strongly against mandatory #3.



Vit

[1]
https://fedoraproject.org/w/index.php?title=PackagingDrafts/Ruby&oldid=270494
Post by Mo Morsi
As a side node, if at all possible, please make sure to cc' both lists
(packaging and ruby-sig) on replies as this discussion is relevant to
both communities. Noticed alot of discussion only on the packaging list
meaning the Fedora ruby community is missing out on alot of this.
I'll try but you may need to pass my message through the ruby-sig
moderation queue.
-Toshio
--
packaging mailing list
packaging at lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/packaging
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.fedoraproject.org/pipermail/ruby-sig/attachments/20120301/58a65a69/attachment-0001.html>
Vít Ondruch
2012-03-02 08:40:41 UTC
Permalink
Toshio,

This discussion begins to be tiring. I'll go last time through this
topic trying to explain every concern you risen in your answer and I
hope not to be very personal :)

Yes, I agree with you that #3 would be perfect in ideal world, but since
we are not living in ideal world, I prefer #2. Since I am maintaining
more than 80 rubygems out of 290 currently available in Rawhide, since I
work with Ruby almost exclusively for more than 5 years, since the
guidelines draft was gone through review of Ruby-SIG members, I believe
you could respect more this opinion.
I've been thinking about this for a week and also waiting for people to
bring forth packages that would be broken by it to see if there's any actual
problems with the strategy. No packages beyond the original one which is
a bug from upstream has been presented. Without that sort of example, I'm
going to vote for #3.
Well it is hard to bring you some example, because there are non.
If there are no problems, then there's no reason not to use it.
There are no problem with approach #3, since nobody uses it. I am not
gonna to convert any of my packages just to prove you that you are
wrong. Prove me that you are right, that rubygem-idn is the only
problematic package and I'll believe you.
There are no
experiences with the #3 except with rubygem-idn, and in this case the simple
approach described in guidelines fails. Try to remove the following lines from
# Avoid "cert_chain must not be nil" error.
sed -i -e "10d" %{gem_name}.gemspec
and you'll see what I am talking about. Please do not ignore these custom lines
I had to come up with to let the #3 work.
I have not ignored this. In the ticket, I stated that this looks like an
upstream bug which, as packagers, it's our job to detect and fix.
cert_chain is being set to nil by the upstream packaging. This is not
supported by the current versions of the build tools (rubygems). We update
so the build script to work with the current version of the tools (and
hopefully submit the change to upstream).
When gcc updates and code stops working, we patch the upstream code to work
with the new gcc. When python updates, we patch upstream code of third
party modules so that they work. There's nothing out of the ordinary with
this at all.
#2 is more complex as people have to keep in mind two sets of instructions.
When a bug fix is needed, it will not be simple to apply it as you'd have to
change the whole structure of the spec file in order to apply the patch and
then revert those changes when the patch is no longer needed. The
temptation for the packager in that situation will be to not apply bug fixes
until after they've made it into an upstream gem release.
Yes, it will not be simple to apply changes, therefore we are moving from
approach #1 which was valid up until now and proposing the approach #2, which
works well for all packages except rubygem-idn
No. You are proposing approach #2 plus approach #3. So your proposal is
twice as complex as approach #3 as there's twice as many recipes for
building rubygems to learn.
You don't have to care about #3 as long as you don't need it. That is
the beauty. You are learning gradually. We agreed that from 290 rubygems
in Fedora you need #3 for one case, yet everybody should know it.

Nobody who comes from Ruby land and want to package his gem for Fedora
knows anything about "gem spec" command. Yes, that is nice that you want
to learn them this command right from beginning, but may be they will go
away, since they would be confused what is going on.
#2 does not fix the problem with rpmbuild --short-circuit.
To be honest, I do not understand what is the point with the --short-circuit.
According to documentation "--short-circuit skip straight to specified stage
(only for c,i)". I do not know how are you using it, but it implies that you
have to do "rpmbuild -bp" to be able to do "rpmbuild -bc --short-circuit". In
that case, what does not work for you? What do you want to see in this step if
it is empty?
rpmbuild -bp kernel.spec
cd kernel-3.0.0
# edit some source files, making backups w/ extension .myfix
cd ..
rpmbuild -bc --short-circuit kernel.spec
# If failure, edit source some more
# If success,: gendiff kernel-3.0.0 .myfix> myfix.patch and add it to spec
# file
Yes, that is good example. I never used it though and will never used it
even if we went with #3.

It is not useful for RubyGems, since I would never build/install the gem
just to try if binary extension builds. I would go to ext dir and fired
"make" (optionally preceded by "ruby extconf.rb"). Never ever "rpmbuild
-bc --shortcircuit".
#3 was compared in the ticket with requiring "Fedora users to always rebuild
each RPM locally, because there might be sometimes broken dependency or
other error." This is a wrong comparison. A packager's job is to deal with
broken dependencies and other errors, not a users. #3 requires packagers to
deal with these issues, not users.
You should understand that RubyGems is packaging system and if I am packaging
the gem for Fedora, I am user of the packaging system from point of view of
RubyGems and I am package maintainer form point of RPM. No need to argue about
"packager's job". I just wanted to find something what would give you better
feeling what is it about and I gave better, more detailed, example along the
lines somewhere in this thread to Rex.
Yes, you are the user of the upstream build system. But it's not
upstream's build system that the end users of Fedora see. So the comparison
that drags in end users is a wrong comparison.
No, it is not. I don't need to know more about RubyGems then necessary
(although it might be beneficial). If you go to package first gem for
Fedora, you are RubyGems user at best and may be you are not RubyGems
user at all. You just want to have something packaged.
Upstream build systems are sometimes easily adaptable to the way that
software should be packaged in rpms but at other times it is something that
we have to work around. Take a look at packaging java code where upstream
has bundled multiple versions of jar files just for their package build as
an example of something even more invasive than what we're talking about
here.
Yes, we are unbundling other gems from gems, or even Java packages, but
that is different topic. And it has nothing to do with packaging system.
I'm also seeing the argument made for #2 that it's less work because
a single command exists to do all the steps. Therefore we should use it.
This is not a compelling argument because the same can be said of other
%setup -n %{srcname}-%{version}
python setup.py install --root %{buildroot}
even "make install" is valid under this argument.
I am not familiar with python, but you cannot compare "make install" with "gem
install". It is something really different. Moreover, "make install" doesn't
necessarily mean that the "configure&& make" will be run before.
The argument I'm seeing is that "Upstream provides me with a command to run
that upstream has told me to use to build and install the package. It does
that in one step instead of spread over three steps like I want. I could
do it a different way using the standard tools and then it will do things in
the proper three steps but I want to do what upstream has asked instead of
what is proper." For that argument, the same can indeed be said about "make
install".
I was forced by you to learn something about packaging of Python
libraries and there is really nothing in common with RubyGems. If you
want to compare gem, you must compare it with RPM, not with *tarball*,
typically used by Python libraries.

There is no "--skip-build" flag available for "gem install" command
which would allow us simple move to #3.
#3 is also (barring someone giving us examples where packages are actually
broken) only a small one-time cost. Changing the rubygem spec files that
are using the old guidelines to use the new guidelines. As stated earlier,
the FPC has always known that the Rubygem guidelines needed to be changed
but was unaware that the time had come when the old guidelines were no
longer needed (lutter hasn't been on the FPC for years). Since you're
proposing changes to the guidelines anyway, you can hardly complain about
a one-time cost. In terms of an ongoing cost, until you show that there's
a non-bug reason that unpacking the source and rebuilding the gem is wrong,
there's no extra cost in maintaining this that package maintainers shouldn't
be responsible for anyhow.
Yes, you see just script which do conversion, but you do not count the
exceptions. Once more, I used #3 for rubygem-idn, the only gem which required
it and in this one case it immediately failed. So for me it is a *100% failure*
of your conversion script.
I've told you earlier why this is not an impressive statistic. Even if
*all* rubygem packages were to fail because they're using features of
rubygems that have been removed in the current version of rubygems, it would
be beneficial that we had detected the problem and submitted patches to the
builds to upstream. So in a way the fact that this bug in rubygem-idn would
have passed unnoticed if you didn't have to fix something else is a failure
in your proposed guidelines as it's not being detected until unrelated
changes are made.
Beneficial for whom? Upstream will care about their .gemspec file in
repository they are using to build the gem. They will care about their
rake taks which can build gem, because they are using them. But they
will not care about any .gemspec provided by "gem spec" command.

You do not understand that during the life of the library, there is
several forms of .gemspec. There is probably the upstream .gemspec file
(but not necessarily, it could be generated from code), which is
actually executable Ruby code. Then there is some metadata in .gem file,
however at this point, it was converted to YAML, hence the the "gem
spec" command without additional parameters provide YAML output. Even if
you used the "--ruby" flag, the .gemspec is already different then the
original file used by upstream. After installing the gem by "gem
install", You will get in the gems/specifications third form of the
.gemspec file.
From last versions of packaging guidelines we moved from #1 to #2 which solves
practical problems. Move from #2 to #3 solves some artificial problems.
That is incorrect. #2 is incomplete by itself. You must include #3 in
order to have a workable guideline. OTOH, #3 is complete in and of itself.
So leaving off #2 halves the number of specfile variations they need to know
if they want to package only rubygems.
I never said that #2 is complete. I said that it is practical and
satisfactory in majority of cases. #3 is exceptional case and should be
handled as all other exceptions. If you want some stats, it is likely
1/290, i.e. 0.35 % chance that you will need #3 for speaking about
current Rawhide, hence it is exception.
Yes, I agree with you that if differs a bit from other packages. Yes, I agree
with you that it could be better. Yes, I proposed that I will work with
upstream to allow better way of patching C extensions, without need to move to
#3. Neither of this seems to satisfy you.
That is correct. A workflow that looks like #3 is the correct way to
package software. You are not proposing to work with upstream to make that
possible but instead working with upstream to kludge their current broken
process to do one more thing (patching C extensions) in a broken manner.
Oh my, how you can say that other packaging system has broken process?
They had different design goals and I am sure that their design goal was
not compatibility with RPM nor mimic RPM. If it was so broken, then
nobody would used it, but the reality is opposite. Everybody is
wondering why not to stick with rubygems, why use RPM. And I see that
sometimes it would be really easier.
The proposed #3 is the workflow that the package build should have. If
you're unsatisfied with that, you should be talking to upstream about how to
adapt their build system to meet that workflow in all cases.
The proposed #3 is the workflow RPM has and RubyGems does not have.
There is nothing wrong with that.
Please note that this [1] was the last proposal coming from me and Ruby-SIG. I
am not against formal changes of the guidelines, as was done by Toshio, if FPC
believes they will be better aligned with other guidelines. I never said
nothing against. I totally support that effort. When I was asked to come up
with the way how to patch the C extensions and I did that. However I am
strongly against mandatory #3.
But you have given no examples that show why it would be wrong. And without
that, you're just spreading fear that a mandatory #3 will cause you issues.
Come on, you gave no example that it will work for majority of cases, so
how you can propose it??? If Ruby-SIG believed it is that important, it
would be already mandatory in the proposal.
If you're so sure that you can get close to a 100% failure rate out of #3
that is due to problems with the procedure rather than bugs in upstream
packaging, you should be able to pick a random rubygem, adapt it to #3 and
then expose some new type of bug that we can examine and determine why it's
unworkable. Failing to do that, the argument that it won't work is just
speculation, not fact.
As same as from your side. Please don't try to manipulate us.


Vit
-Toshio
--
packaging mailing list
packaging at lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/packaging
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.fedoraproject.org/pipermail/ruby-sig/attachments/20120302/982af062/attachment-0001.html>
Vít Ondruch
2012-03-09 13:57:30 UTC
Permalink
Post by Mo Morsi
Post by Vít Ondruch
= Old guidelines, used from the time RubyGems were packaged for Fedora =
%prep
%build
%install
mkdir -p %{buildroot}%{gem_dir}
gem install --local --install-dir %{buildroot}%{gem_dir} \
--force %{SOURCE0}
= What we are proposing =
%prep
%setup -q -c -T
mkdir -p .%{gem_dir}
gem install --local --install-dir .%{gem_dir} \
--force %{SOURCE0}
%build
%install
mkdir -p %{buildroot}%{gem_dir}
cp -a .%{gem_dir}/* \
%{buildroot}%{gem_dir}/
= What FPC is proposing =
%prep
%setup -q -c -T
pushd ..
gem unpack %{SOURCE0}
pushd %{gem_name}-%{version}
gem spec %{SOURCE0} -l --ruby> %{gem_name}.gemspec
gem build %{gem_name}.gemspec
popd
popd
%build
mkdir -p ./%{gem_dir}
gem install --local --install-dir ./%{gem_dir} \
--force ../%{gem_name}-%{version}/%{gem_name}-%{version}.gem
%install
mkdir -p %{buildroot}%{gem_dir}
cp -a .%{gem_dir}/* \
%{buildroot}%{gem_dir}/
All three versions provide the same output, unless I did some mistake,
since I did not tested it (actually the middle one was taken directly
from rubygem-POpen4.spec). So which version you prefer? Please note
that the "gem install" will always "unpack" the gem with some
additional, for our case unimportant, steps. We do not distribute the
.gem file anywhere.
Alternatively we can go with a hybrid of solutions 2 and 3 (your and
FPC's proposals) where the 'gem unpack' and 'gem spec' steps are optional.
As Vit points out, there is no way to implement #2 alone -- it must be
combined with some form of #3 otherwise you just can't patch C Extensions.
A hybrid approach, though, could debate when you decide to implement #3
instead of #2. (ie: all patching, or only when patching C extensions?)
Post by Mo Morsi
The majority of gems do not need an additional modification or patching
to be converted into a RPM. Yes these steps bring things more inline w/
other packages, but at the expense of unnecessary additional work.
If the solution is to suggest gem unpack / gem spec is used while
allowing for it to be omitted (still need to determine if gem install
should still be run in the %build or %install sections), package
maintainers will have a bit more flexibility to run the steps necessary
to build their package w/out any additional work, while at the same time
still being more compliant and in-line w/ other Fedora practices.
Thoughts?
I've been thinking about this for a week and also waiting for people to
bring forth packages that would be broken by it to see if there's any actual
problems with the strategy. No packages beyond the original one which is
a bug from upstream has been presented. Without that sort of example, I'm
going to vote for #3.
#2 is more complex as people have to keep in mind two sets of instructions.
When a bug fix is needed, it will not be simple to apply it as you'd have to
change the whole structure of the spec file in order to apply the patch and
then revert those changes when the patch is no longer needed. The
temptation for the packager in that situation will be to not apply bug fixes
until after they've made it into an upstream gem release.
#2 does not fix the problem with rpmbuild --short-circuit.
#3 was compared in the ticket with requiring "Fedora users to always rebuild
each RPM locally, because there might be sometimes broken dependency or
other error." This is a wrong comparison. A packager's job is to deal with
broken dependencies and other errors, not a users. #3 requires packagers to
deal with these issues, not users.
I'm also seeing the argument made for #2 that it's less work because
a single command exists to do all the steps. Therefore we should use it.
This is not a compelling argument because the same can be said of other
%setup -n %{srcname}-%{version}
python setup.py install --root %{buildroot}
even "make install" is valid under this argument.
#3 is also (barring someone giving us examples where packages are actually
broken) only a small one-time cost. Changing the rubygem spec files that
are using the old guidelines to use the new guidelines. As stated earlier,
the FPC has always known that the Rubygem guidelines needed to be changed
but was unaware that the time had come when the old guidelines were no
longer needed (lutter hasn't been on the FPC for years). Since you're
proposing changes to the guidelines anyway, you can hardly complain about
a one-time cost. In terms of an ongoing cost, until you show that there's
a non-bug reason that unpacking the source and rebuilding the gem is wrong,
there's no extra cost in maintaining this that package maintainers shouldn't
be responsible for anyhow.
Post by Mo Morsi
As a side node, if at all possible, please make sure to cc' both lists
(packaging and ruby-sig) on replies as this discussion is relevant to
both communities. Noticed alot of discussion only on the packaging list
meaning the Fedora ruby community is missing out on alot of this.
I'll try but you may need to pass my message through the ruby-sig
moderation queue.
-Toshio
--
packaging mailing list
packaging at lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/packaging
Hi Toshio,

Today, in attempt to find some compromise solution, we tried to "fix"
RubyGems in a way, that would allow the following workflow:

%prep
%setup -q -c -T
mkdir -p .%{gem_dir}
gem install --local --install-dir .%{gem_dir} \
--force \
--skip-build \
%{SOURCE0}

%build
pushd .%{gem_dir}/ext
./configure && make && make install
popd

%install
mkdir -p %{buildroot}%{gem_dir}
cp -a .%{gem_dir}/* \
%{buildroot}%{gem_dir}/

The main differences are the "--skip-build" flag in %prep section and
the "./configure && make && make install" (or some equivalent) in %build
section. In the first moment, it looked promising. Add the
"--skip-build" flag should be easy as conditionally execution the
extension's build [1], i.e. almost one line change. Unfortunately, it
turned out that things are not as easy as they seem on the first look.

First of all, the '"./configure && make && make install" (or some
equivalent)' is tricky, since there are currently 3 supported build
methods by RubyGems

1) Traditional ./configure && make && make install"
2) The most widely used extconf.rb and make combo
3) Rake builder

It shouldn't be hard to support also other make alternatives. So these 3
possibilities complicate the %build section enough. Moreover, if you
look on the implementation of the builders [2], you'll see that the
builders are patching the Makefiles on several places, provide
additional configuration options etc. There is no good place where to
cut-off the build on some lower layer.

In conclusion, this is dead end. The "--skip-build" option doesn't look
promising ATM unless the RubyGems would undergo some significant changes.


Vit



[1]
https://github.com/rubygems/rubygems/blob/master/lib/rubygems/installer.rb#L229
[2] https://github.com/rubygems/rubygems/tree/master/lib/rubygems/ext
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.fedoraproject.org/pipermail/ruby-sig/attachments/20120309/38969de4/attachment.html>
Vít Ondruch
2012-02-29 15:06:20 UTC
Permalink
Quoting Emanuel Rietveld (2012-02-29 12:18:57)
Post by Vít Ondruch
Pleas do not be mistaken. We are not speaking about building gems from
sources. We are speaking about building from package manager output,
i.e. build gem from gem.
So we are shipping stuff, which is not build from other stuff we ship, but
from magic upstream binaries? Not nice at all.
It is worth noting that .java files compiled into .class files or .jar
files is not the same thing as .rb files. .rb files are not compiled*
However I have seen gem files containing bundled jar files. Not sure if
gem unpacking actually helps things, but it might make it more easy to
spot perhaps. There as easy ways to detect such bundling though, so not
a problem. Just though I'd mention this use case
Yes, there are gems with bundled jar files. There are also gems which
might carry other binaries. For this case, there apply general Fedora's
"No inclusion of pre-built binaries or libraries" and "Duplication of
system libraries" policies, nothing specific is needed for Ruby.

Please note that we proposed to do "gem install" in %prep section which
"unpacks" the gem content among other things, so you can spot such files
easily.


Vit
Vít Ondruch
2012-03-02 08:52:15 UTC
Permalink
Post by Vít Ondruch
Quoting Emanuel Rietveld (2012-02-29 12:18:57)
Post by Vít Ondruch
Pleas do not be mistaken. We are not speaking about building gems from
sources. We are speaking about building from package manager output,
i.e. build gem from gem.
So we are shipping stuff, which is not build from other stuff we ship, but
from magic upstream binaries? Not nice at all.
It is worth noting that .java files compiled into .class files or .jar
files is not the same thing as .rb files. .rb files are not compiled*
However I have seen gem files containing bundled jar files. Not sure if
gem unpacking actually helps things, but it might make it more easy to
spot perhaps. There as easy ways to detect such bundling though, so not
a problem. Just though I'd mention this use case
Yes, there are gems with bundled jar files. There are also gems which
might carry other binaries. For this case, there apply general
Fedora's "No inclusion of pre-built binaries or libraries" and
"Duplication of system libraries" policies, nothing specific is
needed for Ruby.
Please note that we proposed to do "gem install" in %prep section
which "unpacks" the gem content among other things, so you can spot
such files easily.
Actually, Stanislav has a good point. gem install unpacks, builds,
and installs a gem. So when I do a gem install and then do a find . -name
'*.so' or find . -name '*.jar' I don't know right off the bat whether the
files listed were bundled or produced by "gem install". I don't know
whether all of the *.so's were built from source or if there was
a precompiled object file in the gem that was included. So how do you
inspect the results of gem install to determine that there is nothing
bundled?
You can compare results of "gem unpack" and "gem install" for example.
But there is definitely more ways.

Vit
-Toshio
--
packaging mailing list
packaging at lists.fedoraproject.org
https://admin.fedoraproject.org/mailman/listinfo/packaging
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.fedoraproject.org/pipermail/ruby-sig/attachments/20120302/1fa27888/attachment.html>
Loading...