[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: run BISTs for all installed packages
From: |
Andrew Janke |
Subject: |
Re: run BISTs for all installed packages |
Date: |
Tue, 5 Mar 2019 21:39:48 -0500 |
User-agent: |
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:60.0) Gecko/20100101 Thunderbird/60.5.1 |
On 3/5/19 7:18 PM, Mike Miller wrote:
On Tue, Mar 05, 2019 at 10:59:01 -0500, Andrew Janke wrote:
I've put together a list of references to bugs and Octave maintainers
mailing list threads related to enhancing the test suite:
https://github.com/apjanke/octave-testify/blob/e6e9cd4117c6d6e601a8c00487dc75769aa1699f/doc-project/Developer-Notes.md
Any major ones I'm missing?
Looks good, thank you for collecting all of these topics together.
I agree with your project goals.
Thanks!
I also hope you'll preserve some of the
current test suite features I think are important, including being able
to run the test suite pre-install or post-install, read-only, optionally
print full test failures in auto build or CI environments.
I'm gonna preserve ALL of these features, *ALL* of them, ha ha ha!
Though not necessarily with the same invocation syntax. :)
More seriously, my intention for Testify is to be a:
a) more-or-less strict functional superset of existing BIST
functionality, with
b) nicer output and more convenient calling forms, and
c) maybe more structured control over how the BISTs are invoked and
interact with each other and are collected together in summary objects
This Testify project isn't some big idea to make a whole new testing
layer for Octave[1]. I mean, originally all I wanted to do was provide a
concise summary output of failed tests at the end of the
__run_test_suite__() output so you didn't have to scroll back and copy &
paste multiple times to get a list of your failed tests
(https://savannah.gnu.org/bugs/index.php?55522). But all this code is
intertwingled, and one thing led to another, so...
[1] ...although, maybe, I'll throw in a port of the xUnit testing
framework, given some spare time and user interest...
If I have my own list of annoyances / wishlist things I'd like to see
improved, would you rather see them reported on savannah or on your
project?
I'd prefer them reported on my project (one issue report per
annoyance/wishlist item, please, and no annoyance too small), because a)
GitHub has nicer issue/task-tracking tools than Savannah IMHO, and it'll
be easier to grind them down in to separate pieces of work there, b) I'd
probably consider these in-scope for this particular project, and could
iterate on them in GitHub independent of the Octave project management
because Testify is an independent project that's not in Octave Forge
yet, and c) I think in terms of tooling, it's actually easier for us to
work on an independent octave project using git/GitHub and `pkg install`
tooling? Once you get to multiple commits, forks and branches are easier
to manage than Hg changesets stored behind bug attachment URLs.
You should view Testify as "Andrew's wacky idea of where Octave BIST
support should go in the next release or two", and if you submit a
gripe/wishlist item there, I'm likely to pick it up and have an Idea and
do something about it. And then hopefully eventually we can get back
together with core Octave developers, find a way that a compromise
version of Testify can be merged back in to Octave core, do that, and
everybody benefits.
Are you aware of the 'doctest' package [1] to run tests from Octave help
strings? I think it's mostly orthogonal to your goals, just wanted to
make sure you knew it was out there.
>
> [1]: https://github.com/catch22/octave-doctest
Uhh, no! Thanks for the heads-up. I'll look in to it.
At first glance, this actually doesn't look *that* different from BISTs,
it just has fancy ways of specifying the strings it's using as
"expected" values in its assert() logic...
I also maintain a separate shell-based test suite [2] to test Octave's
command interface, command line options, exit status, and so on, also
unrelated to your project goals.
This is interesting. Looks like you're mostly concerned with the
interface between the `octave` command or runtime and the external
system. This is probably something I should learn about some day. :)
But at some point I'd like to see these two supported by Octave to some
extent.
[2]: https://gitlab.com/mtmiller/octave-test-suite