What are we up to?

[DCoder]

I'm bored (uh-oh) and curious. What do you think we (Ares developers, testers and other contributors) are up to these days?

[gordon-creAtive]

Making up quotes for the qdb?

[Graion Dilach]

I've fallen in love. What you guys do... no idea.

[Renegade]

Same thing we do every night, Pinky...wait for someone to actually test Ares.

[4StarGeneral]

For the ten-thousandth time (I think its the ten-thousandth, maybe less...), I would like someone to tell me, pm, email, whathaveyou; what is the workflow in which you start testing, because ever since I started testing before, I never could figure out how to read through the mess, to see what bugs have been tested, and what haven't. I would also like to lend my failing memory of the english language to the manual, and maybe update that. But again, I have no idea on svn's and the like.

As far as on topic, you're up to your (eye)balls in beer. I know this to be true.

[Orac]

I had assumed GTA, mostly.

[Steel Mirage]

Personally, as an "other contributor", I'm up to my eyeballs in school these days which is exactly where I want to be in general.

As far as where everyone else is, its not my place to speculate on where people "are".

What I observe, however, is that everyone appears to be stuck in a prolonged cycle of readjustment. Recently, the Bugtracker made a significant paradigm shift, and until all significant parties have reached the same level of fluency with the current program that was attained under the old SVN regime, this "activity-recession" will continue. Thats neither a positive nor a negative statement, its just the way things are.

That being said...
(queue the groans of "here goes Steel Mirage again)

On the issue of a "Standard Testing Procedure":
In a roundabout way, 4StarGeneral brings up a significant question, one that has been repeatedly asked and answered in the past year:

"As a tester, what the fuck do I do, and why can't you just tell me what to do?"
(or something like that)

In General Terms:

As a unified project, Ares tests the limits of how formal an informal project can be. The most significant limitation on Ares is the fact that, by necessity, there is no single member that can say "do this or do that in this fashion". And should someone attempt to give orders like that, the consequences would be a lot like trying to herd cats: everyone would scatter until the ruckus concluded.
(Renegade this is not me fomenting rebellion, just bear with me on this though process)

The reason for this is that the sole uniting factor of the people working on Ares is the common love for Red Alert 2: Yuri's Revenge. At least, thats what keeps me coming back. That and I enjoy the people here...

Other than that uniting factor, we all have lives and agendas that take precedence over involvement and activity within Ares. If someone were to take a dictatorial stance, the negative consequences would far outweigh the temporary relief that the testing corps would feel about someone giving them direction.

The Specific Controversy:

On an even more fundamental level though, that sort of "directed testing" has been established as also running afoul the "shit to pay" ratio:

The point of endless testing, as deemed necessary by Renegade and legitimized by the acclamation of all significant members of Ares (my perception anyways), is to discover as many possible foul-ups and fuckknuckleries as possible. This is a daunting goal even if applied only to the original programming (see the note in the manual credits regarding programmers and cannons). Ares adds a great deal of complexity to that. If the original programming were flawless, just making sure that Ares was stable would be a giant undertaking by itself. Adding Ares' layer on top of the original "spaghetti derpball" created by Westwood creates a testing twilight zone. Just getting to where we are now with the soup sandwich we were given should, by all rights, be a source of pride to everyone involved.

The point of the Bugtracker is to establish clear communication between the testers and developers in the form of clear cyclical testing and development.

Cyclical processes must have a starting point, and thats where the problem with directed testing, as some testers have demanded, lies. The entire testing process absolutely and fundamentally depends upon the ability of testers to find "unknown unknowns". Once they are known, they are documented in the Bugtracker and the cyclical process can begin. But if all testing is devoted to the "known unknowns", then the cyclical process will become circular, leading constantly back to the same exact point. End result is that no one gets anywhere, nothing gets done and Ares dies a slow heat-death.

This is why directed testing on a large scale is insufficient for the needs of Ares.

The Problem: Expectations

I would like to take a moment right here to thank and congratulate Renegade, DCoder and Alex_B for their commitment and ability.

They understand, sometimes implicitly, everything that I have said here so far. And understanding that, they continue to have patience, be supportive and work on what they can.

I want to be very clear here: All development of Ares absolutely and profoundly depends on the ability and commitment of competent testers to find the "unknown unknowns". Without that crucial element, Ares dies. Its that simple.

Knowing that, Renegade, DCoder and Alex_B (and others, to be fair, but these guys are the big three and we all know that) take pains not to use force to accelerate the progress of the testers. They cajole. They question. They plead. They do not order, they do not force anyone to do anything that they don't want to do. Why? Because this is an informal project, as stated above, and a dictatorial approach would mean the rapid bloody dissolution of Ares.

They are under no illusions of the ability of the testers. They understand that people have lives and that, at best, Ares represents a hobby, and one that gathers dust in most peoples hard-drives until boredom or guilt or re-discovery brings people back in. I am just as guilty of this as anyone, and its nothing to be ashamed of. Its just the nature of the project. And as much as the Big Three would love for rapid progress to occur, its not their place nor their wish to lash a slave force into blind obedience.

Because of that, and because their own efforts as developers (and whatever Renegade does :-D) depend on what the testers unearth from the goopy slime of Westwood's Grand Quagmire, Ares plods on at a very slow pace.

And yet folks have the grand expectations of a finished, release-worthy project now if not sooner (I speak in generalities here; if this offends someone its not my problem, check a mirror if you don't believe me). I'm guilty of it too on occasion. Expectations are a good and natural thing to have; without them, no one would bother to contribute to Ares. Elevated, unreasonable expectations, however, only serve to erode the efforts made by Ares' contributors.

When those unjustified expectations are laid at the feet of the leadership here, however, the earth trembles and lightning flashes in the skies (just read any of Renegades various "walls-o-text"; this essay pales in comparison). Then a rolling clusterfuck of fingerpointing and sometimes namecalling ensues as the pent-up dissatisfaction with progress comes to a fine rolling boil. Eventually we all go into our corners and the vat-of-shit thread is left to fester until referenced in the next rolling clusterfuck.

Nothing gets done, nothing changes, and eventually the process repeats itself.

In my opinion, this is really good and healthy and gives me hope. And usually makes me laugh my ass off, and as we all know, laughter is the best medicine.

A Solution?

Now that I have described the problem, (if ya'll went evidenciary demonstration, look things up on your own time, this essay is long enough on its own), I would like to propose a simple (-ish) solution.

Context: I have just finished a dissertation on the philosophy, ideology, strategies and mechanisms of student identity. My programmes and conclusions would adapt themselves nicely to Ares.

We already have a "Testing Area" forum. I would like to propose a sort of "So you wanna be a tester?" thread that would come at the top of the forum, the contents of which would be:

1. A definition of testing
2. A short description of a testing "process"
3. A short demonstration of testing (maybe a short video?)
4. A list of testing objectives
5. The reason that we test the way we do
6. A process for setting up individual testing strategies
7. Testing Tips and Tricks

This would give all current and prospective testers a one-stop place to get information for how and why they test. It orients testers and gives them a foundation from which to commence testing while at the same time not actually falling prey to the failures of "directed testing" (a strategy that does have a place, but only in specific areas, not the in the abstract general testing that we so badly need).

This provides support for the testers without further draining the time and resources of the organizers and developers. It would provide some unification of testing expectations, lexicon and procedures, which in turn would make communication with the developers more transparent and concise, making their lives easier, which would significantly improve all aspects of the Ares project.

It would also eliminate the need for the same questions to be answered individually again and again. If a tester has a question, he or she (heh, yeah right, just he) could be directed towards the "SoYWaBAT" thread (Maybe we could come up with a better title for a better acronym?). Really good questions and answers could be incorporated into the thread as a resource for future testers.

Even better, this could be a resource that outlives Ares or is useful to other modding organizations, which would bring further traffic to RenProj and make us all (and by us all,I mean Ren) much happier.

Implementation:

I see implementation taking not very long and not very much effort. I certainly wouldn't ask anyone else to put in more effort into it than I would. Even so, this could be up and running within a week with minimal effort.

As I see it:

Part 1: I would do this, after interviewing some folks.
Part 2: I would do this, after interviewing Graion.
Part 3: I would need someone else to do this, someone with experience testing and making videos of their testing. Could probably be done in an afternoon.
Part 4: This would be super simple and would come from Ren, DCoder and Alex_B, seeing as they are "the powers that be". Or they could just zap what they want to me and I could integrate and draft.
Part 5: This would be an excerpt from this thread and others.
Part 6: This would be my wheelhouse, after interviewing some folks.
Part 7: This would be seeded by some experienced testers and be built on over time.

I'm not gonna just unilaterally do this though:
We all have to agree that the status quo is insufficient for our needs.
We all have to agree that something can be done to improve the status quo.
We have to have some sort of consensus that my proposal is appropriate.
We have to understand the obligations that I put forth under "Implementation".

The great part is that once this is done, its done. It would need very little further maintenance.

Unleash the anti-air artillery!!!

[Renegade]

I appreciate the thought you put into this, SM, but unclarity about the testing process really only shows lack of attention, not lack of information.
I've already mentioned why issues-to-test lists are not helpful at this point, we still have one to get at least some form of testing, we have instructions on how to report positive feedback (including more testing directions) as well as a place for it in Answers, etc., etc.

There's really no mystery to it:
Play the damn game. Report any issues you have, or if you have none.
Try out Ares's features. Report any issues you have, or if you have none.
Combine Ares's features. Report any issues you have, or if you have none.
Experiment with Ares's features. Report any issues you have, or if you have none.
Try to break Ares's features. Report any issues you have, or if you have none.

If you must have easier directions, consider testing the way to find the answers to the following three questions:

1. Does feature X work as intended?
   
2. Does feature X integrate properly with the rest of Ares?
   
3. Is Ares as a whole still stable?
   

Answer these three questions for each of Ares's features, and we'll make a giant leap towards release. But apparently, actually modding and playing the game seems too much to ask of the modding community these days.

And before you come with the retarded "Ares has been stable for months *drool*" bullshit, if that were true, we wouldn't still find obvious bugs with frightening regularity...like broken single player campaigns, or Ivan bomb handling having been broken for months.

Quite frankly, the bugs that do get found show us that there hasn't been much searching yet.

A single month of proper testing could be all that's necessary. Really. If we had gotten daily feedback from half a dozen testers over the entirety of January, covering the breadth of Ares, we probably would've released last week.
Instead, the positive feedback thread over at Answers expired, because no one posted in it.


So yeah. There's really no mystery, and really no mechanism needed. We just need people to shut up and actually check Ares for bugs for a change.

[4StarGeneral]

... I never could figure out how to read through the mess, to see what bugs have been tested, and what haven't...

^ Is more my problem. I can test things, no problem, but I can never find what is NEW (what is Feature X), somewhat because of the fact the manual isn't updated often enough. Not that this is any one person's fault.

But if I ever do figure out how to test, I will be sure to contribute by making a video of how to do it effectively, for you SM.

[Steel Mirage]

Thanks 4Star, that would be helpful to us all.

@Renegade: I was headed in a more more meta-question direction with my spiel up there. I don't think the problem is people playing with Ares, I think the problem is that people look at Ares and say "Holy shit, there's a metric fuckload to possibly test, how can I as just one tester hope to ever put a dent into things?"

My solution would be to set up a resource that would help tester set up their own program so that they can set goals for themselves without feeling like they're completely overloaded. To wit:

1. Does feature X work as intended?
2. Does feature X integrate properly with the rest of Ares?
3. Is Ares as a whole still stable?

To me, this looks like a case of "because it looks simple it should feel simple". Unfortunately, because of the scope of Ares, grand strategy simplicity just doesn't translate into trench tactics.

I think what the testers are asking for is a place to start. And as developers with your above stated objections to directed testing, that not only isn't gonna happen, it would actually be bad if it did happen.

Thats what I am hoping the Testers understand.

But that still leaves the testers without something to cling to, without a place to start for themselves.

My idea revolves around a set of simple questions and procedures that the individual testers could answer and follow for themselves, and generate a unique and random chain of testing.

Example:

Code:

1. Do you know how to test?
If yes, then go to 2,
If no, then go to IRC and make ready to use your ears (or something to that effect.  4Stars video would also be handy).
2.  Do you know what you want to test?
If yes, then get to work,
If no, then go to 3.
3.  Are you familiar with the game?
If yes, then go to 4.
If no, then just play the game for a while and come back later
4.  Are you familiar with rulesmd.ini?
If yes, then go to 5.
If no, then familiarize first and then go to 5.
5.  Are you familiar with the manual
If yes, then go to 6.
If no, then familiarize first and then go to 6.
6.  What kind of feature do you want to test

?
Etc, etc, etc.

Typing that literally took me about three minutes, following it takes even less time, and it gives new and old testers a place to start asking themselves questions.

I see the options for step six being unit types, sw's, building types, etc... This would give a tester a place to focus and then they can go from there.

Its this kind of Socratic questioning that not only gives testers a place to start from, but also starts building analytical competence into the testers process.

Thats where I was headed with that last wall o' text.

[Graion Dilach]

And that's the reason why a constantly available Doc Maintainer would be a blessing.

The developers develop. They are programmers. Then can understand each other from the commit messages, the diffs. The Doc maintainer would be the shim to turn that endless amount of stuff into a something understandable by the public.

[Steel Mirage]

I only wish I were constantly available. This latest burst of my participation has been a product of my rough draft thesis going into edit/review. I tried to set up a connection to that thingamajigger a few times so I could edit stuff, but no dice, and I'm not sure why. I think it has to do with having a POS computer...

I don't think its lack of attention, as Renegade says, I think its lack of imagination, and thats what I'm trying to address before I go back into my hole.

[Renegade]

... I never could figure out how to read through the mess, to see what bugs have been tested, and what haven't...

^ Is more my problem. I can test things, no problem, but I can never find what is NEW (what is Feature X), somewhat because of the fact the manual isn't updated often enough. Not that this is any one person's fault.

But if I ever do figure out how to test, I will be sure to contribute by making a video of how to do it effectively, for you SM.

These changes are new. But that's irrelevant, because that's not what we're looking for. The implementation of these changes could have destroyed features that were running fine since pre-0.1.
Hence why I constantly reiterate that targeted testing is bullshit and not helping.

Telling us whether those linked changes behave as intended would be nice and appreciated...but telling us whether the entirety of Ares 0.1 P1 still works as expected inside Ares almost-0.2 would be vastly more helpful.
Because the fact that those two changes went through fine means jack shit if half of our original feature base is in ruins.

Or, to answer your question of "what is Feature X": Feature X is every feature, all the time.

@SM: What is so complicated about "pick any feature and tell us what state it is in" that people need a 12-step-plan to do it?

I understand your reasoning and I understand what you're getting at, it's just a question of cost-benefit calculation: How much education can we provide before it would've been more economical to do the test ourselves, and how little information can be in a report before it becomes worthless to us?

Basically, the more help someone needs with the simple task of trying out a feature and telling us how it went, the higher the chance he's not someone who's gonna be helpful in the first place.

And before you say "beggars can't be choosers", consider that we still have to sift through the reports, helpful or not - no reports means no work, a hundred pointless reports means shitloads of work for no gain.

I'm really not trying to be negative here, I'm just trying to point out that not all testers are created equal, and that not everyone must be turned into a tester at all costs.
We're not literally looking for as many reports as possible. We're looking for as many helpful reports as possible.

That all being said, the stubs in the FAQ section exist for a reason. Of course we'd like to provide as much guidance as possible/necessary. The FAQ would probably greatly benefit from a "How to properly test Ares" entry. But I'm not the one to write it, and I'm not sure the other devs have the time or the necessary point of view to do it.

So while I am in favor of more documentation of the process, right now, I'd be more in favor of testers understanding simple instructions.

[Scott]

I could be just being dumb, but is it basically as simple as "I have a mod using Ares 0.1, shall I just run it with Fat Man and see if it still works?"

'cause if it is, I fail to see why there's not more people doing it. Hell, I'll do just that tomorrow.

Personally, maybe that what this needs, particularly for me, is links to the relevant code without having to hunt through the bugtracker or the manual, especially seems last time I checked, the code for abductor logic wasn't in the manual.

[Graion Dilach]

That's kinda possible, manual is pre-FatMan. Hopefully I can get there nowadays to update it more.